Saturday, 27 July 2019

How to Restore SnapCenter database - SnapCenter Repository Backup Restore

Carrying on from the previous post - Disaster Recovery of SnapCenter Server with IP Address Change - restoring SnapCenter from a repository backup is straightforward.
Note: Using a SnapCenter 4.1 lab.

Restoring with MySQL and SnapManagerCoreService Up and Running


PS> Open-SmConnection -SMSbaseUrl https://SnapCtr.demo.corp.com:8146

cmdlet Open-SmConnection at command pipeline position 1
Supply values for the following parameters:
(Type !? for Help.)
Credential

PS> (Get-SmHost).HostName
DAG1.demo.corp.com
mb1.demo.corp.com
mb2.demo.corp.com
mb3.demo.corp.com
snapctr.demo.corp.com

PS> Get-SmRepositoryBackups

Backup Name
-----------
MYSQL_DS_SC_Repository_snapctr_07-26-2019_09.34.20.3512
MYSQL_DS_SC_Repository_snapctr_07-26-2019_10.35.02.3271
MYSQL_DS_SC_Repository_snapctr_07-26-2019_11.33.04.0570
MYSQL_DS_SC_Repository_snapctr_07-26-2019_12.33.03.8936

PS> Restore-SmRepositoryBackup -BackupName MYSQL_DS_SC_Repository_snapctr_07-26-2019_12.33.03.8936 -HostName SnapCtr.demo.corp.com

SnapCenter respository restored successfully!


Note: Don’t need to use Open-SmConnection and Get-SmHost before running SmRepositoryBackup cmdlets. I just did this to get the correct hostname.

One Slighty Odd Thing

One slightly odd thing is that when you restore, you lose the MySQL dmp from your backup location (perhaps this is to be expected).


PS> Restore-SmRepositoryBackup -BackupName MYSQL_DS_SC_Repository_snapctr_07-26-2019_12.33.03.8936 -HostName SnapCtr.demo.corp.com

Restore-SmRepositoryBackup: Backup files doesn't exist. Please provide a valid location that contains the SnapCenter backup files or specify RestoreFileSystem option to restore from the snapshot.

PS> Get-SmRepositoryBackups

Backup Name
-----------
MYSQL_DS_SC_Repository_snapctr_07-26-2019_09.34.20.3512
MYSQL_DS_SC_Repository_snapctr_07-26-2019_10.35.02.3271
MYSQL_DS_SC_Repository_snapctr_07-26-2019_11.33.04.0570
MYSQL_DS_SC_Repository_snapctr_07-26-2019_12.33.03.8936

PS> Restore-SmRepositoryBackup -BackupName MYSQL_DS_SC_Repository_snapctr_07-26-2019_11.33.04.0570 -HostName SnapCtr.demo.corp.com
SnapCenter respository restored successfully.


Image: After 2 restores, I’ve lost 2 dmp’s!

Q: How to Restore if the Database is Completely Gone?

If the database is completely gone and MySQL won’t start, what to do?

I’ve tried a few things but have not yet found the magic trick using DB restore CLI commands (MySQL or SnapCenter). The easiest way is going to be to revert the VM and database to a known good date.

This link looked promising (not for SnapCenter database though):

Or simply, with SnapManagerCoreService and MySQL57 stopped, replace these files with files from a known good backup -

C:\ProgramData\NetApp\SnapCenter\MySQL Data\Data

ib_buffer_pool
ib_logfile0
ib_logfile1
ibdata1

- restart MySQL57 and SnapManagerCoreService. And – hey presto – should all be back up and running again.

Image: Need to restore these ib files

Disaster Recovery of SnapCenter Server with IP Address Change

Assuming your SnapCenter server is replicated to a DR site, then recovery of SnapCenter including an IP change is very straightforward! (Even more straightforward if you have stretched VLAN and don’t need to re-ip.)

The SnapCenter IP address is not hardcoded anywhere, so once you’ve done the below, everything should just work!
- brought up the replicated SnapCenter in the DR site (not in this blog post)
- re-IP-ed SnapCenter
- updated DNS (make sure all the plugin hosts resolve SnapCenter to the correct IP)
- rebooted the SnapCenter server after re-IP

You would need to check your backups jobs are not trying to backup servers that aren’t available (because of the DR situation). We’d only need to recover from repository backup if there was something wrong with the database.

In the walkthrough below, we run through the steps and show DR of SnapCenter with IP change is nothing to be afraid of. We backup the repository as it is best practice to do so, but don’t use the repository backup (something I’ll blog about later.)

Note: SnapCenter version here is 4.1.

i: A little bit of setup:

Creating the SnapCenter repository backup LUN:


cluster1::>

volume create -vserver svm1 -volume SC_REPO_BKUP -size 10G -space-guarantee none -aggregate aggr1_cluster1_01

lun create -vserver svm1 -volume SC_REPO_BKUP -lun LUN101 -size 50G -ostype windows_2008 -space-reserve disabled -space-allocation enabled

igroup create -vserver svm1 -igroup snapctr -protocol iscsi -ostype windows -initiator iqn.1991-05.com.microsoft:snapctr.demo.corp.com

lun map -vserver svm1 -volume SC_REPO_BKUP -lun LUN101 -igroup snapctr -lun-id 101


ii: Replicating the SnapCenter repository backup LUN:


cluster2::>

volume create -vserver svm2 -volume SC_REPO_BKUP_DR -size 10G -type DP -aggregate aggr1_cluster2_01

snapmirror create -source-path svm1:SC_REPO_BKUP -destination-path svm2:SC_REPO_BKUP_DR -type XDP -policy MirrorAllSnapshots -schedule hourly

snapmirror initialize -destination-path svm2:SC_REPO_BKUP_DR


iii: Configuring SnapCenter Repository Backup:


PS> Open-SmConnection -SMSbaseURL https://snapctr.demo.corp.com:8146

PS> Add-SmHost -HostName snapctr.demo.corp.com -OSType Windows -DiscoverPlugins -RunAsName SCAdmin

PS> Get-SmResources -HostName snapctr.demo.netapp.com -PlugInCode SCW

Completed Discovering Resouces: Job Id [193]

HostResource: R:\
StorageSystemResource: svm1:/vol/SC_REPO_BKUP/LUN101

PS> Protect-SmRepository -HostName snapctr.demo.corp.com -Path R:\BACKUP -RetentionCount 24 -Schedule @{"ScheduleType"="hourly";"StartTime"="7/26/2019 8:33 AM"}

Successfully protected the SnapCenter respository.


0: Proof we have hosts with good status:

Image: SnapCenter host and DAG hosts with green statuses

1: Shut down SnapCenter services prior to Re-IP:


net stop SnapManagerCoreService

net stop MySQL57


2: Get the IP, change it (here we change from 192.168.0.75 to .175) and update DNS as required:


PS> (Get-NetIPAddress | Where-Object {$_.AddressFamily -eq "IPv4"}) | FT IPAddress,InterfaceAlias,InterfaceIndex -AutoSize

IPAddress    InterfaceAlias   InterfaceIndex
---------    --------------   --------------
192.168.0.75 DEMO             12

PS> New-NetIPAddress -InterfaceIndex 12 -IPAddress 192.168.0.175 -PrefixLength 24

PS> Remove-NetIPAddress -InterfaceIndex 12 -IPAddress 192.168.0.75 -PrefixLength 24


3: Restart SnapCenter server after re-IP

4: Verify that SnapCenter Plug-In Hosts DNS has updated:


Pinging snapctr.demo.corp.com [192.168.0.75] with 32 bytes of data:

Reply from 192.168.0.175: bytes=32 time<1ms ttl="128<o:p">


5: Verify SnapCenter Plug-In hosts connectivity is good, and verify that backup jobs are running as expected (except can’t backup hosts caught up in the DR situation)

APPENDIX: SnapCenter Config Files

The IP Address is not hardcoded into any of the below.

C:\Program Files\NetApp\SMCore
SMCoreServiceHost.exe.Config

C:\Program Files\NetApp\SnapCenter\SnapCenter Plug-in for Microsoft Exchange Server
SCEServiceHost.exe.config

C:\Program Files\NetApp\SnapCenter\SnapCenter Plug-in for Microsoft Windows
SnapDriveServices.exe.config
SnapDrive.Nsf.Common.Logging.dll.config
IConfiguration.config

C:\Program Files\NetApp\SnapCenter WebApp
packages.config
Web.config

Thursday, 25 July 2019

Making a SnapMirror Vault Tertiary Copy, Read-Write and Primary: Part 2 of 2: Clustershell Demonstration

Image: START and FINISH

How do we make Prod New the new production volume?

STEPS:

1) Modify Live Snapshot Policy with SnapMirror Labels for All Snapshots
2) Create new Live Snapshot Policy with extended retention
3) Modify Vault Policy with required labels for all SnapShots and Pre_Cutover label
4) Create new MirrorVault SnapMirror policy
5) Terminate client access: CIFS share delete, CIFS session check, volume unmount
6) Create Pre_Cutover Snapshot with Pre_Cutover label
7) Update DR, Break DR, verify DR has Snapshot Policy of none, Delete DR, Release DR
8) Update Vault, Delete Vault, Release Vault
9) Rename original Prod volume and offline
10) Update Mirror of Vault, break Mirror of Vault, set volume Snapshot policy, delete mirror of Vault, release mirror of Vault
11) Rename new Prod volume to original Prod volume name (remove _NEW)
12) Volume mount, CIFS share create, CIFS session check
13) Create DR with MirrorVault policy, resync DR
14) Verify SnapShot retention is working correctly
15) Delete Pre_Cutover Snapshot
16) Delete original Prod volume and SV volume

Clustershell Output:

This is going to be a bit long and my time is limited, so the Clustershell output comes with little explanation.

‌‌Volumes in play, their Snapshot Policy and Snapshot Count. Live Snapshot Policy. SnapMirrors and their type and policy. Vault snapmirror policy.

cluster1::> volume show -volume VOL001* -fields snapshot-policy,snapshot-count
vserver volume snapshot-policy snapshot-count
------- ------ --------------- --------------
SVMA    VOL001 7_1min_8_7min   16
SVMA    VOL001_NEW none        104
SVMB    VOL001_DR none         17
SVMC    VOL001_SV none         104

cluster1::> snapshot policy show -policy  7_1min_8_7min
Vserver: cluster1
              Number of Is
Policy Name   Schedules Enabled Comment
------------- --------- ------- -------
7_1min_8_7min         2 true    -
    Schedule Count Prefix SnapMirror Label
    -------- ----- ------ ----------------
    1min         7 1min   1min
    7min         8 7min   7min

cluster1::> snapmirror show -fields type,policy,schedule
source-path destination-path type schedule policy
----------- ---------------- ---- -------- ------------------
SVMA:VOL001 SVMB:VOL001_DR   XDP  1min     MirrorAllSnapshots
SVMA:VOL001 SVMC:VOL001_SV   XDP  7min     104_7min
SVMC:VOL001_SV SVMA:VOL001_NEW XDP 5min    MirrorAllSnapshots

cluster1::> snapmirror policy show -policy 104_7min
Vserver  Policy   Policy Number         Transfer
Name     Name     Type   Of Rules Tries Priority Comment
-------  -------- ------ -------- ----- -------- -------
cluster1 104_7min vault         1     8  normal  -
  SnapMirror Label: 7min       Keep:     104
                         Total Keep:     104


1) Modify Live Snapshot Policy with SnapMirror Labels for All Snapshots:
(Don’t need to do as this is already so!)

2) Create new Live Snapshot Policy with extended retention:


cluster1::> snapshot policy create -policy 7_1min_104_7min -enabled true -schedule1 1min -count1 7 -snapmirror-label1 1min -schedule2 7min -count2 104 -snapmirror-label2 7min

cluster1::> snapshot policy show -policy 7_1min_104_7min
Vserver: cluster1
                Number of Is
Policy Name     Schedules Enabled Comment
--------------- --------- ------- -------
7_1min_104_7min         2 true    -
    Schedule Count Prefix SnapMirror Label
    -------- ----- ------ ----------------
    1min         7 1min   1min
    7min       104 7min   7min


3) Modify Vault Policy with required labels for all SnapShots and Pre_Cutover label:


cluster1::> snapmirror policy add-rule -policy 104_7min -snapmirror-label 1min -keep 7 -vserver cluster1

cluster1::> snapmirror policy add-rule -policy 104_7min -snapmirror-label pre_cutover -keep 1 -vserver cluster1

cluster1::> snapmirror policy show -policy 104_7min
Vserver Policy    Policy Number         Transfer
Name    Name      Type   Of Rules Tries Priority Comment
------- --------- ------ -------- ----- -------- -------
cluster1 104_7min vault         3     8  normal  -
  SnapMirror Label: 7min              Keep:     104
                    1min                          7
                    pre_cutover                   1
                                Total Keep:     112


4) Create new MirrorVault SnapMirror policy:


cluster1::> snapmirror policy create -policy 7_1min_8_7min -vserver cluster1 -type mirror-vault

cluster1::> snapmirror policy add-rule -policy 7_1min_8_7min -vserver cluster1 -snapmirror-label 1min -keep 7

cluster1::> snapmirror policy add-rule -policy 7_1min_8_7min -vserver cluster1 -snapmirror-label 7min -keep 8

cluster1::> snapmirror policy show -policy 7_1min_8_7min
Vserver Policy         Policy Number         Transfer
Name    Name           Type   Of Rules Tries Priority Comment
------- -------------- ------ -------- ----- -------- -------
cluster1 7_1min_8_7min mirror-vault  3     8  normal  -
  SnapMirror Label: sm_created       Keep:       1
                    1min                         7
                    7min                         8
                               Total Keep:      16


5) Terminate client access: CIFS share delete, CIFS session check, volume unmount:


cluster1::> cifs share show -volume VOL001 -instance

              Vserver: SVMA
                Share: VOL001
                 Path: /VOL001_CIFS_volume
     Share Properties: oplocks
                       browsable
                       changenotify
                       show-previous-versions
   Symlink Properties: symlinks
            Share ACL: Everyone / Full Control
          Volume Name: VOL001
        Offline Files: manual
Vscan File-Op Profile: standard

cluster1::> cifs share delete -vserver SVMA -share-name VOL001

cluster1::> vol unmount -vserver SVMA -volume VOL001


Image: Access to CIFS share is terminated!

6) Create Pre_Cutover Snapshot with Pre_Cutover label:

‌‌
cluster1::> vol snapshot create -vserver SVMA -volume VOL001 -snapshot pre_cutover -snapmirror-label pre_cutover


7) Update DR, Break DR, verify DR has Snapshot Policy of none, Delete DR, Release DR:


cluster1::> snapmirror update -destination-path SVMB:VOL001_DR
Operation is queued: snapmirror update of destination "SVMB:VOL001_DR".

cluster1::> snapmirror show -destination-path SVMB:VOL001_DR -fields status
source-path destination-path status
----------- ---------------- ------
SVMA:VOL001 SVMB:VOL001_DR   Idle

cluster1::> snapmirror break -destination-path SVMB:VOL001_DR
Operation succeeded: snapmirror break for destination "SVMB:VOL001_DR".

cluster1::> snapmirror delete -destination-path SVMB:VOL001_DR
Operation succeeded: snapmirror delete for the relationship with destination "SVMB:VOL001_DR".

cluster1::> snapmirror release -source-path SVMA:VOL001 -destination-path SVMB:VOL001_DR
[Job 168] Job succeeded: SnapMirror Release Succeeded


8) Update Vault, Delete Vault, Release Vault:

‌‌‌
cluster1::> snapmirror update -destination-path SVMC:VOL001_SV
Operation is queued: snapmirror update of destination "SVMC:VOL001_SV".

cluster1::> snapmirror show -destination-path SVMC:VOL001_SV -fields status
source-path destination-path status
----------- ---------------- ------
SVMA:VOL001 SVMC:VOL001_SV   Idle

cluster1::> snapmirror delete -destination-path SVMC:VOL001_SV
Operation succeeded: snapmirror delete for the relationship with destination "SVMC:VOL001_SV".

cluster1::> snapmirror release -source-path SVMA:VOL001 -destination-path SVMC:VOL001_SV
[Job 169] Job succeeded: SnapMirror Release Succeeded


9) Rename original Prod volume and offline:


cluster1::> volume rename -vserver SVMA -volume VOL001 -newname VOL001_OLD
[Job 170] Job succeeded: Successful
cluster1::> vol offline  -vserver SVMA -volume VOL001_OLD
Volume "SVMA:VOL001_OLD" is now offline.


10) Update Mirror of Vault, break Mirror of Vault, set volume Snapshot policy, delete mirror of Vault, release mirror of Vault:


cluster1::> snapmirror update -destination-path SVMA:VOL001_NEW
Operation is queued: snapmirror update of destination "SVMA:VOL001_NEW".

cluster1::> snapmirror show -destination-path SVMA:VOL001_NEW -fields status
source-path    destination-path status
-------------- ---------------- ------
SVMC:VOL001_SV SVMA:VOL001_NEW  Idle

cluster1::> snapmirror break -destination-path SVMA:VOL001_NEW
Operation succeeded: snapmirror break for destination "SVMA:VOL001_NEW".

cluster1::> volume modify -vserver SVMA -volume VOL001_NEW -snapshot-policy 7_1min_104_7min
Volume modify successful on volume VOL001_NEW of Vserver SVMA.

cluster1::> snapmirror delete -destination-path SVMA:VOL001_NEW
Operation succeeded: snapmirror delete for the relationship with destination "SVMA:VOL001_NEW".

cluster1::> snapmirror release -source-path SVMC:VOL001_SV -destination-path SVMA:VOL001_NEW
[Job 171] Job succeeded: SnapMirror Release Succeeded


11) Rename new Prod volume to original Prod volume name (remove _NEW)


cluster1::> vol rename -vserver SVMA -volume VOL001_NEW -newname VOL001
[Job 172] Job succeeded: Successful


12) Volume mount, CIFS share create, CIFS session check:


cluster1::> vol mount -vserver SVMA -volume VOL001 -junction-path /VOL001

cluster1::> cifs share create -vserver SVMA -share-name VOL001 -path /VOL001 -share-properties oplocks,browsable,changenotify,show-previous-versions -symlink-properties symlinks -offline-files manual -vscan-fileop-profile standard


Image: Access to CIFS share is restored!

13) Create DR with MirrorVault policy, resync DR:


cluster1::> snapmirror create -source-path SVMA:VOL001 -destination-path SVMB:VOL001_DR -type XDP -policy 7_1min_8_7min -schedule 1min
Operation succeeded: snapmirror create for the relationship with destination "SVMB:VOL001_DR".

cluster1::> snapmirror resync -destination-path SVMB:VOL001_DR

Warning: All data newer than Snapshot copy 7min.2019-07-25_1021 on volume SVMB:VOL001_DR will be deleted.
Do you want to continue? {y|n}: y
Operation is queued: initiate snapmirror resync to destination "SVMB:VOL001_DR".

cluster1::> snapmirror show -destination-path SVMB:VOL001_DR -field state,status,health
source-path destination-path state        status healthy
----------- ---------------- ------------ ------ -------
SVMA:VOL001 SVMB:VOL001_DR   Snapmirrored Idle   true


14) Verify SnapShot retention is working correctly:


cluster1::> snapshot show -volume VOL001 -snapshot 1min.*
...
7 entries were displayed.

cluster1::> snapshot show -volume VOL001 -snapshot 7min.*
...
104 entries were displayed.

cluster1::> snapshot show -volume VOL001_DR -snapshot 1min.*
...
7 entries were displayed.

cluster1::> snapshot show -volume VOL001_DR -snapshot 7min.*
...
8 entries were displayed.


15) Delete Pre_Cutover Snapshot:


cluster1::> snapshot delete -vserver SVMA -volume VOL001 -snapshot pre_cutover

cluster1::> snapshot delete -vserver SVMB -volume VOL001_DR -snapshot pre_cutover


16) Delete original Prod volume and SV volume:


cluster1::> volume delete -vserver SVMA -volume VOL001_OLD
 [Job 174] Job succeeded: Successful

cluster1::> volume offline -vserver SVMC -volume VOL001_SV
Volume "SVMC:VOL001_SV" is now offline.

cluster1::> volume delete -vserver SVMC -volume VOL001_SV
 [Job 176] Job succeeded: Successful


THE END!

Final check:


cluster1::> volume show -volume VOL001* -fields snapshot-policy,snapshot-count
vserver volume snapshot-policy snapshot-count
------- ------ --------------- --------------
SVMA    VOL001 7_1min_104_7min 112
SVMB    VOL001_DR none         17
2 entries were displayed.

cluster1::> snapmirror show -fields type,policy,schedule
source-path destination-path type schedule policy
----------- ---------------- ---- -------- -------------
SVMA:VOL001 SVMB:VOL001_DR   XDP  1min     7_1min_8_7min