Wednesday, 23 October 2019

Investigating Renaming Snapshots that Fail to be Renamed by SnapCenter (SnapCenter for Exchange)

I’ve been involved with a SnapCenter issue - in conjunction with SnapCenter for Exchange - where SnapCenter will fail to rename the VSS snapshot to the human readable name. This is a big SnapCenter for Exchange environment and it seems like the rename API never actually gets to the cluster (or is dropped). I understand that the reason behind this is that, say 2 rename APIs are sent in the same millisecond, one might get dropped (ZAPIs are low priority, so if the storage is busy doing something else, it drops the ZAPI from the queue.)

Looking at this problem from the perspective of “can/should we do a workaround to rename the snapshots”, I reached the conclusion that there’s no benefit to doing so.

Firstly, to check the Audit log in ONTAP 9+, you can use the clustershell command:

security audit log show


In the audit log you’ll see if the snapshot-rename requests get to the ONTAP cluster (in this case they don’t.)
In the SCW_XXX.log {where XXX is an incrementing number - newer jobs have higher numbers} you’ll see something like the below (from a lab system) - if the rename works:


2019-10-21T03:20:24.8681832-07:00 Verbose SAL PID=[10720] TID=[13720] RenameSnapshot: END: Rename-NcSnapshot -Volume vol_N1Log1 -Snapshot {24171b8f-d400-4664-8e7e-f2c8b04b1fbe} -NewName SCE_10-21-2019_03.20.06.3269
2019-10-21T03:20:24.8681832-07:00 Verbose SAL PID=[10720] TID=[13720] ++RenameSnapshotExtensions::GenerateRenameSnapshotResult
2019-10-21T03:20:24.8681832-07:00 Verbose SAL PID=[10720] TID=[9272] --PSHelper::RunCommand
2019-10-21T03:20:24.8681832-07:00 Verbose SAL PID=[10720] TID=[13720] Rename Snapshot {24171b8f-d400-4664-8e7e-f2c8b04b1fbe} on vol_N1Log1 is successful.


If the snapshot fails to rename there is an error something like (sorry, I don’t have a good example):

Activity 'Renaming VSS Snapshot' failed with error: Skip renaming snapshot.

So, you might think - like I initially did - that it would be easy to write a post-backup script to rename the snapshots. But, no. It’s a bit tricky because the SCW_XXX.log is not on the SnapCenter server, but on the server with the SCW plugin (or - in this case - the Exchange Mailbox server). So, if you wanted to rename the snapshots, you’d have to parse the log from every involved server in your backup (if it’s a large Exchange DAG, that can be a fair few servers and they might be large logs). And, even then, once you’ve renamed the snapshots to their human readable name (the name SnapCenter tried to rename them to), these are still no good for automated restore purposes using SnapCenter as not entered into the SnapCenter database (you can still use the snapshots for manual restore - see this link below SMBR 8 with SnapCenter - how to restore from secondary).

Having a script running in the background (say once a day) to tidy up the VSS named snapshots - that won’t get automatically deleted - when the desired retention expires, makes sense.

SnapCenter Logs Locations

Note: Most of the logs you’ll find on the servers running the SnapCenter plugins, and not the SnapCenter server itself.

On the SnapCenter Server:

C:\Program Files\NetApp\SMCore\job_logs
C:\Program Files\NetApp\SMCore\log

On the Exchange Server running the SnapCenter Plug-in for Windows (most of the logs will be here):

C:\Program Files\NetApp\SnapCenter\SMCore\job_logs
C:\Program Files\NetApp\SnapCenter\SMCore\log
C:\Program Files\NetApp\SnapCenter\SnapCenter Plug-in for Microsoft Exchange Server\job_logs
C:\Program Files\NetApp\SnapCenter\SnapCenter Plug-in for Microsoft Exchange Server\log
C:\Program Files\NetApp\SnapCenter\SnapCenter Plug-in for Microsoft Windows\job_logs

Image: SCW_XXX.log in SMCore\job_logs

Image: SMCore_XXX.log in SMCore\log

Image: SCE_XXX.log in SnapCenter Plug-in for Microsoft Exchange Server\job_logs

Image: SnapMgrContracts_XXX.log in SnapCenter Plug-in for Microsoft Exchange Server\log

Image: SCW_XXX.log in SnapCenter Plug-in for Windows\job_logs

No comments:

Post a Comment