CFT Rollback is Nothing to be Afraid Of

I write this post because I was involved in a real-life CFT rollback - and it was totally my fault! We were consolidating some 7-Mode HA-Pairs to a cDOT HA-Pair, and I messed up checking the disk count, so the CFT process failed because it needed to exceed the destinations platform limit for number of storage devices (assigned disks) - the silver-lining is that a pre-check will be added to 7MTT in the future to prevent this. Most CFT’s are 7-Mode HA-Pair to cDOT HA-Pair (1:1) - I’m not sure how many people are aware of the possibilities for reducing controller count with CFT - so the scenario encountered is very rare.

Image: CFT for Consolidating Multiple 7-Mode HA-Pairs to a single C-Mode HA-Pair

CFT Rollback Steps

1) 7MTT: Click “Go to Rollback Dashboard”

If you encounter an issue where you need to rollback CFT, in the ‘Migration Dashboard’ for the CFT Project, you click the “Go to Rollback Dashboard” button.

Note: Call NetApp support first to see if there is a way forward, or simply if you’re stuck and don’t know what to do.

Image: A successful CFT rollback

2) 7MTT: Click “Rollback Prechecks”

This runs through several pre-checks to confirm you’re good to proceed with rollback.

- Resolve any errors in-order-to proceed to the next step

- Review and remediate warnings as required

- Review the informational messages

3) 7MTT: Click “Generate Rollback Steps”

This generates an output of commands that you need to apply to the Cluster via the Clustershell, and 7-Mode shell commands for later. You can save as CSV.

4) Perform Manual Rollback Commands on Target Cluster

Step 1: Revert the Aggregates

Run the Clustershell commands (diag level)::>

storage transition revert start -session-id {SESSION-ID} -node-name {NODENAME} -kernel-version {VERSION} -aggr-trans-revert-attr {A_LONG_STRING!}

Note i: You need the commands from the CFT Rollback Manual Steps output!

Note ii: There’s a revert for every aggregate that was on 7-Mode.

Note iii: You might need to repeat the command multiple times (if the later show-status doesn’t show “revert_complete”) - but if you’ve tried 3+ times and it’s still not “revert_complete” then something is wrong!

Note iv: All the disks that were in the aggregate need to have been assigned to the correct cDOT nodes for this to work.

Note v: If aggregates on cDOT conflict with names of aggregates on the old 7-Mode controllers, rename the pre-existing cDOT aggregates first, then try again.

The storage transition revert command effectively rolls back the aggregates to their 7-Mode pre-cutover state (and restores the disk RAID label to the correct kernel version.)

Step 2: Make sure the rollback operation is successful for all aggregates

Run the Clustershell commands (diag level)::>

storage transition revert show-status -node {NODENAME} -aggregate {AGGREGATE_NAME} -instance

Note: DO NOT PROCEED if any aggregate does not report revert_complete.

Step 3: Reassign disks back to 7-Mode sysids

Run the Clustershell commands (diag level)::>

disk assign -disk {DISK} -s {SYSID} -force true

Note i: You need the commands from the CFT Rollback Manual Steps output!

Note ii: It can take ~10 seconds per disk assign.

Step 4: Verify Disk Assignment

Verify that the disks have been assigned back to the correct 7-Mode controllers’ sysid::>

storage disk show -fields owner-id

Note: DO NOT PROCEED until disk assignment is correct.

Step 5 (Can be done post rollback to 7-Mode): Remove CFT Restrictions

Remove the Copy-Free Transition restrictions on the target cluster nodes by using the diag level command::>

storage transition pre-commit end -session-id {SESSION-ID}

Note: You need the command from the CFT Rollback Manual Steps output!

Step 6 (Can be done post rollback to 7-Mode): Purge Transition Project Information

Purge the transition project information on the target cluster nodes by using the diag level command::>

storage transition purge-info -session-id {SESSION-ID}

Note: You need the command from the CFT Rollback Manual Steps output!

Step 7: (Can be done post rollback to 7-Mode): Manually remove configurations from target SVMs

5) 7MTT: Click “Confirm” under ‘Execute Commands on Target Cluster (Manual Step)’

6) Perform Re-Cabling Tasks

The shelves must be power-cycled. Here I would suggest:

- Power-off all the 7-Mode shelves

- Re-cable back to the 7-Mode controllers

- Power-on all the 7-Mode shelves

- Wait a few minutes for the shelves to initialize.

7) 7MTT: Click “Confirm” under ‘Cabling (Manual Step)‘

8) Power on 7-Mode Controllers

Power on the 7-Mode controllers.

The 7-Mode controllers will boot to the Boot Menu.

Select (1) Normal Boot.

Image: Normal Boot

When both controllers have booted, on one of the 7-Mode controllers, enable cf, and verify status>

cf enable

cf status

9) 7MTT: Click “Confirm” under ‘Execute Commands on 7-Mode Controllers (Manual Step)’

10) 7MTT: Click “Verify 7-Mode” under ‘Complete Rollback’

Nothing to Be Afraid Of?

This might look a fair few steps, 7MTT packages everything very nicely though (this was version 3.1). 7MTT makes it very straightforward - simple buttons to press, and the “Generate Rollback Steps” output gives you everything you need for a successful rollback (which hopefully you’ll never need to do).