Sunday, 6 May 2018

System Node Migrate-Root - Experience and Tips

I needed to test out ‘system node migrate-root’ in preparation for potentially using it prior to performing a headswap. No physical hardware here just an ONTAP 9.3 HA Simulator lab. These are some observations with a couple of tips.

Documentation

System Node Migrate-Root is documented here:

The syntax is very simple, for example:


cluster1::*>  system node migrate-root -node cluster1-01 -disklist VMw-1.17,VMw-1.18,VMw-1.19 -raid-type raid_dp

Warning: This operation will create a new root aggregate and replace the existing root on the node "cluster1-01". The existing root aggregate will be discarded.
Do you want to continue? {y|n}: y

Info: Started migrate-root job. Run "job show -id 86 -instance" command to
      check the progress of the job.


The process (see below) starts straight away.

The official documentation mentions:

The command starts a job that backs up the node configuration, creates a new aggregate, set it as new root aggregate, restores the node configuration and restores the names of original aggregate and volume. The job might take as long as a few hours depending on time it takes for zeroing the disks, rebooting the node and restoring the node configuration.

The Process and Timings

The SIM has tiny disks of 10.66GB and 28.44GB usable size. The 10.66GB disks were originally used for 3 disk root aggregates, and the migrate-root moved the root aggregate to the slightly larger virtual disks. On a physical system with much bigger than 28.44GB disks, I would expect the timings to be considerably longer than the below. The below timings are taken from acquiring the ‘Execution Progress’ string - from the job show output - every second.

0-27 seconds: Starting node configuration backup. This might take several minutes.
28-146 seconds: Starting aggregate relocation on the node "cluster1-02"
147-212 seconds: Rebooting the node to create a new root aggregate.
213-564 seconds: Waiting for the node to create a new root and come online. This might take a few minutes.
565-682 seconds: Making the old root aggregate online.
683-686 seconds: Copying contents from old root volume to new root volume.
687-864 seconds: Starting removal of old aggregate and volume and renaming the new root.
865-1653 seconds: Starting node configuration restore.
1654-1772 seconds: Enabling HA and relocating the aggregates. This might take a few minutes.
1773 seconds: Complete: Root aggregate migration successfully completed [0]

Nearly 30 minutes for migrate-root on one node of a tiny ONTAP 9.3RC1 HA SIM! And you still need to do a takeover/giveback of the node whose root aggregate was moved (see below).

Tips

1) The process disables HA and Storage Failover, and Aggregate Relocation is used to move the data aggregates to the node that’s staying up. The process does not move data LIFs, these will failover automatically, but I noticed a bit of a delay (my test CIFS share was down for 45 seconds), so I’d recommend moving data LIFs onto the node that’s going to stay up first.

2) I noticed - consistently - that if you run ‘system health alert show’ after the migrate-root completes, you get some weird output. Perform a takeover/giveback of the affected node after the migrate-root completes to correct this.


cluster1::*> system health alert show
This table is currently empty.

Warning: Unable to list entries for schm on node "cluster1-02": RPC: Remote
         system error [from mgwd on node "cluster1-02" (VSID: -1) to schm at
         127.0.0.1].
         Unable to list entries for shm on node "cluster1-02": RPC: Remote
         system error [from mgwd on node "cluster1-02" (VSID: -1) to shm at
         127.0.0.1].
         Unable to list entries for cphm on node "cluster1-02": RPC: Remote
         system error [from mgwd on node "cluster1-02" (VSID: -1) to cphm at
         127.0.0.1].
         Unable to list entries for cshm on node "cluster1-02": RPC: Remote
         system error [from mgwd on node "cluster1-02" (VSID: -1) to cshm at
         127.0.0.1].

cluster1::*> Replaying takeover WAFL log
May 06 14:01:04 [cluster1-01:monitor.globalStatus.critical:EMERGENCY]: This node has taken over cluster1-02.

cluster1::*> system health alert show
This table is currently empty.

cluster1::*>


Image: Remote system error [from mgwd on node...

PowerShell to Record Job Execution Progress Per Second

Sometimes when you’re doing stuff with NetApp ONTAP, you’ll get given a job-id and you’ll be curious to know all the phases the job goes through. So, I wrote this little program to find out just that. In the image - down the bottom - is my use case (which will be the subject of the next blog post.)


############################
## RecordJobExecution.ps1 ##
############################

Import-Module DataONTAP
$C = Read-Host "Enter Cluster FQDN/IP"
[Void](Connect-NcController $C)
$J = Read-Host "Enter Job ID"
while($TRUE){
  $D = Get-Date -uformat %Y%m%d
  $T = Get-Date -uFormat %T
  $P = (Get-NcJob -id $J).JobProgress
  "$D $T $P" >> execution_log.txt
  sleep 1
}


Image: Use Case: Recording per second the job execution progress of system node migrate-root

Saturday, 5 May 2018

One-Liner PowerShell to Record/Log CIFS Share Access Availability

I like one liners where you don’t need a PowerShell script/PS1 file. It might be a bit of a cheat using semi-colons, but you can still paste it into your PowerShell and run it as a one liner.

This one liner can be used to test availability of a CIFS share. Change the path variable as required.

while($TRUE){$T1 = Get-Date; $P = Test-Path "\\192.168.0.120\vol1"; $T2 = Get-Date; $TS = (New-TimeSpan -Start $T1 -End $T2).Seconds; If($TS -gt 1){"DOWN for $TS seconds" >> log.txt}; $D = Get-Date -uformat %Y%m%d; $T = Get-Date -uFormat %T; "$D $T $P" >> log.txt; sleep 1}

It will run until you Ctrl+C to exit. This is what it does:

- Gets a time T1
- Tests the path to the CIFS share (Test-Path has a long timeout)
- Gets a time T2
- If T2 minus T1 is greater than 1 second we log “DOWN for $TS seconds”
- Finally, we log a formatted date with the result of Test-Path
- Sleep for 1 second and repeat

Image: Screenshot of the log showing DOWN caused by intentionally downing the data LIF (IP) in NetApp ONTAP clustershell

Sunday, 29 April 2018

Recovering ARL Headswap Out-of-Quorum (Missing/Unmapped Cluster Ports)

I’ve done a few ARL headswaps and always worked out a strategy to ensure at least one cluster port maps correctly to the new head (example: FAS3220 to FAS6220 ARL headswap by moving one of the FAS3220’s 10 GbE cards to slot 3 and moving cluster ports onto that, then taking the card across to the FAS6220 and slot 3, and post-ARL correcting the cluster ports to follow best practice). I was wondering what I’d do if I absolutely couldn’t come up with a cunning plan to map a cluster port from old controller to new controller. And using a 2-node SIM cluster, I demonstrate what I’d do.

Note: This is of course all unofficial stuff. On production systems, only NetApp support/personnel with a valid support case should be using commands to manipulate the CDB like we do here. Only the absolute minimum required amount of CDB modification is done to get the node back into quorum.

To demonstrate this, I have a 2-node cluster (C91) with nodes C91-01 and C91-02. What I do is ensure epsilon is on node 2 (C91-02), then I shutdown C91-01 and remove the two ports (network adapter 5 and 6 which map to e0e and e0f) used for cluster ports.

Image: Removing cluster ports e0e and e0f from the simulator

1) Gathering a few outputs and halting node 1


C91::*> version

NetApp Release 9.1: Thu Dec 22 23:05:58 UTC 2016

C91::*> cluster show

Node   Health  Eligibility Epsilon
------ ------- ----------- -------
C91-01 true    true        false
C91-02 true    true        true

C91::*> net int show -role cluster

Logical    Status     Network          Current Current Is
Interface  Admin/Oper Address/Mask     Node    Port    Home
---------- ---------- ---------------- ------- ------- ----
C91-01_clus1 up/up    169.254.94.74/16 C91-01  e0e     true
C91-01_clus2 up/up    169.254.94.84/16 C91-01  e0f     true
C91-02_clus1 up/up    169.254.47.70/16 C91-02  e0e     true
C91-02_clus2 up/up    169.254.47.80/16 C91-02  e0f     true

C91::*> net port show -role cluster

Node: C91-01
                                 Speed(Mbps) Health
Port IPspace Broadcast Link MTU  Admin/Oper  Status
---- ------- --------- ---- ---- ----------- -------
e0e  Cluster Cluster   up   1500 auto/1000   healthy
e0f  Cluster Cluster   up   1500 auto/1000   healthy

Node: C91-02
                                 Speed(Mbps) Health
Port IPspace Broadcast Link MTU  Admin/Oper  Status
---- ------- --------- ---- ---- ----------- -------
e0e  Cluster Cluster   up   1500 auto/1000   healthy
e0f  Cluster Cluster   up   1500 auto/1000   healthy

C91::*> halt -node C91-01


2) Remove network adapter 5 and 6 from the simulator

3) Power up node 1

4) Check cluster quorum

Notice that node 1 (C91-01) is out-of-quorum (cluster health = false).


C91::*> node show local -fields node
node
------
C91-01

C91::*> cluster show
Node   Health  Eligibility Epsilon
------ ------- ----------- -------
C91-01 false   true        false
C91-02 false   true        true


Notice that node 2 (C91-02) is in quorum (cluster health = true).


C91::*> node show local -fields node
node
------
C91-02

C91::*> cluster show
Node   Health  Eligibility Epsilon
------ ------- ----------- -------
C91-01 false   true        false
C91-02 true    true        true


5) Fix the problem

We modify ports e0a and e0b to be cluster ports.
Then we modify the cluster LIFs to be on e0a and e0b.
Finally we reboot node 1.


C91::*> net port show -role cluster

There are no entries matching your query.

C91::*> net int show -role cluster

Logical    Status     Network          Current Current Is
Interface  Admin/Oper Address/Mask     Node    Port    Home
---------- ---------- ---------------- ------- ------- ----
C91-01_clus1 up/down  169.254.94.74/16 C91-01  e0e     true
C91-01_clus2 up/down  169.254.94.84/16 C91-01  e0f     true

C91::*> broadcast-domain show

Error: show failed: Cannot run this command because the system is not fully initialized. Wait a few minutes, and then try the command again.

C91::*> set diag

C91::*> network ipspace cdb show

IPspace ID
------- -------
Cluster
        4294967294
Default
        4294967295

C91::*> network port cdb show

                           Auto-Neg Duplex Speed Flowcontrol
Node   Port Role      MTU  Admin    Admin  Admin Admin
------ ---- --------- ---- -------- ------ ----- -----------
C91-01
       e0a  data      1500 true     auto   auto  full
       e0b  data      1500 true     auto   auto  full
       e0c  node-mgmt 1500 true     auto   auto  full
       e0d  data      1500 true     auto   auto  full

Warning: Unable to list entries on node C91-02. RPC: Couldn't make connection

C91::*> network interface cdb show

                     Status Network                   Valid
Node   ID    Name    Admin  Address       Netmask     Id
------ ----- ------- ------ ------------- ----------- -----
C91-01
       1023  C91-01_ up     169.254.94.84 255.255.0.0 true
             clus2
       1024  C91-01_ up     169.254.94.74 255.255.0.0 true
             clus1

Warning: Unable to list entries on node C91-02. RPC: Couldn't make connection

C91::*> net port cdb modify -port e0a -node C91-01 -role cluster -mtu 1500 -flowcontrol-admin none -ipspace-id 4294967294
C91::*> net port cdb modify -port e0b -node C91-01 -role cluster -mtu 1500 -flowcontrol-admin none -ipspace-id 4294967294

C91::*> net port cdb show

                           Auto-Neg Duplex Speed Flowcontrol
Node   Port Role      MTU  Admin    Admin  Admin Admin
------ ---- --------- ---- -------- ------ ----- -----------
C91-01
       e0a  cluster   1500 true     auto   auto  none
       e0b  cluster   1500 true     auto   auto  none
       e0c  node-mgmt 1500 true     auto   auto  full
       e0d  data      1500 true     auto   auto  full

Warning: Unable to list entries on node C91-02. RPC: Couldn't make connection

C91::*> net int cdb show

                     Status Network                   Valid
Node   ID    Name    Admin  Address       Netmask     Id
------ ----- ------- ------ ------------- ----------- -----
C91-01
       1023  C91-01_ up     169.254.94.84 255.255.0.0 true
             clus2
       1024  C91-01_ up     169.254.94.74 255.255.0.0 true
             clus1

Warning: Unable to list entries on node C91-02. RPC: Couldn't make connection

C91::*> net int cdb modify -lif-id 1024 -node C91-01 -home-port e0a -home-node C91-01 -curr-port e0a -curr-node C91-01
C91::*> net int cdb modify -lif-id 1023 -node C91-01 -home-port e0b -home-node C91-01 -curr-port e0b -curr-node C91-01

C91::*> net int show -role cluster

Logical    Status     Network          Current Current Is
Interface  Admin/Oper Address/Mask     Node    Port    Home
---------- ---------- ---------------- ------- ------- ----
C91-01_clus1 up/down  169.254.94.74/16 C91-01  e0a     true
C91-01_clus2 up/down  169.254.94.84/16 C91-01  e0b     true

C91::*> node show local -fields node
node
------
C91-01

C91::*> reboot local


Notice above that even when we’ve got the cluster LIFs on the correct port, they are still marked as operationally down; this is why we have to do the reboot so the CDB can reload correctly.

6) Check everything is okay


C91::*> net int show -role cluster

Logical    Status     Network          Current Current Is
Interface  Admin/Oper Address/Mask     Node    Port    Home
---------- ---------- ---------------- ------- ------- ----
C91-01_clus1 up/up    169.254.94.74/16 C91-01  e0b     true
C91-01_clus2 up/up    169.254.94.84/16 C91-01  e0a     true
C91-02_clus1 up/up    169.254.47.70/16 C91-02  e0e     true
C91-02_clus2 up/up    169.254.47.80/16 C91-02  e0f     true

C91::*> cluster show

Node   Health Eligibility Epsilon
------ ------ ----------- -------
C91-01 true   true        false
C91-02 true   true        true

C91::*> network port broadcast-domain show -ipspace Cluster

IPspace Broadcast                   Update
Name    Domain Name MTU  Port List  Status
------- ----------- ---- ---------- ------
Cluster Cluster     1500
                         C91-01:e0a complete
                         C91-01:e0b complete
                         C91-02:e0e complete
                         C91-02:e0f complete


THE END

How to Convert to a 2-node Switchless Cluster

After spending a little bit of time with the official process to convert a NetApp (Clustered) ONTAP cluster to switchless cluster -

Transitioning to a two-node switchless cluster

- and encountering strange issues (cluster unexpectedly going out-of-quorum), I decided to do it my way (which is better - quicker and simpler - by the way). This is a very simple 9-step step-by-step process.

Image: Starting point: Using 2 cluster switches

1) Cluster checks

Verify the cluster is healthy from the outputs of the below. Cluster LIFs should be on their home ports and should be able to auto-revert. The cluster should be healthy and ping-cluster should be all good. Check discovered-devices/CDPD to ensure the cluster cabling is correct.


set -priv advanced
network options switchless-cluster show
network interface show -role cluster
network interface show -role cluster -fields auto-revert
network port show -role cluster
cluster show
cluster ping-cluster -node {CLUSTER-01}
network device-discovery show


2) Send ASUPs


system node autosupport invoke -node * -type all -message "MAINT=1h BEGIN: Convert to switchless"


3) Wait for the ASUPs to go


system node autosupport history show -node CLUSTER-01
system node autosupport history show -node CLUSTER-02


4) Enable switchless-cluster


network options switchless-cluster show
network options switchless-cluster modify -enabled true
network options switchless-cluster show


Note: All this really does is instruct the cluster to only check cluster communication down the direct connect paths, so disabling the full-mesh checking which is only possible with cluster switches.

5) Un-cable cluster ports from cluster switch 1 and direct connect

Image: Using a direct connect cable and Cluster Switch 2

6) Cluster checks

Verify the cluster is healthy. Verify the cluster ports are healthy. Wait for the Cluster LIFs to auto-revert. After the cluster LIFs auto-revert, verify the cluster is healthy. Finally check cluster communication with ping-cluster.


cluster show
network port show -role cluster
network interface show -role cluster
... WAIT for Cluster LIFs to auto-revert ...
network interface show -role cluster
cluster show
cluster ping-cluster -node {CLUSTER-01}


7) Un-cable cluster ports from cluster switch 2 and direct connect

Image: Using two direct connect cables

8) Cluster checks

Verify the cluster is healthy. Verify the cluster ports are healthy. Wait for the Cluster LIFs to auto-revert. After the cluster LIFs auto-revert, verify the cluster is healthy. Finally check cluster communication with ping-cluster.


cluster show
network port show -role cluster
network interface show -role cluster
... WAIT for Cluster LIFs to auto-revert ...
network interface show -role cluster
cluster show
cluster ping-cluster -node {CLUSTER-01}


9) Send ASUPs


system node autosupport invoke -node * -type all -message "MAINT=END FINISHED: Convert to switchless"


THE END