Sunday, 29 April 2018

Recovering ARL Headswap Out-of-Quorum (Missing/Unmapped Cluster Ports)

I’ve done a few ARL headswaps and always worked out a strategy to ensure at least one cluster port maps correctly to the new head (example: FAS3220 to FAS6220 ARL headswap by moving one of the FAS3220’s 10 GbE cards to slot 3 and moving cluster ports onto that, then taking the card across to the FAS6220 and slot 3, and post-ARL correcting the cluster ports to follow best practice). I was wondering what I’d do if I absolutely couldn’t come up with a cunning plan to map a cluster port from old controller to new controller. And using a 2-node SIM cluster, I demonstrate what I’d do.

Note: This is of course all unofficial stuff. On production systems, only NetApp support/personnel with a valid support case should be using commands to manipulate the CDB like we do here. Only the absolute minimum required amount of CDB modification is done to get the node back into quorum.

To demonstrate this, I have a 2-node cluster (C91) with nodes C91-01 and C91-02. What I do is ensure epsilon is on node 2 (C91-02), then I shutdown C91-01 and remove the two ports (network adapter 5 and 6 which map to e0e and e0f) used for cluster ports.

Image: Removing cluster ports e0e and e0f from the simulator

1) Gathering a few outputs and halting node 1


C91::*> version

NetApp Release 9.1: Thu Dec 22 23:05:58 UTC 2016

C91::*> cluster show

Node   Health  Eligibility Epsilon
------ ------- ----------- -------
C91-01 true    true        false
C91-02 true    true        true

C91::*> net int show -role cluster

Logical    Status     Network          Current Current Is
Interface  Admin/Oper Address/Mask     Node    Port    Home
---------- ---------- ---------------- ------- ------- ----
C91-01_clus1 up/up    169.254.94.74/16 C91-01  e0e     true
C91-01_clus2 up/up    169.254.94.84/16 C91-01  e0f     true
C91-02_clus1 up/up    169.254.47.70/16 C91-02  e0e     true
C91-02_clus2 up/up    169.254.47.80/16 C91-02  e0f     true

C91::*> net port show -role cluster

Node: C91-01
                                 Speed(Mbps) Health
Port IPspace Broadcast Link MTU  Admin/Oper  Status
---- ------- --------- ---- ---- ----------- -------
e0e  Cluster Cluster   up   1500 auto/1000   healthy
e0f  Cluster Cluster   up   1500 auto/1000   healthy

Node: C91-02
                                 Speed(Mbps) Health
Port IPspace Broadcast Link MTU  Admin/Oper  Status
---- ------- --------- ---- ---- ----------- -------
e0e  Cluster Cluster   up   1500 auto/1000   healthy
e0f  Cluster Cluster   up   1500 auto/1000   healthy

C91::*> halt -node C91-01


2) Remove network adapter 5 and 6 from the simulator

3) Power up node 1

4) Check cluster quorum

Notice that node 1 (C91-01) is out-of-quorum (cluster health = false).


C91::*> node show local -fields node
node
------
C91-01

C91::*> cluster show
Node   Health  Eligibility Epsilon
------ ------- ----------- -------
C91-01 false   true        false
C91-02 false   true        true


Notice that node 2 (C91-02) is in quorum (cluster health = true).


C91::*> node show local -fields node
node
------
C91-02

C91::*> cluster show
Node   Health  Eligibility Epsilon
------ ------- ----------- -------
C91-01 false   true        false
C91-02 true    true        true


5) Fix the problem

We modify ports e0a and e0b to be cluster ports.
Then we modify the cluster LIFs to be on e0a and e0b.
Finally we reboot node 1.


C91::*> net port show -role cluster

There are no entries matching your query.

C91::*> net int show -role cluster

Logical    Status     Network          Current Current Is
Interface  Admin/Oper Address/Mask     Node    Port    Home
---------- ---------- ---------------- ------- ------- ----
C91-01_clus1 up/down  169.254.94.74/16 C91-01  e0e     true
C91-01_clus2 up/down  169.254.94.84/16 C91-01  e0f     true

C91::*> broadcast-domain show

Error: show failed: Cannot run this command because the system is not fully initialized. Wait a few minutes, and then try the command again.

C91::*> set diag

C91::*> network ipspace cdb show

IPspace ID
------- -------
Cluster
        4294967294
Default
        4294967295

C91::*> network port cdb show

                           Auto-Neg Duplex Speed Flowcontrol
Node   Port Role      MTU  Admin    Admin  Admin Admin
------ ---- --------- ---- -------- ------ ----- -----------
C91-01
       e0a  data      1500 true     auto   auto  full
       e0b  data      1500 true     auto   auto  full
       e0c  node-mgmt 1500 true     auto   auto  full
       e0d  data      1500 true     auto   auto  full

Warning: Unable to list entries on node C91-02. RPC: Couldn't make connection

C91::*> network interface cdb show

                     Status Network                   Valid
Node   ID    Name    Admin  Address       Netmask     Id
------ ----- ------- ------ ------------- ----------- -----
C91-01
       1023  C91-01_ up     169.254.94.84 255.255.0.0 true
             clus2
       1024  C91-01_ up     169.254.94.74 255.255.0.0 true
             clus1

Warning: Unable to list entries on node C91-02. RPC: Couldn't make connection

C91::*> net port cdb modify -port e0a -node C91-01 -role cluster -mtu 1500 -flowcontrol-admin none -ipspace-id 4294967294
C91::*> net port cdb modify -port e0b -node C91-01 -role cluster -mtu 1500 -flowcontrol-admin none -ipspace-id 4294967294

C91::*> net port cdb show

                           Auto-Neg Duplex Speed Flowcontrol
Node   Port Role      MTU  Admin    Admin  Admin Admin
------ ---- --------- ---- -------- ------ ----- -----------
C91-01
       e0a  cluster   1500 true     auto   auto  none
       e0b  cluster   1500 true     auto   auto  none
       e0c  node-mgmt 1500 true     auto   auto  full
       e0d  data      1500 true     auto   auto  full

Warning: Unable to list entries on node C91-02. RPC: Couldn't make connection

C91::*> net int cdb show

                     Status Network                   Valid
Node   ID    Name    Admin  Address       Netmask     Id
------ ----- ------- ------ ------------- ----------- -----
C91-01
       1023  C91-01_ up     169.254.94.84 255.255.0.0 true
             clus2
       1024  C91-01_ up     169.254.94.74 255.255.0.0 true
             clus1

Warning: Unable to list entries on node C91-02. RPC: Couldn't make connection

C91::*> net int cdb modify -lif-id 1024 -node C91-01 -home-port e0a -home-node C91-01 -curr-port e0a -curr-node C91-01
C91::*> net int cdb modify -lif-id 1023 -node C91-01 -home-port e0b -home-node C91-01 -curr-port e0b -curr-node C91-01

C91::*> net int show -role cluster

Logical    Status     Network          Current Current Is
Interface  Admin/Oper Address/Mask     Node    Port    Home
---------- ---------- ---------------- ------- ------- ----
C91-01_clus1 up/down  169.254.94.74/16 C91-01  e0a     true
C91-01_clus2 up/down  169.254.94.84/16 C91-01  e0b     true

C91::*> node show local -fields node
node
------
C91-01

C91::*> reboot local


Notice above that even when we’ve got the cluster LIFs on the correct port, they are still marked as operationally down; this is why we have to do the reboot so the CDB can reload correctly.

6) Check everything is okay


C91::*> net int show -role cluster

Logical    Status     Network          Current Current Is
Interface  Admin/Oper Address/Mask     Node    Port    Home
---------- ---------- ---------------- ------- ------- ----
C91-01_clus1 up/up    169.254.94.74/16 C91-01  e0b     true
C91-01_clus2 up/up    169.254.94.84/16 C91-01  e0a     true
C91-02_clus1 up/up    169.254.47.70/16 C91-02  e0e     true
C91-02_clus2 up/up    169.254.47.80/16 C91-02  e0f     true

C91::*> cluster show

Node   Health Eligibility Epsilon
------ ------ ----------- -------
C91-01 true   true        false
C91-02 true   true        true

C91::*> network port broadcast-domain show -ipspace Cluster

IPspace Broadcast                   Update
Name    Domain Name MTU  Port List  Status
------- ----------- ---- ---------- ------
Cluster Cluster     1500
                         C91-01:e0a complete
                         C91-01:e0b complete
                         C91-02:e0e complete
                         C91-02:e0f complete


THE END

How to Convert to a 2-node Switchless Cluster

After spending a little bit of time with the official process to convert a NetApp (Clustered) ONTAP cluster to switchless cluster -

Transitioning to a two-node switchless cluster

- and encountering strange issues (cluster unexpectedly going out-of-quorum), I decided to do it my way (which is better - quicker and simpler - by the way). This is a very simple 9-step step-by-step process.

Image: Starting point: Using 2 cluster switches

1) Cluster checks

Verify the cluster is healthy from the outputs of the below. Cluster LIFs should be on their home ports and should be able to auto-revert. The cluster should be healthy and ping-cluster should be all good. Check discovered-devices/CDPD to ensure the cluster cabling is correct.


set -priv advanced
network options switchless-cluster show
network interface show -role cluster
network interface show -role cluster -fields auto-revert
network port show -role cluster
cluster show
cluster ping-cluster -node {CLUSTER-01}
network device-discovery show


2) Send ASUPs


system node autosupport invoke -node * -type all -message "MAINT=1h BEGIN: Convert to switchless"


3) Wait for the ASUPs to go


system node autosupport history show -node CLUSTER-01
system node autosupport history show -node CLUSTER-02


4) Enable switchless-cluster


network options switchless-cluster show
network options switchless-cluster modify -enabled true
network options switchless-cluster show


Note: All this really does is instruct the cluster to only check cluster communication down the direct connect paths, so disabling the full-mesh checking which is only possible with cluster switches.

5) Un-cable cluster ports from cluster switch 1 and direct connect

Image: Using a direct connect cable and Cluster Switch 2

6) Cluster checks

Verify the cluster is healthy. Verify the cluster ports are healthy. Wait for the Cluster LIFs to auto-revert. After the cluster LIFs auto-revert, verify the cluster is healthy. Finally check cluster communication with ping-cluster.


cluster show
network port show -role cluster
network interface show -role cluster
... WAIT for Cluster LIFs to auto-revert ...
network interface show -role cluster
cluster show
cluster ping-cluster -node {CLUSTER-01}


7) Un-cable cluster ports from cluster switch 2 and direct connect

Image: Using two direct connect cables

8) Cluster checks

Verify the cluster is healthy. Verify the cluster ports are healthy. Wait for the Cluster LIFs to auto-revert. After the cluster LIFs auto-revert, verify the cluster is healthy. Finally check cluster communication with ping-cluster.


cluster show
network port show -role cluster
network interface show -role cluster
... WAIT for Cluster LIFs to auto-revert ...
network interface show -role cluster
cluster show
cluster ping-cluster -node {CLUSTER-01}


9) Send ASUPs


system node autosupport invoke -node * -type all -message "MAINT=END FINISHED: Convert to switchless"


THE END

Tuesday, 24 April 2018

Using PowerShell to Create a Correctly Ordered USB Music Key

I have a car which has an Alpine CDE-181RR stereo system. This stereo has a USB slot which you can either use for charging your phone up, or plug in a USB Key full of music. My usage case for this blog post is the latter. I use a 4GB SanDisk thumb drive which is so tiny you barely notice it. The problem with copying music to the USB thumb drive, is that the stereo treats it like a CD i.e. what you copy first is first track in the music list, and what you copy last is the last song.

Image: Alpine CDE-181RR car stereo

Image: SanDisk thumb drive

So, I thought, why not use PowerShell to construct my USB Music Key like a CD burn.
The PowerShell script is below, copy it into a text editor and save as say USB_Music_Key.ps1.
The first thing we need is a correctly formatted CSV with headers:

Include = Put ‘Y’ in the column if you want to include the audio media file
Source Path = Source directory path (need to put ‘\’ on the end)
Source File Name = Source audio media file name
Destination Path = Destination directory path (will be the USB drive, and needs ‘\’ on the end)
Destination File Name = Destination save file name

Once you’ve constructed your CSV file, run the PowerShell script as>

.\USB_Music_Key.ps1 -CSVFilePath {FILENAME/PATH}

This will tell you how much capacity is required on the destination.
Then run as>

.\USB_Music_Key.ps1 -CSVFilePath {FILENAME/PATH} -CalculateOff

- to construct your USB Music Key.

Image: Using USB_Music_Key.ps1

Image: Input CSV

Image: Completed “burnt” USB Music Key (notice capacity is spot on)

Note: I’ve not actually tested this in my car yet. I will update if the order of tracks is not as expected.

The PowerShell Script


<#
.SYNOPSIS
  Creates a correctly ordered USB Music Key.

.DESCRIPTION
  Simple program to copy music files from source to USB key.

.INPUTS
  -CSVFilePath CSVFilePath
    Use this switch to specify where to find the CSV file.
  -CalculateOff
    Use this switch to make the program copy data.
    (Default behaviour is just to calculate how much data will be copied.)
 
  A correctly formatted CSV.
  CSV HEADER 1 = "Include"
  CSV HEADER 1 = "Source Path"
  CSV HEADER 2 = "Source File Name"
  CSV HEADER 3 = "Destination Path"
  CSV HEADER 4 = "Destination File Name"
  Note: Use (Get-ChildItem SOURCE_PATH).Name to obtain list from source path.
 
.OUTPUTS
  Copied data to USB key.

.NOTES
  Creation Date: April 23, 2018
#>

Param(
  [Parameter(Mandatory=$true)][String]$CSVFilePath,
  [Switch]$CalculateOff
)


FUNCTION Wr {
  Param([String]$Echo,[String]$Ink = "WHITE")
  If($Echo){Write-Host $Echo -ForegroundColor $Ink -NoNewLine}
  Else{Write-Host}
}

If(!(Test-Path -LiteralPath $CSVFilePath)){Wr "Cannot find $CSVFilePath!" RED;Wr;EXIT}
[System.Array]$Data = Import-CSV $CSVFilePath -Encoding Default

[Int64]$CR = 0
$Data | Foreach{
  [String]$Include = $_."Include"
  [String]$SourcePath = $_."Source Path"
  [String]$SourceFileName = $_."Source File Name"
  [String]$Source = $SourcePath + $SourceFileName
  [String]$DestinationPath = $_."Destination Path"
  [String]$DestinationFileName = $_."Destination File Name"
  [String]$Destination = $DestinationPath + $DestinationFileName
  If(($Include -eq "Y") -and $CalculateOff){
    If(!(Test-Path -LiteralPath $SourcePath)){
      Wr "Failed test path $SourcePath" RED;Wr
              }Else{
      Wr "Copying Item: $Source to $Destination";Wr
      If(!(Test-Path -LiteralPath $DestinationPath)){
        [Void](New-Item -ItemType directory -Path $DestinationPath)
      }
      Copy-Item -LiteralPath $Source -Destination $Destination
    }
  }elseif($Include -eq "Y"){
    $CR += (Get-Item -LiteralPath $Source).length
  }
}
If(!$CalculateOff){
  Wr "Capacity Required" CYAN;Wr
  Wr ($CR / 1024);Wr " KB";Wr
  Wr ($CR / (1024*1024));Wr " MB";Wr
  Wr ($CR / (1024*1024*1024));Wr " GB";Wr
}