Sunday, 8 October 2017

MetroCluster Installation Workflow

Just some notes from ‘Planning to Implement MetroCluster on Clustered Data ONTAP’. The web based course was a bit old - ONTAP 8.3.1 - but still useful as a refresher/framework.

1) Prepare for the MetroCluster Installation

Q1: What components do we have / do we have all the required components?

Obtain SO/BoM

MetroCluster Configurations:

i) Stretch Configurations
- 2-Node direct-attached (with SAS cables)
- 2-Node bridge-attached (with ATTO bridges)
ii) Fabric-Attached Configurations
- 2-Node (2 x single-node clusters)
- 4-Node (2 x 1 HA-pair clusters)
- 8-Node (2 x 2 HA-pair clusters)

High-Level Components List of 4-Node Fabric-Attached MetroCluster:

     |- Cluster Peering Network
     |- ISLs*
#####|Cluster A Site A
     |- 2 x NetApp FAS/AFF Controllers (HA-Pair)
     |- Cluster ISLs
     |- 2 x Fabric Switches
     |- Storage Shelves
     |- Fiber Channel-to-SAS bridges**
#####|Cluster B Site B
     |- 2 x NetApp FAS/AFF Controllers (HA-Pair)
     |- Cluster ISLs
     |- 2 x Fabric Switches
     |- Storage Shelves
     |- Fiber Channel-to-SAS bridges**

* 2 to 4 per fabric
** 2 per stack

Image: Differences between Clustered Data ONTAP MetroCluster Configurations (in 8.3.1)
* Requires Tie Breaker and PVR

Q2: What software versions are we using?

     |Fabric Switch Firmware
     |ATTO Bridge Firmware

Download required softwares
> Fibre Channel Switch
> Protocol Bridge (for ATTOs)
Download FC Switch Golden Configuration Image files
Download ‘ATTO QuickNAV utility’ to setup ATTOs

Q3: Have we obtained the Hardware Setup Checklist and Software Setup Checklist?

Select your ONTAP version and look for the:
‘MetroCluster Installation and Configuration Guide’

Q4: Device names?

Image: MetroCluster Naming Conventions
2) Configure the MetroCluster Hardware Components

See the ‘MetroCluster Installation and Configuration Guide’

2.1) Install Components and cable the storage controllers
- Rack the components
- Cable the FC-VI and HBA adapters
- Cable the ISLs between MetroCluster sites
- Cable the Cluster Interconnect
- Cable the Cluster Peering connections
- Cable the HA Interconnect
- Cable the Management connections
- Cable the Data ports
- Review the port connections
2.2) Cable and configure storage
- Set shelf IDs
- Configure ATTOs
ATTO Notes:
- Each ATTO bridge must have a unique name
- Configure data rate
- Must set the bridge connection mode to PTP
- Update Firmware
- FibreBridge Installation and Operation Manual available on the ATTO website
ATTO> sastargets
2.3) Configure the FC switches
- FTP Golden Configuration Image files to correct switch
2.4) Repeat for partner cluster

Image: Switch configuration by the NetApp Golden Configuration Image File (Brocade 6505 and 6510)
3) Configure the MetroCluster Software Components

Note: Systems configured in the factory will have much of the below completed.

3.1) Gather the required information
- Switch information
- Cluster creation information
- Node information
- Logical interfaces (LIFs) and ports for cluster peering
- Time server information
- AutoSupport support tool information
- Service processor information
3.2) Verify the disk assignments
Check in Maintenance Mode.
*> storage show shelf
*> storage show disk
*> disk assign -shelf FC_switch_A_1:6.shelf10 -p 0
*> disk assign -shelf FC_switch_B_1:6.shelf20 -p 1
*> disk show -n
*> disk show
3.3) Verify the high-availability (HA) state
*> ha-config show # Should be mcc for all components in 4-node MC, or mcc-2n for 2-node MC
*> ha-config modify controller mcc
*> ha-config modify chassis mcc
3.4) Configure the clusters
- Power up each node
- Enable AutoSupport
- Configure Node Management
::> storage failover show
::> storage failover modify -node localhost -mode ha # If not configured in HA mode
::> net port show -role cluster
::> set adv
::*> network options switchless-cluster modify -enable true # If required
::*> set adm
Note: Recommendation is to use the ‘System Setup tool’ to create the cluster, but do not create aggregates! Or you can use CLI.
3.5) Implement the MetroCluster configuration
- Configure Intercluster LIFs
- Configure network route
CLU1::> cluster peer create -peer-addrs IP1,IP2
CLU2::> cluster peer create -peer-addrs IP1,IP2
::> cluster peer show -instance
::> cluster peer health show
CLU1::> storage aggregate mirror aggr0_controller_A_1
{and repeat for every root aggregate}
CLU1::> storage aggregate create aggr1_controller_A_1 -diskcount X -node controller_A_1 -mirror true
{and repeat for every node - creating a mirrored data aggregate}
::> storage aggregate show-status -aggregate AGGR_NAME
i) Make sure there are 2 non-root mirrored data aggregates present on each cluster (MCC-4 node)
ii) Make sure ha-config state of controller and chassis is mcc (MCC 4 node)> ha-config show
::> metrocluster configure -node-name {NODE_NAME}
Note: This enables metrocluster on all the nodes in the DR group that contains NODE_NAME.
::> metrocluster show
To implement MetroCluster using a Single Data Aggregate
::> set adv
::*> metrocluster configure -allow-with-one-aggregate-true {NODE_NAME}
::*> set adm

4) Prepare to Monitor and Manage the ONTAP MetroCluster Solution

::> metrocluster check run
::> metrocluster check show
::> metrocluster check config-replication show
::> metrocluster check config-replication show -instance
::> metrocluster check config-replication show-aggregate-eligibility
::> metrocluster check node show
::> metrocluster check aggregate show
::> metrocluster check config-replication capture-in-progress show
::> metrocluster check lif show

Run Config Advisor

Send ASUP before testing (on both clusters):
::> system node autosupport invoke -node * -type all -message "Maint=START"

Verify Local HA Operation:
::> storage failover show
::> storage failover takeover controller_A_2
::> storage failover show
::> storage failover giveback controller_A_2
... and repeat ...

Verify Switchover, Healing, and Switchback:
- Run the switchover command
- Confirm that disaster recovery partners have come on line
- Heal the configuration
- Heal the data aggregates
- Heal the root aggregates
- Perform a switchback

Configure the MetroCluster FC Switches for Health Monitoring:
::> storage switch add -switch-ipaddress X.X.X.X
::> storage switch show

Configure the FC-to-SAS Switches for Health Monitoring:
::> storage bridge add -address Y.Y.Y.Y
::> storage bridge show

Protecting Configuration Backup Files Against Disaster (essential for single-node clusters):
::*> system configuration backup settings modify -destination ftp://Z.Z.Z.Z -username admin
::*> system configuration backup settings show

Note 1: Job schedules on both clusters in a MetroCluster configuration must be identical.
Note 2: Both sites of the MetroCluster should be licensed for the same node-locked features.


  1. Vidad, you writes manual Installing the NetApp Software Stack on Windows Server 2008 R2 SP1.doc. how about NetApp Software Stack on Windows Server 2012 R2??

  2. This is a whole informative post which describes the essential things required for installation of metro cluster, this is a useful work and I appreciate that!