Sunday, 29 November 2015

ONTAP-Buttons: Standard Code & GetNodeInfo

In the post Create SecureString File we wrote a tool to create a secure string file. Now we put that SecureString file to use, develop the standard code for "ONTAP-Buttons", and illustrate this with an example to Get Node Information for all nodes in a Clustered Data ONTAP cluster.

Batch File In ...\ONTAP-Buttons

The file GetNodeInfo.BAT has this content:

PowerShell.EXE .\PS\GetNodeInfo.PS1 -User admin -Cluster 192.168.168.200 -GetCredential

PowerShell Script in ...\ONTAP-Buttons\PS

The file GetNodeInfo.PS1 has the following content - any line that starts with <#S#> is standard content:

<#S#> Param(
<#S#>   [String]$Title = "Get Node Info", ## TITLE
<#S#>   [String]$User,
        [String]$Cluster,
<#S#>   [Switch]$SkipPressEnter,
<#S#>   [String]$SecurePasswordString,
<#S#>   [Switch]$GetCredential
<#S#> )
<#S#> FUNCTION Wr {Param([String]$Echo,[String]$Ink = "WHITE"); If($Echo){ Write-Host $Echo -ForegroundColor $Ink -NoNewLine } ELSE { Write-Host }}
<#S#> FUNCTION PressEnterToContinue{If(!$SkipPressEnter){Wr "Press ENTER to continue ..." CYAN; Read-Host}}
<#S#> If($SecurePasswordString){ $Password = $SecurePasswordString | ConvertTo-SecureString }
<#S#> elseif($GetCredential){
<#S#>   [String]$WhoAmi = WhoAmi
<#S#>   [String]$CredsFile = "PS\CREDS\" + $WhoAmi.Replace("\",".") + "." + $User.Replace("\",".") + ".CREDS"
<#S#>   If(!(Test-Path $CredsFile)){Wr "Unable to find credentials file!"; Wr; PressEnterToContinue; EXIT}
<#S#>   $Password = (Get-Content $CredsFile) | ConvertTo-SecureString}
<#S#> FUNCTION Load-DataOntapPSTK { If(Get-Module DataONTAP){ RETURN }
<#S#>   [Void](Import-Module DataONTAP -ErrorAction SilentlyContinue)
<#S#>   If(!(Get-Module DataONTAP)){Wr "Unable to load the DataONTAP PowerShell Toolkit - exiting!" RED; Wr; Wr; PressEnterToContinue; EXIT}
<#S#>   Wr "Loaded the Data ONTAP PowerShell Toolkit!" GREEN; Wr; Wr }
<#S#> FUNCTION Connect-ToCluster { Param([String]$ClusterName)
<#S#>   $Connect = Connect-NcController -Name $ClusterName -Credential $Credential -ErrorAction SilentlyContinue
<#S#>   If(!$Connect){Wr "Unable to connect to $ClusterName!" RED; Wr; Wr; PressEnterToContinue; EXIT}
<#S#>   Wr "Connected to $ClusterName" GREEN; Wr; Wr }
<#S#> Wr "+++++ $Title +++++" MAGENTA; Wr; Wr

Wr "Cluster Login User     : " CYAN; If($User){Wr $User; Wr}else{$User = Read-Host}
Wr "Cluster Login Password : " CYAN; If($Password){Wr "*****"; Wr}else{$Password = Read-Host -AsSecureString}
Wr "Cluster                : " CYAN; If($Cluster){Wr $Cluster; Wr}else{$Cluster = Read-Host}; Wr

<#S#> $Credential = New-Object System.Management.Automation.PsCredential($User,$Password)

[Void](Connect-ToCluster $Cluster)
$Nodes = Get-NcNode; $NodeCount = $Nodes.Count; $i = 1
FUNCTION WrData {
  Param([System.Object]$ObjectIn,[String]$PropertyName,[String]$HealthyResult,[Int]$Pad = 28)
  Wr (("$PropertyName".PadRight($Pad).Substring(0,$Pad)) + ": ") CYAN
  [String]$Property = $ObjectIn.$PropertyName
  If($HealthyResult){If($Property -eq $HealthyResult){Wr $Property GREEN}else{Wr $Property RED}}
  elseif($Property){Wr $Property}; Wr
}
$Nodes | Foreach{
  Wr (">>> Node " + [String]$i + " of " + [String]$NodeCount) MAGENTA; Wr
  WrData $_ "Node"
  WrData $_ "CpuBusytime"
  WrData $_ "CpuFirmwareRelease"
  WrData $_ "EnvFailedFanCount" "0"
  WrData $_ "EnvFailedFanMessage" "There are no failed fans."
  WrData $_ "EnvFailedPowerSupplyCount" "0"
  WrData $_ "EnvFailedPowerSupplyMessage" "There are no failed power supplies."
  WrData $_ "EnvOverTemperature" "False"
  WrData $_ "IsAllFlashOptimized"
  WrData $_ "IsDiffSvcs"
  WrData $_ "IsEpsilonNode"
  WrData $_ "IsNodeClusterEligible" "True"
  WrData $_ "IsNodeHealthy" "True"
  WrData $_ "MaximumAggregateSize"
  WrData $_ "MaximumNumberOfVolumes"
  WrData $_ "MaximumVolumeSize"
  WrData $_ "NodeAssetTag"
  WrData $_ "NodeLocation"
  WrData $_ "NodeModel"
  WrData $_ "NodeNvramId"
  WrData $_ "NodeOwner"
  WrData $_ "NodeSerialNumber"
  WrData $_ "NodeStorageConfiguration"
  WrData $_ "NodeSystemId"
  WrData $_ "NodeUptime"
  WrData $_ "NodeUuid"
  WrData $_ "NodeVendor"
  WrData $_ "NvramBatteryStatus" "battery_ok"
  WrData $_ "ProductVersion"
  WrData $_ "VmhostInfo"
  WrData $_ "VmSystemDisks"
  Wr; PressEnterToContinue; $i ++
}

<#S#> Wr "THE END!" MAGENTA; Wr; Wr; PressEnterToContinue; EXIT

GetNodeInfo in Action

Double-click GetNodeInfo.BAT which has already been specified with Cluster User, Cluster Management IP Address, and we have already generated the credential file -

Image: GetNodeInfo.BAT
- and, bam, node info is displayed!

Image: GetNodeInfo.BAT and GetNodeInfo.PS1 in action (with credentials generated by CreateSecureStringFile.BAT)

Saturday, 28 November 2015

Some Notes from Running a Perfstat against a busy cDOT Cluster

Note i: This post is for the CLI version but there is also a GUI version.
Note ii: As always, this is just a blog post and not support material - the switches may not be optimal for your situation.

1) Make sure you download the right and latest version of PerfStat (which is currently 8.3):


2) Running perstat8.exe:

For scanning one node:

perfstat8.exe --mode=c --nodes={NODE_IP_ADDRESS} -z -t 3 -i 10,0 --preset-file=preset_minimal.txt --exclude="SHELL=SYSTEMSHELL"

Note iii: This is 10 iterations of 3 minutes with 0 gap between iterations, but against a busy node may take 1 hour +

Similarly, to perfstat the whole cluster:

perfstat8.exe {CLUSTER_IP_ADDRESS} --mode=c -z -t 3 -i 10,0 --preset-file=preset_minimal.txt --exclude="SHELL=SYSTEMSHELL"

Explanation of the switches involved:

--mode=c: cDOT Mode
-z: Only filer scan (i.e. don't scan hosts)
-t: Time in minutes (3 is optimal)
-i A,B: A iterations, with B minutes in between iterations (increase iterations to record a longer interval)
--preset-file: Used to exclude certain commands (so the Perfstat doesn't take forever)
--exclude="SHELL=SYSTEMSHELL": skips needing the diag user login

Image: Perfstat8 in action
Contents of the Preset_Minimal.txt File

######################
# Perfstat Preset File
# SECTION       TYPE    HARDWARE        OS      LICENSE SHELL FLAG   SHORTNAME   COMMAND
PRESTATS      CONFIG NODE   >=8.1  *      NGSH                 volume efficiency policy show
PRESTATS      PERF   NODE   >=8.1  *      NGSH                 volume efficiency stat
PRESTATS      PERF   NODE   *      *      NODESHELL                  wafl_susp -z
PRESTATS      PERF   NODE   *      *      NGSH                 statistics show -node local -category latency -object latency -counter nfs-histogram
PRESTATS      PERF   NODE   *      *      NODESHELL                  waffinity_stats zero
PRESTATS      PERF   NODE   *      *      NODERUN              stats start   stats start
PARALLEL      PERF   NODE   *      *      NODERUN              sysstat_M     sysstat -M -c
PARALLEL      PERF   NODE   <8 -c="" -x="" .1.1="" nbsp="" noderun="" sysstat="" sysstat_x_1sec="" time=""> -s 1
PARALLEL      PERF   NODE   >=8.1.1       *      NODERUN              sysstat_x_1sec       sysstat -x -d -c
PARALLEL      PERF   NODE   *      *      NODERUN              statit stutter statit
PARALLEL      PERF   NODE   >=8.0 and <8 -category="" -interval="" -iter="" -node="" .1x="" 1="" local="" nbsp="" nfs3="" ngsh="" periodic="" statistics="" time="">
PARALLEL      PERF   NODE   >=8.1 and <8 -interval="" -iter="" -node="" -object="" .1.1="" 1="" local="" nbsp="" nfs3="" ngsh="" periodic="" statistics="" time="">
POSTSTATS     PERF   NODE   *      *      NGSH                 statistics show -node local -category latency -object latency -counter nfs-histogram
POSTSTATS     PERF   NODE   *      *      NODESHELL                   wafl_susp -w
POSTSTATS     PERF   NODE   *      *      NODESHELL                  wafl_susp -r
POSTSTATS     PERF   NODE   *      *      NODESHELL                  wafl scan status
POSTSTATS     PERF   NODE   *      *      NODESHELL                  wafl scan status -A
POSTSTATS     PERF   NODE   >=8.1  *      NGSH                 volume efficiency stat
POSTSTATS     PERF   NODE   *      *      NODERUN              stats stop    stats stop
POSTSTATS     PERF   NODE   *      *      NODESHELL                  waffinity_stats

Bonus Material - Instructions to FTP out via a Proxy Server

From DOS prompt on the server:

ftp YOUR_PROXY_SERVER

User: anonymous@ftp.netapp.com
Password: YOUR_EMAIL_ADDRESS

ftp> bin
ftp> hash
ftp> cd to-ntap
ftp> put {casenumber}.{filename}

Note iv: In Windows you can drag the file into the command line window from Explorer to paste the filename (the file needs to be on a local drive)
Note v: Instead of using ftp.netapp.com you can use ftp-europe.netapp.com

Button to Convert a List to Comma Separated String

Something I’ve had to do a surprising amount...

Save the following code as say ListToCSV.BAT and double-click to run. Paste in your list of items, and press carriage return after the last item, and a text document with the comma separated string version appears.

Image: Example of ListToCSV.BAT in action
Image: The output of ListToCSV.BAT
The Code

powershell.exe [System.Array]$Outfile = @(); Foreach( $line in (get-content ListToCSV.bat) ){ If($Line.StartsWith('REM ')){ $Outfile += $Line.replace('REM ','') }else{ $Outfile += ('# ' + $Line) } }; Out-File -filepath ListToCSV.ps1 -InputObject $OutFile -Encoding Default; .\ListToCSV.ps1
exit
REM Remove-Item ListToCSV.PS1 -Force -Confirm:$False
REM FUNCTION ListToCSV {PARAM([Parameter(Mandatory=$True)][System.Array]$InArr)
REM   [String]$OutStr = ""; $InArr | Foreach{ $OutStr += $_.trim() + "," }
REM   $OutStr.Substring(0,$OutStr.Length-1)
REM }
REM Write-Host "`nPaste list at the prompt" -ForegroundColor GREEN
REM [String]$ListToCSV = ListToCSV
REM $ListToCSV > "ListToCSV.TXT"
REM Notepad "ListToCSV.TXT"; Sleep 1
REM Remove-Item "ListToCSV.TXT" -Force -Confirm:$False


Tuesday, 24 November 2015

Create SecureString File

The previous post (here) was essentially the start of my ONTAP-Buttons pet project “to have a load of batch files I simply double-click and they do some useful task”. I could construct my buttons (the batch file that runs the PowerShell) with the Secure String in the batch file like I did here (an option I’ll keep), or, I can store these credentials somewhere so they can be shared between my buttons.

Quick note on “ONTAP-Buttons” folder structure:

If my buttons are in the folder ...\ONTAP-Buttons
The PS files are in the folder  ...\ONTAP-Buttons\PS
The creds will be in            ...\ONTAP-Buttons\PS\CREDS

Image: CreateSecureStringFile.BAT in action

Code

## +++++ Create SecureString File +++++ ##
Param([Switch]$NoDisplay)
Function Wr{ Param([String]$Echo,[String]$Ink = "WHITE"); IF($Echo){ Write-Host $Echo -ForegroundColor $Ink -NoNewLine } ELSE { Write-Host } }; Wr
Wr "+++++ Create SecureStringFile +++++" Magenta; Wr; Wr
Wr "......User Name : " Green; $UserName = Read-Host
Wr "User's Password : " Green; $SecureString = (Read-Host -AsSecureString | ConvertFrom-SecureString)

# Replace \ with . (cannot use \ in filename)
[String]$WhoAmI = (WhoAmI).Replace("\",".")
[String]$UserName = $UserName.Replace("\",".")

# Create PS\CREDS dir if they don't exist and write the file
[Void](New-Item -Path "PS" -ItemType directory -Force)
[Void](New-Item -Path "PS\CREDS" -ItemType directory -Force)
# Note: The excution context is where PowerShell is run from (i.e. if using a BAT file, where the BAT file is double-clicked)

[String]$CredsFileName = $WhoAmi + "." + $UserName + ".CREDS"
$SecureString > "PS\CREDS\$CredsFileName"
If(!$NoDisplay){ Notepad "PS\CREDS\$CredsFileName" }


Next thing, a standard way to use these stored credentials (I’m thinking the batch file will need a -GetCredentials switch to actually use them, otherwise it will prompt)...

Monday, 23 November 2015

Snap Reserve Resizer

Mark II of SnapReserveCalculator from this post - Re-sizing Volume Snapshot Reserve Calculator - just a bit prettier and more user-friendly. Still just a calculator though!

Image: SnapReserveResizer in action!
The Script

## +++++ Snapshot Reserve Resizer +++++ ##
Function Wr{ Param([String]$Echo,[String]$Ink = "WHITE"); IF($Echo){ Write-Host $Echo -ForegroundColor $Ink -NoNewLine } ELSE { Write-Host } }
Function TrapOut{ Wr "Not a valid value!" Red; Wr; EXIT }; Wr

Wr "+++++ Snapshot Reserve Resizer +++++" Magenta; Wr; Wr
Wr "To Health Check Snapshot Reserve ::>" Green; Wr
Wr "vol show -snapshot-space-used > 100 -type RW -fields percent-snapshot-space,snapshot-space-used" Cyan; Wr; Wr
Wr "For any volumes with over 100% snapshot reserve, obtain this info ::>" Green; Wr
Wr "vol show -volume VOLNAME -fields size,percent-used,percent-snapshot-space,snapshot-space-used" Cyan; Wr; Wr

Trap {TrapOut}
Wr "..................... Volume size : " Green; [Double]$VolumeSize      = Read-Host
Wr "............. Percent volume full : " Green; [Double]$PercVolSizeUsed = Read-Host
Wr "........ Percent snapshot reserve : " Green; [Double]$PercSnapRes     = Read-Host
Wr "... Percent snapshot reserve used : " Green; [Double]$PercSnapResUsed = Read-Host; Wr
Wr "......... NEW percent volume full : " Cyan;  [Double]$NewVolPercFull  = Read-Host
Wr "NEW percent snapshot reserve used : " Cyan;  [Double]$NewSnapResPerc  = Read-Host; Wr

[Double]$SizeUsedBySnapshots = ($VolumeSize * $PercSnapRes * $PercSnapResUsed) / (100 * 100)
If($PercSnapResUsed -le 100){ [Double]$SnapshotSpill = 0 }
else{ [Double]$SnapshotSpill = $SizeUsedBySnapshots - (($VolumeSize * $PercSnapRes) / 100) }
[Double]$SizeUsedByUserData = (($VolumeSize * $PercVolSizeUsed) / 100) - (($VolumeSize * $PercSnapRes) / 100) - $SnapshotSpill

[Double]$NewSnapResSize   = ($SizeUsedBySnapshots / $NewSnapResPerc) * 100
[Double]$NewVolumeSize    = (($SizeUsedByUserData + $NewSnapResSize) / $NewVolPercFull) * 100
[Double]$NewPercSnapRes   = ($NewSnapResSize / $NewVolumeSize) * 100
[String]$NewVolSizeString = [String]($NewVolumeSize)
[String]$NewSnapResString = [String]($NewPercSnapRes)

Wr "+++ Result +++" Magenta; Wr; Wr
Wr "New Volume Size              = " Cyan; Wr $NewVolSizeString; Wr
Wr "New Percent Snapshot Reserve = " Cyan; Wr $NewSnapResString; Wr; Wr

Wr "Clustershell Commands ::>" Green; Wr
Wr "vol size -vserver VSERVER -volume VOLNAME -new-size SIZE{&UNIT}" Cyan; Wr
Wr "vol modify -vserver VSERVER -volume VOLNAME -percent-snapshot-space PERCENT" Cyan; Wr; Wr
Wr "Press ENTER to exit ..." CYAN; Read-Host; EXIT


Saturday, 21 November 2015

cDOT MetroCluster Links and Resources (8.3.1)

An overview of information resources available for cDOT MetroCluster...

NetApp Library (Library and Videos)

- Benefits of NetApp MetroCluster for Continuous Data Availability (3:51)
- Benefits of MetroCluster (3:09)


NetApp University


- WBT: MetroCluster in Clustered Data ONTAP 8.3 Overview
- ILT: MetroCluster Installation Workshop
- ILT: MetroCluster Troubleshooting

Product Documentation


MetroCluster™ Installation and Configuration Guide
MetroCluster™ Management and Disaster Recovery Guide
MetroCluster™ Service Guide
Tiebreaker Software 1.1 Installation and Configuration Guide*
*Also check out ClusterLion's Tiebreaker option

Metrocluster Installation Express Guide
+ loads more interesting stuff here!


The documentation is still in the process of being written for these, but worth checking for updates:

Field Portal (access permitting)

Loads of stuff if you search for “MetroCluster”*
*recommend searching by Date and look at the material from the last year or so

kb.netapp.com

Search for “MetroCluster” - some example finds:

Data ONTAP PowerShell Toolkit 4.0 MetroCluster cmdlets

Since I’m such a fan of the Data ONTAP PowerShell Toolkit, I couldn’t help pointing out the awesome MetroCluster cmdlets (if you have the PSTK installed - run show-nchelp > Categories > Metrocluster to see more information):

Get-NcMetrocluster
 Get MetroCluster configuration information.
Get-NcMetroclusterAggregateCheck
 Get a list of MetroCluster check aggregate info objects.
Get-NcMetroclusterAggregateEligibilityCheck
 Get information regarding the eligibility of aggregates to host configuration replication volumes.
Get-NcMetroclusterCheck
 Get a list of MetroCluster check info objects.
Get-NcMetroclusterCheckCaptureStatus
 Get information regarding the capture status of replicated metrocluster configuration data.
Get-NcMetroclusterCheckCluster
 Get a list of MetroCluster check cluster info objects
Get-NcMetroclusterConfigDiff
 Get the differences in configuration between two clusters in DR relationship for MetroCluster.
Get-NcMetroclusterConfigReplication
 Get information regarding the cluster-wide storage associated with cross-cluster replication of MetroCluster configuration data.
Get-NcMetroclusterConfigReplicationCheck
 Get the results of the MetroCluster config replication checks.
Get-NcMetroclusterConfigReplicationResyncStatus
 Get MetroCluster configuration replication synchronization information
Get-NcMetroclusterFailedLifPlacement
 Get a list of LIF placement failures in a MetroCluster setup.
Get-NcMetroclusterInterconnectAdapter
 Get a list of MetroCluster interconnect adapter objects.
Get-NcMetroclusterInterconnectMirror
 Iterate over a list of metrocluster-interconnect-mirror objects.
Get-NcMetroclusterInterconnectMirrorMultipath
 Iterate over a list of metrocluster-interconnect-mirror-multipath objects.
Get-NcMetroclusterNode
 Get a list of MetroCluster node objects.
Get-NcMetroclusterNodeCheck
 Get a list of MetroCluster check node info objects.
Get-NcMetroclusterOperation
 Get a list of MetroCluster operations and their details.
Get-NcMetroclusterProgress
 Get a list of MetroCluster progress table info objects.
Get-NcMetroclusterVserver
 Get a list of MetroCluster vserver info objects.
Invoke-NcMetroclusterCheck
 Invoke a check on the MetroCluster configuration.
Invoke-NcMetroclusterHeal
 Heal aggregates and partner roots in preparation for a DR switchback.
Invoke-NcMetroclusterSwitchback
 Initiate the switchback of storage and client access from nodes in the DR site to their home nodes.
Invoke-NcMetroclusterSwitchover
 Initiate the switchover of storage and client access from a disaster stricken site to the DR site.
Register-NcMetrocluster
 Configure a node and its DR partner to mirror NVRAM and be configured to participate in a DR switchover.
Repair-NcMetroclusterLifPlacement
 Verify LIF placement in the destination cluster for the sync-source Vserver LIF in a MetroCluster setup.
Set-NcMetroclusterConfigReplication
 Modify information regarding the cluster-wide storage associated with cross-cluster replication of MetroCluster configuration data.
Sync-NcMetroclusterVserver
 Resynchronize Vserver with it's partner Vserver.
Test-NcMetrocluster
 Test if the cluster is configured for MetroCluster.
Unregister-NcMetrocluster
 Wipe the configuration from nodes in a DR group which are in the local cluster where the API is issued.

Sunday, 15 November 2015

cDOT Mini Projects I’d Like to do if I had the Time

A few cDOT mini projects I’d like to do if I had the time...

1) vCheck for cDOT


2) Apache Perl / IIS PowerShell Web Forms, customizable Web Browser based pre/post change Health Check tool/Auditor

3) Backup and restore of Cluster/SVM config items tool

4) cDOT AV Connector Automated Deployment/Upgrades

5) ONTAP Buttons

In this post I showed it’s possible to have a batch file that is also a PowerShell script. So, I was thinking it would be cool to have a suite of PowerShell applet like buttons that you can invoke by simply double-clicking a file. Some of the things I’d convert to this format I have the basic code for already. Ideas (endless possibilities - hence this is the last item in my list):

- Volume Snapshot Reserve resizer
- Synchronizing cron schedules cluster to cluster
- Tidy up of unused/unnecessary cron schedules
- Synchronizing snapshot policies
- Synchronizing snapshot policies on volumes
- Synchronizing volume efficiency policies
- Synchronizing snapmirror policies
- Storing snapmirror schedules in destination volume comments
- Storing snapmirror schedules in source volume comments
- Storing snapvault policies and schedules in destination volume comments
- Storing snapvault policies and schedules in source volume comments
- Snapmirror schedule re-distributor (i.e. for changing separation of snapmirror schedules from say 10 to 3 minute)
- Synchronizing quota policies and rules
- Synchronizing CIFS shares
- Tidy up of unnecessary shares on DR
- Synchronizing UNIX groups
- Tidy up of unnecessary UNIX groups
- Synchronizing UNIX users
- Tidy up of unnecessary UNIX users
- Synchronizing name-mappings
- Tidy up of unnecessary name-mappings (including rationalization where the default domain name-mapping will work)
- Synchronizing NFS exports and rules
- Tidying up unnecessary export-policies
- Tidying up dead clients in export-policy rules
- Volume Max Autosize Resizer (i.e. SnapVaults have a default 90TB max autosize which sets of OCUM’s over provisioning alert)
- Flexclone split estimator


And on, and on ...

Sunday, 8 November 2015

Synchronizing cDOT Cron Schedules Cluster to Cluster

The following tool can be used to synchronize cron schedules from one Clustered Data ONTAP cluster to another (even if they are on a different ONTAP version.)

The tool can be run in a variety of ways. The example below is run from simply double clicking a BAT file - no prompts need to be answered:

Image: Example running Sync-Cron.ps1 using a BAT file
The Script - Sync-Cron.ps1

## To run in a BAT file (switches optional):
# Powershell.exe .\Sync-Cron.ps1

## To create $SecurePasswordString run:
# PS> (Read-Host -AsSecureString) | ConvertFrom-SecureString
# N.B. This string only works when logged in as the same Windows user that created it.

Param([String]$ClusterUser,[String]$PrimaryCluster,[String]$SecondaryCluster,[String]$SecurePasswordString,[Switch]$SkipPressEnter)
If($SecurePasswordString){ $Password = $SecurePasswordString | ConvertTo-SecureString }

FUNCTION Wr { Param([String]$ToDisplay,[String]$Color = "WHITE")
  IF($ToDisplay){ Write-Host $ToDisplay -ForegroundColor $Color -NoNewLine } ELSE { Write-Host } }

Wr "+++++ Cron Schedule Cluster to Cluster Synchronizer +++++" MAGENTA; Wr; Wr
Wr "Cluster Login User     : " CYAN; If($ClusterUser){ Wr $ClusterUser; Wr }else{ $ClusterUser = Read-Host }
Wr "Cluster Login Password : " CYAN; If($Password){ Wr "*****"; Wr }else{ $Password = Read-Host -AsSecureString }
Wr "Cluster - Primary      : " CYAN; If($PrimaryCluster){ Wr $PrimaryCluster; Wr }else{ $PrimaryCluster = Read-Host }
Wr "Cluster - Secondary    : " CYAN; If($SecondaryCluster){ Wr $SecondaryCluster; Wr }else{ $SecondaryCluster = Read-Host }; Wr
$Credential = New-Object System.Management.Automation.PsCredential($ClusterUser,$Password)

FUNCTION PressEnterToContinue{ If(!$SkipPressEnter){ Wr "Press to exit ..." CYAN; Read-Host }; EXIT }
FUNCTION Load-DataOntapPSTK {
  If(Get-Module DataONTAP){ RETURN }
  [Void](Import-Module DataONTAP -ErrorAction SilentlyContinue)
  If(!(Get-Module DataONTAP)){ Wr "Unable to load the DataONTAP PowerShell Toolkit - exiting!" RED; Wr; Wr; PressEnterToContinue }
  Wr "Loaded the Data ONTAP PowerShell Toolkit!" GREEN; Wr; Wr }
FUNCTION Connect-ToCluster {
  Param([String]$ClusterName)
  $Connect = Connect-NcController -Name $ClusterName -Credential $Credential -ErrorAction SilentlyContinue
  If(!$Connect){ Wr "Unable to connect to $ClusterName!" RED; Wr; Wr; PressEnterToContinue }
  Wr "Connected to $ClusterName" GREEN; Wr; Wr }

[Void](Load-DataOntapPSTK)
[Void](Connect-ToCluster $PrimaryCluster)
$SourceCronJobs = Get-NcJobCronSchedule;
Wr "++ Collected cron jobs from $PrimaryCluster ++"; Wr; Wr
[Void](Connect-ToCluster $SecondaryCluster)
$SourceCronJobs | foreach { $_.NcController = $null; [Void]( $_ | Add-NcJobCronSchedule -ErrorAction SilentlyContinue ) }
$SourceCronJobs | foreach { $_.NcController = $null; [Void]( $_ | Set-NcJobCronSchedule -ErrorAction SilentlyContinue ) }
Wr "++ Applied cron jobs to $SecondaryCluster ++"; Wr; Wr
PressEnterToContinue