Monday, 27 June 2016

Highlights from the ONTAP 9.0 RC1 Release Notes



New and changed features in the ONTAP 9.0 release family

Manageability enhancements

- FIPS 140-2 support for cluster-wide control plane web service interfaces
- Support for enhanced Storage QoS limits
-- up to 12’000 policy groups per cluster
-- can specify a combination of IOPs and MB/s
- Changes in audit configuration operation
- Support for SHA-2 password hash function
- Password security enhancements
-- password complexity enforcement
-- delay after failed login attempt
-- define the account inactive limit
-- expire a user password
-- display password expiry warning
-- send invalid login notifications to audit.log & EMS
- Enhancement for audit log management
- Support for non-English characters in qtree names
- Support for Storage Encryption onboard key management
- Support for cache-retention policies on Flash Pool aggregates
- Support for root-data partitioning enhancement
-- root-data partitioning is supported on AFF platforms
- Support for headroom functionality
-- Assists workflow provisioning and balance
-- Helps you prevent overloading a node

MetroCluster configuration enhancements

- Implementation of node-level QoS
-- reduces node outage by prioritizing the I/O operations needed to complete a disaster recovery (DR) operation
- Support for reestablishing SnapMirror or SnapVault relationships
- Support for eight-node MetroCluster configurations
-- where using the NAS protocol
- Support for unmirrored aggregates in MetroCluster configurations

Networking and security protocol enhancements

- IPv6 support for DDNS
- Increased limits for maximum LIFs per node
-- in clusters where all nodes have at least 20 GB of memory, you can create up to 256 NAS LIFs or 512 SAN LIFs per node
- Automatic detection and isolation of network port failures
- Support for LLDP
-- Link Layer Discovery Protocol - an alternative to CDP
- Support for UC compliance with DSCP marking
-- Differentiated Services Code Point (DSCP) marking is a mechanism for classifying and managing network traffic and is a component of Unified Capability (UC) compliance

Storage resource management enhancements

- Support for inline data compaction
-- Inline data compaction stores multiple user data blocks and files within a single 4 KB block on a system running ONTAP
-- Inline data compaction is enabled by default on AFF, and you can optionally enable it on volumes on FAS systems
-- You can run inline data compaction independently or together with data compression or deduplication
- Support for automated SAN and NAS storage provisioning
-- The following new templates are accessible under the Application Provisioning tab in OCSM:
--- NFS Oracle
--- NFS Oracle RAC
--- SAN Oracle Single
--- SAN Oracle RAC
--- SMB SQL Server
--- SAN SQL Server
--- NAS Virtual Server Infrastructure
--- SAN Virtual Server Infrastructure
--- SAN Virtual Desktop
- Support for relocating root volumes to new aggregates
- Support for rehosting a volume from one SVM to another SVM
- Supported security styles for Infinite Volumes
-- unix, ntfs, mixed, or unified

FlexArray Virtualization (V-Series) enhancements

- Support for 16 TB array LUNs
- FlexArray Virtualization resiliency enhancements

File access protocol enhancements

- Enhanced support for SMB protocols
-- The following enhancements are now supported in SMB 2.0 and later:
--- LDAP signing and sealing
--- Workgroup authentication
--- Large MTU
--- SMB null user and IP qualifiers
-- The following SMB enhancements are now supported in SMB 3.0 and later:
--- Accelerated AES-NI encryption
- Support for CIFS server in workgroup mode
- Enhancements for Kerberos 5
-- Kerberos 5 authentication with privacy service (krb5p) is supported
- Support for LDAP signing and sealing
- LDAP support for SHA2 hashed passwords
- Multiple client match specifications in NFS export rules
-- “provides comparable functionality to standard /etc/exports files in 7-Mode”
- NFS export access cache enhancements
- FPolicy enhancements
-- This release includes the following FPolicy enhancements:
--- Filtering controls
--- Async resiliency
- Support for new CIFS auditing events
-- The new auditing events are as follows:
--- file-share
--- audit-policy-change
--- user-account
--- security-group
--- authorization-policy-change

SAN enhancements

- ODX LUN copy is now supported between clusters
- iSCSI target support for an FQDN response

Data protection enhancements

- Support for triple parity RAID protection (RAID-TEC)
- Support for SnapLock technology
- Support for cloning data protection volumes
-- in an SVM-DR relation, the cluster administrator can clone data protection (DP) volumes from the destination SVM to another SVM in the destination cluster for development testing
- Support for excluding volumes from replication
-- in an SVM-DR relation…
- Support for excluding LIFs from replication
-- in an SVM-DR relation…
- Support for converting volume-level SnapMirror relationships to an SVM disaster recovery relationship
- Support for using a SnapMirror license to enable SnapVault
- Support for NDMP extension
-- NDMP extension 0x2050, which enables Snapshot copy management and backup restart extensions, is supported
- Support for intercluster SVM peer relationships between SVMs with the same name
- Support for SnapMirror and SnapVault global throttling
-- These options are enforced on all nodes in the cluster:
--- Enabling and disabling the node-level throttle
--- Setting the maximum bandwidth for outgoing transfers
--- Setting the maximum bandwidth for incoming transfers
- Support for secondary SnapVault Snapshot copies
-- you can create Snapshot copies on a SnapMirror SnapVault destination for longer term retention without having to create and maintain a corresponding longer term Snapshot copy at the primary destination
- Support for renaming SnapVault Snapshot copies
- Support for all SnapMirror and SnapVault cascade configurations

Antivirus enhancements

- Support for Vscan on-demand scanning
- Support for FQDN configuration of Vscan servers

Upgrade enhancements

- Expanded support for automated nondisruptive upgrades
-- to include major upgrades from Data ONTAP 8.3.x to ONTAP 9.0

Transition enhancements

- Support for transitioning SnapLock volumes
- Support for transitioning a disaster recovery relationship between vFiler units
- Support for transitioning peering networks from IPv4 to IPv6

Unsupported features for ONTAP 9.0 RC1

- LDAP over SSL replaced with LDAP over TLS
- Remote Support Agent (RSA) replaced with AutoSupport On Demand

The 100 Latest NetApp TRs List - July 2016


I wanted to get a list of the latest NetApp TRs (Technical Reports) - i.e. newest/most recently updated - so went to http://www.netapp.com/us/library/, filtered by ‘Document Type: Technical Reports’, sorted by ‘Date: Newest to Oldest’, display the maximum 100 results, and painstakingly constructed this list (and what an outstanding and varied list of TRs there is - credit to the authors for all their hard work):


Note: I could certainly have continued the list - 100 is plenty for now!

Sunday, 19 June 2016

How to Create cDOT Qtrees with Qtree Quotas from a CSV

Question/Scenario:
I have a load of trees that were created on another vendor’s storage (non-NetApp), each tree has a quota; how do I recreate all the qtrees and qtree quotas on cDOT, in readiness for migrating the data off?

Qtree and Quota Creator

The following script was created with the above scenario in mind. We’ve already created the volumes, so need an export of information from the old storage system regarding its Qtrees and Qtree Quotas, and then we can take this information and use it to create our Qtrees and Qtree Quotas on NetApp Clustered Data ONTAP (version 8.3.2 used in the examples).

In the screenshot below is an example of the content of the input CSV. Here we only consider Quota Hard Disk Limits (this could be expanded for other types of Qtree Quota limit.) The column headings in the CSV need to be as in the example (that is: “VOLUME”, “QTREE”, “QUOTA HARD LIMIT”, and “UNIT”).

Image: CSV Input to QtreeAndQuotaCreator

The following is an example of how to run Qtree and Quota Creator in PowerShell>


.\QtreeAndQuotaCreator.ps1 -UserName admin -Password YOURPASS -Cluster 10.0.1.100 -Vserver svm1 -CSVfile quota.csv


An example of the output is in the Appendix beneath the script.

The Script

Note: Formatted for blogger with tabs replaced by two spaces

Copy and paste the below into a text editor and save as QtreeAndQuotaCreator.ps1


############################################
## CREATE CDOT QTREES AND QUOTAS FROM CSV ##
############################################

<# CSV headings processed:
  "VOLUME","QTREE","QUOTA HARD LIMIT","UNIT"
  Note 1: For New-NcQtree, you can also specify
    -Mode -SecurityStyle -Oplocks -ExportPolicy
  Note 2: For Add/Set-NcQuota, you can also specify
    -FileLimit -Threshold -SoftDiskLimit -SoftFileLimit -PerformUserMapping
  Note 3: Units are kb/mb/gb...
#>

Param(
  [Parameter(Mandatory=$true)][String]$UserName,
  [Parameter(Mandatory=$true)][String]$Password,
  [Parameter(Mandatory=$true)][String]$Cluster,
  [Parameter(Mandatory=$true)][String]$Vserver,
  [Parameter(Mandatory=$true)][String]$CSVfile
)

## DISPLAY FUNCTION
Function Wr{
  Param([String]$Echo = "",[String]$Ink = "WHITE",[Switch]$EXIT)
  If($EXIT){ Write-Host $Echo -ForegroundColor RED; EXIT }
  Write-Host $Echo -ForegroundColor $Ink
}

## LOAD THE DATA ONTAP PSTK
If(!(Get-Module DataONTAP)){ [Void](Import-Module DataONTAP -ErrorAction SilentlyContinue) }
If(Get-Module DataONTAP){ Wr "Loaded DataONTAP PSTK" GREEN }
else{ Wr "Failed to load DataONTAP PSTK!" -EXIT }

## CONNECT TO CLUSTER
$SecPass = $Password | ConvertTo-SecureString -asPlainText -Force
$Cred = New-Object System.Management.Automation.PsCredential($UserName,$SecPass)
[Void](Connect-NcController $Cluster -Credential $Cred -ErrorAction SilentlyContinue)
If($Global:CurrentNcController){ Wr "Connected to $Cluster" GREEN}
else{ Wr "Failed to connnect to $Cluster!" -EXIT }

## CHECK VSERVER EXISTS
$GetNcVserver = Get-NcVserver $Vserver
If($GetNcVserver){ Wr "Verified vserver $Vserver" GREEN}
else{ Wr "Vserver $Vserver check failed!" -EXIT}

## GET LIST OF VOLUMES
$VolAttrs = Get-NcVol -Template
[System.Array]$Volumes = (Get-NcVol -Attributes $VolAttrs -VserverContext $Vserver).Name

## GET LIST OF VOLUMES WITH QUOTA ALREADY TURNED ON
$QuotaQuery = Get-NcQuotaStatus -Template
$QuotaQuery.Status = "on"
[System.Array]$VolsWithQuotaOn = (Get-NcQuotaStatus -Query $QuotaQuery -VserverContext $Vserver).Volume

## PROCESS THE CSV
If(Test-Path $CSVfile){ Wr "Tested path $CSVfile OK!" GREEN }
else{ Wr "Path check failed to $CSVfile!" -EXIT }
$CSVinput = Import-CSV -Path $CSVfile

## CREATE QTREES AND QUOTAS
$Quota_Pol = $GetNcVserver.QuotaPolicy # Current Quota Policy
$VolsToTurnQuotaOn = @() # Record volumes where quota needs to be turned on
$VolsToResizeQuota = @() # Record volumes where quota needs resize
$CSVinput | Foreach{
  If($Volumes -Contains $_.VOLUME){
    ## CREATING NEW QTREE
    [Void](New-NcQtree -Volume $_.VOLUME -Qtree $_.QTREE -VserverContext $Vserver -ErrorAction SilentlyContinue)   
    Wr ("Created Qtree " + $_.QTREE + " on volume " + $_.VOLUME) GREEN
    ## CREATING NEW QUOTA
    [Void](Add-NcQuota -Path ("/vol/" + $_.VOLUME + "/" + $_.QTREE) -DiskLimit ($_."QUOTA HARD LIMIT" + $_."UNIT") -Policy $Quota_Pol -VserverContext $Vserver -ErrorAction SilentlyContinue)
    [Void](Set-NcQuota -Path ("/vol/" + $_.VOLUME + "/" + $_.QTREE) -DiskLimit ($_."QUOTA HARD LIMIT" + $_."UNIT") -Policy $Quota_Pol -VserverContext $Vserver -ErrorAction SilentlyContinue)
    Wr ("Set Quota with hard limit of " + $_."QUOTA HARD LIMIT" + $_."UNIT") GREEN
    ## UPDATE VOL QUOTA ON/RESIZE ARRAYS
    If($VolsWithQuotaOn -Contains $_.VOLUME){
      If($VolsToTurnQuotaOn -NotContains $_.VOLUME){
        If($VolsToResizeQuota -NotContains $_.VOLUME){
          $VolsToResizeQuota += $_.VOLUME
        }
      }
    }else{
      $VolsWithQuotaOn += $_.VOLUME
      $VolsToTurnQuotaOn += $_.VOLUME
    }
  }else{
    Wr ("Volume " + $_.VOLUME + " does not exist!") RED
  }
}

## ENABLING QUOTAS
$VolsToTurnQuotaOn | Foreach{
  [Void](Enable-NcQuota -Volume $_ -VserverContext $Vserver)
  Wr "Enabled quota on volume $_" GREEN
}

## RESIZING QUOTAS
$VolsToResizeQuota | Foreach{
  [Void](Start-NcQuotaResize -Volume $_ -VserverContext $Vserver)
  Wr "Started quota resize on volume $_" GREEN
}

## OUTPUT QUOTA REPORT
Get-NcQuotaReport -Vserver $Vserver | Sort-Object QuotaTarget | FT QuotaType,QuotaTarget,DiskLimit -AutoSize
Wr "Newly created quotas can time a time to initialize. If newly created quotas are not in the report, please wait and then run PS>" CYAN
Wr "Get-NcQuotaReport -Vserver $Vserver | Sort-Object QuotaTarget | FT QuotaType,QuotaTarget,DiskLimit -AutoSize" YELLOW


Appendix: Example Output

Image: Example of running Qtree and Quota Creator

Saturday, 18 June 2016

7MTT Multiple Job Parser

This might be handy for large 7 to C transition projects, to be able to get the outputs of all the 7MTT jobs (we’d mostly be interested in pre-checks) and line them all up side by side into one CSV (a column per job for comparison purposes), which makes it easy to filter on pre-check errors/warnings that either aren’t of interest, or are already known about, and then can focus on just the important stuff.

Its super simple to run on your 7MTT box, just do in PowerShell>


.\7MTT_Job_Parser.ps1


It automatically targets the 7MTT jobs folder. I added some other optional switches:

-FileOrFolder
-PrecheckFlag
-TransitionOp
-FileOut

The tool grabs all the .job files, converts them from JSON, and then generates a CSV. I’ve not currently implemented the “AREA” column (i.e. CIFS/NFS/SAN …), and the “Start Time” output could be improved.

An idea for the future would be to automate 7MTT session setup and running pre-checks - using 7MTT CLI - against any number of systems; then this tool can take the results of those pre-check jobs and parse them into one spreadsheet.

The Script

Note: Formatted for blogger with tabs replaced by two spaces

Copy and paste the below into a text editor and save as 7MTT_Job_Parser.ps1


Param(
  [String]$FileOrFolder,
  [String]$PrecheckFlag,
  [String]$TransitionOp,
  [String]$FileOut = "7MTT_JOBS.CSV"
)

## Use 7MTT jobs folder and/or check specified path exists ##
If(!$FileOrFolder){
  $PathTo7MTTbin = $Env:Path.Split(";") | Where{$_ -match "7-Mode Transition Tool"}
  $PathMinusBin  = $PathTo7MTTbin.Substring(0,$PathTo7MTTbin.length - 4)
  $FileOrFolder  = $PathMinusBin + "\data\jobs"
}
If(!(Test-Path $FileOrFolder)){ Write-Host "Path test to $FileOrFolder failed - exiting!" -ForegroundColor Red; EXIT}

## Put job files to check into an array ##
$FilesToCheck = @()
[Boolean]$IsFolder = (Get-Item $FileOrFolder) -is [System.IO.DirectoryInfo]
If(!$IsFolder){ $FilesToCheck += $FileOrFolder }
else{
  Get-ChildItem -Path $FileOrFolder -Filter *.job | Foreach{ $FilesToCheck += ($FileOrFolder + "\" + $_.Name) }
}
If(!$FilesToCheck){ Write-Host "Not found any job files - exiting!" -ForegroundColor Red; EXIT}

## Simplified Add-Member function (saves typing) ##
$R = @{}
Function AM {
  Param($I,$N,$V)
  If(!$R.$I){$R.$I = New-Object PSObject}
  Add-Member -InputObject $R.$I -MemberType NoteProperty -Name $N -Value $V -Force
}

## Create Column and Side Headings ##
# Col 1         Col 2          Col 3            Col 4            Col 5
AM 2 "1" "2"; AM 2 "2" ""  ; AM 2 "3" ""    ; AM 2 "4" ""    ; AM 2 "5" "Job File Name"
AM 3 "1" "3"; AM 3 "2" ""  ; AM 3 "3" ""    ; AM 3 "4" ""    ; AM 3 "5" "Version"
AM 4 "1" "4"; AM 4 "2" ""  ; AM 4 "3" ""    ; AM 4 "4" ""    ; AM 4 "5" "Session"
AM 5 "1" "5"; AM 5 "2" ""  ; AM 5 "3" ""    ; AM 5 "4" ""    ; AM 5 "5" "Pre-Check Flags"
AM 6 "1" "6"; AM 6 "2" ""  ; AM 6 "3" ""    ; AM 6 "4" ""    ; AM 6 "5" "Transition Op"
AM 7 "1" "7"; AM 7 "2" "ID"; AM 7 "3" "AREA"; AM 7 "4" "TYPE"; AM 7 "5" "Start Time"
AM 8 "1" "8"; AM 8 "2" "=="; AM 8 "3" "===="; AM 8 "4" "===="; AM 8 "5" ""

$Col = 5 # Initialize Column count (current column)
$Row = 8 # Initialize Row count (current row)

$RecordedCodes = @()
$CodesToRow    = @{}

$FilesToCheck | Foreach {
  $J = ConvertFrom-JSON -InputObject (Get-Content $_ -Raw)
  ## Allow if no PreCheckFlag or have match ##
  If(!$PreCheckFlag -or ($J.PreCheckFlags -eq $PrecheckFlag)){
    ## Allow if no TransitionOp or have match ##
    If(!$TransitionOp -or ($J.TransitionOp -eq $TransitionOp)){
      ## Construct Header ##
      $Col++
      $FileName = $_.Split("\")[$_.Split("\").Count - 1]
      AM 2 "$Col" $FileName
      AM 3 "$Col" $J.Version
      AM 4 "$Col" $J.SessionName
      AM 5 "$Col" $J.PreCheckFlags
      AM 6 "$Col" $J.TransitionOp
      AM 7 "$Col" $J.StartTime
      AM 8 "$Col" "====="
      $J.JobResultList | Foreach{
        ## Allow if a numeric Result Code ##
        $ResultCode = $_.Result.Code.Trim()
        If($ResultCode -match "^[-+]?([0-9]*\.[0-9]+|[0-9]+\.?)$"){
          ## Check if we've already had this code ##
          If($RecordedCodes -NotContains $ResultCode){
            $RecordedCodes += $ResultCode
            $CodesToRow.$ResultCode = $Row+1
            $Area = "" # NOT IMPLEMENTED YET
            $ResultType = $_.Result.Type
            AM ($Row+1) "1" ($Row+1); AM ($Row+1) "2" $ResultCode; AM ($Row+1) "3" $Area; AM ($Row+1) "4" $ResultType; AM ($Row+1) "5" "Message"
            AM ($Row+2) "1" ($Row+2); AM ($Row+2) "2" $ResultCode; AM ($Row+2) "3" $Area; AM ($Row+2) "4" $ResultType; AM ($Row+2) "5" "Corrective Action"
            AM ($Row+3) "1" ($Row+3); AM ($Row+3) "2" $ResultCode; AM ($Row+3) "3" $Area; AM ($Row+3) "4" $ResultType; AM ($Row+3) "5" "Affected Objects"
            $Row += 3
          }
          $CodeRow = $CodesToRow.$ResultCode
          AM ($CodeRow)   "$Col" $_.Result.Message
          AM ($CodeRow+1) "$Col" $_.Result.CorrectiveAction
          AM ($CodeRow+2) "$Col" ($_.Result.AffectedObjects.ObjectName | Out-String).Trim()
        }
      }
    }
  }
} # End of $FilesToCheck | Foreach

$CSV = @()
For($i=2;$i -le $Row;$i++){ $CSV += $R.$i }
$CSV | FT
$CSV | Where-Object {$_} | Export-CSV -Path $FileOut -NoTypeInformation