Monday, 20 February 2017

Using Python (on Windows) with NetApp ONTAP - Dummy’s Guide

Following on from the previous post, my (Dummy’s) guide for Python with ONTAP!

1) Download Python

This seems like a good place to start: The Hitchhiker's Guide to Python

2) Install Python

Double-click the python-2.7.12.msi and follow the prompts to install.
- Default installation directory is C:\Python27\
- On the Customize Python 2.7.12 screen, I select ‘Add python.exe to Path’ (since I have no other version of Python).

Image: Installing Python for Windows

3) Download the NetApp Manageability SDK (NMSDK)
If not already done so...

> NetApp Manageability SDK > All Platforms

4) NetApp Python Library

Unzip the NMSDK.
To keep my python libraries in one place, I copy the NetApp folder from:

netapp-manageability-sdk-5.6 > lib > python > NetApp

- to -

C:\Python27\Lib

5) Test

Below is a very simple Python script example taken from the NMSDK and ZExplore.

i) Copy and paste into a text editor
ii) Edit CLUSTER_NAME, USER_NAME, and PASSWORD as required.
iii) Save as get-version.py
iv) Run in the DOS command prompt as>
python get-version.py


import sys
sys.path.append("C:/Python27/Lib/NetApp")
from NaServer import *

s = NaServer("CLUSTER_NAME", 1 , 31)
s.set_server_type("FILER")
s.set_transport_type("HTTPS")
s.set_port(443)
s.set_style("LOGIN")
s.set_admin_user("USER_NAME", "PASSWORD")

api = NaElement("system-get-version")

xo = s.invoke_elem(api)
if (xo.results_status() == "failed") :
  print ("Error:\n")
  print (xo.sprintf())
  sys.exit (1)

print ("Received:\n")
print (xo.sprintf())


Note: I was getting the error -
status="failed" reason="[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)" errno="13001"
- and had to change transport_type to HTTP, and port to 80, to get this to work.

Image: Python success!

THE END

Random tip: To change the DOS command prompt to "CMD> ", type in the command prompt>
prompt CMD$g$s

Sunday, 19 February 2017

Using Perl (on Windows) with NetApp ONTAP - Dummy’s Guide

If you’re a regular reader of this blog, you’ll know I love a bit of PowerShell - I’ve tagged 128 posts with PowerShell. Perl - on the other hand - has yet to be tagged on one post. Well, this changes today! The titular “dummy” is me (I’m a Perl noob), and - since I work predominantly from Windows laptop/workstation - this is going to be Perl on Windows...

1) Install Perl

Firstly, we need to install Perl. The following link - http://learn.perl.org/installing/ - contains links to Installing Perl on Windows, OSX, and Unix/Linux.


Image: Padre on Strawberry Perl installed!

Image: Command Prompt/PowerShell and ‘perl -v’ to verify installation.

Image: App-cpanminus installed too.

2) Download the NetApp Manageability SDK (NMSDK)

> NetApp Manageability SDK > All Platforms

3) NetApp Perl Modules

Unzip the NMSDK.
To put all my Perl libraries in one place, I copy the NetApp folder from:

netapp-manageability-sdk-5.6 > lib > perl > NetApp

- to -

C:/Strawberry/perl/lib

4) Test

Below is a very simple Perl script example taken from the NMSDK and ZExplore.

i) Copy and paste into a text editor
ii) Edit CLUSTER_NAME, USER_NAME, and PASSWORD as required.
iii) Save as get-version.pl
iv) Run in the DOS command prompt as>
perl get-version.pl


require 5.6.1;
use lib 'C:/Strawberry/perl/lib/NetApp';
use strict;
use warnings;
use NaServer;
use NaElement;

my $s = new NaServer('CLUSTER_NAME', 1 , 31);
$s->set_server_type('FILER');
$s->set_transport_type('HTTPS');
$s->set_port(443);
$s->set_style('LOGIN');
$s->set_admin_user('USER_NAME', 'PASSWORD');

my $api = new NaElement('system-get-version');

my $xo = $s->invoke_elem($api);
if ($xo->results_status() eq 'failed') {
  print 'Error:\n';
  print $xo->sprintf();
  exit 1;
}

print 'Received:\n';
print $xo->sprintf();


Note: I received an error - errno="13001" - using Cluster Management IP address, so added the name of the cluster to my hosts file and this worked.

Image: Perl success!

THE END

Saturday, 18 February 2017

Treesize for Data ONTAP 7-Mode (Get-NaTreesize.ps1)

Continuing from the previous post...

Since it is pretty trivial to convert Get-NcTreesize.ps1 into a Data ONTAP 7-Mode version, here it is.

A usage example>


$P = Read-Host -AsSecureString

.\Get-NaTreeSize.ps1 -Controller 10.1.7.50 -UserName root -Password $P -ScanStartPath /vol/TVOL001 -OutputFile "7M_TVOL001.CSV"


Image: Example of the raw, unformatted CSV output
The Script


####################
## Get-NaTreesize ##
####################

## A few differences in the 7-Mode version:
# 1) Don't need to specify vFilers!
# 2) There is no -Template with *na* cmdlets.
# 3) There is no Get-NaFileDirectorySecurity.
# N.B.: The APIs in 7-Mode are noticeably slower than in C-Mode!

Param(
  [Parameter(Mandatory=$True)][String]$Controller,
  [Parameter(Mandatory=$True)][String]$UserName,
  [Parameter(Mandatory=$True)][Security.SecureString]$Password,
  # TIP: PS> $P = Read-Host -AsSecureString
  # ... you can use $P after -Password
  [Parameter(Mandatory=$True,HelpMessage="/vol/VOLNAME(/...)")]
  [String]$ScanStartPath,
  [String]$OutputFile = "NaTreesizeReport.CSV",
  [String]$DateFormat = "yyyy.MM.dd"
);""

## GENERIC: Loading PSTK ##
"INFO: DataONTAP PSTK loading ..."
[Void](Import-Module DataONTAP)
If(!(Get-Module DataONTAP)){"ERROR: No DataONTAP PSTK!";EXIT}
"INFO: DataONTAP PSTK loaded.`n"

## GENERIC: Connecting to $Controller ##
"INFO: $Controller connecting ..."
$Global:CurrentNaController = $NULL
$Cred = New-Object System.Management.Automation.PsCredential($UserName,$Password)
[Void](Connect-NaController -Name $Controller -Credential $Cred)
If(!$Global:CurrentNaController){"ERROR: Cannot connect to $Controller!";EXIT}
"INFO: $Controller connected.`n"

## VERIFY $ScanStartPath ##
"INFO: $ScanStartPath verifying ..."
If(!(Read-NaDirectory $ScanStartPath)){"ERROR: $ScanStartPath is incorrect!";EXIT}
"INFO: $ScanStartPath verified.`n"

## GLOBAL VARIABLES ##
[System.Array]$Global:CSV = @() # Output CSV
[Int64]$Global:Order = 0 # For output ordering

## RECURSIVE SCANNING FUNCTION ##
Function GetDirInfoRecursive{
  Param([String]$Path)
  Write-Host "." -NoNewLine # Indicates the program is running.
  $GetDirInfo = Read-NaDirectory $Path
  If(!$GetDirInfo){Return 0} # Return 0 size if failed to read $Path
  $Global:Order ++
  $Local:Order   = $Global:Order
  $Local:DirSize = 0
  [Int]$level    = $Path.Split("/").count - 2
 
  ## CYCLE THROUGH $GetDirInfo ##
  Foreach ($l in $GetDirInfo){          
    If($l.FileType -eq "directory"){
      If($l.Name -eq "."){$CurrentDir = $l}
      If( (".","..",".snapshot" -notcontains $l.Name) -and ($l.IsEmpty -ne "False") ){
        $Local:DirSize += (GetDirInfoRecursive ($Path + "/" + $l.Name))
      }
    }elseif($l.FileType -eq "file"){           
      $Global:Order ++
      $Local:DirSize += $l.FileSize
     
      ## ADD FILE INFORMATION TO CSV ##
      $Global:CSV += [PSCustomObject]@{
        "Order"           = $Global:Order
        "Directory Level" = $Level
        "File Type"       = "file"
        "Name"            = $l.Name
        "Path"            = $Path
        "File Size"       = $l.FileSize
        "Directory Size"  = ""
        "AclType"         = $l.AclType
        "OwnerId"         = $l.OwnerId
        "GroupId"         = $l.GroupId
        "Perm"            = $l.Perm
        "Accessed Time"   = $l.AccessedTimeStampDT.ToString($DateFormat)
        "Changed Time"    = $l.ChangedTimeStampDT.ToString($DateFormat)
        "Creation Time"   = $l.CreationTimeStampDT.ToString($DateFormat)
      }
    }
  }
  ## ADD DIRECTORY INFORMATION TO CSV ##
  $Global:CSV += [PSCustomObject]@{
    "Order"           = $Local:Order
    "Directory Level" = $Level
    "File Type"       = "directory"
    "Name"            = $Path.Split("/")[$level + 1]
    "Path"            = $Path
    "File Size"       = ""
    "Directory Size"  = $Local:DirSize
    "AclType"         = $CurrentDir.AclType
    "OwnerId"         = $CurrentDir.OwnerId
    "GroupId"         = $CurrentDir.GroupId
    "Perm"            = $CurrentDir.Perm       
    "Accessed Time"   = $CurrentDir.AccessedTimeStampDT.ToString($DateFormat)
    "Changed Time"    = $CurrentDir.ChangedTimeStampDT.ToString($DateFormat)
    "Creation Time"   = $CurrentDir.CreationTimeStampDT.ToString($DateFormat)
  }
  Return $Local:DirSize
}

##################
## MAIN PROGRAM ##
##################

("START: " + [String](Get-Date))
[Void](GetDirInfoRecursive $ScanStartPath);""
("END: " + [String](Get-Date) + "`n")
$Global:CSV | Sort "Order" | Export-CSV "$OutputFile" -NoTypeInformation


Friday, 17 February 2017

Treesize for Clustered ONTAP (Get-NcTreesize.ps1)

A colleague asked me whether I knew of anything like “Treesize for NetApp”, I didn’t, so I thought I’d knock up my own version, and here it is.

The recursive function - at the core of the PowerShell script - has been used previously in a fair few posts. The usual ‘Caveat Utilitor’ applies - don’t use this on your production cluster (at least not without serious testing). Check out this post for more background and links "How to reclaim a lot of space and save money" (that wasn’t the original title - it should have been.)

Copy and paste the script to a text document and save as Get-NcTreesize.ps1. The various parameters are detailed at the top of the script. Usage examples:


$P = read-host -AsSecureString

.\Get-NcTreesize.ps1 -Cluster 10.3.5.50 -UserName admin -Password $P -DataSVM SVM1 -ScanStartPath /vol/TEST0001 -OutputFile "TEST0001.CSV"

.\Get-NcTreesize.ps1 -Cluster 10.3.5.50 -UserName admin -Password $P -DataSVM SVM1 -ScanStartPath /vol/TEST0002 -GetFileDirSecInfo -OutputFile "TEST0002.CSV"


Note: -GetFileDirSecInfo substantially slows down Get-NcTreesize. If you don’t want to get file and directory security info, I highly recommend skipping this.

Image: Example of the raw, unformatted CSV output
The Script


####################
## Get-NcTreesize ##
####################

Param(
  [Parameter(Mandatory=$True)][String]$Cluster,
  [Parameter(Mandatory=$True)][String]$UserName,
  [Parameter(Mandatory=$True)][Security.SecureString]$Password,
  # TIP: PS> $P = Read-Host -AsSecureString
  # ... you can use $P after -Password
  [Parameter(Mandatory=$True)][String]$DataSVM,
  [Parameter(Mandatory=$True,HelpMessage="/vol/VOLNAME(/...)")]
  [String]$ScanStartPath,
  [String]$OutputFile = "NcTreesizeReport.CSV",
  [String]$DateFormat = "yyyy.MM.dd",
  [Switch]$GetFileDirSecInfo
);""

## GENERIC: Loading PSTK ##
"INFO: DataONTAP PSTK loading ..."
[Void](Import-Module DataONTAP)
If(!(Get-Module DataONTAP)){"ERROR: No DataONTAP PSTK!";EXIT}
"INFO: DataONTAP PSTK loaded.`n"

## GENERIC: Connecting to $Cluster ##
"INFO: $Cluster connecting ..."
$Global:CurrentNcController = $NULL
$Cred = New-Object System.Management.Automation.PsCredential($UserName,$Password)
[Void](Connect-NcController -Name $Cluster -Credential $Cred)
If(!$Global:CurrentNcController){"ERROR: Cannot connect to $Cluster!";EXIT}
"INFO: $Cluster connected.`n"

## GENERIC: Verifying $DataSVM ##
"INFO: $DataSVM verifying ..."
$vsAttrs = Get-NcVserver -Template
$vsQuery = Get-NcVserver -Template
$vsQuery.VserverType = "data"
[System.Array]$DataSVMs = (Get-NcVserver -Attributes $vsAttrs -Query $vsQuery).VserverName
If($DataSVMs -notcontains $DataSVM){"ERROR: No $DataSVM!";EXIT}
"INFO: $DataSVM verified.`n"
$Global:CurrentNcController.Vserver = $DataSVM

## VERIFY $ScanStartPath ##
"INFO: $ScanStartPath verifying ..."
If(!(Read-NcDirectory $ScanStartPath)){"ERROR: $ScanStartPath is incorrect!";EXIT}
"INFO: $ScanStartPath verified.`n"

## OPTIMIZE READ-NCDIRECTORY ##
# To optimize the process, we only collect the attributes we're interested in.
[System.Array]$L = ,
"AccessedTimestamp",
"ChangedTimestamp",
"CreationTimestamp",
"FileSize",
"FileType",
"IsEmpty",
"Name"
$Attrs = Read-NcDirectory -Template
$L | Foreach{$Attrs.$_ = ""}

## (OPTIONAL) GET JUNCTION PATH FOR FILE DIRECTORY SECURITY ##
If($GetFileDirSecInfo){
  $vAttrs = Get-NcVol -Template
  Initialize-NcObjectProperty -Object $vAttrs -Name VolumeIdAttributes
  $vAttrs.VolumeIdAttributes.JunctionPath = ""
  [String]$Volume = $ScanStartPath.Split("/")[2]
  [String]$JPath = (Get-NcVol -Name $Volume -Attributes $vAttrs).VolumeIdAttributes.JunctionPath
  If(!$JPath){
    "ERROR: Cannot find junction path for $Volume, GetFileDirSecInfo is disabled!`n"
    $GetFileDirSecInfo = $FALSE
  }else{
    $JPath = $JPath.Substring(0,($JPath.Length - 1 - $Volume.Length))
  }
}

## (OPTIONAL) FILE DIRECTORY SECURITY FUNCTION ##
Function Get-NcFileDirSecInfo {
  Param([String]$Path)
  $G = Get-NcFileDirectorySecurity -Path ($JPath + $Path)
  $ACLs = ""
  If($G.Acls){$G.Acls.Split("`n") | Foreach{$ACLs += ($_.Trim(" ") + ";")}}
  $SecInfo = [PSCustomObject]@{
    "DosAttrs"       = $G.DosAttributes
    "DosAttrsText"   = $G.DosAttributesText
    "SecurityStyle"  = $G.SecurityStyle
    "EffectiveStyle" = $G.EffectiveStyle
    "UnixGroupId"    = $G.UnixGroupId
    "UnixModeBits"   = $G.UnixModeBIts
    "UnixUserId"     = $G.UnixUserId
    "ACLs"           = $ACLs
  }
  Return $SecInfo
}

## GLOBAL VARIABLES ##
[System.Array]$Global:CSV = @() # Output CSV
[Int64]$Global:Order = 0 # For output ordering

## RECURSIVE SCANNING FUNCTION ##
Function GetDirInfoRecursive{
  Param([String]$Path)
  Write-Host "." -NoNewLine # Indicates the program is running.
  $GetDirInfo = Read-NcDirectory $Path -Attributes $Attrs
  If(!$GetDirInfo){Return 0} # Return 0 size if failed to read $Path
  $Global:Order ++
  $Local:Order = $Global:Order
  $Local:DirSize = 0
  [Int]$level = $Path.Split("/").count - 2
 
  ## CYCLE THROUGH $GetDirInfo ##
  Foreach ($l in $GetDirInfo){          
    If($l.FileType -eq "directory"){
      If($l.Name -eq "."){ $CurrentDir = $l }
      If(($l.Name -ne ".") -and ($l.Name -ne "..") -and ($l.Name -ne ".snapshot")) {
        If ($l.IsEmpty -ne "False"){
          $Local:DirSize += (GetDirInfoRecursive ($Path + "/" + $l.Name))
        }
      }
    }elseif($l.FileType -eq "file"){           
      $Global:Order ++
      $Local:DirSize += $l.FileSize
     
      ## (OPTIONAL) FILE DIRECTORY SECURITY (FDS) INFO ##
      [System.Object]$FDS = @{}
      If($GetFileDirSecInfo){$FDS = Get-NcFileDirSecInfo ($Path.SubString(4) + "/" + $l.Name)}
     
      ## ADD FILE INFORMATION TO CSV ##
      $Global:CSV += [PSCustomObject]@{
        "Order"           = $Global:Order
        "Directory Level" = $Level
        "File Type"       = "file"
        "Name"            = $l.Name
        "Path"            = $Path
        "File Size"       = $l.FileSize
        "Directory Size"  = ""
        "Accessed Time"   = $l.AccessedTimeStampDT.ToString($DateFormat)
        "Changed Time"    = $l.ChangedTimeStampDT.ToString($DateFormat)
        "Creation Time"   = $l.CreationTimeStampDT.ToString($DateFormat)
        "DosAttrs"        = $FDS."DosAttrs"
        "DosAttrsText"    = $FDS."DosAttrsText"
        "SecurityStyle"   = $FDS."SecurityStyle"
        "EffectiveStyle"  = $FDS."EffectiveStyle"
        "UnixGroupId"     = $FDS."UnixGroupId"
        "UnixModeBits"    = $FDS."UnixModeBits"
        "UnixUserId"      = $FDS."UnixUserId"
        "ACLs"            = $FDS."ACLs"
      }
    }
  }   
 
  ## (OPTIONAL) FILE DIRECTORY SECURITY (FDS) INFO ##
  [System.Object]$FDS = @{}
  If($GetFileDirSecInfo){$FDS = Get-NcFileDirSecInfo $Path.SubString(4)}
 
  ## ADD DIRECTORY INFORMATION TO CSV ##
  $Global:CSV += [PSCustomObject]@{
    "Order"           = $Local:Order
    "Directory Level" = $Level
    "File Type"       = "directory"
    "Name"            = $Path.Split("/")[$level + 1]
    "Path"            = $Path
    "File Size"       = ""
    "Directory Size"  = $Local:DirSize
    "Accessed Time"   = $CurrentDir.AccessedTimeStampDT.ToString($DateFormat)
    "Changed Time"    = $CurrentDir.ChangedTimeStampDT.ToString($DateFormat)
    "Creation Time"   = $CurrentDir.CreationTimeStampDT.ToString($DateFormat)
    "DosAttrs"        = $FDS."DosAttrs"
    "DosAttrsText"    = $FDS."DosAttrsText"
    "SecurityStyle"   = $FDS."SecurityStyle"
    "EffectiveStyle"  = $FDS."EffectiveStyle"
    "UnixGroupId"     = $FDS."UnixGroupId"
    "UnixModeBits"    = $FDS."UnixModeBits"
    "UnixUserId"      = $FDS."UnixUserId"
    "ACLs"            = $FDS."ACLs"
  }
  Return $Local:DirSize
}

##################
## MAIN PROGRAM ##
##################

("START: " + [String](Get-Date))
[Void](GetDirInfoRecursive $ScanStartPath);""
("END: " + [String](Get-Date) + "`n")
$Global:CSV | Sort "Order" | Export-CSV "$OutputFile" -NoTypeInformation


Note: This was tested against Clustered Data ONTAP 8.3.2.