Saturday, 30 August 2014

PowerShell Script to Index Functions

Getting ever more ambitious with PowerShell scripts - some over 1000 lines, and growing all the time - I thought it would be useful to have an index/contents page of functions at the start of the script (my PowerShell editor of choice is NotePad++.)

The below script can be run as:

.\Index-Functions.ps1 filename.ps1

And it puts the index at the start of the PowerShell script/updates the index. It uses a couple of useful additional functions that are included below.

The Script - Index-Functions.ps1

Apologies for the formatting, I really should use something else for posting PS Scripts!

# START OF SCRIPT #

## Function Pad-Right ##
## ================== ##

Function Pad-Right{
       Param([String]$in, [Int]$padR = 30, [String]$pad = " ")
       $in.PadRight($padR,$pad).SubString(0,$padR)
}

## Function Prompt-Menu ##
## ==================== ##

Function Prompt-Menu{

       # DESCRIPTION: Prompts for valid answers to a menu.
       # PARAMETER $args[0]: The question
       # PARAMETER $args[1]/$args[1..X]: An array with answers in / the answers

       $question = $args[0]
      
       # Check if $args[1] is an array (i.e. has more than 1 element)
       If(($args[1].count) -ne 1){
              $answers = $args[1] # Possible answers is array $args[1]
       } else {
              $answers  = $args[1..(($args.count)-1)] # Possible answers are args 1 to ...
       }

       # Checks if the answer ($readIn) is valid
       do {
              $readIn = Read-Host "$question"
              $readIn = $readIn.ToUpper()
              foreach($answer in $answers){
                     if($readIn -eq $answer.ToUpper()){return $readIn}
              }
       } while($true) # An infinite loop!
}

## Function Index-Functions ##
## ======================== ##

Function Index-Functions {

       # DESCRIPTION: Injects/Updates an Index of Functions at the start of the script.
       # PARAMETER filename: Name of the PS script file to be indexed.

       Param([String]$filename)
       If(!$filename){return}             # If no filename paramater - return
       If(!(Test-Path $filename)){return} # If test-path fails - return    
       $contents = Get-Content $filename  # Get contents of the file
       If(!$contents){return}             # If no contents (empty file) - return
      
       # $indexDisplay starts with a basic header
       $indexDisplay = "<#","Line  Function","~~~~  ~~~~~~~~"
      
       # Check for an existing index ($hasIndex = true/false) #
       # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #
             
       If($contents[1] -eq "Line  Function"){  
              $hasIndex    = $true           # We have an index
              $endOfIndex  = $false          # Not found the end of the existing index
              $contentRows = $contents.Count # Count the number of rows in $contents
              $i=3 # Check from 4th line since header is 3 lines (arrays start at 0)
              do {
                     # We're looking for the #> at the end of the index
                     If($contents[$i].contains("#>")){              
                           $endOfIndex = $true # Found the end of the index
                           # Remove the index from the $contents
# ($i+2 because have a blank line after the index)
                           $contents = $contents[($i+2)..$contentRows]
                     }
                     $i++ # Accumlate $i for the next row
              } while (!$endOfIndex) # Stops when found end of index
       }else{$hasIndex = $false} # Othewise there is no index
      
       # Search $contents for functions and create the index #
       # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #
                    
       $i = 1       # Start on line 1 of the script
       $index = @() # Initialize the index array
       foreach ($content in $contents){ 
              $capitalize = $content.ToUpper() # Capitalized version of the row         
              # For a Function we expect FUNCTION at the start of the line
              If($capitalize.StartsWith("FUNCTION")){               
                     $split = $content.Split("{")  # Split row at "{" to get [0] "Function Name"
                     $split = $split[0].Split(" ") # Split at " " to get [1] "Name"
                     $index += $i                  # Add line number to index array
                     $index += $split[1]           # Add Function name to array
              }
              $i ++ # Accumlate $i for the next row
       }
      
       # Construct the index #
       # ~~~~~~~~~~~~~~~~~~~ #
      
       $i = 0
       $linesInIndex = ($index.count / 2) # $linesInIndex has 'line number', 'function name'...
       do {
              # Take line number from $index, add number of lines in the index to it,
# +5 for index header and footer
              $lineNumber = ($index[2*$i] + $linesInIndex + 5).ToString()
              # Add to IndexDisplay line number with a ')', and
# pad it right with " " for 6 columns; then add the function name
              $indexDisplay += (Pad-Right ($lineNumber + ")") 6) + ($index[2*$i+1])
              $i++ # Accumulate $i for the next row
       } while ($i -lt $linesInIndex)
       $indexDisplay += "#>","" # Adds a footer to the index
      
       # Displays the index, prompt (Y/N), inject new/updated index into script file #
       # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #
             
       ""
       $indexDisplay # Display the Index
      
       # Change screen output depending on whether the file had an index before
       If ($hasIndex){$bleep = "replace the index at"}else{$bleep = "be added to"}
       "The above Index will $bleep the start of $filename."
       "The last saved version will be saved with the .backup extension.";""
      
       # Uses Prompt-Menu function with Y/N as the only possible answers
       $answer = Prompt-Menu "Do you want to continue (Y/N)?" "Y" "N"

       If($answer -eq "Y"){
              $backupFile = $filename + ".backup"
              If(Test-Path $backupFile){Remove-Item $backupFile}
              Rename-Item $filename $backupFile
              [Void](New-Item $filename -type file -force)
              $indexDisplay += $contents # Contents is added to index (at start of file)
              $indexDisplay | Out-File $filename
       }
       ""
}

Index-Functions $args[0]

# END OF SCRIPT #

Using PowerShell to Set DFS Override Referral Ordering - UPDATED

Quite a nice update to the script originally posted here. Nice features of this script are that it handles timeouts. And I rather like the Columnize function. No more commentary, now for the script!

#########
# Title #
#########

"";"Set DFSN Override Referral Ordering"
   "===================================";""

#############################################
# Check for Windows Server 2012 requirement #
#############################################

$OSversion = [System.Environment]::OSVersion.Version
if ($OSversion.major -le 6){
       if ($OSversion.minor -lt 2){
              "This script needs to be run on a Windows 2012 Server or later!";"";EXIT}}

#####################################
# Check for DFSN module requirement #
#####################################

if(!(import-module -Name DFSN -PassThru -ErrorAction SilentlyContinue)){
       "Unable to Import-Module DFSN!";"";EXIT}

###################################
# Display some namespace examples #
###################################

$xDomain   = "lab.priv"
$xDFolder  = "ns001"
$xComputer = "fileserver01.lab.priv"
$xCFolder  = "data001"
      
"Domain-based namespace examples:"
"\\$xDomain\$xDFolder";""
"Stand-alone namespace examples:"
"\\$xComputer\$xCFolder";""

###################################################################################################
# This function prompts for namespace type and recursively calls itself if not answered correctly #
###################################################################################################

Function NameSpaceType{
       $readInput = Read-Host "Is this a domain-based or stand-alone namespace (enter D or S)?"
       If($readInput.ToUpper() -eq "D"){Return "D";break}
       ElseIf($readInput.ToUpper() -eq "S"){Return "S";break}
       Else{NameSpaceType}}

$nameSpaceType = NameSpaceType

##############################################
# Request input of domain name/computer name #
##############################################

If($nameSpaceType -eq "D"){
       $xNamespace = "\\$xDomain\$xDFolder"
       $domain = Read-Host "Enter the domain name (e.g. $xDomain)?"
       $ThisJob = Start-Job -Scriptblock {param ($input) Get-DfsnRoot -Domain $input -ErrorAction SilentlyContinue} -ArgumentList $domain}

If($nameSpaceType -eq "S"){
       $xNamespace = "\\$xComputer\$xCFolder"
       $computer = Read-Host "Enter the computer name (e.g. $xComputer)?"
       $ThisJob = Start-Job -Scriptblock {param ($input) Get-DfsnRoot -Computername $input -ErrorAction SilentlyContinue} -ArgumentList $computer}
""

######################################################################
# This function implements a timeout if $ThisJob is taking too long. #
# It uses $ThisJob,$seconds,$exitedJob from outside the function.    #
######################################################################

$seconds = 15
$exitedJob = $false

Function HandleTimeout{
       "This will timeout after $seconds seconds!"
       $i = 1
       $Timer = [System.Diagnostics.Stopwatch]::StartNew()
       While ($ThisJob | Get-Job | where {$_.State -imatch "Running"}){
              If ($Timer.Elapsed.Seconds -ge $seconds){
                     "Job timed out!"
                     $exitedJob = $true
                     $ThisJob | Get-Job | Remove-Job -Force
                     break}
              Start-Sleep -Seconds 1
              "$i seconds has passed..."
              $i++}
       [Void]($Timer.Stop)}

HandleTimeout

####################################################################################
# If Get-DfsnRoot has not timed-out, display the results (if any), otherwise don't #
####################################################################################

$Results = $ThisJob | where {$_.State -inotmatch "failed"} | receive-job

If (!$exitedJob){
       If (!$Results){"No results from Get-DfsnRoot.";""}
       Else {$Results | Format-Table -Autosize}}
Else {"Timed out after $seconds seconds running Get-DfsnRoot.";""}

########################
# Prompt for Namespace #
########################

$namespace  = Read-Host "Enter the DFS Namespace (e.g. $xNamespace)?"
$namespace += "\*"
""

##################################
# Get DFSN Folders in $namespace #
##################################

$ThisJob   = Start-Job -Scriptblock {param ($input) Get-DfsnFolder -path $input -ErrorAction SilentlyContinue} -ArgumentList $namespace
$seconds   = 30
$exitedJob = $false
HandleTimeout
$folders   = $ThisJob | where {$_.State -inotmatch "failed"} | receive-job

If (!$exitedJob){
       If (!$folders){"No results from Get-DfsnFolder.";"";EXIT}}
Else {"Timed out after $seconds seconds running Get-DfsnFolder.";"";EXIT}
""

########################
# Formatting Functions #
########################

Function PadR{
       Param([string]$item,[int]$padR)
       $item.PadRight($padR," ").SubString(0,$padR)}  

Function Columnize{
       $argsCount = $args.count
       $outString = ""
       $i = 0
       do {
              $outString += PadR $args[$i] $args[$i+1] + " "
              $i+=2
       } while ($i -le $argsCount)
       $outString}

######################################
# Output the Folders and TargetPaths #
######################################

$x1 = 30;$x2 = 30;$x3 = 7;$x4 = 21
Columnize "Path" $x1 "TargetPath" $x2 "State" $x3 "ReferralPriorityClass" $x4
Columnize "----" $x1 "----------" $x2 "-----" $x3 "---------------------" $x4

$start = $true            
$folders | foreach{
       $targets = Get-DfsnFolderTarget -path $_.Path
       if ($start){
              $targetsTable = $targets
              $start = $false
       }Else{
              $targetsTable += $targets}
       $targets | foreach{
              Columnize ($_.Path) $x1 ($_.TargetPath) $x2 ($_.State) $x3 ($_.ReferralPriorityClass) $x4}}
""

###########################################
# Prompt for First/Last among all targets #
###########################################

$serverFirst = Read-Host "Enter the server NETBIOS\FQDN to be set as 'First among all targets'"
$serverLast  = Read-Host "Enter the server NETBIOS\FQDN to be set as 'Last among all targets' "
""
Columnize "Path" $x1 "TargetPath" $x2 "State" $x3 "ReferralPriorityClass" $x4
Columnize "----" $x1 "----------" $x2 "-----" $x3 "---------------------" $x4

########################################
# Set GlobalHigh/GlobalLow as required #
########################################

$i=0

$targetsTable | foreach{
       If (($_.TargetPath).contains($serverFirst)){
              [Void](Set-DfsnFolderTarget -Path $_.Path -TargetPath $_.TargetPath -ReferralPriorityClass GlobalHigh)
              Columnize ($_.Path) $x1 ($_.TargetPath) $x2 ($_.State) $x3 "global-high" $x4
              $i++}}

$targetsTable | foreach{
       If (($_.TargetPath).contains($serverLast)){
              [Void](Set-DfsnFolderTarget -Path $_.Path -TargetPath $_.TargetPath -ReferralPriorityClass GlobalLow)
              Columnize ($_.Path) $x1 ($_.TargetPath) $x2 ($_.State) $x3 "global-low" $x4
              $i++}}

""
If ($i -eq 0){"Found no matches!";""}

###########
# THE END #
###########

Tuesday, 19 August 2014

Enabling Compression on a SnapVault Secondary Volume

I was a bit curious about the effects of doing this after having encountered a warning when doing it via the CLI, so did a quick experiment to learn nothing much really happens. Initially, everything is locked away by Snapshots; it is only as new data comes across that the compression can be applied followed by dedupe on the new data which is rehydrated (if required*) from the deduped source.

*If volume efficiency is run on the new source data after the new data has gone to the vault, that data has never been deduped, so doesn’t require rehydration (at least that’s my theory - set volume efficiency on the primary to run after the vault is taken)!

The Lab Output

1) Create a volume efficiency policy on the source:

PRICLU1::> volume efficiency policy create -vserver PRICLU1V1 -policy 0010_for_6hrs -schedule daily -enabled true -qos-policy background -duration 6
PRICLU1::> volume efficiency policy show -vserver PRICLU1V1

                      Job        Duration QoS
Vserver  Policy Name  Schedule   (Hours)  Policy      Enabled
-------- ------------ ---------- -------- ----------- -------
PRICLU1V1
         0010_for_6hrs
                      daily      6        background  true  

2) Show there is currently no deduped data:

PRICLU1::> volume show-space -vserver PRICLU1V1 -volume v_PKD

                        Vserver: PRICLU1V1
                    Volume Name: v_PKD
                      User Data: 253.3MB
              User Data Percent: 12%
                  Deduplication: -
          Deduplication Percent: -
        Temporary Deduplication: -
Temporary Deduplication Percent: -
                     Total Used: 312.3MB
             Total Used Percent: 15%

3) Enable volume efficiency on the source:

PRICLU1::> volume efficiency on -vserver PRICLU1V1 -volume v_PKD

Efficiency for volume "v_PKD" of Vserver "PRICLU1V1" is enabled.
Already existing data could be processed by running "volume efficiency start -vserver PRICLU1V1 -volume v_PKD -scan-old-data true".

PRICLU1::> volume efficiency start -vserver PRICLU1V1 -volume v_PKD -scan-old-data true

Warning: This operation scans all of the data in volume "v_PKD" of Vserver "PRICLU1V1". It may take a significant time, and may degrade performance during that time. Do you want to continue? {y|n}: y
The efficiency operation for volume "v_PKD" of Vserver "PRICLU1V1" has started.                                                                                

PRICLU1::> volume efficiency show -vserver PRICLU1V1

Vserver    Volume           State    Status       Progress           Policy
---------- ---------------- -------- ------------ ------------------ ----------
PRICLU1V1  v_PKD            Enabled  Active       257240 KB Scanned  -

PRICLU1::> volume efficiency show -vserver PRICLU1V1

Vserver    Volume           State    Status       Progress           Policy
---------- ---------------- -------- ------------ ------------------ ----------
PRICLU1V1  v_PKD            Enabled  Idle         Idle for 00:38:54  -

PRICLU1::> volume efficiency modify -vserver PRICLU1V1 -volume v_PKD -policy 0010_for_6hrs
PRICLU1::> volume show-space -vserver PRICLU1V1 -volume v_PKD

                        Vserver: PRICLU1V1
                    Volume Name: v_PKD
                      User Data: 253.3MB
              User Data Percent: 12%
                  Deduplication: 12KB
          Deduplication Percent: 0%
        Temporary Deduplication: -
Temporary Deduplication Percent: -
                     Total Used: 312.5MB
             Total Used Percent: 15%

Note: The dedupe is a tiny 12KB because dedupe was enabled after Snapshots - which had previously been taken. Best practice is to enable dedupe before Snapshots are taken - but it will still dedupe the new stuff if we turn it on after!

4) Create a Snapshot with a Snapmirror label that’s part of the SnapVault policy - so it will be vaulted when we do a SnapMirror update - and run the SnapMirror update from the SnapVault destination:

PRICLU1::> volume snapshot create -vserver PRICLU1V1 -volume v_PKD -snapshot daily.2014-08-19_2058 -snapmirror-label daily

PRICLU2::> snapmirror update *
Operation is queued: snapmirror update of destination "PRICLU2V1:v_PKD".
1 entry was acted on.

PRICLU2::> snapmirror show
                                                                      
Source            Destination  Mirror  Relationship  Total           
Path        Type  Path         State   Status        Progress  Healthy
----------- ---- ------------ ------- -------------- --------- -------
PRICLU1V1:v_PKD
            XDP  PRICLU2V1:v_PKD
                              Snapmirrored
                                      Idle           -         true  

5) Even though the data is deduped on the destination (deduped changes are carried across from the source), dedupe (Volume Efficiency) needs to be enabled so we can see it:

PRICLU2::> volume show-space -vserver PRICLU2V1 -volume v_PKD

                        Vserver: PRICLU2V1
                    Volume Name: v_PKD
                      User Data: 254.3MB
              User Data Percent: 12%
                  Deduplication: -
          Deduplication Percent: -
        Temporary Deduplication: -
Temporary Deduplication Percent: -
                     Total Used: 357.5MB
             Total Used Percent: 17%

PRICLU2::> volume efficiency on -vserver PRICLU2V1 -volume v_PKD

Efficiency for volume "v_PKD" of Vserver "PRICLU2V1" is enabled.
Already existing data could be processed by running "volume efficiency start -vserver PRICLU2V1 -volume v_PKD -scan-old-data true".

PRICLU2::> volume efficiency show

Vserver    Volume           State    Status       Progress           Policy
---------- ---------------- -------- ------------ ------------------ ----------
PRICLU2V1  v_PKD            Enabled  Idle         Idle for 00:00:09  -

PRICLU2::> volume show-space -vserver PRICLU2V1 -volume v_PKD

                        Vserver: PRICLU2V1
                    Volume Name: v_PKD
                      User Data: 254.3MB
              User Data Percent: 12%
                  Deduplication: 12KB
          Deduplication Percent: 0%
        Temporary Deduplication: -
Temporary Deduplication Percent: -
                     Total Used: 357.5MB
             Total Used Percent: 17%

Note: You cannot do a “volume efficiency start” on a SnapVault secondary:

PRICLU2::> volume efficiency start -vserver PRICLU2V1 -volume v_PKD

Error: command failed: Failed to start efficiency on volume "v_PKD" of Vserver "PRICLU2V1": Invalid operation on a SnapVault secondary volume.

6) Enable volume efficiency on the SnapVault destination, and see that even though there’s a very big warning, nothing much happens after running the command (keep in mind the above that we can’t manually start volume efficiency):

PRICLU2::> volume efficiency modify -vserver PRICLU2V1 -volume v_PKD -compression true

Warning: Enabling compression on a secondary volume is strongly discouraged. If compression is enabled on a secondary volume, storage efficiency present on the source will not be preserved during replication. The destination system needs to run offline storage efficiency scanner (compression and dedupe) to achieve storage savings. Additional compression savings on the destination comes at a cost of extra computation resources. In environments where there is a lot of shared data present on the source, (e.g., virtualized environments employing file clones), data inflation during transfer may lead to failed backups due to lack of space on the secondary volume.
Do you want to continue? {y|n}: y
Info: Volume "v_PKD" of Vserver "PRICLU2V1" is a SnapVault destination and the following volume efficiency schedule attributes are not applicable - "Job Schedule", "Duration", and "Enabled".

PRICLU2::> volume efficiency show -vserver PRICLU2V1

Vserver    Volume           State    Status       Progress           Policy
---------- ---------------- -------- ------------ ------------------ ----------
PRICLU2V1  v_PKD            Enabled  Idle         Idle for 00:03:02  -

PRICLU2::> volume show-space -vserver PRICLU2V1 -volume v_PKD

                        Vserver: PRICLU2V1
                    Volume Name: v_PKD
                      User Data: 253.3MB
              User Data Percent: 12%
                  Deduplication: 12KB
          Deduplication Percent: 0%
        Temporary Deduplication: -
Temporary Deduplication Percent: -
                     Total Used: 356.4MB
             Total Used Percent: 17%

7) We create and apply the volume efficiency policy on the destination:

PRICLU2::> volume efficiency policy show
This table is currently empty.

PRICLU2::> volume efficiency policy create -vserver PRICLU2V1 -policy 0010_for_6hrs -schedule daily -enabled true -qos-policy background -duration 6

PRICLU2::> volume efficiency modify -vserver PRICLU2V1 -volume v_PKD -policy 0010_for_6hrs
Info: Volume "v_PKD" of Vserver "PRICLU2V1" is a SnapVault destination and the following volume efficiency schedule attributes are not applicable - "Job Schedule", "Duration", and "Enabled".

PRICLU2::> volume efficiency show
Vserver    Volume           State    Status       Progress           Policy
---------- ---------------- -------- ------------ ------------------ ----------
PRICLU2V1  v_PKD            Enabled  Idle         Idle for 00:19:21  0010_for_6hrs

8) Finally confirm that compression is indeed enabled on the destination but not on the source:

PRICLU2::> volume efficiency show -fields compression
vserver   volume compression
--------- ------ -----------
PRICLU2V1 v_PKD  true

PRICLU1::> volume efficiency show -fields compression
vserver   volume   compression
--------- -------- -----------
PRICLU1V1 v_PKD    false

THE END!