Sunday, 30 July 2017

NetApp Technical Report (TR) PowerShell Collector Script: Version 2


Version 2 Additions

1) Checks for file existence on the TR-XXXX part (not the whole file name)
3) Logs updated files and new files (as well as dead links - the log filename has been changed)

Image: Correctly named PDFs

The Script

Copy and paste into a text editor and save as say NetAppTR_Collector.ps1. The run in PowerShell with>
.\NetAppTR_Collector.ps1


########################################
## NetAppTR_Collector.ps1 (Version 2) ##
########################################

# This program will collect all the TRs from -
# https://www.netapp.com/us/media/tr-XXXX.pdf
# - where XXXX is specified by $RangeStart to $RangeEnd
# Version 2 adds:
# 1) Checks for file existence on the TR-XXXX part (not the whole file name)
# 2) Automatically names the PDFs from http://www.cosonok.com/2017/07/netapp-technical-reports-catalogue.html
# 3) Logs updated files and new files (as well as dead links - the log filename has been changed)

Param(
  [Int]$RangeStart = 4000,
  [Int]$RangeEnd   = 4700,
  [String]$DownloadFolder = "C:\Downloads\NetAppTRs\",
  [String]$LogFile = "NetAppTR_Collector.log"
)

# Checks if download folder path has \ at the end:
If($DownloadFolder -match '.+?\\$'){}
else{$DownloadFolder += "\"}

[String]$LogFilePath = $DownloadFolder + $LogFile
[String]$CatalogDL = "http://www.cosonok.com/2017/07/netapp-technical-reports-catalogue.html"

# Generic display function:
Function Wr{Param($P,$I="WHITE");Write-Host $P -ForegroundColor $I}

##########################
## TITLE EXTRACTER CODE ##
##########################

## Acquire the NetApp TR Catolog if not already acquired
If($Global:NaTrCatalog){}
Else{
  Try{$Global:NaTrCatalog = Invoke-Webrequest $CatalogDL -Method Get}
  Catch{
    Wr "FAILED: Unable to acquire catalog!" YELLOW
  }
}

## Some variables:
If($Global:NaTrCatalog){
  [System.Array]$Arr = ($Global:NaTrCatalog).Content.Split("`n")
  $Catalog = ($Global:NaTrCatalog).Content
  $Count = $Arr.Count 
}
$Position = 0

## Get-Title function:
Function Get-Title{
  If($Catalog.Contains("TR-$($j):")){}
  Else{RETURN "TR-$($j)"}
  For($k=$Position;$k -lt $Count;$k++){
    If($Arr[$k].Contains("TR-$($j):")){
      $Position++
      [String]$Title = $Arr[$k].Split(">")[1]
      $l=1
      While($TRUE){
        If($Title.Contains("<")){
          $Title = $Title.Split("<")[0]
          RETURN $Title
        }else{
          $Title += " "
          $Title += $Arr[$k+$l]
        }
        $l++
        If(($k+$l) -ge $Count){
          Wr "FAILED: Something went wrong here!" RED;PAUSE;EXIT
        }
      }
    }
  }
}

## Create hashtable of titles to number XXXX:
[System.Object]$Titles = @{}
For($j=$RangeEnd;$j -ge $RangeStart;$j--){
  [String]$Titles."$j" = "TR-$($j)"
  If($Global:NaTrCatalog){$Titles."$j" = (Get-Title).Replace(":"," -").Replace("/",",")}
}

#################################
## PREPARE DOWNLOAD FOLDER/LOG ##
#################################

[Void](New-Item -ItemType Directory -Force -Path $DownloadFolder)
If(Test-Path $DownloadFolder){}
else{Wr "FAILED: Test-Path $DownloadFolder" RED;PAUSE;EXIT}
If(Test-Path $LogFilePath){
  Copy-Item $LogFilePath ($LogFilePath + ".bak")
}else{
  [Void](New-Item $LogFilePath -type file -force)
}

######################
## PDF RENAMER CODE ##
######################

Get-ChildItem $DownloadFolder | Foreach{
  $CIname = $_.Name
  If($CIname.StartsWith("TR-","CurrentCultureIgnoreCase")){
    [String]$TRnumber = $CIname.Split("-")[1]
    $TRnumber = $TRnumber.Substring(0,4)    
    If($Titles.$TRnumber){
      If($CIname -ne ($Titles.$TRnumber + ".pdf")){
        Wr "Renaming item: $($DownloadFolder)$($CIname) to $($Titles.$TRnumber).pdf"
        Rename-Item ($DownloadFolder + $CIname) ($Titles.$TRnumber + ".pdf")
      }
    }
  }
}

#######################
## TR COLLECTOR CODE ##
#######################

## Download the TRs:
Import-Module BitsTransfer
For($i=$RangeStart; $i -le $RangeEnd; $i++){
  $Title = "tr-$i"
  $WebUrl = "https://www.netapp.com/us/media/$Title.pdf"
  $SavePath = ($DownloadFolder + $Titles."$i" + ".pdf")

  ## Get the header:
  $hdr = $NULL
  Try{$hdr = Invoke-WebRequest $WebUrl -Method Head}
  Catch{"Dead link: $WebUrl" | Out-File $LogFilePath -Append}

  ## If we have a header:
  If($hdr){

    ## Only download updated versions:
    $Download = $FALSE      
    If(Test-Path $SavePath){
      $SavedLRT = [DateTime]((Get-ChildItem $SavePath).LastWriteTime)
      $WebFileLRT = [DateTime]($hdr.headers."last-modified")
      If($WebFileLRT.Ticks -gt $SavedLRT.Ticks){
        Wr "UPDATED   : $Title" GREEN
        "Updated  : $Title" | Out-File $LogFilePath -Append
        $Download = $TRUE
      }else{
        Wr "NO UPD.   : $Title"
      }
    }else{
      Wr "NEW D/LOAD: $Title" GREEN
      "New D/L  : $Title" | Out-File $LogFilePath -Append
      $Download = $TRUE
    }

    ## Download:
    If($Download){
      Start-BitsTransfer -Source $WebUrl -Destination $SavePath
    }
  }else{
    Wr "DEAD LINK : $WebUrl"
  }
}


4 comments:

  1. This comment has been removed by the author.

    ReplyDelete
  2. This script is very useful.
    I especially like renaming files :).

    The management of the documentation is painful, but the management of the sources and last version are a hell.
    With the release of versions P "x" "without warning.
    Sometimes Netapp recommends Px versions, sometimes not. And there is no listing of these versions.

    I think, i am not the only one confronted with these version problems. Has anyone ever made a script or page referencing the versions of netapp tools (snapdrive, sme, smsql ... etc)?

    ReplyDelete
  3. Hello Guillaume,
    I believe this has changed. NetApp now publish a minimum recommended P-Release, and all the later releases are regarded as being better (the idea is not to have to chase updates.) My rule would be to use the latest P-release that has been out at least 1 month...
    Cheers,
    VC

    ReplyDelete
  4. Hi,

    I agree, the chase updates is not a good thing. I think we all had an unfortunate experience with a last version.

    What bothers me is that there is no announcement (or I have not found) about possible release of P version fixing some problems.
    And sometimes you have to start downloading the version to get a warning about a P release.

    The idea was just to be informed and keep a listing of the versions. Not to chase updates ;).
    In case of a bug, the support always redirects to the link of a P version if the bug is already known / corrected.

    Thank you.
    Best

    Guillaume

    ReplyDelete