Monday, 23 March 2015

How to Compress Old SnapVault Data in Clustered Data ONTAP

The following post shows - by example - a couple of methods by which this can be done.

We have a compressible but not very de-duplicable dataset:

Image: Uncompressed
Image: Compressed (to zip with 7-Zip)
On the Production SVM

Create a volume with Snapshot Policy none on the Production SVM, and a CIFS share ::>

vol create -vserver PRSVM -volume DATAGEN -aggregate aggr01 -size 2.5g -language en -junction-path /DATAGEN -type rw -space-guarantee none -snapshot-policy none

cifs share create -vserver PRSVM -share-name DATAGEN -path /DATAGEN

Add the dataset to the volume, enable dedupe (no compression), scan-old-data, and wait for the volume efficiency process to finish ::>

vol efficiency on -vserver PRSVM -volume DATAGEN

vol efficiency start -vserver PRSVM -volume DATAGEN -scan-old-data true

vol efficiency show -vserver PRSVM -volume DATAGEN

vol show -vserver PRSVM -volume DATAGEN -fields used,dedupe-space-saved,compression-space-saved

volume  used    dedupe-space-saved compression-space-saved
------- ------- ------------------ -----------------------
DATAGEN 555.5MB 436KB              0B

For testing, create a 5minutely snapshot policy and apply to the volume DATAGEN ::>

snapshot policy create -policy 5minutely -enabled true -schedule1 5min -count1 12 -snapmirror-label1 5min

volume modify -vserver PRSVM -volume DATAGEN -snapshot-policy 5minutely

On the SnapVault SVM

Method 1: Using -snapshot-blocks true

Create a SnapVault (DP) volume, SnapVault policy, and create and initialize the SnapVault with a schedule of none ::>

vol create -vserver SVSVM -volume DATAGEN -aggregate aggr01 -size 2.5g -language en -type DP -space-guarantee none

snapmirror policy create -vserver SVSVM -policy 5minutely

snapmirror policy add-rule -vserver SVSVM -policy 5minutely -snapmirror-label 5min -keep 60

snapmirror create -source-path PRSVM:DATAGEN -destination-path SVSVM:DATAGEN -type XDP -policy 5minutely

snapmirror initialize -destination-path SVSVM:DATAGEN

Then we enable compression on the vault, scan-old-data and snapshot-blocks, wait for the volume efficiency task to finish, and get the volume stats ::>

vol efficiency on -vserver SVSVM -volume DATAGEN

vol efficiency modify -vserver SVSVM -volume DATAGEN -compression true

set advanced

vol efficiency start -vserver SVSVM -volume DATAGEN -scan-old-data true -snapshot-blocks true

vol efficiency show -vserver SVSVM -volume DATAGEN

vol show -vserver SVSVM -volume DATAGEN -fields used,dedupe-space-saved,compression-space-saved

volume  used    dedupe-space-saved compression-space-saved
------- ------- ------------------ -----------------------
DATAGEN 652.0MB 508KB              442.5MB

NB: To use -snapshot-blocks true you must specify -scan-old-data true also (the command will error otherwise.)

Method 2: Enable Compression before Creating the SnapVault

The second method is even simpler:

Create a SnapVault (DP) volume and turn on volume efficiency before creating the SnapVault relationship and initializing it ::>

vol create -vserver SVSVM -volume DATAGEN2 -aggregate aggr01 -size 2.5g -language en -type DP -space-guarantee none

vol efficiency on -vserver SVSVM -volume DATAGEN2

vol efficiency modify -vserver SVSVM -volume DATAGEN2 -compression true

snapmirror create -source-path PRSVM:DATAGEN -destination-path SVSVM:DATAGEN2 -type XDP -policy 5minutely

snapmirror initialize -destination-path SVSVM:DATAGEN2

Then wait for volume efficiency to complete (kicks in after the SnapMirror has finished initializing), and get the volume stats ::>

vol efficiency show -vserver SVSVM -volume DATAGEN2

vol show -vserver SVSVM -volume DATAGEN2 -fields used,dedupe-space-saved,compression-space-saved

volume   used    dedupe-space-saved compression-space-saved
-------- ------- ------------------ -----------------------
DATAGEN2 108.0MB 64KB               445.3MB

Further Information

With Method 1 you will need to make sure that the Snapshot holding all the block changes is deleted (perhaps by the SnapMirror update itself), otherwise the volume won’t realize the savings from compression.

Volume used after the post initialization compression (volume efficiency) operation:

volume  used    dedupe-space-saved compression-space-saved
------- ------- ------------------ -----------------------
DATAGEN 652.0MB 516KB              442.7MB
        108.0MB 64KB               445.3MB

Since the SnapMirror had been created with no schedule, we have just these baseline snapshots, so an update will be enough to clear the baseline Snapshot down.

Volume  Snapshot                        State        Size
------- ------------------------------- -------- --------
                                        valid     541.7MB
                                        valid       140KB

After the SnapMirror update for DATAGEN:

volume  used    dedupe-space-saved compression-space-saved
------- ------- ------------------ -----------------------
DATAGEN 111.7MB 516KB              442.7MB
        108.0MB 64KB               445.3MB

Sunday, 22 March 2015

Setting up the Clustered Data ONTAP Simulator 8.2.2P1 on VMware ESXi 5.5

2) Browse down and we’re going to get Simulate ONTAP 8.2.2P1 for VMware ESX for Clustered ONTAP.
3) Click the Clustered-ONTAP link to download the vsim_esx-cm.tgz
4) Once it has downloaded, use 7-Zip or similar to unpack the tgz file
5) Use 7-Zip to unpack the .tar file
6) In your final unpacked folder, you should see a lot of files like this:

Image: vsim_esx-cm.tgz files
7) Via the vSphere Client and the Datastore Browser

Click on the Upload button

Image: vSphere Client Datastore Browser Upload Button
7.1) Click “Upload Folder...”
7.2) Browse to the folder containing the files
7.3) Click OK
7.4) Accept “Yes” to the Upload/Download Operation Warning
7.5) And wait for the folder to upload ...

Image: Waiting for vsim_esx-cm to upload
8) Browse to the newly uploaded folder using the vSphere Datastore Browser

Image: vsim_esx-cm files on the VMware datastore
8.1) Right-click the VMX file
8.2) Select ‘Add to Inventory’ and follow through the wizard

9) Finally, right-click the DataONTAP SIM and ‘Power On’

Image: vSphere Client -> Power on DataONTAP SIM
And we get an error:

Image: Error - Failed to start the virtual machine ... cannot find ... DataONTAP-sim.vmdk
9) Now following the instruction in this community post:

9.1) Remove the disk on IDE (1:1)

Image: Removing IDE (1:1)
Image: Remove from virtual machine and delete files from disk
9.2) And create a new IDE disk on IDE 1:1, 250gb and thin provisioned:

Image: 250GB Thin Provision disk ...
Image: ... on IDE (1:1)
Image: Add Hardware - Ready to Complete ...
10) Take a VMware snapshot before first boot (so we can create a clone from a pre-boot version of the SIM later - and perhaps change the serial number which gets hard set once it has booted beyond boot loader ...)

11) And then power on the VM, and watch it boot!

Image: Booting the DataONTAP SIM in ESXi

Thursday, 12 March 2015

Function Accumulate the Number at the End of the String

Hi there, sorry it’s been a bit quiet on the blog recently, had my head stuck in something big...

Here’s a useful PowerShell function for something I was writing. A few examples of the outputs:

ABCD   --- output ---> ABCD1
ABCD1  --- output ---> ABCD2
999    --- output ---> 1000
ABC999 --- output ---> ABC1000

The function is below (I’m sure there’s a much more elegant way of doing this):

Function AccumulateNumberAtEndOfString {

  [Int]$Pointer             = $InString.Length -1
  [Boolean]$EndsWithNumber  = $False
  [Boolean]$Result          = $False    
  while (!$Result){ 
    If($InString.Substring($Pointer,1) -match "^[0-9]+$") {
      $EndsWithNumber = $True
    } else {
      $Result = $True
    If ($Pointer -lt 0){$Result = $True}              
  If ($EndsWithNumber) {  
    $InputChars  = $InString.Substring(0,$Pointer + 1)
    $InputNumber = [Int64]$InString.Substring($Pointer +1, $InString.Length - $Pointer - 1)
    $InputNumber ++
    Return ($InputChars + [String]$InputNumber)
  } else {
    Return ($InString + "1")