Saturday, 27 December 2014

The End of Year 2014 Post

27th December and feels like time for the End of Year Post.

Another cracking year doing interesting work! The blog’s been mostly 100% NetApp this year, and that’s great because I really really like - might even go as far to say love ... ahem ... - working with NetApp technology, and I know this enthusiasm is shared by so many around the world.

Got loads of projects planned for 2015 (and a lot of stuff I’d hoped to finish in 2014 but didn’t), and hopefully I can share some of the outcomes on this blog. I keep intending to take a sabbatical from blogging, yet it’s such a great motivator; without the blog, after my 9 to 5, there’s a good chance I’d be sucked into the world of fantasy and escapism offered by computer games - the blog keeps me on the straight and narrow, or thereabouts...

Projects

Some planned 2015 IT Projects

- 7 to C Scripted Migration Suite (PowerShell/Excel/???)
- CDOT Quota Reports (PowerShell/???)
- CDOT Synchronization Suite (PowerShell/WFA)
- CDOT DR Suite (PowerShell/WFA)
- CDOT Health Checker (PowerShell/???)
- CDOT Compare V2 (PowerShell)
- CDOT Cluster & SVM Constructor Suite (PowerShell/WFA)
- CDOT Log Viewer (PowerShell)

And I’ll review this list at the end of 2015 to see how I got on.

There are also various personal projects too which will likely take me away from the blog for periods of time...

Blog Stats (as of the 27th December)

Total Pageviews = 1’531’597 (up 521’359 on 2013)
Total Visitors = 709,766 (up 238’071 on 2013)

And Finally ...

Thanks everyone for reading!
And have a great 2015!


7-Mode Head Swap Script

Been meaning to post this for months and months just never got around to it!

This is just a simple but detailed script I concocted for performing 7-Mode Head Swaps - it’s stripped down to bare bones note form, and designed for Notepad/Notepad++/Simple Text Editors. As always with anything on this blog, it’s absolutely not official documentation so caveat lector!

##### COMPANYX
##### FASXXXXA to FASYYYYA
##### For CTRA / CTRB
##### 7-MODE Head Swap Script

# Variables:
# COMPANYX = Company Name
# CTRA / CTRB = Controller A/B
# FASXXXX = Old FAS System
# SYSID_OLDA / SYSID_OLDB = Old FAS Sysid's
# FASYYYY = New FAS System
# SYSID_NEWA / SYSID_NEWB = New FAS Sysid's

#####################################
##### PREPARATION AND PRECHECKS #####
#####################################

# NOTE 1: This script assumes both old and new controllers are on the same version of Data ONTAP
# NOTE 2: Pre-planning for how old ports map to new ports should have been completed!
# NOTE 3: Any new licensing requirements have already been prepared!

###############################
# FASXXXX - CTRA - SYSID_OLDA #
###############################

rdfile /etc/rc
ifconfig -a
rdfile /etc/hosts
rdfile /etc/exports
exportfs
storage show disk -p
aggr status
vol status

disk show -n # OBTAIN/CONFIRM SYSID
disk show -v
disk show -a # SHOULD SEE BOTH CONTROLLERS
sysconfig -a
fcadmin config

rlm status # WORKS FOR OLDER RLMs
sp status  # WORKS FOR NEWER SPs

vfiler status -a

# SAN Checks #

lun show
lun show -v
lun show -m
igroup show

# In the event lun serials change then the following command will be required to reset the lun serials:
# lun serial [-x]lun_path new_lun_serial)

options autosupport

vol status     # LOOK FOR root - vol0?
aggr status    # LOOK FOR root - aggr0?
df -A -g aggr0 # CHECK FREE SPACE
vol size vol0
vol size vol0 250g # TYPICAL VOL0 SIZE in 8.2.1

priv set advanced
ls /etc/software # VERIFY THE CORRECT IMAGE IS THERE (e.g. .tgz for upgrade from 8.X, or .zip from 7.X)

###############################
# FASXXXX - CTRB - SYSID_OLDB #
###############################

rdfile /etc/rc
ifconfig -a
rdfile /etc/hosts
rdfile /etc/exports
exportfs
storage show disk -p
aggr status
vol status

disk show -n # OBTAIN/CONFIRM SYSID
disk show -v
disk show -a # SHOULD SEE BOTH CONTROLLERS
sysconfig -a
fcadmin config

rlm status # WORKS FOR OLDER RLMs
sp status  # WORKS FOR NEWER SPs

vfiler status -a

# SAN Checks #

lun show
lun show -v
lun show -m
igroup show

# In the event lun serials change then the following command will be required to reset the lun serials:
# lun serial [-x]lun_path new_lun_serial)

options autosupport

vol status     # LOOK FOR root - vol0?
aggr status    # LOOK FOR root - aggr0?
df -A -g aggr0 # CHECK FREE SPACE
vol size vol0
vol size vol0 250g # TYPICAL VOL0 SIZE in 8.2.1

priv set advanced
ls /etc/software # VERIFY THE CORRECT IMAGE IS THERE (e.g. .tgz for upgrade from 8.X, or .zip from 7.X)

####################################################
##### IDENTIFY AND RECORD ALL EXISTING CABLING #####
####################################################

## RUN CONFIG ADVISOR ##

##############################
##### UPGRADE: Head Swap #####
##############################

###############################
# FASXXXX - CTRA - SYSID_OLDA #
###############################

aggr scrub status
aggr scrub stop # IF SCRUBS IN PROGRESS
ndmpd status
ndmpd off # IF ON
snapmirror status
snapmirror off # IF ON

cifs terminate
fcp stop
iscsi stop
nfs off

snap create vol0 pre_headswap
config dump pre_headswap
options autosupport.doit "pre_headswap"

###############################
# FASXXXX - CTRB - SYSID_OLDB #
###############################

aggr scrub status
aggr scrub stop # IF SCRUBS IN PROGRESS
ndmpd status
ndmpd off # IF ON
snapmirror status
snapmirror off # IF ON

cifs terminate
fcp stop
iscsi stop
nfs off

snap create vol0 pre_headswap
config dump pre_headswap
options autosupport.doit "pre_headswap"

###############################
# FASXXXX - CTRA - SYSID_OLDA #
###############################

cf disable
cf status
halt

###############################
# FASXXXX - CTRB - SYSID_OLDB #
###############################

cf status
halt

####################################
##### PHYSICAL REMOVAL/INSTALL #####
####################################

# Power off both old FASXXXX controllers and remove chassis from racks
# Physically install new FASYYYY controllers
# Reconnect all disk/back-end cabling as PER REQUIREMENTS
# NOTE: Front End cabling is reconnected later!

####################
##### POWER UP #####
####################

# Power on both FASYYYY controllers
# (can do one after the other, the main thing is that we boot both into maintenance mode)
# IMPORTANT: PRESS CTRL-C AND 5 FOR MAINTENCE MODE BOOT ON BOTH!

####################################################
# FASYYYY - CTRA - SYSID_NEWA # !MAINTENANCE MODE! #
####################################################

disk show -n # NEW SYSID =  SYSID_NEWA
disk show -v
## disk reassign -s -d
disk reassign -s SYSID_OLDA -d SYSID_NEWA
mailbox destroy local
mailbox destroy partner
aggr status
fcadmin config # Check FC ports initiators/targets as required
## fcadmin config -d Xx
## fcadmin config -t initiator Xx
## NOTE: ucadmin command for UTAs
halt

####################################################
# FASYYYY - CTRB - SYSID_NEWB # !MAINTENANCE MODE! #
####################################################

disk show -n # NEW SYSID =  SYSID_NEWB
disk show -v
## disk reassign -s -d
disk reassign -s SYSID_OLDB -d SYSID_NEWB
mailbox destroy local
mailbox destroy partner
aggr status
fcadmin config # Check FC ports initiators/targets as required
## fcadmin config -d Xx
## fcadmin config -t initiator Xx
## NOTE: ucadmin command for UTAs
halt

###############################
# FASYYYY - CTRA - SYSID_NEWA #
###############################

boot_ontap
aggr status
vol status

### NOTE: Check APPENDIX A if getting too much output to the console! ###

rdfile /etc/rc    # CHANGE AS PER REQUIREMENTS (e.g. wrfile /etc/rc)
rdfile /etc/hosts # CHANGE AS PER REQUIREMENTS (e.g. wrfile /etc/hosts)

### >> PASTE OUTPUT OF rdfile /etc/rc << ###

### >> PASTE OUTPUT OF rdfile /etc/hosts << ###

license

### >> PASTE OUTPUT OF license << ###

### May need to add updates licenses with >
## license add LICENSECODE # Can skip if licenses are okay
## license delete INVALID_LICENSES

### software update # IF REQUIRED>

sp setup

## IP Address = ???
## Netmask    = ???
## Gateway    = ???

halt

###############################
# FASYYYY - CTRB - SYSID_NEWB #
###############################

boot_ontap
aggr status
vol status

### NOTE: Check APPENDIX A if getting too much output to the console! ###

rdfile /etc/rc    # CHANGE AS PER REQUIREMENTS (e.g. wrfile /etc/rc)
rdfile /etc/hosts # CHANGE AS PER REQUIREMENTS (e.g. wrfile /etc/hosts)

### >> PASTE OUTPUT OF rdfile /etc/rc << ###

### >> PASTE OUTPUT OF rdfile /etc/hosts << ###

license

### >> PASTE OUTPUT OF license << ###

### May need to add updates licenses with >
## license add LICENSECODE # Can skip if licenses are okay
## license delete INVALID_LICENSES

### software update # IF REQUIRED>

sp setup

## IP Address = ???
## Netmask    = ???
## Gateway    = ???

halt

##########################################################
##### CONNECT FRONT END CONNECTIONS PER REQUIREMENTS #####
##########################################################

###############################
# FASYYYY - CTRA - SYSID_NEWA #
###############################

boot_ontap
version
ifconfig -a
cf status

###############################
# FASYYYY - CTRB - SYSID_NEWB #
###############################

boot_ontap
version
ifconfig -a
cf enable
cf status

###################
##### TESTING #####
###################

### HAS LUNS ###

lun show -v # ON A CHECK SERIALS
lun show -v # ON B CHECK SERIALS

### VFILERS IN USE ###

options licensed_feature.multistore.enable # ON A
vfiler status -a # ON A

options licensed_feature.multistore.enable # ON B
vfiler status -a # ON B

### NOTE: Check APPENDIX B if have issues with vfilers post head-swap and need to re-run vfiler setup ###

cf takeover # ON A
cf giveback # ON A

cf takeover # ON B
cf giveback # ON B

options autosupport # ON A check settings
options autosupport # ON B check settings
options autosupport.doit "post headswap" # ON A
options autosupport.doit "post headswap" # ON B

config diff pre_headswap # ON A
config diff pre_headswap # ON B

options autosupport.doit WEEKLY_LOG # ON A
options autosupport.doit WEEKLY_LOG # ON B

ndmp on # ON A (if was on before)
ndmp on # ON B (if was on before)

snapmirror on # ON A if was on before (NOTE: Check requirements for SnapMirror source and destination versions!)
snapmirror on # ON B if was on before (NOTE: Check requirements for SnapMirror source and destination versions!)

## RUN CONFIG ADVISOR ##

# And verify everything is okay!

#######################
##### THE END !!! #####
#######################

####################################################
##### APPENDIX A: (Optional) QUIETENING SYSLOG #####
####################################################

rdfile /etc/syslog.conf
wrfile /etc/syslog.conf

###########CONSOLE-NONE#################

#*.info   /dev/console
*.info    /etc/messages

###########PREFERRED####################

# Set the alert level for the console
*.err     /dev/console
# Set the alert level for the local messages file
*.info    /etc/messages
# Set the alert level for the @SYSLOG_SERVER_IP
*.info    @X.X.X.X

###########################################################
##### APPENDIX B: (If Issues) RE-RUNNING VFILER SETUP #####
###########################################################

# rdfile /vfiler_path/etc/exports
# rdfile /vfiler_path/etc/hosts
# rdfile /vfiler_path/etc/hosts.equiv
# rdfile /vfiler_path/etc/nsswitch.conf
# rdfile /vfiler_path/etc/resolv.conf
# vfiler create VFILER -r /vol/vol0/VFILER
# vfiler run VFILER setup

More on Tracking Quotas, with Dedupe and Volume Autosize Considerations

1) A correction to this post from July 2014: Volume and Qtrees Quotas in Clustered ONTAP: Tracking Quota

To enable tracking quotas for all Qtrees in a volume (you do not need to create a tracking rule for every qtree), it is simply ::>

volume quota policy rule create -vserver SVMNAME -policy-name POLICYNAME -volume VOLUMENAME -type tree -target "" -threshold "-"
volume quota on -vserver SVMNAME -volume VOLUMENAME

{Where POLICYNAME is the currently active Quota Policy on the SVM, with default policy being default}

When the volume quota has finished initializing (quota show -vserver SVMNAME -volume VOLUMENAME), to get a report for ‘Disk Used’ and ‘Files Used’ for all Qtrees in that volume ::>

volume quota report -vserver SVMNAME -volume VOLUMENAME

2) Quotas and Volume Autosize Considerations

From TR 3505 NetApp Deduplication:

6.4.1 QUOTAS
For deduplicated files, the logical (undeduplicated) size is charged against the quotas. There are several advantages to this scheme as opposed to charging quotas based on the physical (deduplicated) size of the file:
i) This is in line with the general design principle of making deduplication transparent to the end user.
ii) It is easier for system administrators to manage quotas. They can maintain a single quota policy across all volumes, whether or not deduplication is enabled on it.
iii) There are no out-of-space failures when data is being sent using SnapMirror from a volume with deduplication enabled to a destination volume that has deduplication disabled.

To understand the effect of seeing the logical (undeduplicated) size when using Qtree quotas, here are 3 examples. In each example, we have a 1000MB thin-provisioned volume (space guarantee none), with no snapshots, dedupe is enabled on the volume, and Qtree tracking quotas are enabled. We use the same 210 MB folder multiple times (so it de-dupes very well!) The Storage Virtual Machine and AD Machine Account is called cifsv1.

2.1) Example 1: 4 x the same 210MB folder (~840MB) in one Qtree

Image: Example 1 Data Layout
DOS Commands to Map Drives:

net use V: \\cifsv1\vol1
net use Q: \\cifsv1\vol1\qtree1
net use R: \\cifsv1\vol1\qtree2

Image: Displayed via a Windows Client for Example 1
2.2) Example 2: The same 210MB folder (~840MB) in 3 Qtrees and the root of the Volume

Image: Example 2 Data Layout
DOS Commands to Map Drives:

net use V: \\cifsv1\vol2
net use Q: \\cifsv1\vol2\qtree2

Image: Displayed via a Windows Client for Example 2
2.3) Example 3: 5 x the same 210MB Folder (~1050MB) in one Qtree

Image: Example 3 Data Layout
DOS Commands to Map Drives:

net use V: \\cifsv1\vol3
net use Q: \\cifsv1\vol3\qtree3

Image: Displayed via a Windows Client for Example 3
3) Volume Autosize Considerations

Generally, when dedupe and Qtree quotas are in effect, the free space the end user will see is:

End User Free Space = {Volume Size} - {Logical (Un-deduped) Data Size in the Qtree}

- or, if the actual (including dedupe) data size in the volume is larger than the logical data size in the Qtree -

End User Free Space = {Volume Size} - {Actual (Including Dedupe) Data Size in the volume}

We see in example 1 above, the volume share shows the actual free space available in the volume as 797 MB, but the qtree share shows a much smaller free space value of 150 MB. This has implications regards setting volume autosize, in essence:

Volume AutoSize Size Setting = ( {Volume Size} - {Logical (Un-deduped) Data Size in the Largest Qtree} + {Actual (Deduped) Data Size in the Volume} )* {Percentage Autogrow Threshold}

- or, if the actual (including dedupe) data size in the Volume is larger than the logical data size in the largest Qtree, we get the typical formula -

Volume AutoSize Size Setting = {Volume Size} * {Percentage Autogrow Threshold}

In example 1, even though the volume is only 203 MB used out of 1000 MB, if the user is accessing Qtree1, and tries to add more than 150 MB, this will fail (even though there is 797 MB free space in the volume), and the autosize will not have triggered if it were set at 850MB.

Example 1: AutoSize Size Setting = ( 1000 - 850 + 203 ) * 0.85 ~= 300 MB
Example 2: AutoSize Size Setting = 1000 * 0.85 = 850 MB

Note: Interestingly, in Example 3, you might expect to see negative user space when we have 1050 MB logical data in a Qtree in a 1000 MB volume, but that doesn’t happen, the Windows client then shows the deduped data size in the volume!

4) Final Comment

If your end-user is saying his share is full and he needs more space, but the volume is far from being full, nor is the aggregate, and volume autosize has not kicked in as expected, suspect dedupe and quotas!