Saturday, 20 April 2013

How to use NetApp MBRALIGN to Find and Fix Disk Misalignment Issues

The following walkthrough guide runs through an install, setup, and use scenario for NetApp’s fantastic mbralign tool. The lab used here consists of NetApp 8.1.2 7-Mode Controller (SIM), VMware ESXi 5.1 Host, VMware vCenter Server 5.1, and VMware vSphere 5.1 Client. Here we are running VMs on NFS datastores but the theory applies to FC and iSCSI too. Let’s begin!

1. Download and install the NetApp Virtual Storage Console for VMware (if not done already)

The software is available from http://support.netapp.com > Downloads > Software > Virtual Storage Console (VSC). And the latest version of the VSC is 4.1 which we use here. This post is about mbralign hence the VSC install is not covered here; for a slightly old blog post that does, please check here!

2. Identifying Misaligned VMs with the NetApp Virtual Storage Console (VSC)

To run a new scan across whatever datastores are to be interrogated:
- vSphere client > Home > Solutions and Applications > NetApp
- Optimization and Migration > Scan Manager
- select Datastores to be scanned and hit ‘Scan selected’

Image: NetApp VSC 4.1 scanning for misalignment (and more)

Then click on ‘Virtual Machine Alignment’, click on the ‘Misaligned’ folder, and review the results.

Image: NetApp VSC 4.1 showing misaligned VMs

Note: The results of the scan can be seen in various places around the vSphere Client and NetApp tab, and the scan can be scheduled.

3. Download MBR Tools

In the vSphere Client, with either: Datacenter, Cluster, or Host object selected
- select the NetApp tab > Monitoring and Host Configuration > Tools
- and click on Download to download the correct version of the MBR Tools for the VMware host in question (options are ESX 4.x or ESXi 4.x and ESXi 5.x - here we cover ESXi)

Image: VSC 4.1 MBR Tools Download

Note: From ‘Tools’ you can also download Guest OS Tools (an iso file containing a script) to set the correct SCSI I/O timeout values (either 60 or 190 seconds) for various supported guest operating systems (Linux, Windows, Solaris.)

4. Install mbrtools on the ESXi Host

Note: Recommend having mbralign and the VM to be re-aligned on the same ESXi host, this is because mbralign cannot tell power-on status if the VM is on a different host to the one mbralign is being run from.

With the mbrtools_esxi.tgz downloaded, we need to get this to the ESXi host, and the easiest way to do this is to upload to an accessible datastore via the vSphere Client.

Image: vSphere Client - Datastore Browser with Upload button and mbrtools_esxi.tgz highlighed

Next, before using SSH to connect to the ESXi host, the SSH service must be enabled via the Configuration tab > Software: Security Profile > Services and click Properties… > Select SSH and click Options… > click Start and OK

Connect to the host via SSH (e.g. using PuTTY) and type:

cd /vmfs/volumes
ls
cd DATASTORE_CONTAINING_MBRTOOLS
ls
tar xvzf mbrtools_esxi.tgz
cp -r /vmfs/volumes/DATASTORE_GUID_VALUE_DATASTORE/opt/ontap /opt/ontap

The above navigates to the correct datastore, unpacks the mbrtools, and copies them to /opt/ontap.

An example output (abridged):

~ # cd /vmfs/volumes
/vmfs/volumes # ls
…  LONNFSVOL0
/vmfs/volumes # cd LONNFSVOL0
/vmfs/volumes/51086c31-1b392580 # ls
… mbrtools_esxi.tgz
/vmfs/volumes/51086c31-1b392580 # tar xvzf mbrtools_esxi.tgz
opt/ontap/
opt/ontap/libshim.so
opt/ontap/mbrscan
opt/ontap/mbralign
opt/ontap/align.bin
opt/ontap/scan.bin
/vmfs/volumes/51086c31-1b392580 # cp -r /vmfs/volumes/51086c31-1b392580/opt/ontap /opt/ontap

5. Run mbrscan

To check the VM’s VMDK disk alignment status:
- Via the ESXi host CLI, navigate to the folder containing the VMs disks
- Run the command

/opt/ontap/mbrscan VMDISKNAME-flat.vmdk

Example output (abridged) - notice the “aligned:No”:

/vmfs/volumes/7ae8613c-c7ea783d/WIN2K3_32BIT_MS_03 # /opt/ontap/mbrscan WIN2K3_32BIT_MS_03-flat.vmdk
WIN2K3_32BIT_MS_03-flat.vmdk p1 (NTFS)  lba:63  offset:32256    aligned:No

Note: The VM must be powered off before running mbrscan against the vmdk file.

6. Run mbralign

Note i: Before running mbralign, it is not a bad idea to take a SAN snapshot of the datastore/VM folder first.
Note ii: Check out the Appendix section below for using the --sparse option if you don’t want mbralign to produce thin-provisioned VMDKs or not make thin one’s thick!

To run mbralign against a VMDK, via the ESXi host CLI, run the command:

/opt/ontap/mbralign VMDISKNAME-flat.vmdk

Example output (abridaged):

/vmfs/volumes/7ae8613c-c7ea783d/WIN2K3_32BIT_MS_03 # /opt/ontap/mbralign WIN2K3_32BIT_MS_03-flat.vmdk

WARNING: This version of mbralign is intended for ESXi configurations only … This version does not support Windows drive letter re-mapping. It can only align one vmdk file at a time. Since this program will not check for space before creating a new vmdk file of approximately the same size as the original, always check if enough free space is available in the datastore before running mbralign. The Virtual Machine must be powered off before running mbralign against the vmdk file. A backup file of the vmdk file will be created and stored in the datastore alignment occurs in. If an error occurs, manually restore the vmdk file with the backup.

mbrtools esxi version 1.0

 Part    Type          old LBA    New Start LBA      New End LBA     Length in KB
   P1      07               63               64         16755796          8377866

NOTICE: This tool does not check for the existence of Virtual Machine snapshots or linked clones. The use of this tool on a vmdk file that has a snapshot or linked clone associated with it can result in unrecoverable data loss and/or data corruption.
Are you sure that no snapshots/linked clones exist for this vmdk? y
Creating a backup of WIN2K3_32BIT_MS_03.vmdk
Creating a backup of ./WIN2K3_32BIT_MS_03-flat.vmdk
Creating a copy the Master Boot Record
Working on partition P1 (3): Starting to migrate blocks from 32256 to 32768.
12801 read ops in 14 sec.  98.88% read (6.91 mB/s).  98.88% written (6.91 mB/s)
Working on space not in any partition: Starting to migrate blocks.
100.00 percent complete.  100.00 percent written. .
Making adjustments to ./WIN2K3_32BIT_MS_03-flat.vmdk.
Adjusting the descriptor file.

7. Power up the re-aligned VM!

As a final check, re-run mbrscan to verify alignment has worked (the output should say “aligned:Yes”):

/opt/ontap/mbrscan VMDISKNAME-flat.vmdk

Then power on the VM and check all is okay.

Note: There are additional steps that need to be done with Linux VMs as mbralign messes up the grub boot loader.

8. Tidy Up

Mbralign leaves a couple of backup files in the VM folder after the job is complete:
- a VMDK disk descriptor file VMDISKNAME.vmdk-mbralign-backup, and
- a VMDK flat file VMDISKNAME-flat.vmdk-mbralign-backup
Delete these files when they’re no longer needed.

Image: mbralign-backup files

APPENDIX: Useful Notes

A. The --sparse option

Use the sparse option for mbralign to create a thin provisioned VMDK.

/opt/ontap/mbralign --sparse VMDISKNAME-flat.vmdk

Note: See here for a discussion on the --spare option not working (making all aligned vmdk’s thick) with the NFS CopyOffload option enabled.

B. Recovering from mbralign backup

If the mbralign process didn’t work out, simply run mbralign again to restore the original. The restore process is very quick!

/opt/ontap/mbralign VMDISKNAME-flat.vmdk

An example output (abridaged):

/vmfs/volumes/7ae8613c-c7ea783d/WIN2K3_32BIT_MS_03 # /opt/ontap/mbralign WIN2K3_32BIT_MS_03-flat.vmdk
Error: A backup file named ./WIN2K3_32BIT_MS_03-flat.vmdk-mbralign-backup was found.  This may be from a failed realign.
Please remove or rename all files ending in -mbralign-backup.
Shall I restore WIN2K3_32BIT_MS_03.vmdk from the backup I made? y

Note i: This process also removes the -mbralign-backup files.
Note ii: If you accidentally turn a thin VMDK into a thick one, the mbralign restore brings back the thin one!

C. Speeding up mbralign

Final note!
I came across this tip at http://www.deucedaily.org/blog/2012/03/19/speed-up-mbralign/ regards using a Linux management server to mount the NFS datastore with the VMDK to be aligned in, and running mbralign from the Linux management server. Not tried it, but if you’re finding mbralign slow, it might be worth a crack!

1 comment:

  1. Your this blog giving us very much information thanks for share this.
    1337x UK proxy

    ReplyDelete