Tuesday, 19 June 2012

Thoughts on Whether to Use Windows Server 2008 MCS or MPIO for iSCSI

The answer to a question that rarely seems to come up 'Should I Use Windows Server 2008 MCS (Multiple Connected Session) or MPIO for the iSCSI Storage Connection?', depends a lot on if there is a supported DSM (Device Specific Module) for the iSCSI storage vendor in the particular scenario (the Microsoft DSM might come with manufacturer support.) The following post aims to illustrate the differences between MCS and MPIO with configuration examples using either 4-way MCS, or 4-way MPIO, from a Windows 2008 Server with 4 x 1 Gbps dedicated iSCSI interfaces.

MCS is a Many to One Mapping

In the following example, the MCS has been configured (via iSCSI Initiator Properties > Targets tab > Properties > Sessions tab > MCS) with 4 different source portals and only one target portal (it is not possible to select more than one target portal.) This could be used where there is no supported DSM, and the Windows 2008 source uses its 4 x 1 Gbps iSCSI NICs as source portal interfaces, and the Storage target portal is either a virtual interface of port aggregated 1 Gbps NICs (e.g. a Multi-mode VIF on a NetApp FAS2040 utilizing 4 x 1 Gbps ethernet interfaces), or a 10 Gbps interface.

Fig. 1: Multiple Connected Session from 4 source portals to 1 target portal.

Fig. 2: Windows 2008 Server with 4 iSCSI interfaces, connecting via MCS to 1 interface on the Storage.

Note: Needs to be verified with the storage provider regards whether MCS is supported.

MPIO can be a Many to One or Many to Many Mapping

With a supported DSM. From the example lab Windows 2008 Server, the configuration could either use one target portal, and create multiple paths to that target from the dedicated iSCSI NICs (again, the target portal would either be a virtual interface of port aggregated 1 Gbps NICs or a 10 Gbps interface.)

Fig. 3: iSCSI Initiator configured with 1 target portal.

Or could use 4 target portals and establish separate paths from each source portal (e.g. a NetApp FAS2040 with 4 x 1 Gbps ethernet interfaces, each interface configured with a separate IP address.)

Fig. 4: iSCSI Initiator configured with 4 target portals.

Fig. 5: Windows 2008 Server with 4 interfaces, connecting to 4 interfaces on the Storage.

Note: The Multipath I/O feature must first be installed –

Fig. 6: Windows 2008 Server features – Multipath I/O.

– and then the DSM can be installed, and MPIO Multi-Paths can be discovered via MPIOCPL.exe from Administrative Tools.

Fig. 7: MPIOCPL.exe icon in Administrative Tools.

Fig. 8: MPIO Properties > DSM Install.


Admittedly, I have not had much reason to use MCS in practice (would greatly appreciate to hear comments from people that use it,) and with MPIO being more widely supported, this seems to be the way to go if can obtain a supported DSM. Knowing about the option is handy – another weapon to keep in the arsenal of solutions. There is an excellent Microsoft and NetApp White Paper available on the internet entitled “iSCSI 10 Gigabit Ethernet Performance Tuning With Windows Server 2008, Hyper-v and the NetApp FAS 3070” which identifies throughput advantages of MCS over MPIO, at least up to four connections.

Monday, 18 June 2012

Migrating OWA from 2003 to 2010 and Keeping the Same DNS Name with Minimal Service Downtime


A company has recently had an Exchange 2010 Hub/CAS server and Exchange 2010 Mailbox Server implemented, and this is sitting alongside a pre-existing Exchange 2003 Frontend/Backend setup. The customer wants to keep their Exchange 2003 webmail address (owa.test.com in this example,) and the following post give a brief outline of a way to do this.

Preliminary Information Gathering

There is a public DNS A record for owa.test.com.
There is a 1 to 1 NAT on the external firewall which maps the public IP address for owa.test.com to the Exchange 2003 Frontend Server LAN IP address.
OWA is currently working for mailboxes on Exchange 2003, via the Exchange 2003 Frontend server on https://owa.test.com/exchange.


Stage 1

Create a new public DNS A record for say owa2.test.com.
Create a new 1 to 1 NAT on the external firewall to map the public IP address for owa2.test.com to the Exchange 2010 Hub/CAS server's LAN IP address.
Configure the webmail on the Exchange 2010 Hub/CAS for owa2.test.com.
Test https://owa2.test.com/OWA works from the internet for an Exchange 2010 Mailbox.

Stage 2

Create a new public DNS A record for legacy.test.com.
Add an additional IP address to the Exchange 2003 Frontend server's NIC.
Create a new 1 to 1 NAT on the external firewall to map the public IP address for legacy.test.com to the Exchange 2003 Frontend server's secondary LAN IP address.
Test https://legacy.test.com/exchange works from the internet for an Exchange 2003 Mailbox.

Stage 3

Via the Exchange 2010 Management Shell, run the command> Get-OwaVirtualDirectory

[PS] C:\Windows\system32>Get-OwaVirtualDirectory
Name Server OwaVersion
---- ------ ----------
owa (Default Web Site) EXCH2010CAS Exchange2010

Via the Exchange 2010 Management Shell, run the command> Set-OwaVirtualDirectory EXCH2010CAS\OWA* -ExternalUrl https://owa2.test.com/OWA -Exchange2003
URL https://legacy.test.com/exchange

[PS] C:\Windows\system32>Set-OwaVirtualDirectory EXCH2010CAS\OWA* -ExternalUrl https://owa2.test.com/OWA -Exchange2003
URL https://legacy.test.com/exchange

Test https://owa2.test.com/OWA now works from the internet for an Exchange 2003 Mailbox.

Stage 4

Now simply arrange a time with the customer to flip the NAT and update the Exchange 2010 Hub/CAS so that any settings relating to owa2.test.com are changed to owa.test.com.

The NAT flip involves:
i. Remove the NAT for owa2.test.com.
ii. Point the NAT for owa.test.com to the Exchange 2010 Hub/CAS Server.
iii. Re-add the NAT for owa2.test.com to point to the Exchange 2003 Frontend server.

Test https://owa.test.com/OWA works from the internet for Exchange 2003 and Exchange 2010 Mailboxes.

The A record and NAT for  owa2.test.com can be removed if all is working satisfactorily.
When all mailboxes have been moved to Exchange 2010, then the A record and NAT for  legacy.test.com can be removed.

Caveat: This post only considers OWA and it is assumed that a wildcard SSL certificate is in place.
Credit: Thanks for this tip to the mighty Jotmaan Ghins

Is BYOD Over-hyped? No / Yes


BYOD (Bring Your Own Device) is the relatively recent phenomenon where company employees bring personally-owned mobile devices to their place of work to use for work purposes. To find the beginning of time for BYOD, you could perhaps go back as far as when the first practical personal laptops, and/or the first personal mobile phones capable of serving email, appeared on the market.

Related to BYOD is BYOT (Bring Your Own Technology) and BYOA (Bring Your Own App.)

There is a lot of debate in the internet-land regards BYOD, and this following post attempts to pose and answer the 'Is BYOD Over-hyped?' question, as if from a no camp, then from a yes camp, and reach a conclusion at the end.

Is BYOD Overhyped? No

It is important that IT listens to its consumers. There is nothing worse than a strict totalitarian authority that manages to crush all attempts at innovation, and bans things without alternatives being there. Staff should be commended for bringing their own devices into work for work purposes, but BYOD should not be left unchecked – IT should be an enabler for secure BYOD. People are always looking for ways to improve their efficiency and increase productivity, be it via automation or finding things that simply make life easier. The IT department should not be a road-block to new ways of doing things; it is all about working in partnership, but of course someone has got to be ultimately responsible for enterprise data security!

Is BYOD Overhyped? Yes

It is important that employees are given the correct tools to do their job. If employees are coming into the work place to do work things on their own devices, then the corporation is failing to properly equip its staff.

Many a time that a personal device is brought into the workplace, it is maybe used for a little bit of work, and then a big bit of personal recreation – be it accessing websites that are blocked on the corporate internet, playing games, watching movies. The personal device is a way to circumvent corporate IT security lock downs that are (usually) in place for good reason.

Security is a problem. If staff are allowed to download or store work related materials on personal devices, and if these personal devices – that are not secured by any enterprise security systems – are the source of leaks or data loss, this will have repercussions on the corporation whose responsibility it is to protect that data.

Also, a question that needs to be asked before going down the route and expense of BYOD enablement, is how many employees would actually want to bring in to work a personal device to do work on, if it were possible to do this.


Yes, BYOD is cool and it is funky, but the IT department must be responsible for all devices accessing corporate data to maintain enterprise data security.

Saturday, 16 June 2012

How to Configure iSCSI Multipathing (MPIO) in VMware vSphere 5 using ESXCLI (UPDATE)

This is an update of January's post   How to Configure iSCSI Multipathing (MPIO) in VMware vSphere 5 using ESXCLI – with modifications to create just one iSCSI vSphere Standard Switch with two iSCSI VMkernels in, as opposed to a switch per VMkernel. Additionally, it now adds an iSCSI Virtual Machine Port Group. Credit to Stefan Lapers for his help.

The Commands Below Will ...

i. Enable the iSCSI Software Adapter
ii. Create one vSphere standard switch
iii. Create two VMkernel's for iSCSI
iv. Configure the 'Failover and Load Balancing' to have one active physical NIC per iSCSI VMkernel
v. Bind the iSCSI VMkernels to the iSCSI Software Adapter vmhba
vi. Add a send target portal
vii. Set the multi-pathing to default to round-robin
viii. Perform a rescan for storage devices and new volumes
ix. Add an iSCSI Virtual Machine Port Group
The commands can be applied via an SSH connection (PuTTY or similar) to the ESXi 5 host to be configured (Note: to enable SSH access, first start the SSH service on the host using the vSphere Client.) The commands can be run as a batch script by pasting into PuTTY (testing in the lab had a complete iSCSI and MPIO configuration applied in less that 20 seconds!)


1. In practice, Part 1 should be run separately to enable the iSCSI Software Adapter, in order to obtain the iSCSI Initiator name to input into the SAN software to allow access to volumes; also to check the vmhba number.
2. Edit the following variables as per requirements: vmnic4, vmnic5, & subnet , & subnet, vmhba33,
3. The line in Part 6 works for a Storage Array Type of VMW_SATP_DEFAULT_AA (includes HP P4000.) If you want the multipathing to default to round-robin for devices not detected as VMW_SATP_DEFAULT_AA, then this line will need to be modified (for example: Equallogic uses VMW_SATP_EQL.)
4. See http://pubs.vmware.com/vsphere-50/index.jsp for the vSphere CLI Reference.
5. The lines beginning with 'echo' are just there to explain what the line underneath does
6. The below can be modified for than 2-way MPIO (vSphere 5 supports up to 8-way MPIO.)

The ESXCLI Commands

echo @ PART 1 Enable iSCSI Software Adapter
esxcli iscsi software set -e on
echo @ PART 2 Create iSCSI vSwitch and VMkernels
echo @ Add new standard switch called iSCSI_Switch
esxcli network vswitch standard add -v iSCSI_Switch
echo @ Add vmnic4 and vmnic5 to iSCSI_Switch
esxcli network vswitch standard uplink add -u vmnic4 -v iSCSI_Switch
esxcli network vswitch standard uplink add -u vmnic5 -v iSCSI_Switch
echo @ Set vmnic4 and vmnic5 as active adapters for iSCSI_Switch
esxcli network vswitch standard policy failover set -a vmnic4,vmnic5 -v iSCSI_Switch
echo @ Add portgroups iSCSI_VMkernel1 and iSCSI_VMkernel2 to iSCSI_Switch
esxcli network vswitch standard portgroup add -p iSCSI_VMkernel1 -v iSCSI_Switch
esxcli network vswitch standard portgroup add -p iSCSI_VMkernel2 -v iSCSI_Switch
echo @ Add VMkernel interfaces vmk11 and vmk12 to iSCSI_VMkernel portgroups
esxcli network ip interface add -i vmk11 -p iSCSI_VMkernel1
esxcli network ip interface add -i vmk12 -p iSCSI_VMkernel2
echo @ Configure IP Addressing of vmk11 and vmk12
esxcli network ip interface ipv4 set -i vmk11 -I -N -t static
esxcli network ip interface ipv4 set -i vmk12 -I -N -t static
echo @ Apply iSCSI VMkernel failover order
esxcli network vswitch standard portgroup policy failover set -p iSCSI_VMkernel1 -a vmnic4
esxcli network vswitch standard portgroup policy failover set -p iSCSI_VMkernel2 -a vmnic5
echo @ PART 3 Bind iSCSI VMkernels to iSCSI Software Adapter
esxcli iscsi networkportal add -A vmhba33 -n vmk11
esxcli iscsi networkportal add -A vmhba33 -n vmk12
echo @ PART 4 Add portgroup iSCSI_VM_Network to iSCSI_Switch
esxcli network vswitch standard portgroup add -p iSCSI_VM_Network -v iSCSI_Switch
echo @ PART 5 Add send target portal for iSCSI SAN
esxcli iscsi adapter discovery sendtarget add -A vmhba33 -a
echo @ PART 6 Set multipathing to default to round-robin
esxcli storage nmp satp set -P VMW_PSP_RR -s VMW_SATP_DEFAULT_AA
echo @ PART 7 Rescan for storage devices and new volumes
esxcli storage core adapter rescan -a
esxcli storage filesystem rescan

Wednesday, 13 June 2012

Why in Process Monitor is the ReadFile Length 4096 but WriteFile Length 1024?

This question was raised by a client – whilst doing an investigation into the underlying VMware and SAN infrastructure of a couple of systems for a customer where users were reporting poor application performance – as being a potential avenue for analysis.

The application in question reads from an SQL database, then writes the read file to local disk as a temporary file. When Process Monitor (available in the Sysinternals Suite and freely downloadable from Microsoft) is run against the file system to see the transacation taking place, a ReadFile length of 4096, and WriteFile length of 1024 is recorded.

Fig. 1 Process Monitor ReadFile Length 4096

Fig. 2 Process Monitor WriteFile Length 1024

When looking at this, I could find little information on the Net to explain this behaviour, so the following is a hypothesis.

Assuming that the ReadFile length mentioned is 4096 bytes, and the WriteFile length is 1024 bytes, these numbers could come from:

Default Network Packet Size on SQL Server = 4096 bytes

Fig. 3 Network Packet Size setting taken from SQL Server Management Studio: Database Properties > Advanced

NTFS Default Bytes Per FileRecord Segment = 1024 bytes (at least where the Bytes Per Cluster is 4096)

Fig. 4 Bytes Per FileRecord Segment from Windows Command Prompt:\> fsutil fsinfo ntfsinfo C:

Please feel free to comment  – especially if my hypothesis is wrong (which is part of the reason for posting.)

Tuesday, 5 June 2012

NetApp: FAS2040A Walkthrough Setup & Install Notes Part 5/5 – Configuring via the CLI Part 2 (via BMC)

Use PuTTY/SSH client to connect to the BMC's IP Address and login with the root user. And type the system console command to access the Data ONTAP CLI.

Bmc shell -> system console

Updating the Softwares

Note that installing new software can also be done from the boot menu (CTRL+C when boot.) It is beyond the scope of these notes to provide more than one way of doing things.

Download upgrade packages from http://download.netapp.com/ (you will need a NetApp NOW logon.)

Available upgrade packages you might like to download are:
i. Data ONTAP upgrade
ii. Disk Firmware upgrade
iii. Controller Firmware upgrade

Instructions for updating are available in the README.TXT files contained in the download packages. A quick couple of tips:
i. You will need a http server from which the controller will download the update – this can be a Windows 7 (not Home Edition) laptop with the built in IIS 7 installed.
ii. The command from the Data ONTAP CLI is going to be something like:

FASCTRA> software update {package_URL}

Resizing the Root Volume

The rool volume /vol/vol0 is allocated 420GB to start with, and for most scenarios this is too big. Remember that best practice is not to put any user data in the root volume. To set it to 150GB:

To see the total size and used space of volumes:
FASCTRA> df -h

To reallocate files in vol0
FASCTRA> reallocate start -f -p /vol/vol0

To check reallocate status:
FASCTRA> reallocate status

To resize the volume:
FASCTRA> vol status
FASCTRA> vol size vol0 150g


The FAS2040A should come with licenses pre-installed, if not use the license add command to add licenses after obtaining from now.netapp.com:
FASCTRA> license add {license code}

Recommend taking a record of all licenses installed with the output from:
FASCTRA> license

Assigning Disks

To find the unassigned disks:
FASCTRA> disk show -n

To assign a disk to the default pool:
FASCTRA> disk assign {disk_name}

To assign all unassigned disks to the default pool:
FASCTRA> disk assign all

To assign a disk to the aggregate:
FASCTRA> aggr add aggr0 -d {disk_name}

To check the aggregate status (i.e RAID type and if it is online):
FASCTRA> aggr status

To check the aggregate disk usage (i.e which disks are data, parity, dparity, spare):
FASCTRA> aggr status -r

To list available hot spares:
FASCTRA> vol status -s

To set RAID minimum spare count to 0 (to disable low spare warnings):
FASCTRA> options raid.min_spare_count 0

To remove a spare disk from one controller (perhaps if you want to take Controller B's spare disk to re-assign to Controller A):
FASCTRB> priv set advanced
FASCTRB*> disk remove_ownership {disk_name}
FASCTRB*> priv set
FASCTRA> disk assign {disk_name}

With the FAS2040A, you will not need to use the follow command, but useful to be aware of for future scenarios - to disable disk auto assign to the default pool 0:
FASCTRA> options disk.auto_assign off

Creating a NFS Volume and Assigning Root Access (for say VMware Hosts)

To see total space on aggregate, and available space:
FASCTRA> df -g -A

To create a thick provisioned 2TB volume, export as an NFS share for clients on, enable dedupe (SIS), disable volume snapshots:
FASCTRA> vol create NFSVOL -s volume aggr0 2000G
FASCTRA> exportfs -p rw=,root= /vol/NFSVOL
FASCTRA> sis on /vol/NFSVOL
FASCTRA> vol options NFSVOL nosnap on

Note that by default scheduled snapshots are enabled, to delete any that were created (after turning nosnap on):
FASCTRA> snap delete NFSVOL {snapshot name}

Configuring Autosupport

FASCTRA> options autosupport.mailhost {IP of Mail Server}
FASCTRA> options autosupport.from from@test.priv
FASCTRA> options autosupport.to to@test.priv
FASCTRA> options autosupport.support.transport smtp

And to see autosupport settings:
FASCTRA> options autosupport

Cluster Configuration & Testing

To start off with the HA cluster is disabled. On one controller, run the cf enable command to enable:
FASCTRA> cf enable

To check the status:
FASCTRA> cf status

To test failover:
FASCTRB> cf takeover

To undo the takeover
FASCTRB> cf giveback

Audit Log

To see the commands you have inputted:
FASCTRA> rdfile /etc/log/auditlog

Final Note

Some of the above commands will need to be run on both Controller A and Controller B.

NetApp: FAS2040A Walkthrough Setup & Install Notes Part 4/5 – Configuring via the CLI Part 1 (via serial connection)

Note: This requires a laptop/server with a serial port. In the absence of a built-in serial port, a USB to Serial Converter will to the trick!


The first time the FAS2040A is powered on, and a connection is made via the console cable to Controller A, the prompt that will appear is:


Enter the following commands after the prompt, to set the filer to autoboot (so when it reboots next time it does not sit at the LOADER prompt) and boot into Data ONTAP.

LOADER-A> setenv AUTOBOOT true
LOADER-A> boot_ontap


On first boot the NetApp filer will automatically boot into setup.

Note that the setup can be invoked at any time by running the command setup

FASCTRA> setup

The effects of running setup do not apply until reboot, and setup rewrites the following config files:
and saves the original contents of the files in .bak files.

Run through the setup answering prompts as desired.

Note on VIFs (virtual interfaces)

A recommendation is to setup a VIF (virtual interface or IFGRP) with at least two ethernet ports, which are connected to two different switches or switch modules on a stacked switch for resilience. Since this is a FAS2040A, remember to select yes to the option to allow vif to failover to a HA partner.

"Single-mode VIFs act like a fault tolerant team and will fail traffic over to a standby link when the active link goes down. Multi-mode VIFs act like a group of links providing aggregate bandwidth as well as link redundancy. Remember VIFs must have different names for failover to work in a HA setup."

BMC Setup

After running through the setup, configure the BMC using:

FASCTRA> bmc setup

And once complete, reboot the controller with the reboot command

FASCTRA> reboot

NetApp: FAS2040A Walkthrough Setup & Install Notes Part 3/5 – Disk Ownership and RAID Setup

The FAS2040A (Dual Controller Model) utilizes an Active-Active HA pair of controllers. Each controller head must have disk ownership of the disks on which its own root volume sits. In a fail-over situation, the remaining cluster head will take over the 'virtual' identity of the failed head with its igroups, LUN mappings, etcetera.

Out of the factory, the FAS2040A is configured similarly to the diagram below:
A = Controller A
B = ControllerB
D = Data Disk
P = Parity Disk
dP = Double Parity Disk
S = Spare
X = Unowned

Only 2 of the available disks are available for data (1 data disk per controller), and the available data space also contains the root volume /vol/vol0

In the scenario where we have a FAS2040A with 12 disks, and no available disk shelf, below are some options available

Option 1: Utilizing Each Head Equally
In the above option, we can either have 6 disks available for data (3 on each controller) if require to keep a hot spare on each controller, or can have 8 disks available for data (4 on each controller) if it is not required to keep a hot spare on each controller.

Option 2: Going For As Large A Volume As Possible
In the above option, we can have either 7 disks available for data (6 on one controller, 1 on the other) if require to keep a hot spare with the main controllers pool of disks, or can have 8 disks available for data (7 on one controller, 1 on the other) if do not require to keep a hot spare with the main controllers pool of disks.

Note 1: RAID-4 (which has only one parity disk) is not considered here as it is not possible to disable the alarm for no available spare if using RAID-4. Also, for doing no disruptive disk firmware upgrades, need to be on RAID-DP, and it is preferable to have a RAID-DP than a RAID-4 with hot spare.

NetApp: FAS2040A Walkthrough Setup & Install Notes Part 2/5 – Controller Connections

Fig. 1: FAS2040 Controller Port Identification

The above diagram illustrates the 10 connections that are available on a FAS2040 controller

FC 0a & 0b Port
Two 1/2/4 Gb FC Initiator/Target Ports

SAS 0d Port
Integrated SAS port for DS4243 shelf connectivity

Console Port

e0P Port (ACP)
"ACP, or Alternate Control Path, is a protocol that enables Data ONTAP to manage and control the DS4243 disk shelf storage subsystem. It uses a separate network (alternate path) from the data path, so management communication is not dependent on the data path's being intact and available."

BMC Mgmt 10/100 Ethernet Port
Baseboard Management Controller

NetApp: FAS2040A Walkthrough Setup & Install Notes Part 1/5 – Introduction

The following set of notes runs through some considerations and configuration steps involved in a NetApp FAS2040A (dual controller model) implementation. These notes were compiled from in the field installations, and the intention here is to present them in a clear and concise way. It would be impossible to cover every scenario, and NetApp's own documentation and community information is very good, and very detailed; but hopefully these notes will provide a few pointers and a bit of enlightenment to whomsoever reads them.