Tuesday, 26 October 2010

Calculating the RW Value in a VMDK Disk Descriptor File, and Fixing Incorrect Virtual Disk Capacity

*Update: see Formulas to Calculate Dimensionless Hard Disk Size, and a Real World Application to Extending a UFS Partition  posted 20/01/2012

Formula to find R the RW value in the disk descriptor file from G the exact GByte capacity of the disk

R = G x 2097152     {or R = Gx1024x1024x1024/512 }

If you don't know the exact GBbyte capacity of the disk, you can only roughly work it out from number of cylinders C
R = (((C+1)x512xSxH)/1024)-8)x2     {where S is the number of sectors, and H is the number of heads}


Scenario: A virtual guest server has a virtual disk with its capacity totally incorrect, preventing cloning, storage vMotion …

The virtual guest machine in question has a disk that is only 20GB in size (as seen from browsing the datastore, and in Windows,) but the VI Client (this is ESX 3.5U4) connected to either the vCenter or the host, says the disk is 10245.08GB. Investigation shows the RW value in the VMDK disk descriptor file is wrong.


[root@esx01 EXAMPLE]# cat EXAMPLE_0.vmdk

# Disk DescriptorFile
version=1
CID=76cbf27f
parentCID=ffffffff
createType="vmfs"

# Extent description
RW 21485479936 VMFS "EXAMPLE_0-flat.vmdk"

# The Disk Data Base
#DDB

ddb.virtualHWVersion = "4"
ddb.toolsVersion = "7303"
ddb.geometry.cylinders = "2612"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.adapterType = "lsilogic"

[root@esx01 EXAMPLE]#

The RW above should be 20 x 2097152 = 41943040 not 21485479936 (which divided by 2097152 does indeed give the 10245.08 seen via the VI client.)

How to fix:

You might think the above formulas will help, in practice they don't.

The solution:

1: Shutdown the affected machine
2: Detach the affected disk
3: Create another disk of exactly the same size
4: Edit the disk descriptor file for the new VMDK to point to the old -FLAT.VMDK
5: Re-attach the disk using the same SCSI ID as before
6: Boot the affected machine back up

See: http://www.vm-aware.com/2008/06/recreating-missing-vmdk-descriptor-files/

Also see: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1026266

Friday, 22 October 2010

cma.log is filling up most of the space in /var/log (VMware ESX, 4.0.0, 208167)

A full /var/log can cause various problems on an affected ESX host – unable to snapshot a guest, unable to enable HA agent …

One potential culprit for filling up /var/log, can be the cma.log file generated by the HP Insight Management agents.

Example:

[root@esx01 /]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 4.9G 1.9G 2.8G 41% /
/dev/sda2 2.0G 2.0G 0 100% /var/log
/dev/cciss/c0d0p1 198M 68M 121M 36% /boot

Investigation uncovers that the cma.log is taking up nearly all the space:

[root@esx01 /]# cd /var/log/hp-snmp-agents/
[root@esx01 hp-snmp-agents]# ls -s
total 1690984
1690984 cma.log 0 cmaX.log

- or as seen via Veeam Fast SCP (below) -


Fix:

From the folder /var/log/hp-snmp-agents, use the rm command to delete cma.log

[root@esx01 hp-snmp-agents]# rm cma.log
rm: remove regular file `cma.log'? y

(or delete via Veeam Fast SCP)

Then change to the /etc/init.d directory, and run the command ‘service hp-snmp-agents restart’ to restart the HP Management Agents (the cma.log file is invisible, and remains in /var/log until this service is restarted.) This command will not affect live guests in any way.

[root@esx01 hp-snmp-agents]# cd /etc/init.d
[root@esx01 init.d]# service hp-snmp-agents restart
Shutting down NIC Agent Daemon (cmanicd): [ OK ]
Shutting down Storage Event Logger (cmaeventd): [ OK ]
Shutting down FCA agent (cmafcad): [FAILED]
Shutting down SAS agent (cmasasd): [ OK ]
Shutting down IDA agent (cmaidad): [ OK ]
Shutting down IDE agent (cmaided): [ OK ]
Shutting down SCSI agent (cmascsid): [ OK ]
Shutting down Health agent (cmahealthd): [ OK ]
Shutting down Standard Equipment agent (cmastdeqd): [ OK ]
Shutting down Host agent (cmahostd): [ OK ]
Shutting down Threshold agent (cmathreshd): [ OK ]
Shutting down RIB agent (cmasm2d): [ OK ]
Shutting down Rack Infrastructure Info Srv (cpqriisd): [FAILED]
Shutting down Rack agent (cmarackd): [FAILED]
Shutting down Performance agent (cmaperfd): [ OK ]
Shutting down SNMP Peer (cmapeerd): [ OK ]
Starting Health agent (cmahealthd): [ OK ]
Starting Standard Equipment agent (cmastdeqd): [ OK ]
Starting Host agent (cmahostd): [ OK ]
Starting Threshold agent (cmathreshd): [ OK ]
Starting hp-ilo:
Already started hpilo module. [ OK ]
Starting RIB agent (cmasm2d): [ OK ]
Starting hp-ilo:
Already started hpilo module. [ OK ]
Starting Rack Infrastructure Info Srv (cpqriisd): [ OK ]
Starting Rack agent (cmarackd): [ OK ]
Starting Performance agent (cmaperfd): [ OK ]
Starting SNMP Peer (cmapeerd): [ OK ]
Starting Storage Event Logger (cmaeventd): [ OK ]
Starting FCA agent (cmafcad): [ OK ]
Starting SAS agent (cmasasd): [ OK ]
Starting IDA agent (cmaidad): [ OK ]
Starting IDE agent (cmaided): [ OK ]
Starting SCSI agent (cmascsid): [ OK ]
Starting NIC Agent Daemon (cmanicd): [ OK ]

Finally, re-run the df –h to check space on /var/log has been freed up

[root@esx01 init.d]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 4.9G 1.9G 2.8G 41% /
/dev/sda2 2.0G 318M 1.6G 17% /var/log
/dev/cciss/c0d0p1 198M 68M 121M 36% /boot

Thursday, 21 October 2010

Migrating from 32bit Virtual Center to 64bit vSphere 4.1 Virtual Center on a New Database

With the advent of VMware vSphere 4.1, companies with 32bit Virtual Centers will be needing to rebuild these on 64bit systems. There are plenty of articles out there with walkthroughs on how to use the Data Migration tool to migrate from a 32bit to a 64bit Virtual Center, and we will not go into this here. Instead, we shall look at a different scenario:


Scenario:

A small company with 7 hosts in two clusters; 1 cluster of 4 hosts is EVC enabled, the other cluster of 3 hosts is not EVC enabled. The database on the existing vSphere 4 32-bit Virtual Center has grown to over 20GB (in vCenter, estimated space required for 7 hosts and 140 VMs on default settings is just 1.32GB,) so the decision is taken to start afresh with a new database. The decision is also taken to keep the same name and IP address.

Complicating matters – the company uses Veeam Backup and Replication (on v4.1.1,) and has a small 20 seat XenDesktop 4 VDI environment, both of which talk to the Virtual Center server.


Walkthrough and issues encountered:

1) Build Windows 2008 R2 Enterprise server with same name and IP Address as original

2) Install SQL 2008 R2 Express (recommend doing this as comes with a 10GB database limit, whereas the SQL 2005 Express bundled with the vCenter 4.1 installation media only has a 4GB limit)

3) Create a 64bit System DSN for the Virtual Center database and a 32bit System DSN for the Update Manager database

4) Install Virtual Center 4.1

5) Create datacenter and clusters in the new Virtual Centre as a copy of the original

6) Install Update Manager 4.1


Switch over day:

1) Shut down old Virtual Center and reset its account in AD

2) Join the new 64bit Virtual Center to the domain

3) On the non-EVC enabled cluster, the hosts could be added into the new VC without any down time of guest machines

4) On the EVC enabled cluster, hosts could not be added into the new VC without having to shutdown all the guests on the host prior to adding (being in maintenance mode is not a requirement)

Error: The host cannot be admitted to the cluster’s current Enhanced vMotion Compatibility mode


5) Once all the resource pools had been put in place as per the original; time to test the Citrix XenDesktops, and Veeam Backups, and deal with any niggling issues


Citrix XenDesktop 4

These would not initially work as the Citrix Desktop Delivery Controller could not talk to the new VC. The fix was to edit the proxy.xml file on the Virtual Center server. The location of proxy.xml is:

C:\ProgramData\VMware\VMware VirtualCenter

Edited so that the section for type = vim.ProxyService.LocalServiceSpec, and port 8085, and serverNamespace = /sdk - has accessMode changed from httpsWithRedirect to httpAndHttps (see image below):
Change to:
(And restart vCenter Server service to apply changes to proxy.xml)

All the virtual desktops had to be removed from their groups and be re-added, with users reassigned to their desktops.


Veeam Backup 4.1.1

Virtual Machines had to be re-added to their jobs, and the old entry removed. Replicas needed to be kicked off from scratch.


Other Issues:

1) Unable to assign permissions under Domain accounts via the vSphere Client, a search would bring up no accounts

Error -> Call “UserDirectory.RetrieveUserGroups” for object “UserDirectory” on vCenter Server failed


The solution was to have the ‘VMware VirtualCenter Server’ service log on as a domain account (the domain account used must also have permissions to the VIM_VCDB database)

2) In Update Manager, all the settings were greyed out

Possibly a side effect of installing Update Manager on the Virtual Center whilst the machine was still in a workgroup. A few things were tried … Update Manager was uninstalled. When trying to reinstall Update Manager, this error was encountered:

Error 25085.Setup failed to register VMware vCenter Update Manager extension to VMware vCenter Server:


The fix outlined in -

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1024795

was to use ADSI edit to add CN=com.vmware.vcIntegrity which was missing, and then the install ran fine.