Tuesday, 30 October 2012

Tech Round Up 31st October 2012

A quick Tech Round Up whilst have the time before a busy schedule kicks in to play. Random interesting tech stuff from: Cisco, Citrix, Drobo, F5, Fusion-IO, HP, Microsoft, NetApp, Veeam and VMware

## Cisco ##

Vladan Seget recommends the TrainSignal courses in Cisco Nexus 1000v!

## Citrix ##

Important security considerations if using the XenClient

## Drobo ##

A nice storage array with Automated Data-Aware Tiering at a fair price!

## F5 ##

Comes recommended if you’re interested in learning about F5 Networks great products!

## Fusion-IO ##

Interesting discussion on Linkedin regards using SSDs to increase SQL performance; recommendations for just using for tempdb; with some recommending the O/S; considerations that the SQL server caches a lot to memory so won’t actually use the SSD; and thoughts on HA and DR.

## HP ##

Interesting PDF which is pretty much a walkthrough install guide for implementing SCVMM 2012 with HP 3PAR and Hyper-V hosts.

If you lose your datastores after upgrading ESXi 5, check out this article which installs the HP Smart Array P220i Controller after upgrade but before post-upgrade reboot.

“HP Insight Control Storage Module Version 7.1 for vCenter is a plug-in for VMware's vCenter management console, enabling VMware administrators to quickly obtain context-aware information about the HP storage in their VMware environment directly from vCenter.” (a free download)

## Microsoft ##

Considering using Deduplication with Server 2012 then check this Technet link out. Interesting table below:

If you get the chance, try them out!

Even if you’re not excited by Windows 8, you’ll want to check out the final chapter:
Ten Things You’ll Hate about Windows 8 (And How to Fix Them)

## NetApp ##

Slightly biased comparison of the dedupe offerings of Dell, EMC, HDS, HP, IBM, NetApp and startups. Of course NetApp leads the way!

Slightly biased 5 differences between NetApp and EMC!

## Veeam ##

The Veeam vs AppAssure debate is still going strong in the Veeam forums. The link above takes you to a very detailed comparison bmcchesney-nv. Interesting to note that AppAssure uses agents and only recently has started supporting flavours of Linux!

## VMware ##

Excellent how to videos covering – Site Recovery Manager, vCenter Operations Manager, vCloud, vFabic/Spring, vSphere, vSphere Storage Appliance

Nice article with table (below) to ensure you patch your vSphere components in the correct order!

Monday, 29 October 2012

We Can Health Check It for You Wholesale – How to do a Full VMware vSphere Healthcheck!

Introduction

I was going to create a simple checklist with loads of check boxes for arming the savvy but forgetful Technical Consultant with a full arsenal of checks to throw at a VMware infrastructure, then, I thought I’d do something a bit more eccentric!

Image: Arnie from the original Total Recall based on Philip K. Dick’s “We Can Remember It for You Wholesale”

Things we will check for you!

1. Is your networking design to best practice?

With access to your network switches, routers, monitoring tools, topology diagrams, and – where possible – server room and/or datacentre access; we will do a thorough analysis of your networking infrastructure against best practice recommendations for support of your SAN and VMware vSphere infrastructure. This will include some if not all, and is not limited to:

Inventory/audit for make and model, switches with Non-Blocking Backplane design, switching backbone with sufficient bandwidth (2 times Gbps speed of utilized ports for full duplex traffic), network patching, standard of network cabling, support for and use of Inter-Switch Linking (ISL) or dedicated Stacking (recommended) architecture, support for and use of Link Aggregation Groups (LAG), configuration of ports as Trunk ports or Access ports, support for and use of Flow Control (802.3x) on all ports, support for Rapid Spanning Tree Protocol (R-STP), support for and correct use of Jumbo Frames, switches with adequate Buffer Space per switch port (at least 512KB per port), no iSCSI enable on PowerConnect 54xx switches in a SAN of more than one iSCSI arrays, PortFast configured on STP Edge Ports, where STP is acting, Flow Control configured (essential with 10GbE), sufficient packet buffering from 10GbE to 1GbE ports, Speed and Duplex settings hard coded, Storm-Control disabled for iSCSI, MTU set correctly for Jumbo Frames (9000 or 9014) and/or packet fragmentation prevention across all devices (standard MTU 1500), use of VLANs to correctly segregate traffic, requirements for/ use of QoS, resilience of network, goal to minimize number of switches, use of private iSCSI network where appropriate, analysis of switch logs to include packet loss/drop, switch firmware, switch software, support on Hardware Compatibility List (HCL) and Software Compatibility List (SCL), manufacturers best practices, research for known issues, …

2. Is your SAN design to best practice?

With access to your SAN Management consoles, fabric switches, monitoring tools, topology diagrams, and – where possible – server room and/or datacentre access; we will do a thorough analysis of your SAN against known VMware vSpher best practices. This will include some if not all, and is not limited to:

Inventory/audit for make and model, check SAN cabling, provision for dual-power supplies, RAID configuration optimized to support the hosted applications, array firmware (not mixed in groups or clusters), disk firmware, resilience of SAN design, utilisation (sufficient free space – i.e. smaller of 5% or 100GB free on EqualLogic), distribution of load across SAN, disk health and availability of hot-spares, volume access (no read or write access where should not be permitted), volume naming conventions and matching with hosted datastores, initiators being used, flow-control settings, Jumbo Frames settings, utilization of available front-end interfaces (to the network), utilization of available back-end interfaces (to disk shelves), verify connections running at full bandwidth and duplex, check for load-balancing completion, check for RAID build completion, check of inter-switch link congestion, check for management connectivity, check storage latency, use of enhancing software packages/features (examples: SAN HeadQuarters, Multi-pathing Extension Module, Host Integration Tools, AutoSnapshot Manager, VAAI plugins, vSphere vCenter integration), configuration of alerts, logs analysis, support on Hardware Compatibility List (HCL) and Software Compatibility List (SCL), manufacturers best practices, research for known issues, …

3. Is your VMware design to best practice?

With access to your vSphere vCenter and vSphere Client, SSH access to hosts, monitoring tools, topology diagrams, and – where possible – server room and/or datacentre access; we will do a thorough analysis of your VMware vSphere implementation against known best practices. This will include some if not all, and is not limited to:

Inventory/audit for make and model, use of hardware or software HBAs, multi-pathing configuration and load-balancing, VMware Host configuration, NTP configuration, host BIOS settings including unnecessary devices disabled, virtual networking configuration (vSphere Standard Switch and Distributed Switch), resilience to NIC and other component failure, VMware licensing and use of paid for features, storage and adapter queues, storage and adapter latency, cluster configuration (using FQDNs, resources to satisfy HA, DRS settings and rules), datastore capacities and free-space, distribution of VMs on datastores, datastore block size, iSCSI/FC HBA timeout settings for seamless controller failover, obtaining expected storage throughput, use of enhancing software packages/ features/ plug-ins (examples: VAAI plugins, HP Offline Bundle, Dell HIT), configuration of alerts and monitoring, vCenter and Host logs analysis, firmware and software versions, support on Hardware Compatibility List (HCL) and Software Compatibility List (SCL), manufacturers best practices, research for known issues, …

4. Are your Virtual Machines configured to best practice?

With access to your vSphere vCenter; we will do a thorough analysis of your VMware virtual machines against known best practices. This will include some if not all, and is not limited to:

Presence of old or large snapshots and VCB garbage, virtual machine hardware up to date, VMware Tools up to date, CD-ROMs and unnecessary devices disconnected/removed, CPU ready too high, over allocation of vCPUs, VM Swap and Ballooning, guest disk size and free space, thin or thick VMDKs, guest disk-alignment, backup, replication and DR strategy.

Bonus Section 1 - Some Tools We Might Use

vCheck (VMware Analysis)
ESXTOP or RESXTOP (VMware Analysis)
Crystal Disk Mark (Storage Analysis)
VMware VMmark 2.x (Storage Analysis)
VMware I/O Analyzer (Storage Analysis)

Bonus Section 2 - Some Links You Might Want to See

VMware Best Practice Guide:

Gabesvirtualworld.com:

Storage:

Networking:

Final Comment

I will try and keep this updated with new stuff as and when. Thank you for reading!

Sunday, 28 October 2012

Thoughts on Spanned Layer 2 Networks

Introduction

A layer 2 network is by design intended to exist in a local area. A stretched layer 2 network is not considered networking best practice, but there is traction in the industry to provide for it, and make it a work.

There are various solutions out there, for example:

– Citrix NetScaler CloudBridge
– Cisco Pseudowire-Class (Layer 2 Tunnelling Protocol)
– F5 Big-IP
– Or even a straight-forward GRE tunnel

Most of the technologies above have been around for a while.

Considerations

The most important considerations in order to have a functional stretched layer 2 network are:

– Hardware
– Broadcast Traffic and Bandwidth
– Distance and number of hops
– MTU

Hardware:

Not all hardware will support running stretched VLANs, an example is that Cisco Metro switches are recommended if you are considering Cisco Pseudowire-class.

Broadcast Traffic and Bandwidth:

In a local layer 2 network, the broadcast traffic is typically contained across not many switches within a small area. Typically, this broadcast traffic will flow across Gigabit or better networks. When building a stretched layer 2 network, consideration has got to be given regards how much bandwidth will all the broadcast traffic take. A poorly designed stretched layer 2 network could soon find its WAN links saturated by broadcast traffic. Also, the cost of providing enough bandwidth to accommodate all the inter-VLAN and broadcast traffic needs to be carefully measured.

Usual networking best practices apply regards keeping the number of devices on the layer 2 networks down to a reasonable level, so as to reduce the amount of broadcast traffic, and where possible implement VLANs to break up broadcast domains.

Distance and number of hops:

The desired response times from devices in either side of the layer 2 network, will dictate how far it is possible to physically separate the local segments of the stretched layer 2 network.

Example using division by the speed of light:
1000km introduces a minimum round trip (round trip being 2000km) latency of 6.7ms

Every device (hop) in the path from side A to side B, will add an element of jitter, and additional latency.

MTU:

The minimum MTU of devices across the layer 2 network is very important. Every device should be set to the same MTU. If a device sends packets with a larger MTU than another device, this introduces fragmentation of packets, and more delay to their successful delivery.

It is possible to test maximum MTU using the DOS prompt, for example:

C:\>ping www.test.com -f -l 1473

Pinging www.test.com with 1473 bytes of data:
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.

C:\>ping www.test.com -f -l 1472

Pinging www.test.com [174.36.85.72] with 1472 bytes of data:
Reply from 174.36.85.72: bytes=1472 time=153ms TTL=53
Reply from 174.36.85.72: bytes=1472 time=152ms TTL=53
Reply from 174.36.85.72: bytes=1472 time=152ms TTL=53
Reply from 174.36.85.72: bytes=1472 time=153ms TTL=53

In the above example, we see that setting payload size to 1473 causes the sent packet to be fragmented and not delivered. Setting the payload to 1472 meets with success!

Remember that the MTU = max packet size from ping test + 28
(The 28 is for IP and ICMP headers)

A nice little MTU test tool is the free to download mtutest.exe and an example of its output is below:

Final Comment

The industry drive for stretched layer 2 networks comes mostly from the cloud and the desire for on premise networks to be stretched into the cloud for possible migration strategies or to elastically take advantage of provisioning additional resource. Another driver is for moving servers with legacy applications on, and which may not take kindly to having their or other device IP addresses changed.

Personally, I would avoid wherever possible!

Tech Round Up 27th October 2012

It’s been a good six weeks since the last Tech Round Up due to holidays and work commitments; here are a few notes and bookmarks to things found as new, useful, or interesting in October! With stuff from: Cisco, Citrix, EMC, Google, Juniper Networks, Microsoft, NetApp, Veeam, VMware, and Ultimate Storage

## Cisco ##

Cisco have announced a free (Essential Edition) version of their Nexus 1000V 2.1 (currently in beta). Features include: VLAN, ACL, QoS, VXLAN, vPath, LACP, Multicast, Netflow, ERSPAN, Management, vTacker, vCenter Plug-in

## Citrix ##

A very good blog post describing the hard to understand way PVS uses memory for caching disk operations!

Might be handy if you need to do a health check of a Citrix XenApp environment!
“In 4 easy steps you can analyze your log files, profile your XenApp environment and scan for hundreds of known issues. It takes only minutes to deliver clear, actionable advice customized to you.
1. Download and Run Citrix Scout - generates diagnostic log files from your XenApp environment.
2. Upload the log file to Citrix Auto Support – See here for how to generate diagnostic data for other supported products.
3. Wait a few minutes – the file will be parsed and measured against hundreds of parameters.
4. Review the results

XenServer 6.1 is out! Check out the release notes for new features and enhancements.

How NetScaler can help you with your IPv4 to IPv6 transitioning!

## EMC ##

Reminds me that I need to get some EMC certs!

## Google ##


## Juniper Networks ##

An interesting design with:
2x Juniper Networks EX4550 switches = 2u
4x Dell D820 PowerEdge™ Server = 8u
2x Dell PowerVault™ MD3600i 10GbE iSCSI Array & 12 600GB 15K SAS = 4u

And vGW Kernel Interaction:

## Microsoft ##

1. Storage Spaces
2. Hyper-V 3
3. PowerShell 3.0
4. Failover clusters
5. Data deduplication
6. SMB 3.0
7. Scale-out file server

> Can convert and deploy VMs from VMware hosts running either vSphere 4.1 or 5.0 to Hyper-V hosts on Windows Server 2012, by migrating configuration such as memory, virtual processor and by adding virtual network interface cards.
> Converts VMware virtual disks to Microsoft Virtual Hard Disks (VHDs)
> Uninstalls VMware tools prior to conversion and installs Integration Services if the guest OS is Windows Server 2003 SP2 and above
> Supports Windows Server 2003 SP2, Windows Server 2008 and Windows Server 2008 R2 and Windows 7 guest Operating Systems.
> Scriptable command-line interface and wizard-driver GUI

PowerShell scripts to help you build your 2012 IaaS clouds.

A nice review from The Register of System Center 2012 and its components!

Perhaps a bit old now with Exchange 2013 imminent, but a useful interactive tool for planning your upgrade to Exchange 2010 on-premises or in the cloud!

“As I understand it there is no longer a need to split off the various networking duties but finding the references (I heard them in person) is a bit difficult at this time.”

## NetApp ##

Excellent videos on using the NetApp VSC 4.1 (Virtual Storage Console plugin for vSphere and XenCenter)!

## Veeam ##

Veeam Backup & Replication 6.5 is out now!
Including:
> Free e-discovery and item recovery for Microsoft Exchange
> Easy VM recovery from SAN snapshots
> New hypervisor support: vSphere 5.1 and Windows Server 2012 Hyper-V

Veeam Community Forums Digest October 1 – October 7, 2012
“Veeam is building network traffic verification right into the product's engine. With this in place, you can be always sure that the remote backup or replica receives the exact same data as source proxy has sent. And, should the in-transmit data corruption be detected, bad data blocks will be resent automatically without any impact on the job (in the end, no one wants 1TB full backup over WAN to fail at 99%). We will be squeezing this feature right into the upcoming B&R 6.5 release!”

An interesting debate from the Veeam forums!


## VMware ##


“VMware recommends that ESXi has a persistent scratch location available for storing temporary data including logs, diagnostic information, and system swap. (This is not a requirement, however.)”
Which can be configured for the vSphere Client > Advanced Settings > ScratchConfig : ScratchConfig.ConfiguredScratchLocation

Interesting walkthrough to show how it’s done!

Great article for those interested in vCloud Design!

Using the vSphere Web Client 5.1!

Came across this pdf from VMware Professional Services which serves as an interesting template for a professional services engagement! The vCheck health report from here is a very handy tool for this purpose.

Phases of Virtualization:


## Ultimate Storage ##

Interesting if you’re looking for prices for certain (value) storage vendors, including – QNAP, Synology, Quantum, QSAN, Overland Storage …

Friday, 26 October 2012

HP P2000 G3 FC Install Complete with Windows Server 2008R2

The following blog runs through what could be considered to make up a complete install of a HP P2000 MSA and all possible software components when integrating with Windows Server 2008R2.
Note: Not all these components have to be installed; this will depend on use case.

The following reference kit is being considered:
1 x AP846B (HP P2000 G3 FC Dual Controller SFF MSA)
24 x 512547-B21 (HP 146 GB 2.5” 6Gb/s SAS 15k rpm HDD)
2 x AJ764A (HP 82Q 8Gb 2-port PCIe FC HBA)

The FC HBAs will go into two separate physical Windows Server 2008R2 servers, which will be directly connected to the P2000 G3 MSA, in a Failover Cluster setup.

Sources of Information


Contents

Part 1: P2000 Hardware
Part 2: Windows Features Install
Part 3: QLogic FC HBA Hardware Install on Windows Hosts
Part 4: QLogic FC HBA Software Installs on Windows Hosts
Part 5: P2000 Software on Windows Hosts
Part 6: P2000 Software on a Management Station or Windows Host
Part 7: Configure the MSA and Best Practices

Walkthrough

Part 1: P2000 Hardware

1.1 Install equipment in the rack
1.2 Install hardware options
  
1.3 Connect the P2000 G3 MSA System to data hosts

1.4 Connect the P2000 G3 MSA System’s Ethernet network management ports into network switches for management network connectivity.

1.5 Power on components

Part 2: Windows Features Install

2.1 Install Multipath I/O
2.2 Install Storage Manager for SANs
Note 1: Installing Failover Clustering is not considered here.
Note 2: Windows 2008R2 does not need to install an additional DSM.

Part 3: QLogic FC HBA Hardware Install on Windows Hosts

3.1 Install the HP 82Q FC HBA’s into the 2 x physical Windows 2008R2 servers.

Part 4: QLogic FC HBA Software Installs on Windows Hosts

4.1 Install the HP Storage Fibre Channel Adapter Kit (Windows Driver) for the x64 QLogic Storport Driver

4.2 Install the HP Firmware Online Flash for QLogic FC HBAs

4.3 Install the QConvergeConsole Management Utility GUI for HP Branded QLogic based FC Adapters

4.4 Install SANSurfer Manager for HP Branded Qlogic FC HBAs

4.5 Install the HP Insight Management Agents

Part 5: P2000 Software on Windows Hosts

5.1 Install the HP P2000 MSA CAPI Proxy for Windows x64

5.2 Install the HP P2000 MSA VDS Provider for Windows x64

5.3 Install the HP P2000 MSA VSS Provider for Windows x64

5.4 Install the HP P2000 MSA USB Driver (x64)

Part 6: P2000 Software on a Management Station or Windows Host
Note: Of the 4 components below, only the MSA Device Discovery Tool is not supported on virtualized guest operating systems.

6.1 Install the HP P2000 MSA Device Discovery Tool

6.2 Install the HP P2000 MSA Firmware Update Tool for Windows
(Or install the HP P2000 G3 MSA Arrays – Online ROM Flash Component for Windows.)

6.3 Install the HP P2000 MSA HDD Detection Tool

6.4 Install the HP P2000 MSA Performance Tool

Part 7: Configure the MSA and Best Practices

Highly recommended reading:

A few excellent extracts:

The chart below gives information on the expansion capability for the P2000 G3 MSA supported RAID levels

Table 1: An overview of supported RAID implementations

Table 2: Optimizing performance for your application

How to Update a Windows Server 2008 R2 Hyper-V Cluster to SP1

Introduction

The following walkthrough will run through the steps to update a Windows Server 2008 R2 Enterprise Hyper-V Cluster to SP1. The Hyper-V R2 hosts we are using here, have never received a Windows patch since first build.

Note: If you are using SCVMM2008 R2, recommend upgrading this to SCVMM2008 R2 SP1 first!

Preliminary Steps

i. Before starting make sure your backups are secure and you know you can restore those backups.
ii. Download the Service Pack 1 file to your Hyper-V hosts (windows6.1-KB976932-X64.exe)

(Optional) Run a cluster validation test before starting but don’t do the full cluster validation test as this requires some cluster downtime. Recommend omit storage tests.

Fig. 1: Validate This Cluster…
Fig. 2: Run only tests I select
Fig. 3: Deselect Storage
The Service Pack 1 Upgrade

1. Live migrate all the guests from the node you wish to update to SP1
2. Disable antivirus services.
3. In Failover Cluster Manager, Pause the node (pause is typically used when performing updates etcetera)

Fig. 4: Pause node

4. Reboot the node first to make sure it is in a clean state.
5. Double-click to start up the Service Pack 1 file, and follow the prompts to install it.

Fig. 5: Install Windows Server 2008 R2 Service Pack 1
Note: Should take around 30 minutes and multiple reboots might be needed.

Fig. 6:  Windows Server 2008 R2 Service Pack 1 is now installed
6. Verify it has installed

Fig. 7: Using systeminfo to verify Service Pack level
7. (Optional) At this state you might want to apply further Windows and Microsoft updates. At the time of writing there were around 60 post SP1 updates, and to download (on 2 Gbps) and apply the updates takes a good 30-45 minutes. One of the updates post SP1 is IE9, this will wait for a prompt to be acknowledged – yes or no – before applying.

8. Check the event log for any errors

Note 1: In the setup event log you’ll see these messages –
i) Initiating changes for package KB976932. Current state is Absent. Target state is Installed. Client id: SP Coordinater Engine.
ii) Package KB976932 was successfully changed to the Installed state.

Note 2: if an extra reboot is required you’ll see an extra entry in between these stating “A reboot is necessary before package KB976932 can be changed to the Installed state” in which case reboot!

9. Resume node

Fig. 8: Failover Cluster Manager – Resume node
10. Re-enable antivirus.
11. Live migrate guests from the next node to the node already upgraded to SP1 and repeat 1 to 9 until all nodes are upgraded.

*****

12. It is recommended to run a Cluster Validation after the upgrade of all cluster nodes has finished – remember to omit storage tests (see above)!

13. Finally, update guests VMs with the new SP1 version of the Hyper-V Integration Components (Action > Insert Integration Services Setup Disk.) The guests will require a reboot to complete to installation.

Note: If you have SCVMM2008 R2, you can use it to update multiple.

With Hyper-V R2 SP1 installed and the Integration Components updated, you can take advantage of the Dynamic Memory and RemoteFX features.

Credits, Sources, and Further Reading


Lab Note – VMware Workstation 9 with Hyper-V

In the lab we were running Hyper-V inside VMware Workstation 9 – which is perfectly possible, just remember to select/change the Guest operating system to ‘Microsoft Windows Hyper-V (unsupported)’.

Fig. 9: VMware Workstation 9 – Guest O/S = Hyper-V (unsupported)
If not you’ll see the ‘Hyper-V cannot be installed’ message when you attempt to tick the box for the Hyper-V role in server manager.

Fig. 10: Hyper-V cannot be installed
The Hyper-V (unsupported) option is a new feature of VMware Workstation 9. You will not be able to run nested guests under Hyper-V inside VMware Workstation 9 until Service Pack 1 is installed.

Real World Experience on a 3 Node Cluster

One minor issue encountered was that the Hyper-V Virtual Machine Management Service hung on one of the hosts whilst it was shutting down. Left if for about 20 minutes until it became obvious it was stuck, then easy fix was just to kill vmms.exe from the DOS CLI on another server.
The command is>
taskkill.exe /S SYSTEM /IM vmms.exe

Note 1: This is logged in with a domain account with administrative privileges across the hosts.
The full command syntax is>
taskkill.exe /S SYSTEM /U USERNAME /P PASSWORD /IM PROCESS
Also worth knowing for listing running processes on a remote (or the local) host>
tasklist.exe /S SYSTEM /U USERNAME /P PASSWORD

Note 2: The VMM service only hung on one host; this could be because I had not disabled the AV service (Symantec Endpoint Protection) on the host which I did for the others.

Note 3: Because of the time it was taking to apply the service pack (one host took 2 hours), ended up patching two of the three hosts at the same time (two paused hosts,) and this worked fine. Here we had the luxury of being able to shutdown all the guests.

Sunday, 14 October 2012

Tips on How to Troubleshoot a Citrix Provisioning Services (PVS) and XenApp Farm

Introduction

The following notes were compiled prior to a 2 day on-site troubleshooting engagement:

The focus of the troublehooting was:
- Session freezing (single user session)
- Sporadic freezing of XenApp Servers
- The freezes can occur at any time
- When a server locks up, it is unusable even via the VMware vSphere Client
- Issues first occurred about 3 months ago
Note: In the environment in question - PVS is used to stream a large farm of XenApp servers

Citrix Proddocs on Provisioning Server


Major Topics of Investigation

- Hardware – HP (as an example) Static High Performance Power mode enabled?
- Provisioning Services Server – locally attached storage or on a SAN (http://support.citrix.com/article/CTX132848)?
- Provisioning Server with ESX4 (vSphere) Freezing - http://forums.citrix.com/thread.jspa?threadID=269710 (suggests use E1000 NIC cards on VMs)
- PVS host should be physical as per best practices (not a strong requirement anymore), with plenty of processing power and RAM!
- Windows Firewall on Provisioning Services Host / XenApp Server?
- Using Windows NLB or NetScaler LB (better) …?
- Citrix PVS Stream Service reliability?
- Enable Offline Database Support

Best Practices for Configuring Provisioning Server on a Network – (http://support.citrix.com/article/CTX117374)
- Disabling Spanning Tree or Enabling PortFast of PVS and client switchports (Useful PowerConnect and Cisco Switch Commands)
- TCP Large Send Offload disabled on PVS servers and clients?
- Auto Negotiation - recommend hard coding
- Stream Service Isolation – separate VLAN for streaming
+ Configuration of Unicast Storm Control – disable (or raise level) on switch ports connected to PVS servers and clients (http://support.citrix.com/article/CTX121618 - Target Device Performance and Freezing Issue)
+ Multiple 1 Gbps network adapters are better than a single 10 Gbps one (for resilience)

- Recommended Hotfixes for XenApp 6 and Later on Windows Server 2008 R2 - http://support.citrix.com/article/CTX129229
- Problems with 2008R2 SP1 and XenApp 6 - http://support.citrix.com/article/CTX126711
- Hotfix for XenApp 6 - http://support.citrix.com/article/CTX130473 (supersedes 127523 and 1280432)
Note: Latest version (Oct 2012) of Citrix Provisioning Services is 6.1

From the “Best Practices Guide for Provisioning Services and XenApp”

vDisk Options
vDisk – Standard images are a recommended best practice
vDisk Cache – Virtual XenApp Server: Target Device – Shared Storage
vDisk Cache – Physical XenApp Server: Target Device – Local Storage

Network Options
Boot Device Manager where modifying DHCP scope options is not possible
Virtual XenApp server – configure virtual machine to boot from boot image ISO
Physical XenApp server – boot from boot image DVD in physical DVD drive

Operating System Tuning
Events logs – local storage/Citrix EdgeSight/Microsoft Event Collection Services
Auto Update – disable
Group Policy – Computer Configuration > Windows Settings > Security Settings > Local Policies > Security Options > Domain Member: Disable machine account password changes (since maintained by Provisioning Services)
Organizational Units – XenApp servers have their own OU, and each specific XenApp role has its own OU

XenApp Tuning
With XenApp Prep utility managing many of the technical processes that must be done to a XenApp server so it can be provisioned successfully with Provisioning Services.
Application Delivery – all relevant operating system and XenApp hotfixes and service packs should be included in the base image.
Application Delivery – the common operating system and XenApp configuration should be used for the base image (If 80% of the servers require a specific setting while another 20% do not, the base image should include the special setting.)
Application Delivery – The base image should include all XenApp plugins (If application streaming will be used, the streaming plugin should be installed as part of the base image.)
Application Delivery – Depending on the usage of server certificates, the appropriate root certificate should be part of the base image.
Application Cache (considering using for less impact on write cache)
Automate Application Pre-Cache (to improve initial startup time)
Pagefile – leave assigned to C: drive
Multiple Partitions – best practice is to stick with a single partition
Drive Remapping – remapping server drives is not best practice
Web Interface – a common base Web Interface role should be created with all pertinent consistent configurations, and certificates for all WI servers should be on each on
Data Collectors – The primary data collector should be set to Most Preferred and the backup data collector to Preferred (data collectors can be delivered from the same vDisk but a physical data collector is recommended)

Application Tuning
Machine Specific Registry Keys – consider using ID Reset / Personality / Differential Disks

Maintenance
Automatic Updating of vDisks – consider using Workflow Studio

From “Health Monitoring and Alerting for XenApp, XenDesktop, and NetScaler”

- XA Health Monitoring and Recovery Policies
- LBDiag Utility (Load Balancing Diag Utility) (http://support.citrix.com/article/CTX124446)
- CDF Trace (for troubleshooting worker group load balancing issues)
- Citrix Medevac 2.5 (troubleshooting XML, TS, IMA and RPC related issues - http://support.citrix.com/article/CTX107935)
- Power State Tracking with PCM / Hypervisor state tracking (PCM v 6.5)
- Application Deployment / Monitoring with SCCM
- Edgesight Performance Monitoring
- Microsoft’s Performance Monitor (Perfmon - http://support.citrix.com/article/CTX118742)
- HDX Monitor for XA6 / Desktop Director 2.0 for XA6.5
- Monitoring commands: Query, QFARM, QueryDC, QueryHR, QueryDS
- Monitoring Through PowerShell> add-pssnapin citrix.xenapp.commands
+ Complete list of Citrix Support troubleshooting tools http://support.citrix.com/article/CTX126294
+ Wireshark

Further Questions (to the above)

- Infrastructure - networks, hypervisor, server hardware…?
- VMware DRS or similar acting?
- DHCP options or PXE?
- VOIP on the line?

- Active Directory + computer accounts?
- DNS settings (forward/reverse lookup)?
- Time synchronization?

- ICA latency?
- Session reliability?

- RDP enabled?
- HDX Multimedia (client/server side) settings / HDX MediaStream Flash redirection?

- Anti-Virus?
- Applications?
- Particular programs running when experience freezing?

- Licensing?
- SQL Database?

- Web Interface version?
- Where is the web interface?
- CAG or CSG edition/version?
- Going through the CAG/CSG/NetScaler or via Web Interface directly?

- Citrix group policy management installed on a DC?
- Citrix policies?
- AD policies?

Further reading from Citrix Worldwide Consulting Solutions

1: “Best Practices Guide for Provisioning Services and XenApp
2: “XenDesktop and XenApp Best Practices – Reference Guide
3: “XenApp and XenDesktop Policy Planning Guide