Tips on How to Troubleshoot a Citrix Provisioning Services (PVS) and XenApp Farm

Introduction

The following notes were compiled prior to a 2 day on-site troubleshooting engagement:

The focus of the troublehooting was:

- Session freezing (single user session)

- Sporadic freezing of XenApp Servers

- The freezes can occur at any time

- When a server locks up, it is unusable even via the VMware vSphere Client

- Issues first occurred about 3 months ago

Note: In the environment in question - PVS is used to stream a large farm of XenApp servers

Citrix Proddocs on Provisioning Server

http://support.citrix.com/proddocs/topic/technologies/pvs-provisioning.html

Major Topics of Investigation

- Hardware – HP (as an example) Static High Performance Power mode enabled?

- Provisioning Services Server – locally attached storage or on a SAN (http://support.citrix.com/article/CTX132848)?

- PVS cache drive should not be A, B, or C drives - http://forums.citrix.com/thread.jspa?threadID=303783 and http://support.citrix.com/article/CTX122141

- Provisioning Server with ESX4 (vSphere) Freezing - http://forums.citrix.com/thread.jspa?threadID=269710 (suggests use E1000 NIC cards on VMs)

- PVS host should be physical as per best practices (not a strong requirement anymore), with plenty of processing power and RAM!

- Windows Firewall on Provisioning Services Host / XenApp Server?

- Using Windows NLB or NetScaler LB (better) …?

- Citrix PVS Stream Service reliability?

- Enable Offline Database Support

Best Practices for Configuring Provisioning Server on a Network – (http://support.citrix.com/article/CTX117374)

- Disabling Spanning Tree or Enabling PortFast of PVS and client switchports (Useful PowerConnect and Cisco Switch Commands)

- TCP Large Send Offload disabled on PVS servers and clients?

- Auto Negotiation - recommend hard coding

- Stream Service Isolation – separate VLAN for streaming

+ Configuration of Unicast Storm Control – disable (or raise level) on switch ports connected to PVS servers and clients (http://support.citrix.com/article/CTX121618 - Target Device Performance and Freezing Issue)

+ Multiple 1 Gbps network adapters are better than a single 10 Gbps one (for resilience)

- Recommended Hotfixes for XenApp 6 and Later on Windows Server 2008 R2 - http://support.citrix.com/article/CTX129229

- Problems with 2008R2 SP1 and XenApp 6 - http://support.citrix.com/article/CTX126711

- Hotfix for XenApp 6 - http://support.citrix.com/article/CTX130473 (supersedes 127523 and 1280432)

Note: Latest version (Oct 2012) of Citrix Provisioning Services is 6.1

From the “Best Practices Guide for Provisioning Services and XenApp”

vDisk Options

vDisk – Standard images are a recommended best practice

vDisk Cache – Virtual XenApp Server: Target Device – Shared Storage

vDisk Cache – Physical XenApp Server: Target Device – Local Storage

Network Options

Boot Device Manager where modifying DHCP scope options is not possible

Virtual XenApp server – configure virtual machine to boot from boot image ISO

Physical XenApp server – boot from boot image DVD in physical DVD drive

Operating System Tuning

Events logs – local storage/Citrix EdgeSight/Microsoft Event Collection Services

Auto Update – disable

Group Policy – Computer Configuration > Windows Settings > Security Settings > Local Policies > Security Options > Domain Member: Disable machine account password changes (since maintained by Provisioning Services)

Organizational Units – XenApp servers have their own OU, and each specific XenApp role has its own OU

XenApp Tuning

With XenApp Prep utility managing many of the technical processes that must be done to a XenApp server so it can be provisioned successfully with Provisioning Services.

Application Delivery – all relevant operating system and XenApp hotfixes and service packs should be included in the base image.

Application Delivery – the common operating system and XenApp configuration should be used for the base image (If 80% of the servers require a specific setting while another 20% do not, the base image should include the special setting.)

Application Delivery – The base image should include all XenApp plugins (If application streaming will be used, the streaming plugin should be installed as part of the base image.)

Application Delivery – Depending on the usage of server certificates, the appropriate root certificate should be part of the base image.

Application Cache (considering using for less impact on write cache)

Automate Application Pre-Cache (to improve initial startup time)

Pagefile – leave assigned to C: drive

Multiple Partitions – best practice is to stick with a single partition

Drive Remapping – remapping server drives is not best practice

Web Interface – a common base Web Interface role should be created with all pertinent consistent configurations, and certificates for all WI servers should be on each on

Data Collectors – The primary data collector should be set to Most Preferred and the backup data collector to Preferred (data collectors can be delivered from the same vDisk but a physical data collector is recommended)

Application Tuning

Machine Specific Registry Keys – consider using ID Reset / Personality / Differential Disks

Maintenance

Automatic Updating of vDisks – consider using Workflow Studio

From “Health Monitoring and Alerting for XenApp, XenDesktop, and NetScaler”

- XA Health Monitoring and Recovery Policies

- LBDiag Utility (Load Balancing Diag Utility) (http://support.citrix.com/article/CTX124446)

- CDF Trace (for troubleshooting worker group load balancing issues)

- Citrix Medevac 2.5 (troubleshooting XML, TS, IMA and RPC related issues - http://support.citrix.com/article/CTX107935)

- Power State Tracking with PCM / Hypervisor state tracking (PCM v 6.5)

- Application Deployment / Monitoring with SCCM

- Edgesight Performance Monitoring

- Microsoft’s Performance Monitor (Perfmon - http://support.citrix.com/article/CTX118742)

- HDX Monitor for XA6 / Desktop Director 2.0 for XA6.5

- Monitoring commands: Query, QFARM, QueryDC, QueryHR, QueryDS

- Monitoring Through PowerShell> add-pssnapin citrix.xenapp.commands

+ Citrix Scout http://support.citrix.com/article/CTX130147

+ Complete list of Citrix Support troubleshooting tools http://support.citrix.com/article/CTX126294

+ Wireshark

Further Questions (to the above)

- Infrastructure - networks, hypervisor, server hardware…?

- VMware DRS or similar acting?

- DHCP options or PXE?

- VOIP on the line?

- Active Directory + computer accounts?

- DNS settings (forward/reverse lookup)?

- Time synchronization?

- ICA latency?

- Session reliability?

- RDP enabled?

- HDX Multimedia (client/server side) settings / HDX MediaStream Flash redirection?

- Anti-Virus?

- Applications?

- Particular programs running when experience freezing?

- Licensing?

- SQL Database?

- Web Interface version?

- Where is the web interface?

- CAG or CSG edition/version?

- Going through the CAG/CSG/NetScaler or via Web Interface directly?

- Citrix group policy management installed on a DC?

- Citrix policies?

- AD policies?

Further reading from Citrix Worldwide Consulting Solutions

1: “Best Practices Guide for Provisioning Services and XenApp”

2: “XenDesktop and XenApp Best Practices – Reference Guide”

3: “XenApp and XenDesktop Policy Planning Guide”