Sunday, 14 October 2012

Tips on How to Troubleshoot a Citrix Provisioning Services (PVS) and XenApp Farm

Introduction

The following notes were compiled prior to a 2 day on-site troubleshooting engagement:

The focus of the troublehooting was:
- Session freezing (single user session)
- Sporadic freezing of XenApp Servers
- The freezes can occur at any time
- When a server locks up, it is unusable even via the VMware vSphere Client
- Issues first occurred about 3 months ago
Note: In the environment in question - PVS is used to stream a large farm of XenApp servers

Citrix Proddocs on Provisioning Server


Major Topics of Investigation

- Hardware – HP (as an example) Static High Performance Power mode enabled?
- Provisioning Services Server – locally attached storage or on a SAN (http://support.citrix.com/article/CTX132848)?
- Provisioning Server with ESX4 (vSphere) Freezing - http://forums.citrix.com/thread.jspa?threadID=269710 (suggests use E1000 NIC cards on VMs)
- PVS host should be physical as per best practices (not a strong requirement anymore), with plenty of processing power and RAM!
- Windows Firewall on Provisioning Services Host / XenApp Server?
- Using Windows NLB or NetScaler LB (better) …?
- Citrix PVS Stream Service reliability?
- Enable Offline Database Support

Best Practices for Configuring Provisioning Server on a Network – (http://support.citrix.com/article/CTX117374)
- Disabling Spanning Tree or Enabling PortFast of PVS and client switchports (Useful PowerConnect and Cisco Switch Commands)
- TCP Large Send Offload disabled on PVS servers and clients?
- Auto Negotiation - recommend hard coding
- Stream Service Isolation – separate VLAN for streaming
+ Configuration of Unicast Storm Control – disable (or raise level) on switch ports connected to PVS servers and clients (http://support.citrix.com/article/CTX121618 - Target Device Performance and Freezing Issue)
+ Multiple 1 Gbps network adapters are better than a single 10 Gbps one (for resilience)

- Recommended Hotfixes for XenApp 6 and Later on Windows Server 2008 R2 - http://support.citrix.com/article/CTX129229
- Problems with 2008R2 SP1 and XenApp 6 - http://support.citrix.com/article/CTX126711
- Hotfix for XenApp 6 - http://support.citrix.com/article/CTX130473 (supersedes 127523 and 1280432)
Note: Latest version (Oct 2012) of Citrix Provisioning Services is 6.1

From the “Best Practices Guide for Provisioning Services and XenApp”

vDisk Options
vDisk – Standard images are a recommended best practice
vDisk Cache – Virtual XenApp Server: Target Device – Shared Storage
vDisk Cache – Physical XenApp Server: Target Device – Local Storage

Network Options
Boot Device Manager where modifying DHCP scope options is not possible
Virtual XenApp server – configure virtual machine to boot from boot image ISO
Physical XenApp server – boot from boot image DVD in physical DVD drive

Operating System Tuning
Events logs – local storage/Citrix EdgeSight/Microsoft Event Collection Services
Auto Update – disable
Group Policy – Computer Configuration > Windows Settings > Security Settings > Local Policies > Security Options > Domain Member: Disable machine account password changes (since maintained by Provisioning Services)
Organizational Units – XenApp servers have their own OU, and each specific XenApp role has its own OU

XenApp Tuning
With XenApp Prep utility managing many of the technical processes that must be done to a XenApp server so it can be provisioned successfully with Provisioning Services.
Application Delivery – all relevant operating system and XenApp hotfixes and service packs should be included in the base image.
Application Delivery – the common operating system and XenApp configuration should be used for the base image (If 80% of the servers require a specific setting while another 20% do not, the base image should include the special setting.)
Application Delivery – The base image should include all XenApp plugins (If application streaming will be used, the streaming plugin should be installed as part of the base image.)
Application Delivery – Depending on the usage of server certificates, the appropriate root certificate should be part of the base image.
Application Cache (considering using for less impact on write cache)
Automate Application Pre-Cache (to improve initial startup time)
Pagefile – leave assigned to C: drive
Multiple Partitions – best practice is to stick with a single partition
Drive Remapping – remapping server drives is not best practice
Web Interface – a common base Web Interface role should be created with all pertinent consistent configurations, and certificates for all WI servers should be on each on
Data Collectors – The primary data collector should be set to Most Preferred and the backup data collector to Preferred (data collectors can be delivered from the same vDisk but a physical data collector is recommended)

Application Tuning
Machine Specific Registry Keys – consider using ID Reset / Personality / Differential Disks

Maintenance
Automatic Updating of vDisks – consider using Workflow Studio

From “Health Monitoring and Alerting for XenApp, XenDesktop, and NetScaler”

- XA Health Monitoring and Recovery Policies
- LBDiag Utility (Load Balancing Diag Utility) (http://support.citrix.com/article/CTX124446)
- CDF Trace (for troubleshooting worker group load balancing issues)
- Citrix Medevac 2.5 (troubleshooting XML, TS, IMA and RPC related issues - http://support.citrix.com/article/CTX107935)
- Power State Tracking with PCM / Hypervisor state tracking (PCM v 6.5)
- Application Deployment / Monitoring with SCCM
- Edgesight Performance Monitoring
- Microsoft’s Performance Monitor (Perfmon - http://support.citrix.com/article/CTX118742)
- HDX Monitor for XA6 / Desktop Director 2.0 for XA6.5
- Monitoring commands: Query, QFARM, QueryDC, QueryHR, QueryDS
- Monitoring Through PowerShell> add-pssnapin citrix.xenapp.commands
+ Complete list of Citrix Support troubleshooting tools http://support.citrix.com/article/CTX126294
+ Wireshark

Further Questions (to the above)

- Infrastructure - networks, hypervisor, server hardware…?
- VMware DRS or similar acting?
- DHCP options or PXE?
- VOIP on the line?

- Active Directory + computer accounts?
- DNS settings (forward/reverse lookup)?
- Time synchronization?

- ICA latency?
- Session reliability?

- RDP enabled?
- HDX Multimedia (client/server side) settings / HDX MediaStream Flash redirection?

- Anti-Virus?
- Applications?
- Particular programs running when experience freezing?

- Licensing?
- SQL Database?

- Web Interface version?
- Where is the web interface?
- CAG or CSG edition/version?
- Going through the CAG/CSG/NetScaler or via Web Interface directly?

- Citrix group policy management installed on a DC?
- Citrix policies?
- AD policies?

Further reading from Citrix Worldwide Consulting Solutions

1: “Best Practices Guide for Provisioning Services and XenApp
2: “XenDesktop and XenApp Best Practices – Reference Guide
3: “XenApp and XenDesktop Policy Planning Guide

6 comments:

  1. Nice artical Vidad. I surrely use it Thanx

    ReplyDelete
  2. Awesome tips ! Very useful. Thank you very much for sharing !

    ReplyDelete
  3. Great in sequence! There is something wonderful about "Tips on How to Troubleshoot a Citrix Provisioning Services (PVS) and XenApp Farm". I am fearful by the excellence of information on this website. There are a bundle of good quality resources here. I am sure I will visit this place another time soon.

    ReplyDelete
  4. Thanks for share this nice information! Your Sound is really good about "Tips on How to Troubleshoot a Citrix Provisioning Services (PVS) and XenApp Farm". I am intimidated by the excellence of information. There are a bundle of good funds here. I am sure I will visit this position again soon.
    I think, Computer Troubleshooters Service Review takes a look deeper into industry which revolves around technology.
    Here is some information about computer troubleshooters.

    ReplyDelete
  5. These are indeed great tips! Those who provide outsourced NOC services will surely find this post helpful. Thank you for posting.

    ReplyDelete