Date post: | 05-Dec-2014 |
Category: |
Technology |
Upload: | vmworld |
View: | 982 times |
Download: | 4 times |
Horizon View Troubleshooting:
Looking under the Hood
Matt Coppinger, VMware
Jack McMichaels, VMware
EUC5291
#EUC5291
2
Disclaimer
This session may contain product features that are
currently under development.
This session/overview of the new technology represents
no commitment from VMware to deliver these features in
any generally available product.
Features are subject to change, and must not be included in
contracts, purchase orders, or sales agreements of any kind.
Technical feasibility and market demand will affect final delivery.
Pricing and packaging for any new technologies or features
discussed or presented have not been determined.
3
Agenda
Common Issues
What can go wrong?
Domain 1: View Client Connectivity issues
Domain 2: Desktop Not Available!
Domain 3: Broken Broker
Domain 4: Why Is My Desktop So Slow!?
Resources
4
Top 6 GSS tickets
1. SSL, SSL, SSL – View 5.1/5.2 introduces security improvements
• "Obtaining SSL Certificates for VMware View Servers” PDF on vmware.com
2. Persona – Make sure permissions are set correctly, and avoid
this common misconfiguration error.
http://kb.vmware.com/kb/2019502
3. Parent VM issues – Do NOT P2V! Clean VM please…
4. Connectivity – Understand the View network requirements!
5. PCoIP sizing – PCoIP Calculator -
http://myvirtualcloud.net/?page_id=1562
6. PCoIP Log Viewer - http://mindfluxinc.net
5
Generating a Certificate Signing Request – the Right Way
Read the Manual!
Create a config file
Generate cert signing
request (CSR)
Validate CSR and Private
Key
Send CSR to CA
Receive Signed Cert
from CA
Import Cert Configure
View Server to use Cert
Test!
6
The Ultimate Resource
• http://pubs.vmware.com/view-52/topic/com.vmware.ICbase/PDF/horizon-view-52-obtaining-certificates.pdf
Obtaining SSL
Certificates
for Horizon View
7
Understanding View Connectivity Requirements
8
Broker Specifications
Common Issues resulting from undersizing a broker:
• Memory Heap Issues
• Threading issues
• Latency Issues
Recommendations:
• Always size your broker right from the beginning, avoid downtime or
issues later.
• Minimum recommended specifications:
• 2 vCPU / 10 GB RAM for normal sizing (50-500 VMs)
• 4 vCPU / 16+ GB RAM for large sizing (500+ VMs)
9
Troubleshooting Keys
Check View Administrator Dashboard
• Shows system health and any issues
Understand the client connection process (where most problems lie)
Set the Logging Level on CS
Check Connection Broker Logs
• Match and Filter SessionID, User, FSP
Check View Agent Logs
Check View Desktop PCoIP Logs
Use kb.vmware.com!
10
Tracking Sessions
Use BareTail/BareGrep or similar 3rd party text parsers
11
You’ve Deployed View. What Could Go Wrong?
You’ve got a problem:
• I can’t even connect to View
• I get disconnected randomly!
• Why is the display so bad?
• Why is my desktop not available?
• I’m seeing an error in View, what does it mean?
• vCenter is reporting an error
• My desktop is slow…
12
Identifying the Problem Domain
View Client
Network
View Manager
View Composer
vCenter Server
Compute
Storage
13
Domain 1: View Client Connectivity Issues
Common challenges
• View Client can’t connect
• Logon failure
• Black screen
• Poor quality display
• Randomly disconnected session
14
Domain 1: Failure to Communicate…
Where to look
• Connection Broker logs
• Windows 2008 - <DriveLetter>:ProgramData\Application Data\VMware\VDM\logs
• Event Database
What to look for –
• (Client connects) [SimpleAJPService] (ajp:broker:Request9) Request from
/192.168.2.1: POST /broker/xml
• (Broker authentication) [WinAuthFilter] (SESSION:7072-***-a79c mattc) Attempting
to authenticate user 'mattc' in domain 'FUTUREOFFICE’
• (User has authenticated to Broker) [AuthorizationFilter] (SESSION:7072-***-a79c)
User FUTUREOFFICE\mattc has successfully authenticated to VDM
• (Audit Entry) [Audit] (SESSION:7072-***-a79c)
BROKER_LOGON:USER:FUTUREOFFICE\mattc;USERSID:S-1-5-21-326850759-
2560684469-1780228732-1113;USERDN:CN=S-1-5-21-326850759-2560684469-
1780228732-1113,CN=ForeignSecurityPrincipals,DC=vdi,DC=vmware,DC=int;
• Event Database: BROKER_USERLOGGEDIN
15
User Experience Issues
Black screen of death – instead of desktop!
• PCoIP port blocked (TCP and UDP 4172) or SVGA Driver issue
• pcoip_server/client logs - C:\Users\All Users\VMware\VDM\logs
• Error attaching to SVGADevTap, error 4000: EscapeFailed
• MGMT_SCHAN :scnet_client_open: tera_sock_connect returned error 10060 -
Connection timed out!
• Incorrect PCoIP External URL configured for Security/Connection Servers
16
User Experience Issues
Poor quality display
• Bandwidth, latency, or QoS
• Pcoip_server logs report
• VGMAC :Stat frms: Loss=0.45%/0.21% (R/T)
• MGMT_PCOIP_DATA :BW: Decrease (loss) old = 234.9982 new = 176.8438
Randomly disconnected session?
• 15 min after established - wssm process hasn't started on desktop
• View Agent logs (<DriveLetter>:\ProgramData\VMware\VDM\logs)
• PENDING_EXPIRED
• Sometimes caused by daisy-chaining the GINA (WinXP)
17
Domain 2: Desktop Not Available
Common Issues
• No Desktop Available
• Pool provisioning issues – customization
• Agent not communicating with broker
• Stuck at desktop login screen (SSO)
Where to look
• Connection Broker/ View Agent logs
• Event Database
What to look for
• Broker returns list of desktops available to client
• [DesktopsHandler] (SESSION:7072-***-a79c) For user [S-1-5-21-326850759-2560684469-1780228732-1113] and pool [cn=gold-np,ou=server groups,dc=vdi,dc=vmware,dc=int] DesktopTracker returned 2 guest DNs
18
Desktop Not Available
What to look for… (walk through successful connection)
Client requests desktop
• Event Database: BROKER_DESKTOP_REQUEST
Broker allocates session to user
• [FarmImp] (SESSION:7072-***-a79c) cn=3f974017-409f-4912-83bc-
2ee794f22fab,ou=servers,dc=vdi,dc=vmware,dc=int, total session count: 0
• [FarmImp] (SESSION:7072-***-a79c) allocateNewSession - identified server
for application CN=GOLD-NP,OU=Applications,DC=vdi,DC=vmware,DC=int
• Event Database: BROKER_MACHINE_ALLOCATED
Broker attempts SSO
• [FarmImp] (SESSION:7072-***-a79c) Using domain for SSO:
FUTUREOFFICE**
• User won’t be logged on to the VM without this!
19
ADSI Edit – Check Desktop Allocated
20
ADSI Edit – Common Key Values to Inspect
pae-DisplayName
• VM name as displayed in View Admin
pae-DirtyForNewSessions
• Indicates whether the VM is “Dirty” and can be re-used in a non-persistent pool
pae-SVIVMSnapshot
• Indicates the current Snapshot that is in use
pae-VmPath
• Indicates the full Path to the VM in vCenter
pae-VmState
• Indicates the current state of the Desktop – some states are a combination of
this value and other values
21
ADSI Edit – Searching for a Desktop
Find VMs with a Snapshot:
• (&(objectClass=pae-VM)(pae-SVIVmSnapshot=/Baseline/Snapshot1/Snapshot2))
Find VMs with a Name:
• (&(objectClass=pae-VM)(pae-DisplayName=Desktop-234))
22
Desktop Not Available
What to look for…
Broker starts session on VM
• [DesktopSessionImp] (SESSION:7072-***-a79c) startSession – sending StartSession message
Agent responds…
• "DesktopManager got a StartSession message”
• Client Info should be in Agent Log along with PCoIP launch
Event Database: AGENT_PENDING
[DesktopSessionImp] (SESSION:7072-***-a79c) startSession completed:
[DesktopTracker] User FUTUREOFFICE\mattc connected to machine gold-np-2 for desktop gold-np
Client connects to VM (Agent)
• “PCoIPCnx::OnConnectionComplete Begin (PCOIP)”
• “WTS_SESSION_LOGON”
• Event Database: AGENT_CONNECTED
23
Event Database
24
Desktop Not Available
What to look for… Pool Provisioning
Desktops not available due to provisioning error?
• Check View Administrator for Pool status, check datastore capacity
• Check Event Database - BROKER_PROVISIONING_ERROR_*
• Check View Composer has network access to ESX hosts
Desktop not available due to customization?
• Check Desktop status – AGENT UNAVAILABLE
• Check View Dashboard
• Desktop Status > Preparing Desktops OR Problem Desktops
• Check Desktop connectivity to DNS/AD/Connection Server
25
Desktop Not Available
Desktop not available due to VM reset/crash?
• Check Desktop status – ALREADY USED
• Typical on refresh-on-logoff or delete-on-use desktops
• Broker never received an explicit logout message from the agent
• Missing AGENT_ENDED event in DB for VM
View Composer Issues associated with incorrect domain
credentials
• C:\ProgramData\VMware\View Composer\Logs\
• FATAL CSvmGaService - [svmGaService.cpp, 116] Domain join failed Error 5 (0x5):
Access is denied.
26
Domain 3: Broken Broker
Common Issues
• Cannot connect to vCenter
• View Composer errors/issues
• JMS connectivity
• ADAM replication failure
Where to look
• View Administrator
• Event Database
• Windows Event Logs
• View Composer Logs
• Connection Server Logs
27
Broken Broker
What to look for
ADAM Replication
• Check the Connection Broker Window Event Logs
• ADAM (VMwareVDMDS)-log
• Error: ADAM Replication
• Check ADAM replication status on Connection Server
• C:\WINDOWS\adam\repadmin.exe /showrepl localhost:389 DC=vdi,DC=vmware,DC=int
28
Broken Broker
vCenter Server Connectivity
• Admin UI will show RED status
• Check Event Database
• VC_DOWN events
• Impacts provisioning and power operations ONLY
• Check Connectivity from Connection Server to vCenter Server
• Check credentials used to connect to vCenter Server
• Attempt to login in directly to vCenter using vSphere Client
29
Broken Broker
What to look for…
View Composer
• VMs stuck in DELETING status
• VMs have been manually deleted – then pool/desktop deleted
• Causes Composer DB and VC DB to get out of sync
• Composer thinks VM already exists
• Orphaned VMs – KB-2015112 (kb.vmware.com)
• Desktop Composer Fault: 'Virtual Machine with Input Specification already exists‘
JMS Connectivity
• Split site architecture / firewall causes “split brain”
• View Dashboard shows RED status
• Connection Server Logs
• tracker REJOIN messages – JMS connectivity
• tracker RESYNC messages – messages being delayed
30
Cleaning Up Broken / Orphaned Pools
BACKUP
Disable Provisioning on Broker
Stop View Composer
• Remove Composer Database Objects
• Remove ADLDS Servers, Server Groups, and Applications
• Remove AD Computer Entries
• Remove vCenter objects
• Unprotect replicas with sviconfig
• SviConfig -operation=RemoveSviClone -VmName=replica-<guid> -AdminUser=administrator
-AdminPassword=passowrd -ServerUrl=https://localhost:18443/SviService/v2_0
Active Directory View Composer
AD LDS
31
One Query To Rule Them All (Proceed with Extreme Caution!)
Delete One VM
DELETE FROM dbo.SVI_VM_NAME WHERE NAME='replaceMe'
DELETE FROM dbo.SVI_COMPUTER_NAME WHERE NAME='replaceMe'
DELETE FROM dbo.SVI_SC_PDISK_INFO WHERE PARENT_ID=(SELECT ID FROM dbo.SVI_SIM_CLONE WHERE VM_NAME='replaceMe')
DELETE FROM dbo.SVI_SC_BASE_DISK_KEYS WHERE PARENT_ID=(SELECT ID FROM dbo.SVI_SIM_CLONE WHERE VM_NAME='replaceMe')
DELETE FROM dbo.SVI_TASK_STATE WHERE SIM_CLONE_ID=(SELECT ID FROM dbo.SVI_SIM_CLONE WHERE VM_NAME='replaceMe')
DELETE FROM dbo.SVI_REQUEST WHERE ID=(SELECT REQUEST_ID FROM dbo.SVI_TASK_STATE WHERE SIM_CLONE_ID=(SELECT ID FROM
dbo.SVI_SIM_CLONE WHERE VM_NAME='replaceMe'))
DELETE FROM dbo.SVI_SIM_CLONE WHERE VM_NAME='replaceMe'
Delete ALL VMs and Pools
DELETE FROM dbo.SVI_VM_NAME
DELETE FROM dbo.SVI_COMPUTER_NAME
DELETE FROM dbo.SVI_SC_PDISK_INFO
DELETE FROM dbo.SVI_SC_BASE_DISK_KEYS
DELETE FROM dbo.SVI_TASK_STATE
DELETE FROM dbo.SVI_REQUEST
DELETE FROM dbo.SVI_SIM_CLONE
DELETE FROM dbo.SVI_REPLICA
DELETE FROM dbo.SVI_DG_CUST_PROP
DELETE FROM dbo.SVI_DEPLOYMENT_GROUP
Note: Composer will auto clean replicas when no dependent SIM_CLONE objects are available.
32
Domain 4: Why is My Desktop So Slow?
Common Issues
• Storage IO bottleneck
• Memory contention
• CPU contention
• Network issues
Where to look
• vCenter Server
• ESXTOP
• vCops for View
• 3rd Party Tools?
33
Why is My Desktop So Slow?
What to look for
CPU
• Cluster/Host utilization < 90%
• VM utilization - %USED (ESXTOP)
• VM %RDY Time (ESXTOP) < 10
Memory
• Host utilization < 85%
• VM utilization
• Swapping / Ballooning SWCUR > 1 / MCTLSZ > 1 (ESXTOP)
Storage
• Disk Read Latency < 25ms
• ESXTOP DAVG or KAVG < 25ms (ESXTOP)
34
ESX Performance Snapshots
Use the following command to collect performance metrics for
8 hours on a host:
• for i in `seq 8`;do esxtop -a -b -d 5 -n 720 > $i.<hostname>.csv;done
The above command will create eight 100mb files consisting of
1 hour's worth of ESXTOP snapshots.
• -d = delay in seconds
• -n = iterations
• (-d 5 x -n 720 = 3600 seconds or 1 hour).
Useful for replaying performance data over wide time periods
for support to analyze!
35
CPU
36
Memory
37
Storage
38
vSphere Performance Management Tools – vCenter Ops Mgr
vCenter Operations Manager
• Aggregates metrics into workload, capacity and health scores
• Relies on dynamic thresholds
39
VCOPs – My Favorite Tool D
ata
sto
re L
ate
ncy
Data
sto
re T
hro
ughput
40
Getting Help
Read the product documentation
Double check your configuration!
Check kb.vmware.com for your issue
http://communities.vmware.com
Run the Support.bat to extract the logs
• Notice the diagnostic tests that run…
• http://kb.vmware.com/kb/1017939
• Check the product documentation for using VDMADMIN command for creating
various Data Collection Tool bundles
Submit a Support Request
41
Summary
Understand where the issue may lie
• Client? Agent? Server? Composer? vCenter? ESX?
Know the problem domains
Check the View Dashboard and Event Database
Identify the issue – know what a successful connections looks like
Check the logs
Use Performance Tools – ESXTOP, vCenter, vCenter Operations
for View
Get Help
42
Questions
43
Other VMware Activities Related to This Session
HOL:
HOL-MBL-1301
Horizon View from A to Z
Group Discussions:
EUC1006-GD
View with Andre Leibovici
EUC5291
THANK YOU
Horizon View Troubleshooting:
Looking under the Hood
Matt Coppinger, VMware
Jack McMichaels, VMware
EUC5291
#EUC5291