Advanced XenServer Training CCAT Q1 - 2013. © 2012 Citrix | Confidential – Do Not Distribute...

Advanced XenServer TrainingCCAT

Q1 - 2013

© 2012 Citrix | Confidential – Do Not Distribute

Coursework Agenda

• XenServer Overview• XenServer Components Deep-Dive

ᵒ Lab 1: Navigating Dom0, Logs and the CLI

• XenServer Storage Primitives (Caveman-ish?)ᵒ Lab 2: Playing with Storage

• XenServer Networkingᵒ Lab 3: NICs, NICs and more NICs

• XenServer Troubleshootingᵒ Lab 4: How do I fix this stuff?

Hypervisor OverviewWhat is XenServer?


Topics Covered

• Comparisons to other hypervisorsᵒ VMware vSphere 5.1ᵒ Microsoft Hyper-V 2012

• Real world applications for XenServerᵒ XenAppᵒ XenDesktopᵒ PVS

Hypervisor OverviewXenServer vs. The Competition


Market Share and Growth Comparison

Company Hypervisor Platform

2011 Growth Current Estimated Market Share

Microsoft Hyper-V 62% 25%

Citrix XenServer 25% 15 – 20%

VMware ESX 21% 50%


VMware vSphere

• Current Version – 5.1

• Consists of two components: VMware ESX and vCenter Server ᵒ ESXi is the hypervisor and installs on bare metal hardware. ᵒ vCenter Server provides centralized management and allows administrators to

configure and monitor ESXi hosts, provision virtual machines, storage, networking, and much more.

• Management mechanism: vSphere Client ᵒ Windows application that acts as a single pane of glass to manage either a standalone

ESXi host directly or an entire datacenter though vCenter.


vSphere Architecture diagram


vSphere VM Architecture diagram


Hyper-V

• Current Version – Hyper-V 2012

• Originally released in Server 2008

• Bundled with Windows Server licenses

• Behind VMware, Hyper-V retains the most market share in server virtualization

• Consists of single component: Windows Server with Hyper-V roleᵒ Windows Server is the base hypervisor

• Management mechanism: SCVMMᵒ SCVMM (System Center Virtual Machine Manager) provides centralized management

and allows administrators to configure and monitor hosts, provision virtual machines, storage, networking, etc.

ᵒ Can tie in directly with MOM/SCOM and Systems Management Server


Hyper-V Architecture diagram


Hyper-V VM Architecture diagram


Why is Hyper-V important?

• Bundled in with Windows Server 2012, Hyper-V provides free virtualization, and enterprise features at no additional cost

• Bridging functionality gap with VMware

• Tight integration with Windows infrastructure

• Microsoft has had much success with midmarket companies (<1000 users), winning 30 – 40% of the time


XenServer

• Current Version – 6.1

• Consists of single component: XenServer Coreᵒ CentOS-based hypervisor

• Management mechanism: XenCenterᵒ XenCenter console which is Windows only is the management tool. It directly connects

to the current Pool master in order to send commands to the rest of the XS pool ᵒ Has APIs for monitoring and management (To be discussed in more detail later).


XenServer VM Architecture diagram


Limitations Comparison

Parameter Hyper-V 2012 XenServer 6.1 vSphere 5.1

Host Cores 320 160 160

RAM 4 TB 1 TB 2 TB

VMs 1024 150 (60 w/HA) 512

vCPUs 900 2048 (25/core)

Cluster/Pool VMs 4000 1024 (600/SR) 4000

Hosts 64 16 32

VM CPUs 64 16 64

RAM 1 TB 128 GB 1 TB

NUMA Yes Host Only Yes


Feature Comparison

Feature Hyper-V 2012 XenServer 6.1 vSphere 5.1

Incremental Backups Yes Yes (VM Prot & Reco) Yes

NIC Teaming Yes Yes Yes

Integrated HA Yes Yes Yes

OS App Monitoring Yes No API-Only

Failover Prioritization Yes Yes Yes

Affinity & Rules Yes API-Only (SCVMM) Yes

Cluster-Aware Updating Yes 50/50 (Pool Upgrade is not very smart)

Yes

Hypervisor OverviewXenServer 6.1 - What’s new?

-Thanks to Ryan McClure for donating content


XenServer 6.1 – Key features

• Link Aggregation Control Protocol (LACP) support

• Storage XenMotion

• Integrated StorageLink (iSL) support for EMC VNX series arrays


XenServer 6.1 – Virtual Appliances

• Demo Linux Virtual Appliance

• Workload Balancing 6.1.0 Virtual Appliance

• vSwitch Controller Virtual Appliance

• Web Self Service 1.1.2 Virtual Appliance

• Citrix License Server VPX v11.10

• Citrix XenServer Conversion Managerᵒ Allows up to 500 VM conversions from VMware vSphere to XenServer


What’s New? – Management and Support


• Live VDI Migration

• Emergency Network Reset

• XenServer Conversion Manager

• Enhanced Guest OS Support

• Supported VM Count Increased to 150


• LACP Support

• VLAN Scalability Improvements

• IPv6 Guest Support

• Updated OvS

What’s New? – Networking


Storage XenMotion and VDI Migration

Storage XenMotion Live VDI Migration


XenServer 6.1 – Feature(s) that didn’t get lime light

• Performance Monitoring Enhancements Supplemental Packᵒ Extended XenServer monitoring available (stored in RRD – Round Robin Databases)

Metric AvailabilityAverage physical CPU usage over all CPUs (%) NewTime physical CPU spent in [C,P]-state (%) NewHost memory reclaimed by dynamic memory control (B) New

Maximum host memory that could be reclaimed by dynamic memory control (B) New

Host aggregate, physical network interface traffic received/sent (B/s) New

I/O throughput read/write/total per PBD (MiB/s) NewAverage I/O latency (milliseconds) NewI/O average queue size NewNumber of I/O requests currently in flight NewIOPS, read/write/total (requests/s) NewTime spent waiting for I/O (%) New


XenServer Roadmap

• Sarasotaᵒ Dom0 Disaggregationᵒ Increased Scalabilityᵒ Fewer Reboots due to Hotfixesᵒ vGPU

• Jupiterᵒ Storage QoS

• Tallahassee – Private Releaseᵒ NetScaler SDX Updateᵒ XenDesktop Interoperability and

Usability Enhancementsᵒ Hardware Monitoring and Reporting

• Clearwaterᵒ Windows 8 Guest Supportᵒ UI Enhancements

• Naplesᵒ User Experience and Third Party

Enhancementsᵒ VHDX Disk Format Support


Additional Resources

• CTX118447 XenServer Administrator’s Guide• CTX118448 XenServer Installation Guide• CTX118449 XenServer Virtual Machine Installation Guide• CTX118451 XenServer Release Notes• CTX119472 XenConvert Guide • CTX118450 XenServer Software Development Guide

Advanced XenServer TrainingXenServer Under The Hood


Agenda

• Core Componentsᵒ Hardwareᵒ Hypervisorᵒ Dom0ᵒ DomU

• Beyond Single Serverᵒ Resource Poolsᵒ HAᵒ Meta Dataᵒ XenMotion/Live Migration

• Workload Balancing

XenServer OverviewCore Components


Terminology to Remember

• Host

• Resource Pool

• Pool Master

• Dom0

• XAPI (XENAPI)

• XenServer Tools

• Live VDI Migration



Behind The Scenes

• Native 64-bit bare metal hypervisor – Is this true?

• Cent OS

• Based on Linux 2.6.32 kernel

• Drivers are SUSE Linux Enterprise Server v11 SP1

• Xen hypervisor and Domain 0 manage physical server resources among virtual machines


Legacy Virtualization Architecture

UserApps

UserApps

Hardware

HALT SAFE HALT

Legacy Hypervisor

“Binary Translation”Slow, expensive software

emulation layer

http://en.wikipedia.org/wiki/Image:Tux.png


UserApps

UserApps

Hardware

HALTHYPERCALL

HALT

XenServer

VT/AMD-V

Paravirtualized guests make high-speed calls

directly to the hypervisor

Paravirtualization

• Relies on “enlightened” operating systems

• Kernel and I/O paths know they are being virtualized

• Cooperation provides best performance



Hardware-Assisted Virtualization

UserApps

UserApps

Hardware

HALTHYPERCALL

HALT

XenServer

VT/AMD-V

Other guests benefit from hardware-accelerated call

translation

• Hardware-assist allows high performance without emulation



Understanding Architectural Components

The Xen hypervisor and Domain 0 manage physical server resources among virtual machines


Understanding the Hardware Component

Hardware layer contains the physical server components, including memory, CPU and storage


Understanding the Hypervisor Component

Xen hypervisor is a thin layer of software that runs right on top of the hardware


Understanding the Domain 0 Component

Domain 0, or the Control Domain, is a Linux VM that manages the network and storage I/O of all guest VMs


Dom0 Resourcing

• Linux-based guests• virtual network interfaces• virtual disks• virtual CPUs

• Windows-based guests• virtual network interfaces• virtual disks

• Resource Control• Processor, network, disk I/O priority• Machine Power-States


Why This Model Kinda Sucks

• Control Requests - XS Console / Direct CLIᵒ Queuingᵒ Confusion

• Dom0 Overburdening

• Distributed Domino Effect

• Limited space due to pre-allocation

• Log Bug

• Not built to Scale out of the Box ᵒ Dom0 Memoryᵒ Dom0 CPU


Why This Model Kinda Sucks

• Control Requests - XS Console / Direct CLIᵒ Queuingᵒ Confusion

• Dom0 Overburdening

• Distributed Domino Effect

• Limited space due to pre-allocation

• Log Bug

• Not built to Scale out of the Box ᵒ Dom0 Memoryᵒ Dom0 CPU


DOM0 Tuning

• Supports 4 vCPUs (Default 1 Prior to 6.0)ᵒ More is betterᵒ IRQBALANCE (http://irqbalance.org/)

• Memory Tweaking (Default 752)ᵒ 2940M Tested for 130/16.25 VMsᵒ Warning: Single Server!

• XenHeap (5.5 Only)ᵒ Set xenheap_megabytes to (12 + max-vms/10)ᵒ Xc.Error("creating domain failed: hypercall 36 fail: 12: Cannot allocate memory (ret -

1)")

http://irqbalance.org/

http://irqbalance.org/


Understanding the DomU Linux VM Component

Linux VMs include paravirtualized kernels and drivers


Understanding the DomU Windows VM Component

Windows VMs use paravirtualized drivers to access storage and network resources through Domain 0


Real-time Resource Adjustment

• Linux-based guestsᵒ Add/remove virtual network interfacesᵒ Add virtual disksᵒ Add/remove virtual CPUs

• Windows-based guestsᵒ Add/remove virtual network interfacesᵒ Add/remove virtual disks

• Resource QoS Controlᵒ Processor, network, disk I/O priority


Why This is Cool

• Paravirtualization is key to performanceᵒ Hypervisor and OS cooperation provides best performance

• Utilizes hardware-assisted virtualizationᵒ Fully supports Intel VT and AMD-V enabled processorsᵒ Hardware-assist allows high performance without emulation

• Future: Paravirtualization with Hardware-Assist

• Benchmarks vs running on nativeᵒ Linux can expect .5 - 4% overheadᵒ Windows can expect 2 - 7% overhead

• VMware (EULA prevents publishing)ᵒ 5-10% Overhead seen on internal testing


Lessons Learned: The other DOM - DOMU

• We can only push around 350 Mb/s through a single Windows VM.

• We can 1 to 1.1 Gb/s on a single Linux VM • To push more traffic through a single Windows

VM: ᵒ Manually add additional netback processes

• xen-netback.netback_max_groups=<#Procs> under /boot/extlinux.conf in section labeled xe-serial just after the assignment console=hvc0.

ᵒ Manually pinning vCPUs to specific VMs• xe vm-param-set uuid=<vm-uuid> VCPUs-

params:mask=<CPUs to pin>

XenServer OverviewBeyond Single Server


Management Architecture Comparison

VMWare XenServer

Traditional ManagementArchitecture

• Single backend management server

DistributedManagement Architecture

• Clustered management layer


Resource Pools

• Join multiple physical servers into one logical pool

• Enables shared configurations, including storage

• Required for Automated VM Placement and XenMotion

Xen Hypervisor

Xen Hypervisor

Xen Hypervisor


High Availability

Xen HypervisorXen HypervisorXen Hypervisor

Shared Storage

Prot

ecte

dPr

otec

ted

Prot

ecte

dD

o no

t re

star

t


Xen HypervisorXen HypervisorXen Hypervisor

Shared Storage

XenMotion - Live Migration


XenMotion – Live VM Movement

• XenMotion allows minimal downtime movement of VMs between physical systems

• Generally 150-200ms of actual “downtime”

• Most of the downtime is related to network switch moving IP traffic to new port


XenMotion CPU Requirements

• XenMotion requires systems XS have similar CPUs

• Must be the same manufacturer

• Must be the same type

• Can be different speed

• Example Xeon 51xx series chipsᵒ Could have a mix of 5130 and 5120 chips


• Systems verify correct storage and network setup on destination server• VM Resources Reserved on Destination Server

Pre-Copy Migration: Round 1

Source Virtual Machine Destination



• While source VM is still running XenServer copies over memory image to destination server• XenServer keeps track of any memory changes during this process






• After first pass most of the memory image is now copied to the destination server• Any memory changes during initial memory copy are tracked



• XenServer now does another pass at copying over changed memory





• Xen still tracks any changes during the second memory copy• Second copy moves much less data• Also less time for memory changes to occur





• Xen will keep doing successive memory copies until minimal differences between source and destination

Pre-Copy Migration


• Source VM is paused and last bit of memory and machine state copied over• Master unlocks storage from source system and locks to destination system• Destination VM is unpaused and attached to storage and network resources• Source VM resources cleared

XenMotion: Final


The Concept of MetaData

• The metadata contains information about the VMs:ᵒ Name, description, Universally Unique Identifier (UUID), VM configuration, and

information about the use of resources on the host or Resource Pool such as Virtual Networks, Storage Repository, ISO Library.

• Most metadata is written when the VM, SR and network are created and is updated if you make changes to the VM configuration.

• If the metadata is corrupted, the resource pool and/or VMs may become unusable.

• Always backup metadata during:ᵒ Resource pool and/or host upgradeᵒ Before patchingᵒ Migration from one storage repository to another


Backing up the MetaData• It is not necessary to perform daily

exports of all the VM metadata.

• To export the VM metadata:1. Select Backup, Restore and

Update from the menu.2. Select Backup Virtual Machine

Metadata.3. If prompted, log on with root

credentials.4. Select the Storage Repository

where the VMs you want to back up are stored.

5. After the metadata backup is done, verify the successful completion on the summary screen.

6. In XenCenter, on the Storage tab of the SR selected in step 3, a new VDI should be created named Pool Metadata Backup.


Restoring MetaData

• Prerequisites: ᵒ Set up/re-attach the vDisk Storage Repository ᵒ Virtual Networks are set up correctly by using the same names

• From the console menu:1. Select Backup, Restore and Update from the menu.2. Select Restore Virtual Machine Metadata.3. If prompted, log on with root credentials.4. Select the Storage Repository to restore from.5. Select the Metadata Backup you want to restore.6. Select if you want to restore only VMs on this SR or all VMs in the pool.7. After the metadata restore is done, verify the summary screen for errors.8. Your VMs are now available in XenCenter and can be started at the new site.


Data Replication for DR


Simplifying Disaster Recovery

Xen Hypervisor

Xen Hypervisor

Xen Hypervisor

Shared Storage

Xen Hypervisor

Xen Hypervisor

Xen Hypervisor

Shared Storage

Production Site DR Site

Automated backup of VM metadata to SR

Replication of SR includes Virtual Disks and VM metadata

Attach replicated SR

Restore of VM metadata will recreate VMs

1

2

3

4

1

2

3

4

XenServer OverviewWorkload Balancing


Workload Balancing

• Highly granular, continuous guest and host performance profiling

• Policy-driven workload balancing across XenServer pools and hosts

• Historical analysis and trending for planning purposes

• Reports available in XenCenter


Workload Balancing – Critical Thresholds

• Components included in WLB evaluation:• CPU• Memory• Network Read• Network Write• Disc Read• Disk Write

• Optimization recommendation is being triggered if a threshold is reached


Workload Balancing – Weights

• Weights are all about the recommended target host

• Simple interface to determine which factors are most relevant for the VM


Workload Balancing

• Ideal placement recommendations at VM start-up, resume and live relocation

• Ideal VM relocation recommendations for host maintenance


Workload Balancing – Placement Strategies

• Maximize Performance• Default setting• Spread workload evenly across all physical

hosts in a resource pool• The goal is to minimize CPU, memory, and

network pressure for all hosts

• Maximize Density• Fit as many virtual machines as possible

onto a physical host• The goal is to minimize the number of

physical hosts that must be online


Workload Balancing Server

• All components can run on one (virtual) machine

• Multiple server deployment recommended for larger, multi-pool environments

• Data store can be hosted on existing DB platform

• Architecture look familiar?

Data Store

Data Collection Manager service

Web Service Host

Analysis Engine service


Workload Balancing - Components

• Workload Balancing Components• Data Collection Manager service• Analysis Engine service• Web Service Host• Data Store• XenServer• XenCenter

Data StoreXenServer

Resource PoolXenServer

Resource Pool

Data Collection Manager service

Web Service Host

XenCenter

Performance Metrics

Perform

ance Metric

s

Recommendations

Analysis Engine service

Hypervisor OverviewLab 1 – Intro Lab


In this lab… 30 minutes

• Create a XenServer Resource Pool

• XenServer Resource Pool Management (switching pool master, Dom0 tweaks)

• Upgrading from XenServer 6.0.2 to 6.1

• Backing up and Restoring Metadata

XenServer StorageDeep Dive


Agenda

• Introduction to XenServer Storage• SRs – Deep Dive• Xen Storage – Command line• Storage XenMotion Under The Hood• Debugging Storage Issues• Key Takeaways and References


Before we get started…

• Lets familiarize ourselves with the following terms:ᵒ HBAᵒ LUNᵒ Fabricᵒ Storage Processorᵒ SRᵒ VDIᵒ PBD and VBDᵒ Multipathingᵒ Fast Clone, Linked Cloneᵒ XenMotion and Storage XenMotion

XenServer StorageIntroduction to Storage


Disk Types in common use

• SATA (Serial ATA) disksᵒ Size up to 3 TB (4TB + due for release)ᵒ Spin speed 7200 RPMᵒ Transfer rate ~ 30-40 MB per secondᵒ IOPS 70-80ᵒ Pros: Inexpensive, large Storage. Cons: Relatively slow, long rebuild time

• SAS (Serial Attached SCSI) Disksᵒ Size up to 600 GB (900+ GB due for release)ᵒ Spin speed 15000 RPMᵒ Transfer rate ~ 80-120 MB per secondᵒ IOPS 160-180ᵒ Pros faster access time, redundancy Cons Higher cost per GB

• SSD (Solid state Disk)ᵒ Size 100GB upwards (enterprise class)ᵒ Extremely fast access speedsᵒ Transfer rate ~ up to full bus speedᵒ IOPS read > 40,000 write > 10,0000ᵒ Pros Extremely fast access times low power Cons Very expensive, Limited life.


Disk Sizing

Disk Size In MB Expected MB Diff Reported Size in windows

100 GB 100,000,000,000 107,374,182,400 7.37% 93.1 GB, 95,367 MB

1 TB 1,000,000,000,000 1,099,511,627,776 9.95% 931 GB, 953,674 MB

Latency TimeThe faster the disk spins, the shorter the wait for the computer. A hard drive spinning at 7,200 rotations per minute has an average latency of 4.2 thousandths of a second. This is approximately one-half the time it takes for the disk to turn one revolution.Latency ConsequencesWhile 4.2 thousandths of a second seems inconsequential, a computer might process many thousands of data requests. The computer's CPU is fast enough to keep up with the workload but the hard disk's rotational latency creates a bottleneck.


RAID (redundant array of independent disks)

• Way of presenting multiple disks so that they appear to the server as a single larger disk.

• Can also provide for redundancy in the event of a single/multiple disk failure(s)

• Has multiple types each with its advantages and disadvantages.


Raid Sets

Pros – cheapestCon – A failure of any disk will fail all of the data on the disk

Pros – a failure of any drive will still provide full access to dataOnly 1 additional write to secure dataCon - Double the amount of disks needed


Pros – Data is striped across multiple disks with parityCan support a single disk failure without data loss

Cons – when a disk fails, performance is reduced because the controller has to calculate the missing disks data.Full performance is not recovered until the failed disk is replaced and rebuilt.Write intensive applications can run slowly due to the overhead of calculating and writing the parity blocks

Pros – Data is striped across multiple disks with double parityCan support two disk failures without data loss

A single disk failure does not affect performance

Cons – Write intensive applications can run slower because of the overhead of calculating and writing two parity blocks

Additional 2 disks per raid set required to handle parity requirements.


Hybrid RAID types• RAID 0+1• RAID 10• RAID 50


IOPS (Input/Output Operations per second)

Measurement Description

Total IOPS Total number of I/O operations per second (when performing a mix of read and write tests)

Random Read IOPS Average number of random read I/O operations per second

Random Write IOPS Average number of random write I/O operations per second

Sequential Read IOPS Average number of sequential read I/O operations per second

Sequential Write IOPS Average number of sequential write I/O operations per second

IOPS are primarily a benchmarking figure that is used to find/compare the performance of storage solutions.

IOPS are affected by many factors – such as RAID type, stripe size, Read/Write ratios,Random/sequential balance, cache sizes on controller and disk.

Quoted maximum IOPS from manufactures seldom translate to real world applications.


What is a SAN?

• SAN (Storage Area Network)

• Dedicated network that provides access to consolidated, block level data storage.

• Can be a single device or Multiple Devices

• Does not provide any file level functionality

• However file systems that make use of storage on a SAN will allow for file locking sharing etc.

• Provide access to a pool of shared storage using Either ISCSI, Fibre Channel, FCOIP (fibre channel over IP) for connectivity.

• Is (ideally) on a separate network from other traffic.



NAS (Network Attached Storage)

A NAS differs from a SAN in that it offers remote Access to a filesystem as if it were a fileserver whereas a SAN offers remote access to Block level storage as if it was a hard disk controller.

NAS normally allows connection to its storage via NFS (Unix/Linux), SMB/CIFS (Windows), AFP (Apple)

Some NAS can be used for other protocols such as FTP, HTTP, HTTPS,NDMP, and various media streaming protocols.

A NAS Head (or Filer) can have single or multiple nodes A NAS essential replaces one or more fileservers


What are SAN/NAS used for

• Both provide access to SHARED storage.

• Both can provide high availability/fault tolerance by having multiple Disk Controllers / NAS Heads

• A SAN is used to provide BLOCK level storage to multiple Hosts.

• Connection to a SAN is Normally through Fibre channel or ISCSI (1GB/10GB) over Ethernet.

• A NAS is used to replace FILE servers by providing CIFS / NFS/FTP etc. features.

• Connection to a NAS is normally via Ethernet.

XenServer StorageSRs, SR types and more…


XenServer Storage Objects

• Describes a particular storage target in which Virtual Disk Images (VDIs) are stored.

• Flexible—supports a wide variety of storage types.

• Centralized—easier to manage, more reliable with a XenServer pool.

• Must be accessible to each XenServer host.


Types of SRs supported in XenServer

• Block based – LVMᵒ Fiber Channel (FC)ᵒ iSCSIᵒ Local LVM

• File basedᵒ Ext (local)ᵒ NFSᵒ CIFS (ISOs only)

© 2012 Citrix | Confidential – Do Not Distribute102

Protocols

• NFS – inexpensive, easiest to implement, 1 or 10 Gb, overhead*

• iSCSI – fairly cheap and easy, HW and SW flavors, 1 or 10 Gb

• FCoE – pretty expensive and difficult, requires CNAs, 10 Gb only

• FC – most expensive and difficult to implement, 1/2/4/8/16** Gb• FC is most common in the enterprise, but 10 Gb+ networking is making

NFS and iSCSI more and more common

• In terms of performance, typically you will find that 8 Gb FC will outperform 10 Gb iSCSI or NFS

• Ask the customer if they have “storage tiers”ᵒ Ex: 1. EMC 8 Gb FC 2. HP 10 Gb iSCSI 3. NetApp 1 Gb NFS


Comparisons to other Hypervisors

Facet XenServer VMware vSphere Microsoft Hyper-V

VM Storage LVM, File-level File-level CSV, SMB 3.0 (CIFS)

Storage plugins Storagelink (integrated)

PSP, NMP, Vendor supported

N/A

Multipathing Yes Yes Yes

NFS Support Yes Yes No


xapi

kernel

dom0

Xen Hypervisor

Linuxstoragedevices

LinuxNetworkdevices

Storage Network Domainsdb

SM i/fscripts:

xenstore

Blktap/blkbackNetback

qemu-dm

vm consoles

XML/RPC; HTTP

XenAPIstreamingservices

Hardware devices

VM1

Shared memory pages

VM1

Shared memory pages

XenServer – under the hood


XenServer - under the hood

• I/O (Network and Storage) depends significantly on Dom0ᵒ This changes in the future releases of Xenserver (disaggregation of Dom0)

• All VMs use para-virtualized drivers (installed with Xen tools) to communicateᵒ Blkfront, netfront components in VMs send I/O traffic to Dom0’s Blkback, netbackᵒ Hence the SLOW performance and scalability limits with XenServer


The competition

• ESXi - file based architecture

• VMFS - clustered filesystem

• VMFS 5 - can be shared between 32 hosts

• Custom plugins can be installed based on shared filesystem



• XenServer Storage Objectsᵒ SRs, VDIs, PBDs and VBDs

• Virtual Disk Data Formatsᵒ File-based VHD, LVM and StorageLink


XenServer Storage ObjectsVDIs, PBDs, VBDs

• Virtual Disk Images are a storage abstraction that is presented to a VM.

• Physical Block Devices represent the interface between a physical server and an attached SR.

• Virtual Block Devices are connector objects that allow mappings between VDIs and VMs.


PBD

PBD

PBD

SR

VDI

VDI

VDI VBD

VBD

VBDXenServer Host

XenServer Host

XenServer Host

Virtual Machine

Virtual Machine



Virtual Disk Data FormatsLogical Volume (LVM)-based VHDs

• The default XenServer block device-based storage inserts a Logical Volume manager on a disk. VDIs are represented as volumes within the Volume manager.

• Introduced LVHD in XenServer 5.5ᵒ Enhances LVM for SRsᵒ Hosts VHD files directly on LVM volumesᵒ Adds Advanced Storage features like Fast Cloning and Snapshotsᵒ Fast and simple upgradeᵒ Backwards compatible


Virtual Disk Data FormatsFile-based VHD

• VM images are stored as thin-provisioned VHD format files on either a local non-shared file system (EXT type SR) or a shared NFS target (NFS type SR).

• What is VHD?ᵒ A Virtual Hard Disk (VHD) is a file formatted to be structurally identical to a physical

Hard Disk Drive.ᵒ Image Format Specification was created by Microsoft in June, 2005.


Virtual Disk Data FormatsStorageLink (LUN per VDI)

• LUNs are directly mapped to VMs as VDIs by SR types that provide an array-specific plug-in (NetApp, Equallogic or StorageLink type SRs). The array storage abstraction therefore matches the VDI storage abstraction for environments that manage storage provisioning at an array level.


Virtual Disk Data FormatsStorageLink Architecture

ᵒ XenServer calls direct to Array API‘s to provision and adjust storage on demand.

ᵒ Fully leverages array hardware capabilities.ᵒ Virtual disk drives are individual LUNs.ᵒ High performance storage model.ᵒ Only the server running a VM connects to the

individual LUN(s) for that VM.


LVM vs. StorageLink

Storage Repository

LUN

LVM Volume Group

LVMLogicalVolume

VHD header

VHD header

XenServer 6.1 iSCSI / FC

LUN

Storage Repository

XenServer 6.1 iSCSI / FC + integrated

StorageLink

LUN

VM Virtual Disk

LVMLogicalVolume

XenServer StorageCommand line vs. XenCenter


Xen Storage – command line vs. XenCenter

• Can you trust XenCenter info?ᵒ The answer is – NOᵒ XenCenter relies on the database integrity of XenServer resource poolᵒ Each entity of XenServer (Resource pool data, CPU, NIC and Storage) are stored in

XenServer’s built-in databaseᵒ From a Storage perspective:

• PBD• SR• VDI• VBD

ᵒ All the above entities are stored in each XenServer’s databaseᵒ And now…why should I remember this?

• If a VM fails to boot – the above information will come in handy during troubleshooting• If SR disappears from XenServer (oh yes it happens!), you will need these commands to

recreate the storage path


Xen Storage – Command line

• All Xen commands start with xe <Object>-<Action> <parameters>ᵒ For example:# xe vbd-list vm-name-label=WIN7The above command will list all the disks attached to the VM1. Object can be SR, VBD, VDI, PBD2. Action(s) can be create, delete, list, probe, scan etc.3. Parameters are specific to the command

• Entire list of Xen Commands can be found here

http://www.pablolegorreta.com/wp-content/uploads/2012/01/XenServer-6.0-XE-Command-Reference-1.1.pdf

XenServer StorageStorage XenMotion


Storage XenMotion – under the hood

• New with XenServer 6.1

• Supported to migrate VMs across multiple resource pools

• Also supports Storage migration between local storage and shared storage

• Please keep in mind – its version 1.0


Storage XenMotion and VDI Migration

Storage XenMotion Live VDI Migration


Feature Overview: Use cases

1. Upgrade a storage array

2. Upgrade a pool with VMs on local storage

3. Provide tiered storage arrays

4. Rebalance VMs between XenServer pools, or CloudStack clusters

“The Cloud” was the major use case we had in mind when designing this.


Feature Overview: Bird’s-eye view

Cross-pool migration and VDI migration consist of the following:

1. Synchronously mirror VDIs between source and destination

2. Create new VM object on destination pool (new ref, same uuid)

3. When copy complete, migrate VM as usual

Note: VDI migrate implemented with “localhost” cross-pool migrate!


Feature Overview: Limitations

• No more than 6 VDIs, and no more than 1 VM snapshot

• No more than 3 concurrent ops per host

• No VMs with PCI pass-through enabled

• The following are untested and unsupported:ᵒ HA and WLB on source or destination poolᵒ DVSC integrationᵒ VDI-migrate between multiple local SRs


Feature Overview: Caveats

• Minimum network or storage throughput requirements? This is currently unknown, but we’ll begin investigations soon.

• Can’t check whether destination SR has space available for incoming VDIs – if you fill up an SR, your migration will fail to complete.

• Extra temporary storage is required on the source (and destination) SR, so you must be careful when migrating VMs off of a full SR.

• IO performance inside guest will be reduced during migration because of synchronous mirroring of VDIs.


Feature Overview: API walkthrough

Host.migrate_receivehost:refnetwork:refoptions:map

Result = receive_token

VM.migrate_sendvm:refreceive_tokenlive:boolvdi_sr:mapvif_network:mapoptions:map

Result = None

VM.assert_can_migratevm:refdest:maplive:bool

vdi_sr:mapvif_network:mapoptions:map

Result = None

VDI.pool_migratevdi:refsr:refoptions:map

Result = vdi:ref


Feature Overview: CLI walkthrough

• xe vm-migrateᵒ New params: remote-address, remote-username, remote-password, remote-network, vif, vdi

• Extends the original vm-migrate command• Bold params are required to enable cross-pool migration• vif and vdi map VIFs to target networks and VDIs to target SRs• remote-network specifies the network used for data transfer• Can use host/host-uuid to specify host on pool to send VM

• xe vdi-pool-migrateᵒ Params: uuid, sr-uuid

• uuid of target VDI• sr-uuid of destination SR


Feature Overview: GUI walkthrough


Feature Overview: GUI walkthrough


Architecture: VDI operations

• For each VDI:ᵒ Snapshot VDI and synchronously mirror all subsequent writes to destination SRᵒ Copy the snapshot to destination SRᵒ Finally, compose those writes onto the snapshot on the destination SR

• Continue to mirror all new writes

• Each of these operations occurs sequentially for each VDI, but each VDI mirror continues until the VM migration is complete

• VM memory is copied only after final VDI compose is complete

VDI 1: snapshot & start mirroring

VDI 1: copy snapshots

VDI 2: snapshot & start mirroringVDI 2: copy snapshots

Copy VM memory


VDI mirroring in picturesSOURCE DESTINATION

mirror

copy

root

VM VM

no color = emptygradient = live


VMware Storage VMotion In Action

132

Source Destination

“Fast suspend/resume” VM to start running on new home and disks

4

Copy all remaining disk blocks

4

Copy VM home to new location

1

Delete original VM home and disks

5

Start changed block tracking

2

Pre-copy disk to destination(multiple iterations)

3

XenServer StorageDebugging Storage Issues


Troubleshooting tips

• Check logs: /var/log/{xensource.log,SMlog,messages}ᵒ Note: xensource.log and messages both implemented with syslog, so they now have

consistent timestamps!

• xn command: CLI to xenopsdᵒ Try ‘xn help’ for documentation.

• tap-ctl commandᵒ Could be useful for diagnosing problems (can’t describe usage here)


Logging/Debugging

• All backend drivers use the Smlog to record storage eventsᵒ /var/log/Smlog

• Logs are rotated, same as system message log, xensource.log files

• In the python module util.py there is a helper log function -> util.SMlog(STRING)

• On retail edition, all python backend files are editable for logging/debugging purposes

• Some things to look for in SMLog (tail –f /var/log/SMLog):ᵒ Slow SR Probing: Check for breaks in the SMLog timestamps or for

SR_BACKEND_FAILURE_46, this is likely an issue with multipathingᵒ Storage Issue: Look for the word FAILURE within the log, this is likely an issue

communicating with the storage or a corrupted SRᵒ Metadata Corruption: Look for SR_BACKEND_FAILURE_18, this is likely an issue

with metadata being corrupted due to special characters being used in VM namesᵒ VDI Unavailable: This is typically caused by faulty clones/snapshots but could also

point to misbehaving Storage.


Debugging tips

• Use tail –f on any log file while actively debugging a system

• Correlate logs between xensource.log, SMLog and messages

• Verify IP settings, firewall config for any IP based storage connections

• Check status of dependent objects, such as PIFs, VIFs, PBDs etc...

XenServer StorageLab 2 – Storage Lab



• Create a local SR

• Create a shared SR with Integrated Storagelink

• Storage XenMotion Lab

• (Advanced Lab) Create shared SR using NetApp SIM

XenServer NetworkingFoundational Concepts


Agenda

• Networking Overview• XenServer Networking• XenServer 6.1 – What’s New?

ᵒ Link Aggregation (LACP)

• Debugging Networking Issues• Key Takeaways and References


Before we get started…

• Lets familiarize ourselves with the following terms:Common Enterprise Networking terminologies:ᵒ NICᵒ Switch – Coreᵒ Router, Hubᵒ Firewallᵒ DHCP, PXE, TFTP, TSBXenServer:ᵒ LACPᵒ VLANᵒ VIFᵒ PIF


Internet

Server Farm

Backbone Switch

Access Switch

Example Enterprise Switching Environment

Distribution Switch

Firewall


Example WAN Environment


Network Vendors

Vendor Components

Cisco Switches, ACE (load balancing), ASA (Firewall), Routers

Juniper Network Security Solutions

Brocade Switches, Routers

Citrix NetScalerF5River Bed

Access Gateway, WAN Accelerators,Load Balancing

XenServer Advanced TrainingIntroduction to Enterprise Networking


XenServer Networking Conceptual Model

Xen Hypervisor

Control Domain(Dom 0)

Hardware

Virtual Machine

Network Card

Linux Driver

PIFvNIC

BridgeVIF

Netback Netfront


XenServer Networking Configurations

Linux NIC Drivers

Linux Config Files

XenServer PoolDB

Network Card

XAPI

Command Line

XenCenter


Virtual Switches

PIF (eth0)

VIF

VIF

VIF

Virtual Machine

Virtual Machine

XenServer Network Terminology

Network 0 (xenbr0)

Private(xapi1)

Network Card


Virtual Switches

PIF (eth1)

PIF (eth0)

VIF

VIF

VIF

Virtual Machine

Virtual Machine


Network 1 (xenbr1)

Network 0 (xenbr0)

Network Card

Network Card


PIF (bond0)

PIF

VIF

VIF

Virtual Machine

Virtual Machine


Network Card

Network Card

VIF

Bond 0+1 (xapi2)

PIF (eth0)

PIF (eth1)


VLAN 55

Virtual Switches

PIF

PIF

VIF

VIF

VIF

Virtual Machine

Virtual Machine


Network 1

Network 0

VLAN 25

Network Card

Network Card


PIFs

VIF

VIF

VIF

Virtual Machine

Virtual Machine


VLAN 25

Network Card

Network Card

VLAN 55

Bond0+1


Bonding Type (Balance SLB)

Virtual Machine

Network Card

Network Card

Virtual Machine

Bond

0:00 SEC0:10 SEC0:20 SEC0:30 SEC

Stacked Switches

Virtual Machine


What about faster iSCSI/NFS?

Virtual Machine

Network Card

Network Card

Virtual Machine

Dom0 with iSCSI or NFS software

Bond

0:00 SEC0:10 SEC0:20 SEC0:30 SEC

iSCSI/NFS

iSCSI/NFS

Advanced XenServer TrainingDistributed vSwitch


Open Virtual Switch for XenServer

VM

Hypervisor

VM

VM

VM

VM

Hypervisor

VM

VM

VM

VM

Hypervisor

Visibility· Resource control · Isolation · Security

VM

VM

• Open Source Virtual Switch maintained at www.openvswitch.org• Rich layer 2 feature set (in contrast to others on the market)• Ships with XenServer 5.6 FP1 as a post-install configuration option

http://www.openvswitch.org/


Hypervisor

Distributed Virtual Switch Controller

HypervisorHypervisorHypervisor

DVS

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

DVS Controller is a XenServer Virtual Appliance that controls multiple Open vSwitches


Distributed Virtual Switch

HypervisorHypervisorHypervisor

Built-in policy-based ACLs move with VMs

DVS

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

Virtual Interface (VIF) {MAC, IP} ACLspermit tcp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 192.168.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 172.16.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 192.168.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 172.16.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq 123

Virtual Interface (VIF) {MAC, IP} ACLspermit tcp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 192.168.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 172.16.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 192.168.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 172.16.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq 123

XenServer NetworkingLACP Support – Finally!!


Overview

• Existing bonding modes and related issues:• active-active (balance-slb)

• ARP table confusion for some switches (due to MAC flapping)• active-passive (active-backup)

• only one link used at a time• LACP (802.3ad)

• not supported• working only on Linux Bridge

• The modern way – LACP bonding mode:• Link aggregation on both server and switch side.• Sides communicate with LACP protocol.• Better load balancing.


NIC 1

VM 1

Virtual interface (VIF)

LACP Bond

NIC 2

App 1

App 2

VM traffic

Virtual interface (VIF)

VM 2

App 1

App 2

StackedSwitches


Architecture

• For given NICs, we bond the PIFs and set mode to LACP.

• Bond object is created on xapi level.

• Command bond-create executes vSwitch commands.

• LACP is configured on the switch ports.

• Switch and server exchange LACP frames.

• Switch chooses the active members.

• Switch and server balance their outgoing traffic independently.


Load balancing

• Hash-based traffic balancing:ᵒ vSwitch computes flow number (0-255) for each packetᵒ Each flow (hash) is assigned to an active linkᵒ Flows can be moved between links (re-balancing every 10s)

• Two hashing algorithms:ᵒ tcpudp_ports: based on IP and port of source and destination

• default algorithm for LACP• source MAC also taken into account• VM traffic from different applications can use more than one link

ᵒ src_mac: based on source MAC• vSwitch uses the same mechanism as for active-active• traffic from one VIF will not be split


Configuration

• Two new bond modes in XenCenter wizard:ᵒ LACP with load balancing based on IP and port of source and destinationᵒ LACP with load balancing based on source MAC address

• CLI commands:xe bond-create mode=lacp network-uuid=… pif-uuids=… \ properties:hashing-algorithm=src_mac

xe bond-param-set uuid=… \ properties:hashing_algorithm=tcpudp_ports

Create LACP bond

specify hashing algorithm


Switch configuration

• Switch must support 802.3ad (LACP)

• Choose unused Link Aggregation Group (LAG)

• For each NIC being bonded:ᵒ Find the switch port/unit connected to the NICᵒ Set LAG membership for the portᵒ Enable LACP for the port

• If necessary, bring up the LAG interface

• If required, configure the VLAN settings on the LAG


Side note: Static LAG

• Static LAG:ᵒ Ports are members of a Link Aggregation Group.ᵒ Ports do not have LACP set.ᵒ All links within the LAG are active.

• Static LAG is not LACP.ᵒ Use balance-slb, not LACP bond mode for static LAG.


Limitations and changes

• LACP for Linux Bridge not supported.If used anyway, note that it was renamed from 802.3ad to lacp.

• EtherChannel/Cisco Port Aggregation Protocol (PAgP) not supported.

• Switches must be stacked.

• Up to four NICs per bond for all bond modes.

• Only 2 NICs per bond supported for Linux Bridge.

• Improvements for bonding planned in near future:ᵒ Switching off dynamic load balancing (preventing MAC flapping)ᵒ Choice of active link for active-passive bonds


Troubleshooting

• Frequent surprises:ᵒ Wiring misinformation – wrong ports might be aggregatedᵒ Switch rules – and it can decide to use just one link as activeᵒ No HCL – 802.3ad is a standardᵒ Longer set-up – more time might be required for LACP and DHCP ᵒ One ‘fat’ flow still will not be splitᵒ In ovs commands, ‘bond mode’ means ‘hashing algorithm’

• Mismatching settingsᵒ LACP only on server side – we should have active-active modeᵒ LACP only on switch side – you are on the mercy of the switch


Debugging

• CLI command: xe bond-param-list

• vSwitch commands: ᵒ ovs-vsctl list port ᵒ ovs-appctl bond/show bond0 (lacp_negotiated: false/true)ᵒ ovs-appctl lacp/show bond0 (actor and partner information, hashes)

• xensource.log – network daemon entries, including DHCP

• /var/log/messages – vSwitch entries, including hash shifting

XenServer NetworkingSR IOV and more


SR-IOV

• SR-IOV – VM to talk directly to a NIC rather than having to pass network traffic through Dom0

• SR-IOV virtualizes the network in hardware, rather than in software

• benefits and disadvantages:ᵒ good: there has generally a significant speed improvement in per-VM bandwidth (and

usually in latency) when using SR-IOV instead of Dom0-mediated networkingᵒ good: partitioningᵒ good: avoiding the switchᵒ bad: avoiding the switchᵒ bad: migration issues


SR-IOV Pre-requisites

• SR-IOV capable network device (i.e. Intel 82599 10 Gb-E Controller)

• The SR-IOV NIC cannot be used as the management. ᵒ A second physical NIC must be installed on the system

• iommu must be enabled on the XS hostᵒ /opt/xensource/libexec/xen-cmdline --set-xen iommu=1


XenServer Networking – without SR-IOV

Xen Hypervisor


Hardware

Virtual Machine

Network Card

Linux Driver

PIFvNIC

BridgeVIF

Netback Netfront


XenServer Networking – with SR-IOV

Xen Hypervisor


Hardware

Virtual Machine

Network Card

Linux Driver

PIFvNIC

BridgeVIF

Netback VF Driver

SR-IOV Aware Network Card

© 2012 Citrix | Confidential – Do Not Distribute175

Using Multi-Queue NICs

Driver Domain

Guest Domain

Backend Driver

Frontend Driver

Xen

Physical Driver

HardwareMQ NIC

I/O Channels

Incoming Pkt

IRQ

event6 8

Post grant on I/O channel

1

Map bufferpost buf on dev

queue

DMA5

UnMap buffer7

9Push into the network stack

gr3

2

• Advantage of multi-queue• Avoid data copy• Avoid software

bridgeOne RX queue

per guest

guestMAC addr

demux4


VLAN Overview

• A VLAN allows a network administrator to create groups of logically networked devices that act as if they are on their own independent network, even if they share a common infrastructure with other VLANs.

• Using VLANs, you can logically segment switched networks based on functions, departments, or project teams.

• You can also use a VLAN to geographically structure your network to support the growing reliance of companies on home-based workers.

• These VLANs allow the network administrator to implement access and security policies to particular groups of users.


VLAN Overview


VLAN in details

• A VLAN is a logically separate IP subnetwork.

• VLANs allow multiple IP networks and subnets to exist on the same switched network.

• For computers to communicate on the same VLAN, each must have an IP address and a subnet mask that is consistent for that VLAN.

• The switch has to be configured with the VLAN and each port in the VLAN must be assigned to the VLAN.


VLAN in details

• A switch port with a singular VLAN configured on it is called an access port.

• Remember, just because two computers are physically connected to the same switch does not mean that they can communicate.

• Devices on two separate networks and subnets must communicate via

a router (Layer 3), whether or not VLANs are used.


Dom0 – Limitations with VLAN

• Dom0 doesn’t support VLANsᵒ Management network needs to be on its on NICᵒ XenMotion cannot be separate from management NICᵒ iSCSI, NFS can be on VLANsᵒ Dom0 supports NIC bonding

XenServer NetworkingLab 3 – Networking Lab



• Create a LACP bond

• Create a distributed virtual switch

• Explore VIFs, PIFs and eth# interfaces


Resource Links

• XS Administrator’s Guide – revised section on bonding

• The standard:

http://standards.ieee.org/getieee802/download/802.1AX-2008.pdf

• Internal wiki page: http://confluence.uk.xensource.com/display/engp/Bonding+and+LACP

• Feature lead: Kasia Borońska ([email protected])

http://standards.ieee.org/getieee802/download/802.1AX-2008.pdf

http://confluence.uk.xensource.com/display/engp/Bonding+and+LACP

mailto:[email protected]

XenServer HealthMonitoring & Troubleshooting


Agenda

• Managing & Monitoring this wonderful piece of software• It’s broken, How do I fix it?• Show me Your Commands!

XenServer Monitoring & TroubleshootingManaging & Monitoring this wonderful piece of Software


Using Console Folder View

• Manage XenServer environment in your own way

• Selection styles:• SRs (Remote and Local)• Snapshots• Networks• VMs• Pools• Servers

• Custom Folders • Simple drag & drop functionality


Delegated Management (Web Self Service)

• Delegate VM level access to end-users

• View consolidated virtual machine guests from multiple resource pools

• Basic life cycle operations such as Start, Stop, Suspend and Reset on virtual machine

• Remote login (VNC for Linux Guests and RDP for Windows Guests) to the virtual machine guests

• Force Shutdown & Force Reboot (Only with XS Tools)

• Fully AD Integrated


Web Self Service Roles

XenServer Roles WSS Roles

Pool Admin WSS Admin

Pool Operator WSS Operator

VM AdminVM OperatorVM Power AdminRead Only

WSS User

No Role (Default) WSS User


Web Self Service OdditiesSharing Through Tags? What?


Web Self Service Oddities

• Authentication Can’t be changed – Better Make up your mind

• Integrated Auth is pretty annoying

• Not Really AD, XS AD!

• Process is still manual for AD?ᵒ Must Add every User unless auto-login is enabled


Legacy Performance Monitoring

• In Citrix XenServer 5.6+ you are no longer able to collect Host and VM performance data via the “legacy” API calls. Instead they direct you to use the web services via URL


RRDs (Round Robin Databases)

• RRDs are the OpenSource industry standard, high performance data logging and graphing system for time series data.

• XenServer’s Performance monitoring file is actually an RRDTool File, go figure.

• Download format is: http://<username>:<password>@<host>/rrd_updates?start=<secondssinceJan1,1970>ᵒ Use &host=true suffix to get RRD for a XS Host

• XML downloaded contains every VM's data. It is not possible to query a particular VM on its own or a particular parameter. ᵒ To differentiate, the 'legend' field is prefixed with the VM's uuid.

• Free RRDTool available at: http://oss.oetiker.ch/rrdtool/


TAAS (Previously XenoScope)

• XenServer Metrics:ᵒ Uses RRD Metricsᵒ Can be drilled down by domain, time frame, graph size. ᵒ Host memory usage Piechartᵒ Host Hardwareᵒ Pool hosts Overviewᵒ Pool Hardware Comparisonᵒ Installed Hotfixes Comparisonᵒ Running Virtual Machines Overviewᵒ CPU Flags Overview and Comparisonᵒ Recommendationsᵒ Server Log review tools


TAAS (Previously XenoScope)A Very Clean Interface (Also for XA/XD/NS)


TAAS (Previously XenoScope)A Very Clean Interface (Also for XA/XD/NS)


DoubleRev’s Management Pack (http://www.doublerev.com/ )

• Pros:ᵒ Automatic and agent less mass update of XenToolsᵒ DVD Manager ᵒ Storage Analyzer ᵒ Mass VM-Copyᵒ Re-convert templates to virtual machinesᵒ VM-Inventory Exportᵒ Seamless XS Console Integration

http://www.doublerev.com/



DoubleRev’s Management Pack (http://www.doublerev.com/ )

• Cons:ᵒ CLI Commandsᵒ Limited Monitoringᵒ Most of these features probably will come in Future XS Consoleᵒ Costs $$$

• Verdict:ᵒ Should only be considered if you are extremely lazy as most of these features are

nothing terribly difficult to accomplish



XenServer Monitoring & TroubleshootingIt’s Broken, How do I fix it?


Methodology For Recovery Overview


Step 1 - Is the Pool Master down?

• Then the Master server cannot be contacted and is likely non-functional.ᵒ If connection to the XenServer pool is not possible from XenCenterᵒ If issuing commands (xe host-list) at CLI of a surviving host returns “Cannot perform

operation as the host is running in emergency mode”

• Under some circumstances (such as power outages) there may have been multiple XenServer host failures in the pool. In this case, the pool master should be attempted to be recovered first.


Who’s the Pool Master Anyways?

• Determined through pool.conf located under /etc/xensourceᵒ Master for PoolMasterᵒ slave:<pool master IP> for all others

• Important Commands when in Emergency Mode:ᵒ xe pool-emergency-reset-masterᵒ xe pool-emergency-transition-to-master ᵒ Follow up with: xe pool-recover-slavesᵒ Check that default-SR is valid: xe pool-param-list


Step 2 - Recover Pool operations

If recovery of the pool master is not possible within a short timeframe, it will be necessary to promote a member server to a master to recover pool management and restart any failed Virtual Machines.

1. Select any running XenServer within the pool that will be promoted. 2. From the server’s command line, issue the following command: xe pool-emergency-

transition-to-master3. Once the command has completed, recover connections to the other member

servers using the following command: xe pool-recover-slaves4. Verify that pool management has been restored by issuing a test command at the CLI

(xe host-list)


Step 3 - Verify which XenServer(s) failed

This step verifies which XenServer(s) in the pool have failed

1. Issue the following command at the Command line of a surviving pool member: xe host-list params=uuid,name-label,host-metrics-live

2. Any servers listed as host-metrics-live = false have failed.

If it was necessary to recover the Pool Master with Step 2, the Master will now show as true.

3. Note down the first few characters of the UUID of any failed servers (or the UUID of the Master server if it originally failed).


Step 3 - Verify which XenServer(s) failed

ᵒ Result of xe host-list params=uuid,name-label,host-metrics-live


Step 4 - Verify which Virtual Machines have failed

• Regardless of whether a Master or Slave server fails, virtual machines running on other hosts continue to run.

1. Issue command at the CLI of a surviving server. Use the UUIDs from Step 3, type first 3 digits of the UUID, and press [tab] to complete: xe vm-list is-control-domain=false resident-on=UUID_of_failed_server

2. Note that the power state of these VMs is still “running” even though the server has failed. Step 5 will take care of this problem.

3. Repeat this step if necessary using the UUID of any other failed servers (including master).


Step 4 - Verify which Virtual Machines have failed

• Result of xe vm-list is-control-domain=false resident-on=UUID_of_failed_server


Step 5 - Reset power state on failed VMs

• To restart VMs after a host failure, it is necessary to reset their power state.

1. Issue the following command at the command line of a surviving server: xe vm-reset-powerstate resident-on= UUID_of_failed_server --force –multiple

ᵒ Alternately, you can reset VMs individually.

2. Verify that there are no VMs still listed as resident on the failed server by repeating step 4. The vm-list command should now return no results.


Step 6 - Restart VMs on another XenServer

1. Load XenCenter and verify that each VM that was originally running on the failed server is now marked as halted (Red icon next to the VM)

Note: VMs which have a home server assigned will not appear in XenCenter – Why?

ᵒ xe vm-param-set uuid=<uuid of vm to change> affinity=<uuid of new home server>

2. Restart each VM on a surviving pool member


Collecting Crash Dumps (Regular Style)

1. Configure System for Memory Dump: http://support.microsoft.com/kb/927069

2. Find the Universally Unique Identifier (UUID) of the VM from which you would like to collect memory dump using: xe vm-list

3. Based on the UUID of the virtual machine, find a domain ID in the current system. The Domain ID changes every time the VM starts: list_domains

4. Use the Domain ID (1st number listed) to crash the host using: /usr/lib/xen/bin/crash_guest <VM Domain>

http://support.microsoft.com/kb/927069

http://support.microsoft.com/kb/927069


Collecting Crash Dumps (Ninja Style)

• Crashdump redirection to dom0 filespaceᵒ When streaming PVS guests, where you cannot save the crashdump to the Pagefile

after a crash due to the network being disabled automaticallyᵒ When the root drive doesn't have enough space to store a Pagefile as large as the

guest's memory size

• Consider mounting either an iscsi lun or nfs, cifs filesystem under /crashdumps to store large dumps since root partition typically has <1.5GB available space



1. Open /opt/xensource/libexec/qemu-dm-wrapper with vi

2. Search for "SDL_DISABLE_WAITMAPPED“ and modify by adding the following at the line immediately following it:

ᵒ qemu_args.append("-priv")ᵒ qemu_args.append("-dumpdir")ᵒ qemu_args.append("/crashdumps")ᵒ qemu_args.append("-dumpquota")ᵒ qemu_args.append("1024")

3. Create the directory where the dumps will be stored: mkdir -p /crashdumps



4. Reboot the VM

5. use the crash_guest util to force the crashdump. If the condition leads to the BSOD, just wait for the streaming to finish and check the file at /crashdumps. Example:

Find dom ID: xe vm-list name-label="Windows Server 2003" params=dom-id

Result: dom-id ( RO) : 3To crash the domain id 3: /usr/lib/xen/bin/crash_guest 3

6. Copy the dumps outside the VM with a tool such as winscp.


VM Needs Imaginary Storage

• While working on-site with a client, XenServer decided to subscribe to the cult of Jacob Marley and refused to turn on VM saying ‘This VM needs storage that cannot be seen from that host’


VM Needs Imaginary Storage

• Caused by CIFS/DVD drives on XenServer and the machine where the ISOs/DVD resided failed.

1. xe vm-cd-eject –multiple (Gets rid of Faulty CD/DVD media)


File System on Control Domain Full



• Result of the Dom0 Storage allocation becoming Full

• Typically caused by overgrown Log Filesᵒ XenServer logrotate feature can break - it cannot parse the size parameter.

1. Check Disk Space to validate issue: df –h

2. Check what’s causing the problem: find {/path/to/directory/} -type f -size +{size-in-kb}k -exec ls -lh {} \; | awk ‘{ print $9 “: ” $5 }’

ᵒ Specify the path and the size in kb

3. In order to determine root cause lower the log rotations and enable compression (/etc/logrotate.conf)



4. In Case of Total Loss (XAPI died): xe-toolstack-restart

5. Typically this issue is due to TCPOffload freaking out causing unnecessary entries in logfiles.

6. Run Script to disable TCP/Checksum Offloadᵒ Disabling TCP Offload and checksum means that the main CPU handles all the

load which was previously handled directly by the network card


XenServer Unable To See NICs

• The XenServers are experiencing problems detecting their network cards. The network cards are registered in the XenServer OS, however, they do not display in the XenServer console.


XenServer Unable To See NICs

• Issue is caused by disconnecting the pool master while it has attached slave servers.

1. Log into the local XenServer, opening the file pool.conf located under /etc/xensource, deleting the contents of this file and replacing them with the word master.

2. Set the management console’s NIC to a valid IP/subnet

3. Run xe-toolstack-restart

4. Edit the pool.conf again to read slave:<pool master IP>


Frozen VM

• My VM(s) Froze and now XenCenter refuses to respond because it simply does not care about how troubled my life already is


Frozen VM

• This is typically caused by conflicting commands being sent through the Xen Center console resulting in erroneous metadata

• To force reboot VM (Like a Boss)

1. Find out the target VM UUID, type the command line : xe vm-list (Hint: Use | more when there are too many VMs)

2. Find your VM(s) UUID

3. Run the command: xe vm-reset-powerstate uuid=<Machine UUID> force=true


XenServer Installation Fails

• I started installing XenServer on a bunch of machines, went out to a super relaxing lunch only to come back to my XenServer installation stuck at 49%


XenServer Installation Fails

• Some versions of XenServer do not include the firmware for the Qlogic QLA2100, QLA2200, QLA2300 and QLA2322 series NICs

1. Turn on the host, using the XenServer installation media

2. At the boot prompt, type: shell

3. You will now be presented with a command prompt. Type the following: rmmod qla2xxx; rmmod qlge

4. Type: Exit

5. Install XS normally


Commands CheatSheet

• XenServer Host and Dom0: xentop

• Hostᵒ xe-toolstack-restart to restart XAPIᵒ eject to eject physical CD from server.ᵒ cat /etc/xensource-inventory to see your host information.

• Networkᵒ ifconfig to see what interfaces are up for networking.ᵒ ethtool –p eth0 60 for make NIC flash for identification.ᵒ ethtool eth0 to check the status of the interface.ᵒ ethtool –i eth0 to check the driver type and version of NIC.


Commands CheatSheet

• Diskᵒ fdisk –l to view local disk information.ᵒ df –h to see how much space you have left in root disk.

• Multipathᵒ multipath -ll to view the current mulitpath topology as presented by control domain.

• VMᵒ xe vm-reboot vm=<VM Name> force=true to hard-reboot a VM.

• Logsᵒ xen-bugtool –yestoall to get the logs for support.ᵒ tail –f /var/log/messages to view events in messages log.

XenServer Monitoring & TroubleshootingLab 4 -Show Me Your Commands!



• Install Performance monitoring pack (new with XenServer 6.1)

• Configure Performance VM and run sample workloads (Discuss results)

• TAAS.CITRIX.COM – Upload server status reports

• (Advanced) – Break/Fix lab

Advanced XenServer TrainingQuestions, Yes?

Work better. Live better.

Date post:	26-Dec-2015
Category:	Documents
Upload:	alvin-sutton
View:	242 times
Download:	12 times

Advanced XenServer Training CCAT Q1 - 2013. © 2012 Citrix | Confidential – Do Not Distribute...

Documents