Date post: | 26-Mar-2015 |
Category: |
Documents |
Upload: | adam-combs |
View: | 218 times |
Download: | 0 times |
HP Technology Forum & ExpoGet connected. People. Technology. Solutions.
Security Agility Reduced Security Agility Reduced CostCost
Best Practices using Xen Virtualization with Red Hat
Enterprise Linux 5
Jan Mark [email protected] Engineer Emerging Technology GroupCTO Office
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability3
Outline • Best Practices
• Performance Expectations
• Key directories and files
• Some common examples
• Q&A
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability4
Basic Information before you start
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability5
Before you start• Read the documentation and QuickStart guides
− RHEL5 Virt Guide• http://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/
Virtualization-en-US/index.html• http://www.redhat.com/docs/
− Fedora Core 6 Xen QuickStart• http://fedoraproject.org/wiki/FedoraXenQuickstartFC6
− Kbase articles• http://kbase.redhat.com• http://kbase.redhat.com/faq/topten_108_0.shtm for virtualization specific
topics
− Virtualization “brain dump”• http://et.redhat.com/~jmh/docs/
− Look for Installing_RHEL5_Virt.pdf/odt− Red Hat Summit Slides
• http://et.redhat.com/~jmh/RH_Summit_2007/
− XenWiki• http://wiki.xensource.com/xenwiki/
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability6
RHEL5/Xen Architecture
Hardware
Hypervisor
dom0
Device Driver
Back End
domU/VM1
Front end
domU/VM2
Front end
Virtual CPU & Memory
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability7
Guest/Hypervisor Matrixwith PAE
32bit PAE paravirt Guest
32bit HVM Guest
64bit paravirt Guest
64bit HVM Guest
32bit (PAE) Hypervisor / dom0
64bit Hypervisor / dom0
For Paravirtual, the guest has to be equal to dom0 For HVM, the guest has to be equal or less than dom0 RHEL5/Xen Hypervisor itself must be EQUAL to dom0
Goal to support 32bit para-virt on 64bit dom0/HV for RHEL 5.1
8HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability
Platform/Guest Release Support in RHEL5 Fully supported by Red Hat
Para-virtualized guest support
RHEL5 at GA (Q1/2007)
RHEL4 U5 / RHEL 4.5
SMP and UP support for all RHEL para-virt guests
Fully-virtualized support (with appropriate hardware features(VTx or AMD-V)
All existing and supported RHEL releases
RHEL3, RHEL4 (SMP and UP support)
“Will/Should” work operating systems (best effort support by Red Hat)
Fully virt support only for any “foreign” operating systems
SUSE SLES9/SLES10
OpenSolaris
Windows
Other x86/x86_64 OS
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability9
Basic Information and Hints
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability10
Prerequisites for RHEL5/Xen Use RHEL5 Xen packages, DO NOT install Xen RPMs from XS !! Apply Day “0” errata updates for RHEL5 and Xen components
Fix for 'dual' cursor mouse in virt-manager
Need Intel-VT/AMD-V capabilities for HVM/Fully-Virt guests Install VNC and VSFTP packages
VNC for remote desktop access VSFTP for access to local/remote para-virt installation tree
At least 2GB of memory for a “working” system Use at least 300MB for guest installations
More memory will speed up the installation process 256MB may potentially cause installation failures
Plan for enough disk space on the host Rough guideline for a guest installation is ~5GB Guest image includes encapsulated swap space
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability11
Installing RHEL5/Xen Considerations
Secure RHEL5 platform layer before installing any virtual machines or applications
Enable SElinux to run in 'enforcing' mode Remove or disable any unwanted services
AutoFS, NFS, FTP, WWW, NIS, telnetd, sendmail etc... Only add minimum number of user accounts needed for platform
management Avoid running applications on dom0/Hypervisor Running applications in dom0 may impact virtual machine
performance Use central location for virtual machine installations
Will make it easier to move to shared storage later on If laptop with wireless adapter it used as virt platform need to perform
special steps to make Xen networking functional Setup dummy network interface and NAT traffic through WiFi
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability12
Planning for a Xen Infrastructure Naming conventions for Virtual Machines, Network Interfaces,
Filesystems, Volume Groups At least choose different name for dom0/host VolGroup00
Consider shared storage to store Virtual Machine images Foundation for live migration Make /var/lib/xen/images a seperate filesystem and as of RHEL5.1
use GFS2 even on a single/local host Make /var/lib/xen/dumps a seperate filesystem to avoid out-of-space
for VM core dumps Private network for Xen traffic
Secure migration traffic and improve performance Central location for installation sources
Network based installation tree for Para-Virt installations ISO file(s) or DVD/CD driver for HVM/Fully-virt installations
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability13
RHEL5/Xen Considerations
Most of the standard Linux/RHEL tools for troubleshooting still work in a virtualized environment
top, iostat, vmstat, lsof, etc... Very few Xen specific tools (mostly xm and virsh commands)
“xm top” for HyperVisor view of performance data XenOprofile for profiling of active and inactive domains
Leverage RHEL5/Xen capabilities to streamline provisioning Golden Images for VMs and applications Fast Provisioning with Kickstart files
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability14
/etc/xen xend-config.sxp
xend configuration file Domain / guest config files
/etc/xen/scripts (customizable) control scripts
Device management -eg. Networking /etc/xen/auto
Softlinks to guests that should start automatically xendomains service starts these guests
Typically disabled by default
Configuration – Key files and directories
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability15
/var/lib/xen dump
Kernel dumps generated with “xm dump-core <Domain>”
images Guest images (covered by SElinux policy)
xend-db Xen internal database
/var/lib/xend Sockets used by xend (xend-socket / relocation-socket)
/var/lib/xenstored xend database (tdb) Can be “read” with xenstore-ls
Configuration – Key files and directories
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability16
/var/log/xen xend.log
Logfile used by xend for logging Primary file to review in case of problems
xend-debug.log Debug output from xend
xend-hotplug.log Logfile for hotplug events Will record information in case of hotplug failure of devices
qemu-dm.{PID}.log Logfile used by the qemu-dm process
All logfiles are human readbale and can be view with the standard Linux utilities (or “xm log” for xend.log)
Logging – Key files and directories
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability17
/proc/cpuinfo shows whether pae, svm, vmx enabled CPU
/proc/xen various information about xen
/sys/hypervisor more information about domain, including UUID
Xen Python scripts are logging to debug facility (not enabled by default)
Add *.* /var/log/debug.log to syslog config file
Logging/Capabilities – Key files and directories
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability18
• Scripts are stored in /etc/xen/scripts
• Configured in /etc/xen/xend-config.sxp
Networking
(network-script network-bridge) (vif-script vif-bridge)
This configuration declares that
• “network-bridge” should handle the network setup
• “vif-bridge” should handle each domU add/remove
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability19
• Multi Network Bridge Solution : − Modify /etc/xen/xend-config.sxp
• Replace call to network-bridge script with call to custom script (ie multi-network-bridge)
• Create custom script in /etc/xen/scripts− Inside the custom script call network-bridge script with addtl
parameters/interfaces
Networking / Multi-bridge config
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability20
Networking / Example for multi-bridge config
#!/bin/sh# network-xen-multi-bridgeset -e# First arg is the operation.OP=$1shiftscript=/etc/xen/scripts/network-bridge.xencase ${OP} in start)
$script start vifnum=0 bridge=xenbr0 netdev=eth1 $script start vifnum=1 bridge=xenbr1 netdev=eth2 $script start vifnum=2 bridge=xenbr2 netdev=eth3 ;; stop) $script stop vifnum=0 bridge=xenbr0 netdev=eth1 $script stop vifnum=1 bridge=xenbr1 netdev=eth2 $script stop vifnum=2 bridge=xenbr2 netdev=eth3 ;; status)
$script status vifnum=0 bridge=xenbr0 netdev=eth1 $script status vifnum=1 bridge=xenbr1 netdev=eth2 $script status vifnum=2 bridge=xenbr2 netdev=eth3 ;; *) echo 'Unknown command: ' ${OP} echo 'Valid commands are: start, stop, status' exit 1esac
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability21
Migration
• Requires shared storage− Both dom0's must access same disk image or
physical device
• Using Physical device− LUN on SAN− Exported block device
• ISCSI, GNBD− Both dom0's should use same name for device
•eg. /dev/sdg•Use UDEV rules for mapping if required
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability22
Migration
• Using File based disk image− Disk image stored on shared file system
•NFS *•Samba•GFS
− Should be mounted on same mount point•Both domain 0's see same directory structure
− eg. /var/lib/xen/images
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability23
Migration• Enable relocation server
/etc/xen/xend-config.sxp(xend-relocation-server yes)(xend-relocation-port 8002)(xend-relocation-address '')(xend-relocation-hosts-allow '')
• Enables relocation server
• Listens on port 8002 (default)
− Use lsof to verify relocation port is active
[root@grumble]$ sudo lsof -i :8002
python 3873 root 5u IPv4 12329 TCP *:teradataordbms (LISTEN)
• Binds to all IP addresses
• Allows any host to migrate to this domain 0
− Authorized hosts can be listed by IP or name
− '^myserver.mydomain.com$ ^.*\.redhat\.com$'
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability24
Basic Xen commands
•Once you have your first guest installed you can use the following commands for some basic management•To startup a guest
# /usr/sbin/xm create -c GuestName −Where GuestName is the name you gave for your guest during the installation−The -c will attach a xen console to your vm
•A variety of other commands are available via xm including# /usr/sbin/xm help or # virsh help
−For a list of commands that can be run−Use '--long' in addition for extended help text−You can also use
#/usr/sbin/xm help –help −for a specific command
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability25
Basic Xen commands contd.
# /usr/sbin/xm list (--long) or # virsh listList running domains/guest and their status/accumulated CPU time
# /usr/sbin/xm topfor a display showing what your virtual machines are doing similar to that provided by top
# /usr/sbin/xm shutdown GuestName or # virsh shutdown GuestName
to nicely shut down a guest OS where foo is the name of your guest.
# /usr/sbin/xm destroy GuestName or # virsh destroy GuestNameTo power down a guest (hard reset)
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability26
Basic Xen commands (Suspend/Resume)
# /usr/sbin/xm save GuestName GuestName.sav or # virsh save GuestName GuestName.sav
to save the state of the guest 'GuestName' to the file GuestName.restore
# /usr/sbin/xm restore GuestName.sav or # virsh restore GuestName.sav
to restore the above saved guest
# /usr/sbin/xm pause GuestName or # virsh pause GuestNameto suspend a running guest (release CPU cycles but retain memory footprint)
# /usr/sbin/xm unpause GuestName or # virsh unpause GuestName
to resume a previously suspended guest
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability27
Basic Xen commands (Resource Management)# /usr/sbin/xm vcpu-set <dom> <value> or # virsh setvcpus <dom> <vcpus>
set the number of CPUs available to <dom> to <value> (only works for dom0/paravirtualized guests)
# /usr/sbin/xm vcpu-list or # virsh vcpuinfoList the physical-virtual CPU bindings
# /usr/sbin/xm mem-set <dom> <value> or # virsh setmem <dom> <value>
balloon <dom> up or down to <value> (only works for dom0/paravirtualized guests)
# /usr/sbin/xm sched-credit -d <DomainID>Display credit schedule information and set cap/weight for individual domain
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability28
Other Useful Commands and ToolsBasic Xen commands
xm logxm dmesgxm info / virsh nodeinfoxm top
virsh commands (extract)virsh dominfo <dom>virsh domstate <dom>Virsh dumpxml <dom>
Tools strace, lsof, iostat/vmstatSystemtap/var/log/messages/var/log/xenAVC messages (setroubleshoot)sosreport has a plugin that automatically gathers all of the above information
Plugin not included in RHEL-5 GA, should be included in day0 errata
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability29
RHEL5 Virt Performance Data
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability30
Red Hat Virt Single-guest Performance
0
50
100
150
200
250
300
350
400
00.0050.010.0150.020.0250.030.0350.040.0450.05
0.017391304347826
0.0353982300884956
0.0418250950570342
0.04682622268470340.0454545454545455
domU vs dom0 RHEL5 RC1 2.6.18-7 3.2 ghz em64T 1GB mem
dom0domU(1)dom0/domU
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability31
Red Hat Virt Performance Para-Virt vs Fully Virt VTx
Copy 10 MB/sec
Linpackd 1kx1k Mflops
GeoMean 1k-1G
External Mbit/sec
Internal MB/sec
0
100
200
300
400
500
600
700
800
900
1000
RHEL5, R4 PV, R4/R3 FV Performance Xen 2.6.17-1.2519.1 Woodcrest
R5-FVR4U4 PVR4U4 FVR3 U7 FV
Perf
orm
ance
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability32
Red Hat Virt Multi-guest Performance
0
50
100
150
200
250
300
350
400
Multiiple XenU Scalability 1,2,4 on RHEL5 GA 2.6.18-8
3.2 Ghz 2-cpu/2ht em64T 4GB memory, 1GB/xen guest
xenU-1xenU-2XenU-4
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability33
Red Hat Virt Multi-guest Efficiency
0
0.2
0.4
0.6
0.8
1
1.2
Efficiency of Multiple XenU 1,2,4 on RHEL5 GA 2.6.18-8
3.2 Ghz em64T 4GB memory, 1GB/xen guest
XenU-1 BaseXen2/Xen1Xen4/Xen1
Sca
ling
eff
icie
ncy
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability34
Red Hat Virt Java Multiguest
Performance
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability35
Red Hat Virt Multiple Guest
Performance
Ora TPM(Fiber)0
10
20
30
40
50
60
70
80
Oracle 10G tpm RHEL5 Multi-Instance Xen1,2,4
RHEL5 GA 2.6.18-8 4-cpu 3 ghz em64T 1GB /vm
RHEL4 U4RHEL5 RC1R5 – Dom0Xen1Xen2Xen4
Tra
ns/
min
(tp
m)
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability36
SMP in Guest OSes
• Takes great care to get good performance while remaining secure
• Paravirtualized approach yields many important benefits− Avoids many virtual IPIs− Enables ‘bad preemption’ avoidance− Auto hot plug/unplug of CPUs
• SMP scheduling (at hypervisor)− Strict gang scheduling not optimal− Credit Scheduler in Xen 3.0.2 (RHEL5)
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability37
Red Hat Virt SMP guest Performance
0
5000
10000
15000
20000
25000
30000
RHEL5 GA Aim 4-cpu, 4GB Woodcrest
r5xen 1-cpur5xen 2-cpur5xen 4-cpur5-Base
Processes
Jo
bs/m
in
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability38
Red Hat Virt SMP Performance
1 2 40
5000
10000
15000
20000
25000
30000
35000
0.82
0.83
0.84
0.85
0.86
0.87
0.88
0.89
Oracle 10G tpm SMP Scaling - R4, R5, Xen
2.6.18-8 4-cpu 3 ghz em64T 1GB/vm
RHEL4 U4RHEL5 B1R5 – Dom0R5 – Xen%diff Xen/R5
Number of CPUs
OA
ST -
TPM
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability39
1 20
5
10
15
20
25
30
35
40
0.81
0.815
0.82
0.825
0.83
0.835
0.84
0.845
0.85
0.855
Sybase 12.5 tpm SMP Scaling - R4, R5, Xen
2.6.18-8 4-cpu 3 ghz em64T 1GB/vm
RHEL4 U4RHEL5 B1R5 – Dom0R5 – Xen%diff Xen/R5
Number of CPUs
TPM
(k)
Red Hat Virt SMP guest Performance
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability40
Red Hat Virt Network Performance
Dom0->External (raw) Dom0->-Guest External->Guest
0
20
40
60
80
100
120
RHEL5 RC1 NetPerf Performance x86_64 (@3k = ave size of SPECweb2005)
PV5 3k
PV4.5 3k
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability41
Red Hat Virt Network Performance
Guest-Dom0 Guest->Guest Guest->External0
100
200
300
400
500
600
RHEL5 RC1 NetPerf Performance x86_64
(@3k = ave size of SPECweb2005)
PV5 3kPV4.5 3k
Th
rou
gh
pu
t M
B/s
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability42
Red Hat Xen FV Performance
Linpackd 1kx1k Mflops Copy MB/sec0
500
1000
1500
2000
2500
3000
3500
Fully Virtualized VT Performance Xen 2.6.18-7 Woodcrest
FC5-BaseR4U4 i386R4U4 x86_64R3 U7
Perf
orm
an
ce
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability43
Red Hat Virt Storage Alternatives
Fibe
r(xvd
)
iSCSI
(xvd
)
ISCSI
(gue
st)
NFSfiler
(xvd
)0
5000
10000
15000
20000
25000
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
RHEL5 RC1 Xen Application Perfor-mance
w/ various Storage (2-cpu AMD64 2.2)
R5(base)R5 – Dom0R5 – Xen4-cpu %diff Xen/Dom0
Ora
cle 1
0G
Tra
ns/
min
(tp
m)
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability44
Tuning using PCI domain
Hypervisor can dedicate PCI-domain's to individual guest.
Default Xen shared HBA / NICs within guest config.
Disk – tested up to 6 Fiber Channel HBAs Network – tested up to 3 1-Gibit network
cards Pros - Dedicate physical PCI resource to guest
Done in guest configuration file Improve I/O performance 10-15%
Cons - Limit throughput when VM's are idle. Limit configuration if planning to use
migration
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability45
Red Hat Virt Performance
Enhance CPU Perf w/ Vcpu-pin Tuning
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability46
Tuning using PCI domain
47HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability
A few examples
48HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability
Installation Issues
Require different installation source for PV vs FV installations
PV requires network based install treeFV require local boot.iso or DVD
In upcoming version of virt-manager/virt-install can specify a network tree for boot.iso
VM image not in /var/lib/xen/images with SElinux enabled in enforcing mode
Will cause backend device not being “connected” and generate a hotplug eventInstallation seems to hang
If iSCSI is used for root device for VM need to create a local boot slab
Also needs a custom network script if iSCSI is not reachable via default network
49HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability
Installation Issues
If multiple Xen bridges are configured use virt-install to specify specific bridge
virt-manager will use the bridge attached to the default network/route
50HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability
Installation Issues / VT/AMD-V Extensions
Intel/VT extensions not enabled in BIOSUnable to install HVM guest (box grayed out)
You can use the following commands to verify whether the virtualization extensions have been enabledOn an Intel/VT based system, look for “VMX”
[root@woodie ~]# xm dmesg | grep VMX(XEN) VMXON is done(XEN) VMXON is done
and to verify the CPU flags have been set [root@woodie ~]# cat /proc/cpuinfo |grep vmxflags: fpu tsc msr pae mce cx8 apic mtrr mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall lm constant_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr lahf_lmYou should have a VMXON for each reported processor. If you have any other messages visit your BIOS settingsThere is no reason to go any further until you have VMXON reported - it just isn't going to work
51HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability
Installation Issues / VT/AMD-V Extensions
AMD/AMD-V extensions not enabled in BIOSUnable to install HVM guest (box grayed out)
On an AMD-V based system, look for “SVM”[root@perf3 ~]# xm dmesg|grep SVM(XEN) AMD SVM Extension is enabled for cpu 0(XEN) AMD SVM Extension is enabled for cpu 1and to verify the CPU flags have been set
[root@perf3 ~]# cat /proc/cpuinfo |grep svmflags : fpu tsc msr pae mce cx8 apic mtrr mca cmov pat pse36
clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni cx16 lahf_lm cmp_legacy svm cr8legacy ts fid vid ttp tm stc
You should have a AMD SVM for each reported processor. If you have any other messages visit your BIOS settingsThere is no reason to go any further until you have VMXON reported - it just isn't going to work
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability52
Troubleshooting (Memory Ballooning)
Failed domain creation due to memory shortage, ie unable to balloon domainA domain might fail to start if there's either not enough memory available or if dom0 has not ballooned down enough to provide space for the newly created/started guestA typical error message in your /var/log/xen/xend.log would be
[2006-11-21 20:33:31 xend 3198] DEBUG (balloon:133) Balloon: 558432 KiB free; 0 to scrub; need 1048576; retries: 20.[2006-11-21 20:33:52 xend.XendDomainInfo 3198] ERROR (XendDomainInfo:202) Domain construction failed
You can verify the amount of memory currently used by dom0 using the command “xm list Domain-0” , if dom0 indeed is not ballooned down you can use the command “xm mem-set Domain-0 NewMemSize” (where NewMemSize should be a small(er) value
53HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability
Troubleshooting (non-Xen/PAE kernel)
Wrong kernel image (ie non-Xen kernel in a para-virt guest)If you try to boot a non-xen kernel in a para-virtualized guest you will see the following error message
[root@grumble]# xm create testVMUsing config file "./testVM".Going to boot Red Hat Enterprise Linux Server (2.6.18-1.2839.el5)kernel: /vmlinuz-2.6.18-1.2839.el5initrd: /initrd-2.6.18-1.2839.el5.imgError: (22, 'Invalid argument')
In the above example you can see that the kernel line shows that it's trying to boot a non-xen kernel. The correct entry would be ”kernel: /vmlinuz-2.6.18-1.2839.el5xen”
54HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability
Troubleshooting (networking/bridging)
Wrong bridge configured in guest configuration file causing Xen Hotplug scripts to timeoutIf you have moved configuration files between different hosts you may to make sure your guest configuration files have been updated to reflect any change in network topology/configuration such as Xen bridge numbering etc
[root@grumble xen]# xm create r5b2-mySQL01Using config file "r5b2-mySQL01".Going to boot Red Hat Enterprise Linux Server (2.6.18-
1.2747.el5xen)kernel: /vmlinuz-2.6.18-1.2747.el5xeninitrd: /initrd-2.6.18-1.2747.el5xen.imgError: Device 0 (vif) could not be connected. Hotplug scripts not
workingIn /var/log/xen/xen-hotplug.log you will see the following error being logged
bridge xenbr1 does not exist!
55HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability
Troubleshooting (networking/bridging) contd
In /var/log/xen/xend.log you will see the following messages being logged[2006-12-14 15:07:08 xend 3874] DEBUG (DevController:143)
Waiting for devices vif.[2006-12-14 15:07:08 xend 3874] DEBUG (DevController:149)
Waiting for 0.[2006-12-14 15:07:08 xend 3874] DEBUG (DevController:464)
hotplugStatusCallback /local/domain/0/backend/vif/2/0/hotplug-status.
[2006-12-14 15:07:08 xend 3874] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vif/2/0/hotplug-status.
[2006-12-14 15:08:48 xend 3874] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vif/2/0/hotplug-status.
[2006-12-14 15:08:48 xend 3874] DEBUG (DevController:464)
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability56
Zombie Domains
When shutting down a domain or migrating, it doesn't actually die but ends up leaving a domain named Zombie-<dom>
Typically caused by restarting xend while domains are running Xend does not properly reconnect certain devices (most notably frame
buffers), so when the domain goes to shut down, xend does not know how to find all of the resources for the domain to destroy
If it happens during migration likely cause is either networking problem or remote relocation server not enabled (status can also change to “migrating-<dom>”
[root@dhcp78-237 ~]# xm list
Name ID Mem(MiB) VCPUs State Time(s)
Domain-0 0 14627 8 r----- 18883.9
Zombie-rhel4u5pv01 9 511 1 -p---- 1072.2
rhel5gapv01 4 511 1 -b---- 1961.1 To resolve the issue
Reboot (only reliable solution) Balloon down Zombie domain to make room for other domains
57HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability
Storage Troubleshooting (Filebased)
Unable to start more than 8 file based guests If file based guest images are used one may have to increase
the number of configured loop devices. The default configuration allows up to 8 loop devices to be active, if more than 8 file based guests/loop devices are needed the number of loop devices configured can be adjusted in /etc/modprobe.conf
Simply edit /etc/modprobe.conf and add the following line to it
options loop max_loop=64 You can substitute the number '64' with a number which fits
your local configuration
58HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability
Storage Troubleshooting (Filebased)
Check if tapdisk process is actually running
[root@dhcp78-237 ~]# ps auxw|grep tap
root 3338 0.0 0.0 95644 680 ? Ssl Apr29 0:00 blktapctrl
root 1053 0.0 0.0 60268 708 pts/7 S+ 06:09 0:00 grep tapd
root 5238 0.0 0.0 30176 612 ? Sl Apr29 0:04 tapdisk /dev/xen/tapctrlwrite2
/dev/xen/tapctrlread2
root 6843 0.0 0.0 30172 612 ? Sl Apr29 0:03 tapdisk /dev/xen/tapctrlwrite3
/dev/xen/tapctrlread3
59HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability
Storage Troubleshooting (Filebased) For each disk there should be a corresponding entry in your domain
configuration[root@woodie ~]# virsh dumpxml 4
<domain type='xen' id='4'>
<name>rhel5gapv01</name>
<os>
<type>linux</type>
<kernel>/var/lib/xen/vmlinuz.Q0Psrs</kernel>
<initrd>/var/lib/xen/initrd.B_e0Gm</initrd>
<cmdline>ro root=/dev/VolGroup00/LogVol00 rhgb quiet</cmdline>
<disk type='file' device='disk'>
<driver name='tap' type='aio'/>
<source file='/var/lib/xen/images/rhel5gapv01.dsk'/>
<target dev='xvda'/>
</disk>
<console tty='/dev/pts/3'/>
</devices>
</domain>
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability60
Other random bits
A fully virtualized x86_64 guest fails to boot with “Your CPU does not support long mode. Use a 32bit distribution”.
Resolution: Make sure “pae=1” is set in the domain configuration file
Over time, the /var/lib/xen directory becomes full of kernel.xxxxx and initrd.xxxxx files
Xen tools don't always clean up after errors Resolution:
Safe to just delete all of the files
Message: “FATAL: Module microcode not found” during domain boot
It's not really a fatal error, it is just warning that the virtual machine couldn't update the CPU microcode
Resolution: Nothing required To get rid of error message, disable microcode_ctl service
61HP Technology forum June 2007 / RHEL5/Virt Workshop
Product features subject to change prior to availability
Consider the possibilities...
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability62
Resources
• Red Hat − http://www.redhat.com/
• Virtualization Infocenter− http://www.openvirtualization.com/
• Libvirt− http://www.libvirt.org/
• Virt-Manager− http://virt-manager.et.redhat.com/
• Red Hat Cluster Suite− http://www.redhat.com/solutions/gfs/
• Red Hat Emerging Technology Group− http://et.redhat.com/
63HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability
QuestionQuestions?s?
ThanksThanks
HP Technology Forum & ExpoGet connected. People. Technology. Solutions.
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability65
Highly Available RHEL5 Host
RHEL5
Host A
Guest
RHEL5
Host B
Shared
Storage
Guest running as a RHCS serviceGuest 1
Guest 2
Guest X
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability66
Highly Available RHEL5 Host
RHEL5
Host A
Guest
RHEL5
Host B
Shared
Storage
Guest running as a RHCS serviceGuest 1
Guest 2
Guest X
Automatic failover upon Hypervisor failure
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability67
Highly Available RHEL5 Host
RHEL5
Host A
Guest
RHEL5
Host B
Shared
Storage
App
Guest 2
Guest XGuest 1Guests running as independent cluster
Hypervisor clustered via RHCS
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability68
Highly Available RHEL5 Host
RHEL5
Host A
RHEL5
Host B
Shared
Storage
Guest 2
App
Guest XGuest 1Guests running as independent cluster
Hypervisor clustered via RHCSApplication failover upon hosts/guest failure
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability69
Highly Available RHEL5 Host
RHEL5
Host A
Guest
RHEL5
Host B
Shared
Storage
Guest 2
App
Guest XGuest 1Guests running as independent cluster
Hypervisor clustered via RHCSApplication failover upon hosts/guest failure
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability70
Highly Available RHEL5 Host
RHEL5
Host A
RHEL5
Host B
Shared
Storage
App
Guest 2
Guest X
Guests running as independent cluster
Hypervisor and bare metal host clustered via RHCS
RHEL5
Host C
Guest 1 Guest X
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability71
Highly Available RHEL5 Host
RHEL5
Host A
RHEL5
Host B
Shared
Storage
Guest 2
Guest X
Guests running as independent cluster
Hypervisor and bare metal host clustered via RHCS
RHEL5
Host C
App
Guest 1 Guest XApplication can migrate to another Guest/VM
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability72
Highly Available RHEL5 Host
RHEL5
Host A
RHEL5
Host B
Shared
Storage
Guest 2
Guest X
Guests running as independent cluster
Hypervisor and bare metal host clustered via RHCS
RHEL5
Host CApp
Guest 1 Guest X
Application can migrate to a bare metal system
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability73
RHEL5 Disaster Recovery
RHEL5
Site A
RHEL5
Site B
Shared
Storage
Guest 1
Guest 2
Guest XGuestImage
GuestImage
GuestImage
GuestImage
GuestImage
GuestImage
XP/CA, EVA/CA
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability74
RHEL5 Disaster Recovery
RHEL5
Site A
RHEL5
Site B
Shared
Storage
Guest 1
Guest 2
Guest XGuestImage
GuestImage
GuestImage
GuestImage
GuestImage
GuestImage
XP/CA, EVA/CA
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability75
RHEL5 Disaster Recovery
RHEL5
Site A
RHEL5
Site B
Shared
Storage
Guest 1
Guest 2
Guest X
GuestImage
GuestImage
GuestImage
GuestImage
GuestImage
GuestImage
XP/CA, EVA/CA
HP Technology Forum June 2007 / RHEL5/Virt Best Practices
Product features subject to change prior to availability76
RHEL5 Disaster Recovery
RHEL5
Site A
RHEL5
Site B
Shared
Storage
Guest 1
Guest 2
Guest X
GuestImage
GuestImage
GuestImage
XP/CA, EVA/CA