1 Virtualization Management the oVirt way
Introduction to Ovirt
Chris Henderson, RHCE RHCVA Technical Account Manager, Red Hat
2 Virtualization Management the oVirt way
oVirt history oVirt installation oVirt internals Getting familiar with oVirt logs Reporting an issue
Agenda
3 Virtualization Management the oVirt way
History
● February 2007 KVM accepted into Linux kernel 2.6.20
● Red Hat Acquired Qumranet 9/2008
http://www.redhat.com/about/news/press-archive/2008/9/qumranet
4 Virtualization Management the oVirt way
Installation
http://www.ovirt.org/Quick_Start_Guide
http://www.ovirt.org/Troubleshooting
http://www.ovirt.org/Quick_Start_Guidehttp://www.ovirt.org/Troubleshooting
5 Virtualization Management the oVirt way
Complex project Multiple technologies Distributed components
Deep integration with the operating system Storage (LVM, multipath, iSCSI) Network (VLAN, bond, bridge)
Bugs (Yes, we have some)
6 Virtualization Management the oVirt way
Let’s meet the characters: UI/API – “the beauty” Engine – “the brains” Hypervisor – “the muscle”
VDSM, Libvirt & KVM
oVirt under the hood
7 Virtualization Management the oVirt way
Web-Admin-Portal Engine Hypervisor
VDSM
Libvirt
QEMU-KVM
Kernel
Fedora
DB
Fedora
NAS\SAN
FC
HB
A
disk
TCP:8443
XMLRPC
TCP:54321
LUN
LUN
oVirt basic architecture
8 Virtualization Management the oVirt way
Storage Pool
Data (master)
Block\NFS\Posix\Local NFS
Disk Disk Disk
Data Export ISO
Disk+
OVF
ISO+
VFD
Domain function
Domain Type
usage
Managed by SPM
Overview of oVirt storage concepts
9 Virtualization Management the oVirt way
Cross-system actions are controlled by the engine Started by either user (UI/API) or system (internal) Can be traced using correlation-ID
Flows under the hood
10 Virtualization Management the oVirt way
Web-Admin-Portal Engine Hypervisor
VDSM
Libvirt
QEMU-KVM
Kernel
Fedora
Local Storage
Run VM
DB
FedoraTCP 443
XMLRPC
TCP:54321
SPICE
Run VM
11 Virtualization Management the oVirt way
Run VM flow
RunVmCommand(BLL)
ISOPrefixVdsCommand(IRS)
Run and protect getIsoList()
CreateVMVDSCommand(VDS)
Run and protect vmCreate()
Run and protect prepareImage()
Preapre disks, build xml and send to
libvirt
GetVmStats(VDS)
Run and protect vmGetStats()
12 Virtualization Management the oVirt way
Engine SPM
VDSM
Libvirt
Fedora
LUN
NAS\SAN
FC
DB
Fedora QEMU-KVM
Kernel
Create storage domain
LUN
VGPV
LV (metadata)
HB
AH
BA
Create storage domain (block)
13 Virtualization Management the oVirt way
Create storage domain flow
AddSANStorageDomainCommand(BLL)
CreateVGVDSCommand(VDS)
CreateStorageDomainVDSCommand
(VDS)
Run and protect createVG()
Run and protect createStorageDomain()
GetStorageDomainStats(VDS)
Run and protect getStorageDomainStats()
GetVGInfo(VDS)
Run and protect getVGInfo()
ConnectStorageServerVDSCommand (VDS)
DiscoverSendTargets()(VDS)
Run and protect discoverSendTargets()
Run and protect connectStorageServer()
GetDeviceList (VDS) Run and protect getDeviceListr()
14 Virtualization Management the oVirt way
Extend storage domain
Engine SPM
VDSM
Libvirt
Fedora
HSM
VDSM
Libvirt
QEMU-KVM
Kernel
Fedora
ids metadata leases
inbox outbox
Volume Group
Physical Volume
Storage Domain
LUN
Physical Volume
NAS\SAN
FC
HB
A
DB
Fedora
LUN
LUN
LUN
QEMU-KVM
Kernel
Extend Storage domain
15 Virtualization Management the oVirt way
Extend storage domain flow
UpdateStorageDomainCommand(BLL)
ConnectStorageServerVDSCommand (VDS)
RefreshStoragePoolVDSCommand (VDS)
Run and protect refreshStoragePool()
Run and protect connectStorageServer()
GetDeviceListVDSCommand (VDS)
Run and protect getDeviceLiist()
ConnectAllHostsVDSCoammd(BLL)
ConnectStorageServerVDSCommand (VDS)
Run and protect connectStorageServer()
GetDevicesVisibility (VDS) Run and protectgetDevicesVisibility()
ExtendStorageDomainVDSCommand (IRS)
Run and protect extendVG()
Sent only to SPM
Sent to all hosts in the pool
Sent to all hosts in the pool
Sent only to HSM
16 Virtualization Management the oVirt way
Getting familiar with oVirt logs
oVirt - backend
Engine log /var/log/ovirt-engine/engine.log Console log /var/log/ovirt-egine/console.log Server log /var/log/ovirt-engine/server.log
VDSM - hypervisor
VDSM log /var/log/vdsm/vdsm.log Libvirt log /var/log/libvirtd.log QEMU log /var/log/libvirt/qemu/.log
17 Virtualization Management the oVirt way
First look at the engine All System actions are logged Easy to understand & parse (grep is your friend)
2013-01-04 17:58:07,177 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-3-thread-4) [73b0d2b1] START, CreateVmVDSCommand(HostName = master-vds11, HostId = 11340fbe-fefc-11e0-ac21-00145e832c40, vmId=ed0807af-00de-4523-a96b-fcab28ca656c, vm=org.ovirt.engine.core.common.businessentities.VM@563944b1), log id: 26607971
Engine log - intro
18 Virtualization Management the oVirt way
Log format: Date & time Log message severityClass (code path)ThreadCorrelation idState (start\finish)User\system aggregated commandArguments
Engine log - format
19 Virtualization Management the oVirt way
Update Virtual Machine name
2013-01-04 18:13:28,150 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (http-/0.0.0.0:8443-16) [6bfe2800] START, UpdateVMVDSCommand( storagePoolId = 132859ec-ef83-4c01-b411-0cea6d3e1ed6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 00000000-0000-0000-0000-000000000000, infoDictionary.size = 1), log id: ea2d99e
2013-01-04 18:13:28,227 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (http-/0.0.0.0:8443-16) [6bfe2800] FINISH, UpdateVMVDSCommand, log id: ea2d99e
Engine log - example
20 Virtualization Management the oVirt way
External vs. internal commands Communicating with hypervisor CanDoAction START & FINISH sequence
Engine log - concepts
21 Virtualization Management the oVirt way
Standard formatting Helps to:
Understand the flow Correlate between user action to system operation Correlate with hypervisor
Look for ERROR
Engine log - summary
22 Virtualization Management the oVirt way
Easy to understand & parse (grep is your friend)Service is always running in debug modeLow level commands are logged as executed
Thread-3870::DEBUG::2013-01-04 19:14:36,645::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/dd iflag=direct if=/dev/0748bcbd-2b41-4431-aad9-a93a57fbde67/metadata bs=4096 count=1’
VDSM log - intro
23 Virtualization Management the oVirt way
Log format: ThreadLog message severityDate & timeClass (python module)Entry point (API)Command (function)ArgumentsCorrelation id
VDSM log – format
24 Virtualization Management the oVirt way
Extend storage domain
Thread-485093::INFO::2013-01-03 11:29:36,317::logUtils::37::dispatcher::(wrapper) Run and protect: extendStorageDomain(sdUUID='0748bcbd-2b41-4431-aad9-a93a57fbde67', spUUID='132859ec-ef83-4c01-b411-0cea6d3e1ed6', devlist=['360a98000572d45366b4a6f594f644175'], options=None)
Thread-485093::INFO::2013-01-03 11:29:39,473::logUtils::39::dispatcher::(wrapper) Run and protect: extendStorageDomain, Return response: None
VDSM log - example
25 Virtualization Management the oVirt way
Helps to: Understand flows Understand errors Low level call commands Different components interaction
VDSM log – summary
26 Virtualization Management the oVirt way
Events tab Capture error message
Get correlation id
oVirt-engine Search correlation id in logs
Search for error
Understand command (internal\external)
Hypervisor Match the time with engine
Look for ‘Run and protect’ command
Look for error
Look for call command
System correlation – best practice
27 Virtualization Management the oVirt way
Basic oVirt concepts and terminology oVirt components Common oVirt flows and how they work Location and format of oVirt logs
Summary
28 Virtualization Management the oVirt way
Engine side: Events tab (UI)
Task manager (UI)
oVirt-Shell (CLI)
Hypervisor side: VDSM CLI (vdsClient)
Dump-storage-table tool
Libvirt CLI (virsh)
GNU-debugger (gdb)
Additional tools
29 Virtualization Management the oVirt way
Search for existing bug in www.bugzilla.com Search for existing issue in mailing list arch Send email to [email protected] Open a bug
Report an issue
http://www.bugzilla.com/mailto:[email protected]://bugzilla.redhat.com/enter_bug.cgi?product=oVirt
30 Virtualization Management the oVirt way
Title: describe the flow and not the error Description: write as much details as possible Repro steps: clear steps causing the issue System info:
All relevant package information Type of storage
Attachment: all relevant logs (from previous slide)
File a new bug
31 Virtualization Management the oVirt way
Pirates
32 Virtualization Management the oVirt way
Q&A
33 Virtualization Management the oVirt way
THANK YOU !
http://[email protected]
http://www.ovirt.org/