Heterogeneous Live Migration of Virtual Machines
Pengcheng Liu, Ziye Yang, Xiang Song, Yixun Zhou, Haibo Chen, and Binyu Zang
Parallel Processing Institute, Fudan Universityhttp://ppi.fudan.edu.cn
The market of Virtualization is booming
• Virtual data-center, grid computing, and cloud computing
• “By 2010, at least 30%of non-desktop IT infrastructure will be virtualized (up from less than 5 percent in 2007).”– www.cio.com/article/170653
• What can virtualization do?– Sever consolidation– Mobility
• Virtual machine live migration– …
VM Live Migration
• Benefits– Load balancing– Online maintenance – Fault tolerance
23/4/18 PPI, Fudan University 3
VirtualMachine
Virtual Machine
VMMHardware Hardware
VMM
War between different VMM venders
• The battle in virtualization area is and will be continuing – Vmware, Microsoft, kvm, …
– They will likely co-exisit in a long time• Linux kernel 2.6.23 embeds different vmm support
– Xen, lguest, kvm
• Cloud Computing• More resources brings inevitably heterogeneity
Virtual Machine
VirtualMachine
Motivation
• Heterogeneity of different VMMs– Break uniformity and mobility– Downgrade the applicability of live migration
23/4/18 PPI, Fudan University 5
VMM 2VMM 1Hardware Hardware
Contribution: Vagrant
• A preliminary study aim at– Bridging the heterogeneity of different VMMs and
support live migration
• Goal– Provide common migration protocol– Use common format of VM abstraction– Provide a pool of memory migration algorithms
• Preliminary results– Support VM migration across Xen and KVM– Accept able performance
Outline
• Heterogeneity among VMMs
• Vagrant Design and Implementation
• Evaluation
• Conclusion
23/4/18 PPI, Fudan University 7
Heterogeneity in Abstracting Resources
• CPU– Internal virtual CPU format
• Memory– Memory management information
• I/O Devices– Virtual I/O Device format
23/4/18 PPI, Fudan University 8
Heterogeneity in Migration Algorithms
• Stop-and-copy
• Pre-copy
• Push and pull
23/4/18 PPI, Fudan University 9
Heterogeneity in Migration Protocols
• Different set of control commands
• Example – Xen VMM initiates migration by sending a string
”receive” to the target– KVM sends a special integer to start migration
23/4/18 PPI, Fudan University 10
Outline
• Heterogeneity among VMMs
• Vagrant Design and Implementation
• Evaluation
• Conclusion
23/4/18 PPI, Fudan University 11
Architecture of Vagrant
• Vagrant acts as a middle man between different VMMs
23/4/18 PPI, Fudan University 12
VMM A
VM
Management VM
VMM B
Vagrant Agency
StateTransformer
Algorithm pool
Management VM
Vagrant Agency
StateTransformer
Algorithm pool
Bridging the Heterogeneity
• CPU State– Intercept the state from local virtual CPU module
– Essential CPU state is transformed into Vagrant format
– Requirements :• VMMs can expose same CPU features
23/4/18 PPI, Fudan University 13
Bridging the Heterogeneity (2/2)
• I/O Devices– Intercept the state from local device module• H/W based virtualization relies on device emulation
– Eg. QEMU used by Xen, KVM and VirtualBox
– Requirements:• VMMs can support all I/O devices existing in the
migrated VM
23/4/18 PPI, Fudan University 14
Migration Protocol
• Initiate and Control the migration process– Common migration protocol and commands– Negotiation before the migration• Ensure the same capability of pre- and post- migration
environment– CPU features, supported I/O devices
23/4/18 PPI, Fudan University 15
VMM A VMM B
Hey,I want a RTL8169 NIC,
Can you support?
Stop and Copy
23/4/18 PPI, Fudan University 17
VirtualMachine
Virtual Machine
VMMHardware Hardware
VMM
Simply stop the VM and send
VM state
May cause very long
downtime, but total
time is short
Pre-copy Based Algorithm
• Sending memory pages while the VM is still running
23/4/18 PPI, Fudan University 18
VirtualMachine
Virtual Machine
VMMHardware Hardware
VMM
Sending memory pages
VM is still running
Stop the VM
Send whole VM state to destination
Pre-copy Reduces
downtime
Algorithm Pool
• Algorithm pool– At least one common algorithm is needed– Choose the algorithm by negotiation
• Ideally, support a set of different algorithms– Choose appropriate algorithm in different
scenarios– eg, the user wants to finish the migration as soon
as possible, stop and copy is better
23/4/18 PPI, Fudan University 19
Implementation
• A prototype based on– X86 with VT-x extensions– Xen 3.1.0 and KVM-48
– CPU• Both expose most CPU features
– I/O Device• Both rely on QEMU• VMM support for some devices, such as PIC/APIC/IOAPIC
23/4/18 PPI, Fudan University 20
Outline• Heterogeneity among VMMs
• Vagrant Design and Implementation
• Evaluation– Migration time– Application performance
• Conclusion23/4/18 PPI, Fudan University 21
Experimental Setup(1/2)
• Machine setup:– Migration between pair of PCs with
• 2.33 GHz Intel Core Duo CPU, 2GB RAM• RTL8169 NIC• 250 GB 7200 RPM SATA disk
– NFS server• 3.0 GHz Pentium IV with 1GB RAM• Intel Pro 100/1000 Ethernet NIC• 250 GB 7200 RPM SATA disk
– Machines were connected via switched Gigabit Ethernet
23/4/18 PPI, Fudan University 22
Experimental Setup (2/2)
• VM configuration– 128 MB of RAM– 5GB disk image with installation of Fedora Core 6
• Migration time under different workload– Idle, kernel-compile, memtest86, apache benchmark (ab)
• Application performance– Migrating a running Web Server
• Continuously serves a 256 KB file to a remote client
23/4/18 PPI, Fudan University 23
Questions for Evaluation
• Whether Vagrant has an acceptable total migration and minimal downtime?
• Whether applications during the migration can still provide acceptable performance?
• Is Vagrant’s performance comparable to other migration framework, eg. Xen and KVM?
23/4/18 PPI, Fudan University 24
Total Migration Time
23/4/18 PPI, Fudan University 25
Vagrant adopts batched memory transfer while KVM uses one-by-one manner
Migration Downtime
23/4/18 PPI, Fudan University 26
Downtime in Vagrant in comparable to homogeneous live migration
Migrating a Running Web Server
23/4/18 PPI, Fudan University 27
Memory Pre-copy, about 10% downgrade
Less than 1 second
Related Work
• Homogenous live migration over LAN– Vmotion, Xen (Clark et al. ’05) and KVM
• Live wide-area migration– Bradford et al. (vee’07)– “VM Turntable” (FGCS’06)
• Tons of virtualization related work
23/4/18 PPI, Fudan University 28
Conclusion
• Heterogeneous live VM migration framework– General abstraction of VMs and migration protocol
– No changes to hosted OS and application
– Minor changes to VMM abstraction
– Comparable downtime and end-to-end time
23/4/18 PPI, Fudan University 29
Heterogeneous Live Migration of Virtual Machines
Pengcheng Liu, Ziye Yang, Xiang Song, Yixun Zhou, Haibo Chen, and Binyu Zang
Parallel Processing Institute, Fudan Universityhttp://ppi.fudan.edu.cn
Backup Slides
• Control commands in Vagrant
• CPU state in Vagrant
• I/O devices configured for the VM
• Memory migrating algorithm
23/4/18 PPI, Fudan University 32
Control Commands in Vagrant
• Common set of control commands– Each has a 3 seconds timeout
23/4/18 PPI, Fudan University 33
CPU State in Vagrant
• The necessary information to be transferred includes– GPR, control regs, segment regs, debug regs, msr,
fpu regs, sysenter registers– Time stamp counter– Pending events
23/4/18 PPI, Fudan University 34
I/O devices configured for the VM
• The migrated VM is configured with– Keyboard and mouse– Rtc– PIC/APIC/IOAPIC– PIT– IDE disk– NIC– Serial port– VGA
23/4/18 PPI, Fudan University 35