1

The KVM Weather Report

Jes Sorensen <Jes.Sorensen@redhat.com>LinuxCon Japan, June 1st 2011

2

Agenda

Foundation for Cloud

Future

Virtualization Overview

3

Virtualization Overview: History

1967 IBM CP-40 1972 Goldberg Arch Principles for VCS

2006 libvirt launched 2007 KVM integrated into Linux 2.6.20

2011 KVM goes to 110

2

4

6

8

10

12

KVM goes to 11

4

Virtualization Overview: History

5

Virtualization Overview: KVM TodayNative vs. Virtualized: 1% Overhead in KVM

SAP Linux Certification SuiteCPU intensive reports phaseRHEL 6 + KVM and Intel E7-8800

6

KVM: The Kernel-based Virtual Machine

7

KVM features

VT-x/AMD-V

EPT/NPT

SMP Guests

Live Migration

OvercommitCPU, Memory, I/O

Hotplug

High Performance I/O

SR-IOV

VT-d/IOMMUKSM

NUMASPICE

Power Management

Massive scalability

sVirtSVVP & WHQL

PCI Device Assignment

Live snapshot

TransparentHuge Pages

Native Windowsdrivers

8

Why reinvent the wheel?

Focus on virtualization.

9

KVM Architecture

10

KVM Architecture

11

KVM Architecture

12

KVM in the Real World

● RAS

● Security Isolation

● Resource management

● Performance improvements● CPU, Block, Net, Memory

● SPECvirt

13

RAS

● timekeeping

● QMP, qdev, VMState, SeaBIOS

● vmchannel

● stable guest HW ABI● machine type● stable PCI topology

● PCI Device Assignment improvements

● Live migration with large memory guests

14

Isolation: sVirt Security (or similar w/ AppArmour)

● Applying security labels to individual guest virtual machines and their resources

● Guest Isolation achieved withSELinux Mandatory AccessControls (MAC)

● Protect against untrustedGuest VM

● Protect against Hostmisconfiguration

● Prevents unauthorizedaccess of Guests/Host

● Builds on existing, provensecurity mechanisms & controls

15

Isolation: Resource Management

Virt Guest A50% CPU50% Mem

Virt Guest B25% CPU25% Mem

Application Z25% CPU25% Mem

Network40% net 40% net 20% net

60% 20%20% I/O

Storage

● Control Group (Cgroups) for CPU/Memory/Network/Disk

● Benefit: guarantee Quality of Service

● Ideal for: Virtualization/Cloud deployments

16

● Scale to 64 vcpus! And same pcpu scaling as Linux

● Guest spin lock-holder preemption sol'n

● KVM efficiency● User return notifiers...get lazier

● x2apic● Use MSR access to limit mmio accesses to the irq chip

KVM: CPU Performance

17

Performance: SMP Scalability

1-cpu 4-cpu 8-cpu 16-cpu 32-cpu 64-cpu0

10000

20000

30000

40000

50000

60000

70000

80000

64 CPU Scalability - Stream BenchmarkIntel EX 64-cpu, 128GB, FC

RHEL 6 PhysicalRHEL 6 Virtual

Mem

ory

BW

(M

B/s

ec)

Excellent, linear scalability; minimal virtualization overhead

Note: With this h/w, at 32 CPUs socket bandwidth is saturated

18

KVM Block Architecture

19

KVM: Block Performance

● qemu vectored I/O (preadv/pwritev) and native AIO

● virtio● flush + fua● MSI support● ioeventfd

● qcow2 cache

20

KVM Performance – RHEL6 aio=native Win2k8 Intel 24cpu, 64GB, FC IOmeter

2k 4k 8K 16K 32K 64K0

50

100

150

200

250

300

350

400

450

IO - Sequential Reads

r5kvm r6aio_native metal

MB

/se

c

2k 4k 8K 16K 32K 64K0

10

20

30

40

50

60

70

80

90

100

CPU - Sequential Reads

r5kvm r6aio_native metal

%C

PU

21

KVM Performance – RHEL6 aio=native Win2k8 Intel 24cpu, 64GB, FC IOmeter

2k 4k 8K 16K 32K 64K0

10

20

30

40

50

60

70

80

90

100

CPU - Sequential Writes

r5kvm r6aio_native metal

%C

PU

2k 4k 8K 16K 32K 64K0

50

100

150

200

250

300

350

400

IO - Sequential Writes

r5kvm r6aio_native metal

MB

/se

c

22

KVM Network Architecture

23

Performance: Networking● vhost-net

● virtio backend in kernel● transparent to guest● reduce latency● better throughput● lower CPU consumption

● SR-IOV for near native performance

● VEPA/VNLink

24

virtio network architecture – detail

25

vhost_net Efficiency

32 64128

256512

10242048

40928192

1638432768

655070

50

100

150

200

250

300

350

400

8 Guest Scale Out RX Vhost vs Virtio - % Host CPU

Mbit per % CPU netperf TCP_STREAM

VhostVirtio

Message Size (Bytes)

Mb

it /

% C

PU

(b

igge

r is

be

tte

r)

26

Latency comparison – RHEL 6

1 64765

61478 21 35125

195384

10272048

409312288

2457332771

655360

50

100

150

200

250

300

350

400

Network Latency by guest interface method

Guest Receive (Lower is better)

host RX virtio RX vhost RX SR-IOV RXMessage Size (Bytes)

La

ten

cy (

use

cs)

27

Performance: Memory

● Transparent Huge Pages● Benefit for bare metal apps (4.2%)● Huge benefit for KVM VMs (25%)

● KSM working with THP

28

Performance

What does it add up to?

29

SPECvirt

30

SPECvirt_sc2010 results*

RHEL 5.5 (KVM) IBM X3650 M3 (12 cores, 144GB, 12 tiles)

VMware ESX 4.1 HP DL380 G7 (12 cores, 192GB, 13 tiles)

RHEL 6 (KVM) IBM HS22V (12 cores, 288GB, 14 tiles)

RHEL 5.5 (KVM) IBM x3690 X5 (16 cores, 512GB, 14 tiles)

RHEL 6 (KVM) IBM x3690 X5 (16 cores, 1TB, 18 tiles)

VMware ESX 4.1 U1 HP BL620c G7 (20 cores, 512GB, 19 tiles)

RHEL 6.1 (KVM) HP BL620c G7 (20 cores, 512GB, 19 tiles)

VMware ESXi 4.1 Bull SAS (32 cores, 512GB, 28 tiles)

VMware ESXi 4.1 IBM x3850 x5 (32 cores, 1TB, 28 tiles)

VMware ESX 4.1 HP DL580 G7 (40 cores, 1TB, 38 tiles)

RHEL 6 (KVM) IBM x3850 X5 (64 cores, 2TB, 56 tiles)

RHEL 6 (KVM) IBM x3850 X5 (80 cores, 2TB, 72 tiles)

0

1000

2000

3000

4000

5000

6000

7000

8000

1169 1221 1367 13691763 1811 1820

2721 2742

3723

5466

7067

*As of May 4, 2011. See http://www.spec.org/virt_sc2010 for details

31

Agenda

Foundation for Cloud

Future

Virtualization Overview

32

Cloud

● Computing as a Service (Anything as a Service)

● Utility model

● Self provisioning

● Dynamic resources

33

KVM in Cloud

● Increase density

● Improve efficiency

● Multi-tenancy (security, isolation)

34

KVM Is The Cloud

35

Agenda

Foundation for Cloud

Future

Virtualization Overview

36

Roadmap

● Scaling and performance

● NUMA

● Lock holder preemption

● Zero copy networking

● Multiqueue virtio-net

● Debugging and profiling

● Further cgroup integration

● virt-agent (guest agents)

● QCOW2 performance

● Multi-tenant network isolation

● Memory management (async page faults, hinting, compcache)

● USB 2.0 support

● Performance Monitoring Unit (PMU) support

Weather forecast...

37

KVM development activity

2.6.20 2.6.21 2.6.22 2.6.23 2.6.24 2.6.25 2.6.26 2.6.27 2.6.28 2.6.29 2.6.30 2.6.31 2.6.32 2.6.33 2.6.34 2.6.35 2.6.36 2.6.37 2.6.38 2.6.390

50

100

150

200

250

300

350

KVM commits per release

Release

Co

mm

its

38

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 240

100

200

300

400

500

600

QEMU commits per month

Months since start

Co

mm

its

QEMU development activity

39

Cloudy with a chance of

total world domination