18 August 2014

Xen on ARMHow fast is it, really?

Stefano Stabellini

Status

© 2013 Citrix | Confidential – Do Not Distribute

Xen Project 4.4 release: statusFeatures:● 64-bit guest support in ARMv8● stable hypercall ABI● basic lifecycle operations● memory ballooning● scheduler configurations, vcpu pinning, cpu pools

HCL (not inclusive):● ARMv8 FVP● Arndale board● Allwinner Sunxi (Cubieboard 2)● TI OMAP5● Applied Micro X-Gene

© 2014 Citrix

© 2013 Citrix | Confidential – Do Not Distribute

Linux v3.13+: status

● dom0 and domU● 32-bit and 64-bit● SMP● swiotlb-xen● PV drivers

○ disk○ network○ console○ framebuffer○ keyboard○ mouse

© 2014 Citrix

© 2013 Citrix | Confidential – Do Not Distribute

Coming in Xen 4.5● irq latency reduction: no maintenance interrupts● support for interrupt priorities● irq migration: physical follow virtual● multi-grants support● SMMU support● Device Assignment on ARM● Xen UEFI host support● GICv3 support● support for up to 1TB of guest RAM● super pages support● support for sparse memory map● support for 48-bit physical addresses● QEMU PV backend on ARM: framebuffer,

mouse, keyboard, multi-console© 2014 Citrix

● MiniOS on ARM● PSCI v0.2

© 2013 Citrix | Confidential – Do Not Distribute

Coming in Xen 4.5● irq latency reduction: no maintenance interrupts● support for interrupt priorities● irq migration: physical follow virtual● multi-grants support● SMMU support● Device Assignment on ARM● Xen UEFI host support● GICv3 support● support for up to 1TB of guest RAM● super pages support● support for sparse memory map● support for 48-bit physical addresses● QEMU PV backend on ARM: framebuffer,

mouse, keyboard, multi-console© 2014 Citrix

● MiniOS on ARM● PSCI v0.2

© 2013 Citrix | Confidential – Do Not Distribute

xen-devel traffic

© 2014 Citrix

© 2013 Citrix | Confidential – Do Not Distribute

Maintainers & reviews

© 2014 Citrix

© 2013 Citrix | Confidential – Do Not Distribute

Coming in Xen 4.5: performance improvements

● irq latency reduction: no maintenance interrupts● support for interrupt priorities● irq migration: physical follow virtual● multi-grants support● super pages support● support for up to 1TB of guest RAM● support for sparse memory map● support for 48-bit physical addresses

© 2014 Citrix

© 2013 Citrix | Confidential – Do Not Distribute

irq latency reduction: no maintenance interrupts

DomU

Xen

irq 109

virq 109

DomU

Xen

EOI

DomU

Xen

Maintenance interrupt

GICH_LRWrite

GICH_LRClear

© 2014 Citrix

© 2013 Citrix | Confidential – Do Not Distribute

irq latency reduction: no maintenance interrupts

DomU

Xen

irq 109

virq 109

DomU

Xen

EOI

DomU

Xen

Maintenance interrupt

GICH_LRWrite

GICH_LRClear

© 2014 Citrix

© 2013 Citrix | Confidential – Do Not Distribute

Coming in Xen 4.5: performance improvements

● irq latency reduction: no maintenance interrupts● support for interrupt priorities● irq migration: physical follow virtual● multi-grants support● super pages support● support for up to 1TB of guest RAM● support for sparse memory map● support for 48-bit physical addresses

© 2014 Citrix

© 2013 Citrix | Confidential – Do Not Distribute

irq migration: physical follow virtual

vcpu0 vcpu1

pcpu0 pcpu1

irq 109

virq 109

© 2014 Citrix

© 2013 Citrix | Confidential – Do Not Distribute

irq migration: physical follow virtual

vcpu0 vcpu1

pcpu0 pcpu1

irq 109

virq 109

© 2014 Citrix

© 2013 Citrix | Confidential – Do Not Distribute

irq migration: physical follow virtual

vcpu0 vcpu1

pcpu0 pcpu1

irq 109

virq 109

© 2014 Citrix

© 2013 Citrix | Confidential – Do Not Distribute

Coming in Xen 4.5: performance improvements

● irq latency reduction: no maintenance interrupts● support for interrupt priorities● irq migration: physical follow virtual● multi-grants support● super pages support● support for up to 1TB of guest RAM● support for sparse memory map● support for 48-bit physical addresses

© 2014 Citrix

© 2013 Citrix | Confidential – Do Not Distribute

multi-grants support

© 2014 Citrix

Backend Frontend

PagePage

Page

Grant

Grant

© 2013 Citrix | Confidential – Do Not Distribute

multi-grants support

© 2014 Citrix

Backend Frontend

PagePage

Page

Grant

Grant

mfn

==

mfn

© 2013 Citrix | Confidential – Do Not Distribute

multi-grants support

© 2014 Citrix

Backend Frontend

PagePage

Page

Grant

Grant

mfn

==

mfn

mfn_to_pfn

??

© 2013 Citrix | Confidential – Do Not Distribute

multi-grants support

© 2014 Citrix

Backend Frontend

PagePage

Page

Grant

Grant

Pagepfn == mfn

mfn

==

mfn

mfn_to_pfn

??

© 2013 Citrix | Confidential – Do Not Distribute

Coming in Xen 4.5: performance improvements

● irq latency reduction: no maintenance interrupts● support for interrupt priorities● irq migration: physical follow virtual● multi-grants support● super pages support● support for up to 1TB of guest RAM● support for sparse memory map● support for 48-bit physical addresses

© 2014 Citrix

© 2013 Citrix | Confidential – Do Not Distribute

Coming in Xen 4.5: hardware and interfaces

● SMMU support● Xen UEFI host support● PSCI v0.2● GICv3 support

© 2014 Citrix

© 2013 Citrix | Confidential – Do Not Distribute

Coming in Xen 4.5: features

● (Non-PCI) Device Assignment on ARM● QEMU PV backend on ARM: framebuffer, mouse, keyboard, multi-console● MiniOS on ARM

© 2014 Citrix

© 2013 Citrix | Confidential – Do Not Distribute

Looking ahead

● save/restore and live-migration● PCI device assignment

● IOMMU emulation for remote processors● GICv2m support● SMMUv2 support● ACPI support

© 2014 Citrix

© 2013 Citrix | Confidential – Do Not Distribute

Embedded and Automotive

● Embedded and Automotive Xen Project initiativehttp://xenproject.org/about/events/viewevent/146-webinar-virtualization-in-the-automotive-industry-2.html

● Embedded and Automotive PV drivers projecthttp://xenproject.org/developers/teams/embedded-and-automotive.htmlhttp://wiki.xenproject.org/wiki/Embedded_and_Automotive_PV_Drivers/Roadmap

© 2014 Citrix

Benchmarks

© 2013 Citrix | Confidential – Do Not Distribute

Execution environment

● Linux v3.15-rc8dom0, domU and native kernel

● Xen v4.5-unstable+

● QEMU-KVM v2.0.0

● OpenSUSE 13.1

© 2014 Citrix

Applied Micro X-Gene Intel Xeon CPU X5650

ARMv8 64-bit 8 cores 2.4 Ghz x86_64 2 sockets x 6 cores 2.67 Ghz

© 2013 Citrix | Confidential – Do Not Distribute

Kernbenchvirtualization overhead (percentage) - Lower is better

© 2014 Citrix

© 2013 Citrix | Confidential – Do Not Distribute

PBZIP2virtualization overhead (percentage) - Lower is better

© 2014 Citrix

© 2013 Citrix | Confidential – Do Not Distribute

FIO rand-rwvirtualization overhead (percentage) - Lower is better

© 2014 Citrix

© 2013 Citrix | Confidential – Do Not Distribute

SPECjbb2005virtualization overhead (percentage) - Lower is better

© 2014 Citrix

© 2013 Citrix | Confidential – Do Not Distribute

PGBenchvirtualization overhead (percentage) - Lower is better

© 2014 Citrix

© 2013 Citrix | Confidential – Do Not Distribute

PGBenchvirtualization overhead (percentage) - Lower is better

© 2014 Citrix

© 2013 Citrix | Confidential – Do Not Distribute

PGBenchvirtualization overhead (percentage) - Lower is better

© 2014 Citrix

© 2013 Citrix | Confidential – Do Not Distribute

PGBenchvirtualization overhead (percentage) - Lower is better

© 2014 Citrix

120

Benchmarks by

© 2013 Citrix | Confidential – Do Not Distribute

SpecINT 2006Xen on ARM virtualization overhead (percentage) - Lower is better

2%

© 2014 Citrix

Conclusions

© 2013 Citrix | Confidential – Do Not Distribute

● many new features coming● good performance results● still lots of room for improvement

Conclusions

© 2014 Citrix

© 2013 Citrix | Confidential – Do Not Distribute

● need more reviewers

Conclusions

© 2014 Citrix

Questions?

© 2013 Citrix | Confidential – Do Not Distribute

More information

● http://www.xenproject.org

● Xen on ARM @wiki.xenproject.org http://goo.gl/FKNXe

● Xen on ARM whitepaper http://goo.gl/vWMn7D

● http://lists.xenproject.org/mailman/listinfo/xen-devel

© 2014 Citrix

© 2013 Citrix | Confidential – Do Not Distribute

Work better. Live Better.

Citrix

© 2014 Citrix

Slide parking lot

© 2013 Citrix | Confidential – Do Not Distribute

IPerf

Linerate

© 2014 Citrix

© 2013 Citrix | Confidential – Do Not Distribute

FIOXen on ARM virtualization overhead (percentage) - Lower is better

© 2014 Citrix

© 2013 Citrix | Confidential – Do Not Distribute

Reading xen-devel

© 2014 Citrix