31
1 A Few Words on the ARM Ecosystem Thomas Molgaard Director of Software Marketing Systems & Software Group
1
A Few Words on the ARM Ecosystem
Thomas Molgaard Director of Software Marketing
Systems & Software Group
2
About ARM
Global leader in the development of semiconductor IP
R&D outsourcing for semiconductor companies (partners)
Partners utilize ARM IP designs to create and manufacture system-on-chip designs
Innovative business model Upfront licence fee – flexible licensing models
Ongoing royalties – typically based on a percentage of chip price
Technology reused across multiple applications
3
ARM Technology Across Segments Including next generation processors, graphics, video, display and physical IP technologies
4
Why Virtualization on ARM?
Server
•Software management requires efficient virtualization mechanisms •Software maintenance costs dominate the TCO
Automotive
•Safety requiring separation of manufacturer’s code from user’s code •Contemporary user interface experiences
Networking
•Software management and topology configuration •Separation of traffic for different clients
Mobile
•Separation of valuable content from client applications •BYOD support for enterprise clients
Digital Home
•Separation of content for different screens •Separation of system management from user applications
Scalability: The ARM architecture ranges from the smartwatch to the big server, and virtualization is supported across the spectrum.
This opens new possibilities for virtualization, new use cases.
What about stuff nobody has thought of just yet? What about the next crazy idea?
Migrating a VM running on my phone to my workstation or to the cloud…
5
Par
tner
Enab
lem
ent
ARM and Partners Enable the Ecosystem ARM SW Strategy and Ecosystem
Applications
Middleware
Driver & OS
Security
Tools
Models
100,000s
1,000s
100s
10s
10s
A collective effort from ARM and partners
ARM focuses on enabling key building blocks
The partnership ensures the entire ecosystem is enabled
is all about this
AR
M e
nab
lem
ent
6
Applications
Management and Orchestration
Base enabling SW
OpenDataPlane Implementation
Partner I/O and Accelerators
ARM Implementations (A57, GIC-500, MMU-500…) Partner Implementations
Architecture (CPU, SBSA, SMMU, GIC)
I/O, CPU, Memory Virtualization Moving Parts Using Networking as an example
So
ftware
H
ard
ware
ARM
Silicon Partner
ARM + Linaro/ Community
Linaro/ Community
7
ARM is committed to a vibrant ecosystem, full of choice
The Xen Project is a damn good example of this (Welcome, Alibaba!)
Linux on ARM happens in partnership with Linaro This includes virtualization for Xen, KVM, QEMU
Scalability of the ARM architecture opens new possibilities for virtualization, new use cases.
We need the Xen community to help us explore these use cases
A Call to ARMs…
Migrating a VM running on my phone to my workstation or to the cloud… hmmm
Who are you?
Linux kernel hacker for 20+ yearsFirst contribution merged in 1996 (RAID)
With ARM for the past 5 years
Small team of awesome engineers
Focusing on architecture and IP enablementPersonally looking after:
VirtualizationAnything related to interrupts
Co-maintaining KVM/ARM - with Christoffer Dall [Linaro]Allows us to focus on a single projectAlso used as a prototyping vehicle for the architecture
2
Why is Xen important to ARM
We are in the middle of a very diverse ecosystemWide spectrum of requirements coming from partnersCovers all angles of what is possible to do with virtualization
Mono-culture is generally a bad ideaDifferent approaches exercise different aspects of thearchitecture
ARM doesn’t contribute to Xen. Yet.Indirect contributions (from the Linux code base)IOMMU, interrupt controller code
Very good relationship with the Xen/ARM maintainersBeing both located in Cambridge UK helps a lotTechnical issues being hashed out on IRC...
3
Why is Xen important to ARM
We are in the middle of a very diverse ecosystemWide spectrum of requirements coming from partnersCovers all angles of what is possible to do with virtualization
Mono-culture is generally a bad ideaDifferent approaches exercise different aspects of thearchitecture
ARM doesn’t contribute to Xen. Yet.Indirect contributions (from the Linux code base)IOMMU, interrupt controller code
Very good relationship with the Xen/ARM maintainersBeing both located in Cambridge UK helps a lotTechnical issues being hashed out on IRC...... or even better at the local pub
3
The ARM® Architecture: a Rapid EvolutionThe architecture has had a fast paced evolution, while preserving full binary compatibility:
ARMv6 (ARM1136™, ARM1176™,ARM11™MPCore™)
SMPSecurity extensions
ARMv7-A (Cortex™-A5,A8,A9)HW TLB broadcastHW cache coherencyThumb®-2 ISA
ARMv7-A with VE (Cortex-A7,A15,A17)Large Physical Address ExtensionsVirtualization extensionsGeneric timers
ARMv8-A (Cortex-A53,A57,A72)New AArch64 execution modeNew A64 instruction set64bit all the wayFull 32bit compatibility
ARMvx-A is the revision of the architectureCortex-Ayz is an implementationNot only ARM produces implementations of the architecture
4
ARMv8-A Privilege Model
Supports both AArch64 and AArch32 execution modes32-64bit inter-working limited to exception boundariesAArch64 always has a higher privilege than AArch32AArch64 state is a superset of the AArch32 state
5
ARM Architecture Virtualization ExtensionsIntroduced with the latest revision of the ARMv7 architectureNew hypervisor execution state (EL2 or HYP)Non-secure world, higher privilege than EL1Second stage translation
Adds an extra level of indirection between guests and physical memoryA form of nested page tables, as implemented by many other architecturesTLBs are tagged by Virtual Machine ID (VMID)
Ability to trap access to most system registersThe hypervisor decides what it wants to trap
Can handle IRQs, FIQs and asynchronous abortsThe guest doesn’t see physical interrupts firing, for example
Guests can call into HYP mode (HVC instruction)Allows for para-virtualized services
Standard architecture peripherals are virtualization-awareGIC and timers have specific features to help virtualization
6
EL2: Not EL1++EL2 is not a superset of NS-EL1
Actually an orthogonal mode to EL1Allows multiplexing of NS-EL1 guests on the hardware.
Own translation regimeSeparate Stage 1 translation, no Stage 2 translationBroadly follows the LPAE formatOnly one Translation Table Base Register (TTBR0_EL2)
It would be very difficult to run Linux in EL2Requires too many changes to be practical
Instead, the EL2 mode can be used as a “world switch”Between guests (bare metal hypervisor / Type-1)Between host and guests (hosted hypervisor / Type-2)This makes the host a form of specialized guest.
7
Hypervisor architectureA hypervisor is expected to context switch the state of guests:
Stage 2 translation table
Trap configuration
GP registers
System control registers
Floating point
Interrupt configuration
Timers
EL1
EL0
EL2
Guest Kernel
Hypervisor
Guest Kernel Guest Kernel
Guest
Userspace Userspace UserspaceGuest Guest
8
Hypervisor architecture
A hypervisor is expected to context switch the state of guests:
Stage 2 translation table
Trap configuration
GP registers
System control registers
Floating point
Interrupt configuration
Timers
EL1
EL0
EL2
Dom0 Kernel
XEN kernel
DomU Kernel DomU Kernel
Dom0
Userspace Userspace UserspaceDomU DomU
8
Hypervisor architecture
A hypervisor is expected to context switch the state of guests:
Stage 2 translation table
Trap configuration
GP registers
System control registers
Floating point
Interrupt configuration
Timers
EL1
EL0
EL2
KVM
Host Kernel
Switching Code
Guest Kernel Guest Kernel
Host
Userspace Userspace UserspaceGuest Guest
8
VM PreemptionA typical hypervisor forces a guest VM exit on a few key events:
Physical interrupt deliveryAll the guest interrupts are virtualPhysical interrupts must be handled by the host
Stage 2 faultCan be either a translation, permission, or access fault
HVC instructionFor hypercalls (PSCI call for power management, for example)
SMC instructionFor secure mode services
WFI (Wait For Interrupt) instructionWhen the guest is waiting for an interrupt
Blocking WFE (Wait For Event) instructionTo efficiently implement directed yield on blocking spinlock
A few privileged system registers and operations
9
Memory ManagementBasic page size of 4kB, 16kB or 64kB
Sections of 2MB, 32MB, 512MBGuests in total control of their own Stage 1 page tables (VA → IPA)
Any page size supported by the HWJust like on a bare metal systemCan be AArch32 or AArch6440-bit IPA on ARMv7, up to 48-bit on ARMv8No trapping for memory managementNo shadow page tables
The hypervisor controls their Stage 2 translation tables (IPA → PA)All TLBs are tagged by VMID (8 bit wide, up to 256 VMs running at the same time)Requires invalidation on reuse (extremely rare)
Easy to map a simple device into the guest’s “physical” spaceVirtual devices do not have a valid Stage 2 translation
Access will generate a translation faultEmulation done by the hypervisor
10
InterruptsInterrupts are under control of the Generic Interrupt Controller (GIC).
Physical interrupts are always taken by the hypervisorIt is the hypervisor duty to deliver them to the guestsCan be a wired interrupt or a MSI
Hardware assisted virtual interrupt deliveryUsing the virtual GIC extensionsVirtual CPU interface mapped into the guest...... or system register access for GICv3
The hypervisor controls the guest view of injected interruptsDeals with prioritiesCan enable trap-less EOI of HW interrupts from the guest
The GIC revision constrains the size and capability of the VM:GICv2: maximum 8 vCPUs per VMGICv3: zillions of vCPUs per VM, device isolationGICv4: direct injection of MSI
11
About time...The ARM architecture offers a few features time virtualization a bit easier
Global counter, with per exception-level viewAlways on, fixed frequencyAn EL2 viewA physical EL1 view, similar to its EL2 counterpartA virtual EL1 view, from which a per-CPU virtual offset is subtracted
Per-CPU EL2 timerTwo per-CPU EL1 timer
A physical timerA virtual timer, with the same virtual offset
This allows a great deal of flexibilityTime offsetting for migrationSteal time accounting
12
IOMMU
The story wouldn’t be complete without a look at IOMMUs
Two revisions of the architectureSMMUv2SMMUv3
Supports the same page sizes as the CPU
Secure/non-secure devicesNo need for interrupt remapping
GICv3/v4 provides the necessary isolation/indirection
SMMUv3 improvementsLarger number of devices (descriptors in memory)Support for ATS/PRI
13
A necessary evil: Firmware
x86 has a notion of “standard platform”This doesn’t exist on ARM - at all
Each SoC vendor puts together a number of IPs that are specific to their marketNo coordination between SoC vendorsNeed to embrace the diversity of the ecosystemFirmware is crucial to expose platform topology
Device TreeInspired by Open Firmware (and dumbeddown)Mature, robust solutionExtremely flexibleAble to describe the most complicatedplatformLittle standardizationScares x86 people
14
A necessary evil: Firmware
x86 has a notion of “standard platform”This doesn’t exist on ARM - at all
Each SoC vendor puts together a number of IPs that are specific to their marketNo coordination between SoC vendorsNeed to embrace the diversity of the ecosystemFirmware is crucial to expose platform topology
Device TreeInspired by Open Firmware (and dumbeddown)Mature, robust solutionExtremely flexibleAble to describe the most complicatedplatformLittle standardizationScares x86 people
14
A necessary evil: Firmware
x86 has a notion of “standard platform”This doesn’t exist on ARM - at all
Each SoC vendor puts together a number of IPs that are specific to their marketNo coordination between SoC vendorsNeed to embrace the diversity of the ecosystemFirmware is crucial to expose platform topology
Device TreeInspired by Open Firmware (and dumbeddown)Mature, robust solutionExtremely flexibleAble to describe the most complicatedplatformLittle standardizationScares x86 people
ACPIA simplified version of the x86/ia64 thingVery new on ARM (embryonic supportmerged in Linux v4.1)Rather inflexibleAble to describe a more standardized set ofplatforms (SBSA)Committee drivenScares everyone else
14
Beyond ARMv8
The ARM architecture is in constant evolution to better fit the software ecosystem.The upcoming ARMv8.1 revision contains a number of points of interest:
New atomic instructions
New SIMD instructions
Memory management improvements
Performance monitoring improvements
VMIDs expanded to 16 bits
New Virtualization Host extensions, designed for Type-2 hypervisors
Unsurprisingly, this last item is very interesting for KVM
15
ARMv8.1: an Improved EL2
The Virtualization Host (VH) Extensions expand the capabilities of EL2:
Designed to improve the support of Type-2 hypervisors
Allows the host OS to be run at EL2
The host OS requires minimal changes to run at EL2
User-space still runs at EL0
Guest OSes run at EL1
Host has no software running at EL1
AArch64 specific
EL2 becomes a strict superset of EL1
16
ARMv8.1: Impacts on the Linux Kernel
With the VH extensions enabled, the Linux kernel can now be run at EL2.
Changes required for the kernel itself:
Prevent the kernel from switching to EL1
Use HYP timer interrupt instead of the Guest’s
And nothing else
The KVM changes are more invasive:
New method to enter the switch code
New way to address guest’s system registers
Vectors are switched depending on the role (hostkernel or hypervisor)
EL1
EL0
EL2
Guest Kernel Guest Kernel
Host
Userspace Userspace UserspaceGuest Guest
Host Kernel + KVM
The expanded addressing capability could also be useful to Xen!
17
A great architecture for virtualization
The ARM architecture offers a robust set of virtualization featuresNot just about CPU virtualizationCovers the whole system architecture
Scalable architecturePowers IoT-like devices...... all the way to server-grade systems
An open architectureDiversity of implementationsDiversity of solutionsIt is also an architecture in motionConstant evolution (ARMv8.1, with an improved EL2)
A great architecture for XenAnd a great team of Xen hackers to back itKudos to Ian, Stefano and Julien!
18
Thank You
The trademarks featured in this presentation are registered and/or unregistered trademarks of ARM limited (or its subsidiaries) in the EUand/or elsewhere. All rights reserved. All other marks featured may be trademarks of their respective owners.
19
