XPDS13 : PVH Technical Deep Dive - George Dunlap, Citrix

transcript

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

PVH technical deep-dive

George Dunlap

Edinburg – 21-23 October, 2013

Intro: PVH

Edinburg – 21-23 October, 2013 PVH technical deep-dive 2 / 25

I Give you a technical overview of PVH, so that...

I you can understand the characteristics, advantages, anddisadvantages...

I and approach the code to improve and fix

Overview

I PV, HVM, and PVH

I PVH from Xen’s perspective

I PVH from Linux’s perspective

I Issues in PVH

Overview

I PV, HVM, and PVH

I Issues in PVH

Overview

I PV, HVM, and PVH

I Issues in PVH

Overview

I PV, HVM, and PVH

I Issues in PVH

PV and HVM

I Issues with PV

I PVMMU in LinuxI 64-bit hypercalls

I Issues with HVM (PVHVM)

I Qemu processI Legacy bootI Devices emulated in Xen

PV and HVM

I Issues with PVI PVMMU in Linux

I 64-bit hypercalls

PV and HVM

I Issues with PVI PVMMU in LinuxI 64-bit hypercalls

PV and HVM

I Issues with HVM (PVHVM)I Qemu process

I Legacy bootI Devices emulated in Xen

PV and HVM

I Issues with HVM (PVHVM)I Qemu processI Legacy boot

I Devices emulated in Xen

PV and HVM

I Issues with HVM (PVHVM)I Qemu processI Legacy bootI Devices emulated in Xen

I “PV in an HVM container”

I Goal: take the best aspects of PV and HVM

I Written by Mukesh Rathor @ Oracle

I Significant revisions by George Dunlap @ Citrix

PVH from Xen’s perspective

I Begin with HVM guest

I Disable HVM-specific things not needed

I Start & pin in 64-bit paged mode

I Enable PV path for a handful of things

Xen: Things disabled

I Device model (qemu)

I Emulated hardware (APIC, PIT, etc)

I Nested HVM

I MSIX

I Nested HVM

I MSIX

I Nested HVM

I MSIX

I Nested HVM

I MSIX

Xen: Pinning in 64-bit

I Set initial values ofr cr0, cr4, EFER

I Certain paging things have to happen at start of day

I Disable writes to EFER

I Don’t allow guest to change paging-related cr0 bits

Xen: PV paths

I Enable PV hypercalls

I PV e820 map

I PVH VCPU boot

I PV CPUID

I PV PIO

Xen: PV paths

I PV e820 map

I PVH VCPU boot

I PV CPUID

I PV PIO

Xen: PV paths

I PV e820 map

I PVH VCPU boot

I PV CPUID

I PV PIO

Xen: PV paths

I PV e820 map

I PVH VCPU boot

I PV CPUID

I PV PIO

Xen: PV paths

I PV e820 map

I PVH VCPU boot

I PV CPUID

I PV PIO

PVH from Linux’s perspective

I xen hvm domain() false, xen pv domain() true

I Act natural (get rid of PV special cases)

I Autotranslate side effects

I Use PVHVM callback vector setup

I VCPU bring-up

Linux: Things disabled

I No PV IDT

I No PV irq ops

I No PV CPUID

I Native Syscall/sysenter

I No VM assists

I No event / failsafe callbacks

I No need to set IOPL

I No PV IDT

I No PV irq ops

I No PV CPUID

I No VM assists

I No PV IDT

I No PV irq ops

I No PV CPUID

I No VM assists

I No PV IDT

I No PV irq ops

I No PV CPUID

I No VM assists

I No PV IDT

I No PV irq ops

I No PV CPUID

I No VM assists

I No PV IDT

I No PV irq ops

I No PV CPUID

I No VM assists

I No PV IDT

I No PV irq ops

I No PV CPUID

I No VM assists

MMUOPS:

I No need to pin pagetables

I No pfn/mfn conversion

I No need to special-case PT protections

I Only pvmmu op: flush tlb others

MMUOPS:

Linux: Autotranslate

I Pagetables controlled by XenI Real MFN in pagetablesI “Map this page here” hypercall

I Pagetables controlled by guestI gpfn in pagetablesI Every mapped page must be in the p2m

I PVI Pagetables controlled by Xen

I Real MFN in pagetablesI “Map this page here” hypercall

I PVI Pagetables controlled by XenI Real MFN in pagetables

I “Map this page here” hypercall

I PVI Pagetables controlled by XenI Real MFN in pagetablesI “Map this page here” hypercall

I PVHI Pagetables controlled by guest

I gpfn in pagetablesI Every mapped page must be in the p2m

I PVHI Pagetables controlled by guestI gpfn in pagetables

I Every mapped page must be in the p2m

I PVHI Pagetables controlled by guestI gpfn in pagetablesI Every mapped page must be in the p2m

I “Special pages”: grant frame, make a hole in the p2m

I Need to add p2m entries to map foreign pages

I Grant table: similar to HVM (but not the same)

Linux: VCPU bring-up

I Brought up via hypercalls (PV)

I Guest in control of IDT, Xen can’t guarantee loading properly

I Only set GS by default

I All other state must be set in on-cpu boot code

PVH: Not yet working

I 32-bit

I vtsc

I shadow mode

I vcpu hotplug

I 32-bit

I vtsc

I shadow mode

I vcpu hotplug

I 32-bit

I vtsc

I shadow mode

I vcpu hotplug

I 32-bit

I vtsc

I shadow mode

I vcpu hotplug

Issues: HVM paths

I Original idea: “Lightweight” container

I Reality: Large amount of code necessary for minimumfunctionality

I Very large amount of code duplication

I Current patch: use HVM paths, with special cases for PVH

Issues: HVM paths

Issues: PIO instructions

I Direct access permitted

I pv PIT

I cmos idx

I pci config spaceI “Abusing” IO with SMM

I Execute exact instruction with guest GPRs

I pv PIT

I cmos idx

I pv PIT

I cmos idx

I pv PIT

I cmos idx

I pci config space

I “Abusing” IO with SMM

I pv PIT

I cmos idx

I pv PIT

I cmos idx

I Problems

I Ugly changes to allow PV paths to emulate for PVH guestsI Race condition because of double checking

I Possibilities

I Need 2 sets of access controls: For user procs, and for guestI PV only has oneI PVH: Have two, so may not need to emulate anymore

I ProblemsI Ugly changes to allow PV paths to emulate for PVH guests

I Race condition because of double checking

I Possibilities

I ProblemsI Ugly changes to allow PV paths to emulate for PVH guestsI Race condition because of double checking

I Possibilities

I PossibilitiesI Need 2 sets of access controls: For user procs, and for guest

I PV only has oneI PVH: Have two, so may not need to emulate anymore

I PossibilitiesI Need 2 sets of access controls: For user procs, and for guestI PV only has one

I PVH: Have two, so may not need to emulate anymore

I PossibilitiesI Need 2 sets of access controls: For user procs, and for guestI PV only has oneI PVH: Have two, so may not need to emulate anymore

HAP vs Shadow

I Page table update

I HAP: Memory writeI Shadow: Trip through Xen

I TLB effectiveness

I HAP: Superpages allow 2MiB / 1GiB TLB entriesI Shadow: Still only 4k TLB entries

I TLB replacement cost

I Shadow: Worst case, 4 memory readsI HAP: 9-16 memory reads

I Does this matter?

I Kernel build: Shadow 30% slower than HAPI SpecJBB: Shadow 30% faster than HAP

HAP vs Shadow

I Page table updateI HAP: Memory write

I Shadow: Trip through Xen

I TLB effectiveness

I Does this matter?

HAP vs Shadow

I Page table updateI HAP: Memory writeI Shadow: Trip through Xen

I TLB effectiveness

I Does this matter?

HAP vs Shadow

I TLB effectiveness

I Does this matter?

HAP vs Shadow

I TLB effectivenessI HAP: Superpages allow 2MiB / 1GiB TLB entries

I Shadow: Still only 4k TLB entries

I Does this matter?

HAP vs Shadow

I TLB effectivenessI HAP: Superpages allow 2MiB / 1GiB TLB entriesI Shadow: Still only 4k TLB entries

I Does this matter?

HAP vs Shadow

I Does this matter?

HAP vs Shadow

I TLB replacement costI Shadow: Worst case, 4 memory reads

I HAP: 9-16 memory reads

I Does this matter?

HAP vs Shadow

I TLB replacement costI Shadow: Worst case, 4 memory readsI HAP: 9-16 memory reads

I Does this matter?

HAP vs Shadow

I Does this matter?

HAP vs Shadow

I Does this matter?I Kernel build: Shadow 30% slower than HAP

I SpecJBB: Shadow 30% faster than HAP

HAP vs Shadow

I Does this matter?I Kernel build: Shadow 30% slower than HAPI SpecJBB: Shadow 30% faster than HAP

Overview

I PV, HVM, and PVH

I Issues in PVH

Overview

I PV, HVM, and PVH

I Issues in PVH

Overview

I PV, HVM, and PVH

I Issues in PVH

Overview

I PV, HVM, and PVH

I Issues in PVH

Questions

Questions?

Feedback? george.dunlap@citrix.com

Check out our blog: http://blog.xen.org/

Xen: PVH hypercalls

I memory op

I console io

I vcpu op

I mmuext op

Xen: PVH hypercalls

I memory op

I console io

I vcpu op

I mmuext op

Xen: PVH hypercalls

I memory op

I console io

I vcpu op

I mmuext op

Xen: PVH hypercalls

I memory op

I console io

I vcpu op

I mmuext op

XPDS13 : PVH Technical Deep Dive - George Dunlap, Citrix

Technology