Date post: | 19-May-2015 |
Category: |
Technology |
Upload: | the-linux-foundation |
View: | 1,139 times |
Download: | 2 times |
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
PVH technical deep-dive
George Dunlap
Edinburg – 21-23 October, 2013
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Intro: PVH
Edinburg – 21-23 October, 2013 PVH technical deep-dive 2 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Goal
I Give you a technical overview of PVH, so that...
I you can understand the characteristics, advantages, anddisadvantages...
I and approach the code to improve and fix
Edinburg – 21-23 October, 2013 PVH technical deep-dive 3 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Goal
I Give you a technical overview of PVH, so that...
I you can understand the characteristics, advantages, anddisadvantages...
I and approach the code to improve and fix
Edinburg – 21-23 October, 2013 PVH technical deep-dive 3 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Goal
I Give you a technical overview of PVH, so that...
I you can understand the characteristics, advantages, anddisadvantages...
I and approach the code to improve and fix
Edinburg – 21-23 October, 2013 PVH technical deep-dive 3 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Overview
I PV, HVM, and PVH
I PVH from Xen’s perspective
I PVH from Linux’s perspective
I Issues in PVH
Edinburg – 21-23 October, 2013 PVH technical deep-dive 4 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Overview
I PV, HVM, and PVH
I PVH from Xen’s perspective
I PVH from Linux’s perspective
I Issues in PVH
Edinburg – 21-23 October, 2013 PVH technical deep-dive 4 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Overview
I PV, HVM, and PVH
I PVH from Xen’s perspective
I PVH from Linux’s perspective
I Issues in PVH
Edinburg – 21-23 October, 2013 PVH technical deep-dive 4 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Overview
I PV, HVM, and PVH
I PVH from Xen’s perspective
I PVH from Linux’s perspective
I Issues in PVH
Edinburg – 21-23 October, 2013 PVH technical deep-dive 4 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
PV and HVM
I Issues with PV
I PVMMU in LinuxI 64-bit hypercalls
I Issues with HVM (PVHVM)
I Qemu processI Legacy bootI Devices emulated in Xen
Edinburg – 21-23 October, 2013 PVH technical deep-dive 5 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
PV and HVM
I Issues with PVI PVMMU in Linux
I 64-bit hypercalls
I Issues with HVM (PVHVM)
I Qemu processI Legacy bootI Devices emulated in Xen
Edinburg – 21-23 October, 2013 PVH technical deep-dive 5 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
PV and HVM
I Issues with PVI PVMMU in LinuxI 64-bit hypercalls
I Issues with HVM (PVHVM)
I Qemu processI Legacy bootI Devices emulated in Xen
Edinburg – 21-23 October, 2013 PVH technical deep-dive 5 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
PV and HVM
I Issues with PVI PVMMU in LinuxI 64-bit hypercalls
I Issues with HVM (PVHVM)
I Qemu processI Legacy bootI Devices emulated in Xen
Edinburg – 21-23 October, 2013 PVH technical deep-dive 5 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
PV and HVM
I Issues with PVI PVMMU in LinuxI 64-bit hypercalls
I Issues with HVM (PVHVM)I Qemu process
I Legacy bootI Devices emulated in Xen
Edinburg – 21-23 October, 2013 PVH technical deep-dive 5 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
PV and HVM
I Issues with PVI PVMMU in LinuxI 64-bit hypercalls
I Issues with HVM (PVHVM)I Qemu processI Legacy boot
I Devices emulated in Xen
Edinburg – 21-23 October, 2013 PVH technical deep-dive 5 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
PV and HVM
I Issues with PVI PVMMU in LinuxI 64-bit hypercalls
I Issues with HVM (PVHVM)I Qemu processI Legacy bootI Devices emulated in Xen
Edinburg – 21-23 October, 2013 PVH technical deep-dive 5 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
PVH
I “PV in an HVM container”
I Goal: take the best aspects of PV and HVM
I Written by Mukesh Rathor @ Oracle
I Significant revisions by George Dunlap @ Citrix
Edinburg – 21-23 October, 2013 PVH technical deep-dive 6 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
PVH
I “PV in an HVM container”
I Goal: take the best aspects of PV and HVM
I Written by Mukesh Rathor @ Oracle
I Significant revisions by George Dunlap @ Citrix
Edinburg – 21-23 October, 2013 PVH technical deep-dive 6 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
PVH
I “PV in an HVM container”
I Goal: take the best aspects of PV and HVM
I Written by Mukesh Rathor @ Oracle
I Significant revisions by George Dunlap @ Citrix
Edinburg – 21-23 October, 2013 PVH technical deep-dive 6 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
PVH
I “PV in an HVM container”
I Goal: take the best aspects of PV and HVM
I Written by Mukesh Rathor @ Oracle
I Significant revisions by George Dunlap @ Citrix
Edinburg – 21-23 October, 2013 PVH technical deep-dive 6 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
PVH from Xen’s perspective
I Begin with HVM guest
I Disable HVM-specific things not needed
I Start & pin in 64-bit paged mode
I Enable PV path for a handful of things
Edinburg – 21-23 October, 2013 PVH technical deep-dive 7 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
PVH from Xen’s perspective
I Begin with HVM guest
I Disable HVM-specific things not needed
I Start & pin in 64-bit paged mode
I Enable PV path for a handful of things
Edinburg – 21-23 October, 2013 PVH technical deep-dive 7 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
PVH from Xen’s perspective
I Begin with HVM guest
I Disable HVM-specific things not needed
I Start & pin in 64-bit paged mode
I Enable PV path for a handful of things
Edinburg – 21-23 October, 2013 PVH technical deep-dive 7 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
PVH from Xen’s perspective
I Begin with HVM guest
I Disable HVM-specific things not needed
I Start & pin in 64-bit paged mode
I Enable PV path for a handful of things
Edinburg – 21-23 October, 2013 PVH technical deep-dive 7 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Xen: Things disabled
I Device model (qemu)
I Emulated hardware (APIC, PIT, etc)
I Nested HVM
I MSIX
Edinburg – 21-23 October, 2013 PVH technical deep-dive 8 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Xen: Things disabled
I Device model (qemu)
I Emulated hardware (APIC, PIT, etc)
I Nested HVM
I MSIX
Edinburg – 21-23 October, 2013 PVH technical deep-dive 8 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Xen: Things disabled
I Device model (qemu)
I Emulated hardware (APIC, PIT, etc)
I Nested HVM
I MSIX
Edinburg – 21-23 October, 2013 PVH technical deep-dive 8 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Xen: Things disabled
I Device model (qemu)
I Emulated hardware (APIC, PIT, etc)
I Nested HVM
I MSIX
Edinburg – 21-23 October, 2013 PVH technical deep-dive 8 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Xen: Pinning in 64-bit
I Set initial values ofr cr0, cr4, EFER
I Certain paging things have to happen at start of day
I Disable writes to EFER
I Don’t allow guest to change paging-related cr0 bits
Edinburg – 21-23 October, 2013 PVH technical deep-dive 9 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Xen: Pinning in 64-bit
I Set initial values ofr cr0, cr4, EFER
I Certain paging things have to happen at start of day
I Disable writes to EFER
I Don’t allow guest to change paging-related cr0 bits
Edinburg – 21-23 October, 2013 PVH technical deep-dive 9 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Xen: Pinning in 64-bit
I Set initial values ofr cr0, cr4, EFER
I Certain paging things have to happen at start of day
I Disable writes to EFER
I Don’t allow guest to change paging-related cr0 bits
Edinburg – 21-23 October, 2013 PVH technical deep-dive 9 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Xen: Pinning in 64-bit
I Set initial values ofr cr0, cr4, EFER
I Certain paging things have to happen at start of day
I Disable writes to EFER
I Don’t allow guest to change paging-related cr0 bits
Edinburg – 21-23 October, 2013 PVH technical deep-dive 9 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Xen: PV paths
I Enable PV hypercalls
I PV e820 map
I PVH VCPU boot
I PV CPUID
I PV PIO
Edinburg – 21-23 October, 2013 PVH technical deep-dive 10 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Xen: PV paths
I Enable PV hypercalls
I PV e820 map
I PVH VCPU boot
I PV CPUID
I PV PIO
Edinburg – 21-23 October, 2013 PVH technical deep-dive 10 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Xen: PV paths
I Enable PV hypercalls
I PV e820 map
I PVH VCPU boot
I PV CPUID
I PV PIO
Edinburg – 21-23 October, 2013 PVH technical deep-dive 10 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Xen: PV paths
I Enable PV hypercalls
I PV e820 map
I PVH VCPU boot
I PV CPUID
I PV PIO
Edinburg – 21-23 October, 2013 PVH technical deep-dive 10 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Xen: PV paths
I Enable PV hypercalls
I PV e820 map
I PVH VCPU boot
I PV CPUID
I PV PIO
Edinburg – 21-23 October, 2013 PVH technical deep-dive 10 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
PVH from Linux’s perspective
I xen hvm domain() false, xen pv domain() true
I Act natural (get rid of PV special cases)
I Autotranslate side effects
I Use PVHVM callback vector setup
I VCPU bring-up
Edinburg – 21-23 October, 2013 PVH technical deep-dive 11 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
PVH from Linux’s perspective
I xen hvm domain() false, xen pv domain() true
I Act natural (get rid of PV special cases)
I Autotranslate side effects
I Use PVHVM callback vector setup
I VCPU bring-up
Edinburg – 21-23 October, 2013 PVH technical deep-dive 11 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
PVH from Linux’s perspective
I xen hvm domain() false, xen pv domain() true
I Act natural (get rid of PV special cases)
I Autotranslate side effects
I Use PVHVM callback vector setup
I VCPU bring-up
Edinburg – 21-23 October, 2013 PVH technical deep-dive 11 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
PVH from Linux’s perspective
I xen hvm domain() false, xen pv domain() true
I Act natural (get rid of PV special cases)
I Autotranslate side effects
I Use PVHVM callback vector setup
I VCPU bring-up
Edinburg – 21-23 October, 2013 PVH technical deep-dive 11 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
PVH from Linux’s perspective
I xen hvm domain() false, xen pv domain() true
I Act natural (get rid of PV special cases)
I Autotranslate side effects
I Use PVHVM callback vector setup
I VCPU bring-up
Edinburg – 21-23 October, 2013 PVH technical deep-dive 11 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Linux: Things disabled
I No PV IDT
I No PV irq ops
I No PV CPUID
I Native Syscall/sysenter
I No VM assists
I No event / failsafe callbacks
I No need to set IOPL
Edinburg – 21-23 October, 2013 PVH technical deep-dive 12 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Linux: Things disabled
I No PV IDT
I No PV irq ops
I No PV CPUID
I Native Syscall/sysenter
I No VM assists
I No event / failsafe callbacks
I No need to set IOPL
Edinburg – 21-23 October, 2013 PVH technical deep-dive 12 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Linux: Things disabled
I No PV IDT
I No PV irq ops
I No PV CPUID
I Native Syscall/sysenter
I No VM assists
I No event / failsafe callbacks
I No need to set IOPL
Edinburg – 21-23 October, 2013 PVH technical deep-dive 12 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Linux: Things disabled
I No PV IDT
I No PV irq ops
I No PV CPUID
I Native Syscall/sysenter
I No VM assists
I No event / failsafe callbacks
I No need to set IOPL
Edinburg – 21-23 October, 2013 PVH technical deep-dive 12 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Linux: Things disabled
I No PV IDT
I No PV irq ops
I No PV CPUID
I Native Syscall/sysenter
I No VM assists
I No event / failsafe callbacks
I No need to set IOPL
Edinburg – 21-23 October, 2013 PVH technical deep-dive 12 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Linux: Things disabled
I No PV IDT
I No PV irq ops
I No PV CPUID
I Native Syscall/sysenter
I No VM assists
I No event / failsafe callbacks
I No need to set IOPL
Edinburg – 21-23 October, 2013 PVH technical deep-dive 12 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Linux: Things disabled
I No PV IDT
I No PV irq ops
I No PV CPUID
I Native Syscall/sysenter
I No VM assists
I No event / failsafe callbacks
I No need to set IOPL
Edinburg – 21-23 October, 2013 PVH technical deep-dive 12 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Linux: Things disabled
MMUOPS:
I No need to pin pagetables
I No pfn/mfn conversion
I No need to special-case PT protections
I Only pvmmu op: flush tlb others
Edinburg – 21-23 October, 2013 PVH technical deep-dive 13 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Linux: Things disabled
MMUOPS:
I No need to pin pagetables
I No pfn/mfn conversion
I No need to special-case PT protections
I Only pvmmu op: flush tlb others
Edinburg – 21-23 October, 2013 PVH technical deep-dive 13 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Linux: Things disabled
MMUOPS:
I No need to pin pagetables
I No pfn/mfn conversion
I No need to special-case PT protections
I Only pvmmu op: flush tlb others
Edinburg – 21-23 October, 2013 PVH technical deep-dive 13 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Linux: Things disabled
MMUOPS:
I No need to pin pagetables
I No pfn/mfn conversion
I No need to special-case PT protections
I Only pvmmu op: flush tlb others
Edinburg – 21-23 October, 2013 PVH technical deep-dive 13 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Linux: Autotranslate
I PV
I Pagetables controlled by XenI Real MFN in pagetablesI “Map this page here” hypercall
I PVH
I Pagetables controlled by guestI gpfn in pagetablesI Every mapped page must be in the p2m
Edinburg – 21-23 October, 2013 PVH technical deep-dive 14 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Linux: Autotranslate
I PVI Pagetables controlled by Xen
I Real MFN in pagetablesI “Map this page here” hypercall
I PVH
I Pagetables controlled by guestI gpfn in pagetablesI Every mapped page must be in the p2m
Edinburg – 21-23 October, 2013 PVH technical deep-dive 14 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Linux: Autotranslate
I PVI Pagetables controlled by XenI Real MFN in pagetables
I “Map this page here” hypercall
I PVH
I Pagetables controlled by guestI gpfn in pagetablesI Every mapped page must be in the p2m
Edinburg – 21-23 October, 2013 PVH technical deep-dive 14 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Linux: Autotranslate
I PVI Pagetables controlled by XenI Real MFN in pagetablesI “Map this page here” hypercall
I PVH
I Pagetables controlled by guestI gpfn in pagetablesI Every mapped page must be in the p2m
Edinburg – 21-23 October, 2013 PVH technical deep-dive 14 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Linux: Autotranslate
I PVI Pagetables controlled by XenI Real MFN in pagetablesI “Map this page here” hypercall
I PVH
I Pagetables controlled by guestI gpfn in pagetablesI Every mapped page must be in the p2m
Edinburg – 21-23 October, 2013 PVH technical deep-dive 14 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Linux: Autotranslate
I PVI Pagetables controlled by XenI Real MFN in pagetablesI “Map this page here” hypercall
I PVHI Pagetables controlled by guest
I gpfn in pagetablesI Every mapped page must be in the p2m
Edinburg – 21-23 October, 2013 PVH technical deep-dive 14 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Linux: Autotranslate
I PVI Pagetables controlled by XenI Real MFN in pagetablesI “Map this page here” hypercall
I PVHI Pagetables controlled by guestI gpfn in pagetables
I Every mapped page must be in the p2m
Edinburg – 21-23 October, 2013 PVH technical deep-dive 14 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Linux: Autotranslate
I PVI Pagetables controlled by XenI Real MFN in pagetablesI “Map this page here” hypercall
I PVHI Pagetables controlled by guestI gpfn in pagetablesI Every mapped page must be in the p2m
Edinburg – 21-23 October, 2013 PVH technical deep-dive 14 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Linux: Autotranslate
I “Special pages”: grant frame, make a hole in the p2m
I Need to add p2m entries to map foreign pages
I Grant table: similar to HVM (but not the same)
Edinburg – 21-23 October, 2013 PVH technical deep-dive 15 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Linux: Autotranslate
I “Special pages”: grant frame, make a hole in the p2m
I Need to add p2m entries to map foreign pages
I Grant table: similar to HVM (but not the same)
Edinburg – 21-23 October, 2013 PVH technical deep-dive 15 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Linux: Autotranslate
I “Special pages”: grant frame, make a hole in the p2m
I Need to add p2m entries to map foreign pages
I Grant table: similar to HVM (but not the same)
Edinburg – 21-23 October, 2013 PVH technical deep-dive 15 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Linux: VCPU bring-up
I Brought up via hypercalls (PV)
I Guest in control of IDT, Xen can’t guarantee loading properly
I Only set GS by default
I All other state must be set in on-cpu boot code
Edinburg – 21-23 October, 2013 PVH technical deep-dive 16 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Linux: VCPU bring-up
I Brought up via hypercalls (PV)
I Guest in control of IDT, Xen can’t guarantee loading properly
I Only set GS by default
I All other state must be set in on-cpu boot code
Edinburg – 21-23 October, 2013 PVH technical deep-dive 16 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Linux: VCPU bring-up
I Brought up via hypercalls (PV)
I Guest in control of IDT, Xen can’t guarantee loading properly
I Only set GS by default
I All other state must be set in on-cpu boot code
Edinburg – 21-23 October, 2013 PVH technical deep-dive 16 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Linux: VCPU bring-up
I Brought up via hypercalls (PV)
I Guest in control of IDT, Xen can’t guarantee loading properly
I Only set GS by default
I All other state must be set in on-cpu boot code
Edinburg – 21-23 October, 2013 PVH technical deep-dive 16 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
PVH: Not yet working
I 32-bit
I vtsc
I shadow mode
I vcpu hotplug
Edinburg – 21-23 October, 2013 PVH technical deep-dive 17 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
PVH: Not yet working
I 32-bit
I vtsc
I shadow mode
I vcpu hotplug
Edinburg – 21-23 October, 2013 PVH technical deep-dive 17 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
PVH: Not yet working
I 32-bit
I vtsc
I shadow mode
I vcpu hotplug
Edinburg – 21-23 October, 2013 PVH technical deep-dive 17 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
PVH: Not yet working
I 32-bit
I vtsc
I shadow mode
I vcpu hotplug
Edinburg – 21-23 October, 2013 PVH technical deep-dive 17 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Issues: HVM paths
I Original idea: “Lightweight” container
I Reality: Large amount of code necessary for minimumfunctionality
I Very large amount of code duplication
I Current patch: use HVM paths, with special cases for PVH
Edinburg – 21-23 October, 2013 PVH technical deep-dive 18 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Issues: HVM paths
I Original idea: “Lightweight” container
I Reality: Large amount of code necessary for minimumfunctionality
I Very large amount of code duplication
I Current patch: use HVM paths, with special cases for PVH
Edinburg – 21-23 October, 2013 PVH technical deep-dive 18 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Issues: HVM paths
I Original idea: “Lightweight” container
I Reality: Large amount of code necessary for minimumfunctionality
I Very large amount of code duplication
I Current patch: use HVM paths, with special cases for PVH
Edinburg – 21-23 October, 2013 PVH technical deep-dive 18 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Issues: HVM paths
I Original idea: “Lightweight” container
I Reality: Large amount of code necessary for minimumfunctionality
I Very large amount of code duplication
I Current patch: use HVM paths, with special cases for PVH
Edinburg – 21-23 October, 2013 PVH technical deep-dive 18 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Issues: PIO instructions
I Direct access permitted
I pv PIT
I cmos idx
I pci config spaceI “Abusing” IO with SMM
I Execute exact instruction with guest GPRs
Edinburg – 21-23 October, 2013 PVH technical deep-dive 19 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Issues: PIO instructions
I Direct access permitted
I pv PIT
I cmos idx
I pci config spaceI “Abusing” IO with SMM
I Execute exact instruction with guest GPRs
Edinburg – 21-23 October, 2013 PVH technical deep-dive 19 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Issues: PIO instructions
I Direct access permitted
I pv PIT
I cmos idx
I pci config spaceI “Abusing” IO with SMM
I Execute exact instruction with guest GPRs
Edinburg – 21-23 October, 2013 PVH technical deep-dive 19 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Issues: PIO instructions
I Direct access permitted
I pv PIT
I cmos idx
I pci config space
I “Abusing” IO with SMM
I Execute exact instruction with guest GPRs
Edinburg – 21-23 October, 2013 PVH technical deep-dive 19 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Issues: PIO instructions
I Direct access permitted
I pv PIT
I cmos idx
I pci config spaceI “Abusing” IO with SMM
I Execute exact instruction with guest GPRs
Edinburg – 21-23 October, 2013 PVH technical deep-dive 19 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Issues: PIO instructions
I Direct access permitted
I pv PIT
I cmos idx
I pci config spaceI “Abusing” IO with SMM
I Execute exact instruction with guest GPRs
Edinburg – 21-23 October, 2013 PVH technical deep-dive 19 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Issues: PIO instructions
I Problems
I Ugly changes to allow PV paths to emulate for PVH guestsI Race condition because of double checking
I Possibilities
I Need 2 sets of access controls: For user procs, and for guestI PV only has oneI PVH: Have two, so may not need to emulate anymore
Edinburg – 21-23 October, 2013 PVH technical deep-dive 20 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Issues: PIO instructions
I ProblemsI Ugly changes to allow PV paths to emulate for PVH guests
I Race condition because of double checking
I Possibilities
I Need 2 sets of access controls: For user procs, and for guestI PV only has oneI PVH: Have two, so may not need to emulate anymore
Edinburg – 21-23 October, 2013 PVH technical deep-dive 20 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Issues: PIO instructions
I ProblemsI Ugly changes to allow PV paths to emulate for PVH guestsI Race condition because of double checking
I Possibilities
I Need 2 sets of access controls: For user procs, and for guestI PV only has oneI PVH: Have two, so may not need to emulate anymore
Edinburg – 21-23 October, 2013 PVH technical deep-dive 20 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Issues: PIO instructions
I ProblemsI Ugly changes to allow PV paths to emulate for PVH guestsI Race condition because of double checking
I Possibilities
I Need 2 sets of access controls: For user procs, and for guestI PV only has oneI PVH: Have two, so may not need to emulate anymore
Edinburg – 21-23 October, 2013 PVH technical deep-dive 20 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Issues: PIO instructions
I ProblemsI Ugly changes to allow PV paths to emulate for PVH guestsI Race condition because of double checking
I PossibilitiesI Need 2 sets of access controls: For user procs, and for guest
I PV only has oneI PVH: Have two, so may not need to emulate anymore
Edinburg – 21-23 October, 2013 PVH technical deep-dive 20 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Issues: PIO instructions
I ProblemsI Ugly changes to allow PV paths to emulate for PVH guestsI Race condition because of double checking
I PossibilitiesI Need 2 sets of access controls: For user procs, and for guestI PV only has one
I PVH: Have two, so may not need to emulate anymore
Edinburg – 21-23 October, 2013 PVH technical deep-dive 20 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Issues: PIO instructions
I ProblemsI Ugly changes to allow PV paths to emulate for PVH guestsI Race condition because of double checking
I PossibilitiesI Need 2 sets of access controls: For user procs, and for guestI PV only has oneI PVH: Have two, so may not need to emulate anymore
Edinburg – 21-23 October, 2013 PVH technical deep-dive 20 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
HAP vs Shadow
I Page table update
I HAP: Memory writeI Shadow: Trip through Xen
I TLB effectiveness
I HAP: Superpages allow 2MiB / 1GiB TLB entriesI Shadow: Still only 4k TLB entries
I TLB replacement cost
I Shadow: Worst case, 4 memory readsI HAP: 9-16 memory reads
I Does this matter?
I Kernel build: Shadow 30% slower than HAPI SpecJBB: Shadow 30% faster than HAP
Edinburg – 21-23 October, 2013 PVH technical deep-dive 21 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
HAP vs Shadow
I Page table updateI HAP: Memory write
I Shadow: Trip through Xen
I TLB effectiveness
I HAP: Superpages allow 2MiB / 1GiB TLB entriesI Shadow: Still only 4k TLB entries
I TLB replacement cost
I Shadow: Worst case, 4 memory readsI HAP: 9-16 memory reads
I Does this matter?
I Kernel build: Shadow 30% slower than HAPI SpecJBB: Shadow 30% faster than HAP
Edinburg – 21-23 October, 2013 PVH technical deep-dive 21 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
HAP vs Shadow
I Page table updateI HAP: Memory writeI Shadow: Trip through Xen
I TLB effectiveness
I HAP: Superpages allow 2MiB / 1GiB TLB entriesI Shadow: Still only 4k TLB entries
I TLB replacement cost
I Shadow: Worst case, 4 memory readsI HAP: 9-16 memory reads
I Does this matter?
I Kernel build: Shadow 30% slower than HAPI SpecJBB: Shadow 30% faster than HAP
Edinburg – 21-23 October, 2013 PVH technical deep-dive 21 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
HAP vs Shadow
I Page table updateI HAP: Memory writeI Shadow: Trip through Xen
I TLB effectiveness
I HAP: Superpages allow 2MiB / 1GiB TLB entriesI Shadow: Still only 4k TLB entries
I TLB replacement cost
I Shadow: Worst case, 4 memory readsI HAP: 9-16 memory reads
I Does this matter?
I Kernel build: Shadow 30% slower than HAPI SpecJBB: Shadow 30% faster than HAP
Edinburg – 21-23 October, 2013 PVH technical deep-dive 21 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
HAP vs Shadow
I Page table updateI HAP: Memory writeI Shadow: Trip through Xen
I TLB effectivenessI HAP: Superpages allow 2MiB / 1GiB TLB entries
I Shadow: Still only 4k TLB entries
I TLB replacement cost
I Shadow: Worst case, 4 memory readsI HAP: 9-16 memory reads
I Does this matter?
I Kernel build: Shadow 30% slower than HAPI SpecJBB: Shadow 30% faster than HAP
Edinburg – 21-23 October, 2013 PVH technical deep-dive 21 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
HAP vs Shadow
I Page table updateI HAP: Memory writeI Shadow: Trip through Xen
I TLB effectivenessI HAP: Superpages allow 2MiB / 1GiB TLB entriesI Shadow: Still only 4k TLB entries
I TLB replacement cost
I Shadow: Worst case, 4 memory readsI HAP: 9-16 memory reads
I Does this matter?
I Kernel build: Shadow 30% slower than HAPI SpecJBB: Shadow 30% faster than HAP
Edinburg – 21-23 October, 2013 PVH technical deep-dive 21 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
HAP vs Shadow
I Page table updateI HAP: Memory writeI Shadow: Trip through Xen
I TLB effectivenessI HAP: Superpages allow 2MiB / 1GiB TLB entriesI Shadow: Still only 4k TLB entries
I TLB replacement cost
I Shadow: Worst case, 4 memory readsI HAP: 9-16 memory reads
I Does this matter?
I Kernel build: Shadow 30% slower than HAPI SpecJBB: Shadow 30% faster than HAP
Edinburg – 21-23 October, 2013 PVH technical deep-dive 21 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
HAP vs Shadow
I Page table updateI HAP: Memory writeI Shadow: Trip through Xen
I TLB effectivenessI HAP: Superpages allow 2MiB / 1GiB TLB entriesI Shadow: Still only 4k TLB entries
I TLB replacement costI Shadow: Worst case, 4 memory reads
I HAP: 9-16 memory reads
I Does this matter?
I Kernel build: Shadow 30% slower than HAPI SpecJBB: Shadow 30% faster than HAP
Edinburg – 21-23 October, 2013 PVH technical deep-dive 21 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
HAP vs Shadow
I Page table updateI HAP: Memory writeI Shadow: Trip through Xen
I TLB effectivenessI HAP: Superpages allow 2MiB / 1GiB TLB entriesI Shadow: Still only 4k TLB entries
I TLB replacement costI Shadow: Worst case, 4 memory readsI HAP: 9-16 memory reads
I Does this matter?
I Kernel build: Shadow 30% slower than HAPI SpecJBB: Shadow 30% faster than HAP
Edinburg – 21-23 October, 2013 PVH technical deep-dive 21 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
HAP vs Shadow
I Page table updateI HAP: Memory writeI Shadow: Trip through Xen
I TLB effectivenessI HAP: Superpages allow 2MiB / 1GiB TLB entriesI Shadow: Still only 4k TLB entries
I TLB replacement costI Shadow: Worst case, 4 memory readsI HAP: 9-16 memory reads
I Does this matter?
I Kernel build: Shadow 30% slower than HAPI SpecJBB: Shadow 30% faster than HAP
Edinburg – 21-23 October, 2013 PVH technical deep-dive 21 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
HAP vs Shadow
I Page table updateI HAP: Memory writeI Shadow: Trip through Xen
I TLB effectivenessI HAP: Superpages allow 2MiB / 1GiB TLB entriesI Shadow: Still only 4k TLB entries
I TLB replacement costI Shadow: Worst case, 4 memory readsI HAP: 9-16 memory reads
I Does this matter?I Kernel build: Shadow 30% slower than HAP
I SpecJBB: Shadow 30% faster than HAP
Edinburg – 21-23 October, 2013 PVH technical deep-dive 21 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
HAP vs Shadow
I Page table updateI HAP: Memory writeI Shadow: Trip through Xen
I TLB effectivenessI HAP: Superpages allow 2MiB / 1GiB TLB entriesI Shadow: Still only 4k TLB entries
I TLB replacement costI Shadow: Worst case, 4 memory readsI HAP: 9-16 memory reads
I Does this matter?I Kernel build: Shadow 30% slower than HAPI SpecJBB: Shadow 30% faster than HAP
Edinburg – 21-23 October, 2013 PVH technical deep-dive 21 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Overview
I PV, HVM, and PVH
I PVH from Xen’s perspective
I PVH from Linux’s perspective
I Issues in PVH
Edinburg – 21-23 October, 2013 PVH technical deep-dive 22 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Overview
I PV, HVM, and PVH
I PVH from Xen’s perspective
I PVH from Linux’s perspective
I Issues in PVH
Edinburg – 21-23 October, 2013 PVH technical deep-dive 22 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Overview
I PV, HVM, and PVH
I PVH from Xen’s perspective
I PVH from Linux’s perspective
I Issues in PVH
Edinburg – 21-23 October, 2013 PVH technical deep-dive 22 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Overview
I PV, HVM, and PVH
I PVH from Xen’s perspective
I PVH from Linux’s perspective
I Issues in PVH
Edinburg – 21-23 October, 2013 PVH technical deep-dive 22 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Goal
I Give you a technical overview of PVH, so that...
I you can understand the characteristics, advantages, anddisadvantages...
I and approach the code to improve and fix
Edinburg – 21-23 October, 2013 PVH technical deep-dive 23 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Goal
I Give you a technical overview of PVH, so that...
I you can understand the characteristics, advantages, anddisadvantages...
I and approach the code to improve and fix
Edinburg – 21-23 October, 2013 PVH technical deep-dive 23 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Goal
I Give you a technical overview of PVH, so that...
I you can understand the characteristics, advantages, anddisadvantages...
I and approach the code to improve and fix
Edinburg – 21-23 October, 2013 PVH technical deep-dive 23 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Questions
Questions?
Feedback? [email protected]
Check out our blog: http://blog.xen.org/
Edinburg – 21-23 October, 2013 PVH technical deep-dive 24 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Xen: PVH hypercalls
I memory op
I console io
I vcpu op
I mmuext op
Edinburg – 21-23 October, 2013 PVH technical deep-dive 25 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Xen: PVH hypercalls
I memory op
I console io
I vcpu op
I mmuext op
Edinburg – 21-23 October, 2013 PVH technical deep-dive 25 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Xen: PVH hypercalls
I memory op
I console io
I vcpu op
I mmuext op
Edinburg – 21-23 October, 2013 PVH technical deep-dive 25 / 25
Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides
Xen: PVH hypercalls
I memory op
I console io
I vcpu op
I mmuext op
Edinburg – 21-23 October, 2013 PVH technical deep-dive 25 / 25