Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

MySQL and VirtualizationPercona Live MySQL Conference and Expo 2013

Raghavendra Prabhuraghavendra.prabhu@percona.com

Percona

23rd April, 2013

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

Outline

Introduction

Virtualization

MySQL

Symbiosis

Instrumentation/Debugging

Instrumentation II

Application

Conclusion

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

Overview

I WhyI Why notI Performance v/s EfficiencyI Overcommit!I High AvailabilityI Testing/Instrumentation

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

Popular solutions

I Xen, KVM, VirtualBoxI Why KVM

I Massive reuse of kernel and userspaceI Compatibility in case of para-virt

I Most applies to Xen as wellI Important difference

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

Native against Emulation

I Emulation: TCGI Valgrind

I KVMI Kernel module

I VT-x (Intel), AMD-VI Guest - Host context

I Exit - Enter latencyLazy approach

I Para-virtualization

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

Development

I QEMUI Most active

I In-kernelI Native client

I More hypervisorsI KVM API - ioctl /dev/kvm

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

Considerations

I PerformanceI Scaling - Horizontal or VerticalI Eggs and the BasketI Moore’s law

I CostI Saturation

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

Memory

I KSMI How it works and is usedI How it can be used: MADV_MERGEABLE

I Memory Ballooning: VirtioI OvercommitI Memory statistics: QOM

I CachingI Affects I/OI Write{back,through}, none, directsync and unsafe

I Snapshot mode & commit: cache=unsafe

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

Memory III KSM for MySQL

I Multiple buffer poolsI Shared storage - VirtFS

I With NUMAI Pinning + KSM = !

I ksmtuned and numadI Ballooning: In host’s shoes

I WhereI Migration & Upgrade - Hot/Cold

I With QGAI Feedback from MySQL: Para-virtualized process!

I CacheI InnoDB + cache=none = true O_DIRECT: for NFSI =writethrough: app fails to grok fsyncI =writeback: speed and benefits of none

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

CPU

I Processor pinningI Cgroups: Better

I Applies to othersI Dynamic

I HotplugI Overcommitting - smp

I Context switches

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

NetworkI Types

I VDEI TAP and BridgeI OpenvswitchI DumpI Multicast with UDP

I PerformanceI virtio-net

I Para-virtualized constructI Multi-queue

I vhost-netI Zero-copy transmit: DMAI No Qemu: kernel and KVMI Not just network

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

Security

I MasqueradingI Isolation

I Namespaces

I Random NumbersI Why?I Hardware / Virtio RNG

I Resource exhaustion: DDoSI OOM: Cgroups

I Seccomp sandbox: Safety is only an illusion!I Guest to host vulnerabilities

I Encryption

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

I/O and StorageI The weakest linkI Virtio

I virtio-blk-pci data planeI Separate I/O thread

I PCI PassthroughI PCI devices in generalI IOMMU (AMD) & VT-d (Intel)I SR-IOV: multiple devices without having

I VFIO: in userspaceI No longer exclusiveI More secure: very importantI Interrupts closer to guest

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

I/O and Storage III VirtFS

I Filesystem passthroughI 9p protocol

I PossibilitiesI Fragmentation

I How: Hole punchingI Fallocate, pre-allocation Qcow

I Thin provisioningI Qcow lazy provisioningI Device mapper target

I Throttling: Errant guests!block_set_io_throttle

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

I/O and Storage III

I BackendsI RawI QcowI Block device attachment

I Performance close to bare metalI GlusterFSI Sheepdog

I Distributed StorageI iSCSII NBD

I Networked storageI Shared disk image

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

I/O and Storage IVI Scheduler

I DeadlineI CFQ - quantum and slice

I DiscardI Pass discard requests to host filesystemI Fully async

I AIOI pthread v/s native

I SwapI SwappinessI Less preferred to Ballooning and KSMI Poor with EPT

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

Others

I Transparent Huge PagesI Issues?

I Possible fixes - MADV_NOHUGEPAGEI How can it help

I ivshmemI Fastest message passing

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

Reliability

I What everyone wants: Uninterrupted Sleep - ‘D’ stateI What they usually get: Nagios alertsI Its like InsuranceI What can be doneI Backups

I Backup the original imageI Blockdev snapshotI guestfs-fsfreeze-{freeze,thaw} and FTWRL

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

Reliability III High Availability

I System uptime v/s Application uptimeI Difference

I MigrationI Live migration: RequirementI Online block streaming/commit

I Drive MirroringI Live block copy

I Snapshots

I VM Snapshots: complete state

I Qcow2 snapshotsRaghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

Upgrades

I Downtime

I Ballooning the buffer pool

I Migration

I Preserves connectionsI OpenvSwitch

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

TestingI Cluster

I PXC

I TopologiesI Army of nodesI Forest: Cluster of clusters

Data centers in microcosm

I Memory

I NUMA

I Page cache and Memory Pressure

I Dump guest memory: coreI Examine online

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

Fault injection

I NMI and MCE

I Pulling the power cord!

I Filesystem

I Interesting cases

I Ext4I XFS

I PCI AER

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

Others

I Jenkins and Vagrant

I Watchdog

I “Houston, we have a problem”

I With MySQL: Turtles all the way

I systemd

I ACPI tables

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

perf-kvm

I perf toolsuite

I Strace?

I Caveats

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

GDB monitor

I Remote Target and Single Stepping

I Kernel development

I bzImage and initrd

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

Qemu Monitor (QMP/HMP)

I Ballooning

I Throttle

I PCI Hot-add

I Examining memory

I JSON API

I Endless possibilities with scriptingI Python scripts in repo

I Info commands

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

TracingI Ftrace

I LatencyI Virtio I/O

I Dtrace

I With systemtap

I LTTng

I Simple

I Stderr

I Dprintf

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

Development

I QCOW

I Backing file hierarchy

I block stream

I Advantages

I Copy on read

I Lazy Refcounts

I Encryption and Compression

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

Development III Caveats

I Raw v/s Qcow: Performance

I Rendering Qcow unusable

I Filesystem passthrough

I Share datadir

I Common code-tree / development directories

I Pass the entire root R/O!I Lightweight SnapshotI Useful to have newer FHS with /usr

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

WrappersI Scripting

I libvirt

I virsh / virt-manager

I Openstack

I Proxmox

I Candy

I Spice

I VNC

I oVirtI VDSM

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

Thoughts?

I Questions

I Usage and Experience

Raghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

AppendixI Best practices: http://goo.gl/LY3tf

I Virtio Introduction: http://goo.gl/B9bCA

I KVM Kernel API: http://goo.gl/5A4Ma

I linux-kvm.org: http://goo.gl/yWPiU

I Internals blog: http://goo.gl/z07cy

I Qemu wiki: http://goo.gl/IGoKa

I RedHat Virtualization docs: http://goo.gl/rXRKc

I Docs from Qemu git tree: http://goo.gl/qD7nj

I KSM internals: http://goo.gl/ypyhkRaghavendra Prabhu Percona

MySQL Virtualization

Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion

Me

I Raghavendra Prabhu

I http://wnohang.net/about

I Software Engineer at Percona and Product Lead of PXC

I https://launchpad.net/percona-xtradb-cluster

I Slides downloadable at http://goo.gl/jsult

Raghavendra Prabhu Percona

MySQL Virtualization