Netbus: A Transparent Mechanism for Remote Netbus: A Transparent Mechanism for Remote Device Access in Virtualized SystemsDevice Access in Virtualized Systems
Sanjay KumarSanjay KumarPhD StudentPhD Student
Advisor:Advisor:
Prof. Karsten SchwanProf. Karsten Schwan
Remote Device AccessRemote Device Access
Transparent access to remote devices is becoming Transparent access to remote devices is becoming essential in various computing environmentsessential in various computing environments– datacenters, clusters and even in personal computingdatacenters, clusters and even in personal computing– Remote disks, backup drives, movie on a thin client etc.Remote disks, backup drives, movie on a thin client etc.
Essential in virtualized environment for VM Essential in virtualized environment for VM migrationmigration– Easy for networked devicesEasy for networked devices
iSCSI, NAS, SANiSCSI, NAS, SANBut what about non-networked devices?But what about non-networked devices?
– NDB, DRBD etc. for block devicesNDB, DRBD etc. for block devicesNo support during VM migrationNo support during VM migration
Netbus: Provides transparent and generic access Netbus: Provides transparent and generic access to remote devices to remote devices – Virtual device migration and device hot-swappingVirtual device migration and device hot-swapping
Network
Network
Backup DVD-drive
IPod on networked dockingstation
Blade-Server Blade-Server
Netbus Software ArchitectureNetbus Software Architecture
Targets high-bandwidth, low-latency and reliable Targets high-bandwidth, low-latency and reliable networks with single administrative domainnetworks with single administrative domain– Datacenters, blade-servers, clusters, home and office LANDatacenters, blade-servers, clusters, home and office LAN
Similar to channels in pub/sub systemsSimilar to channels in pub/sub systems– Server exports the device (channel), Client connects (subscribe) Server exports the device (channel), Client connects (subscribe)
to itto it
Logical extension of frontend-backend approach to Logical extension of frontend-backend approach to device virtualizationdevice virtualization– Frontend can communicate with a remote backendFrontend can communicate with a remote backend– Application layer client establishes the connection with serverApplication layer client establishes the connection with server
The fast path is inside kernelThe fast path is inside kernel– Abstraction of a Abstraction of a network busnetwork bus
Generic mechanism with device specific callback Generic mechanism with device specific callback functionsfunctions– Common Netbus header followed by device specific dataCommon Netbus header followed by device specific data– Data passed to device specific callbacks for further processingData passed to device specific callbacks for further processing
Netbus: Software Architecture (contd.)Netbus: Software Architecture (contd.)
Hypervisor
Service VM ServiceVM
Hypervisor
GuestVM
L-BEFE
vdevice device
R-BE
device driver
Local Machine Remote Machine
Netbus
Network
Client
Add remote Dev. to VM
Server
Hypervisor
Service VMGuestVM
BEFE
vdevice
device driver
Local Machine
Add Deviceto VM
device
User
Kernel
Netbus: Software Architecture (contd.)Netbus: Software Architecture (contd.)
Virtual Device MigrationVirtual Device Migration– Migrate virtual devices along with VMsMigrate virtual devices along with VMs– Provides continuous access to VM’s devices after VM Provides continuous access to VM’s devices after VM
migration through Netbusmigration through Netbus– How to deal with pending I/O operations?How to deal with pending I/O operations?
Bring the device into a quiescent state before Bring the device into a quiescent state before migrationmigration– No pending I/O operationsNo pending I/O operations– I/O operations get queued into the FE and complete I/O operations get queued into the FE and complete
after VM migrationafter VM migration
Hypervisor
Service VM ServiceVM
Hypervisor
GuestVM
L-BEFE
vdevice device
R-BE
device driver
Local Machine Remote Machine
Netbus
Network
Client
Add remote Dev. to VM
Server User
Kernel
GuestVM
FE
device driver
vdevice
VM Migration
Virtual Device Migration and Device Hot-swappingVirtual Device Migration and Device Hot-swapping
device
Netbus: Software Architecture (contd.)Netbus: Software Architecture (contd.)
Device Hot-swappingDevice Hot-swapping– Replace the remote device with an ‘equivalent’ local Replace the remote device with an ‘equivalent’ local
device while in operationdevice while in operation– Increases throughput, removes network dependence, Increases throughput, removes network dependence,
hardware maintenancehardware maintenance– Useful for rarely migrated VMsUseful for rarely migrated VMs– How to deal with pending IO operationsHow to deal with pending IO operations
Same as with virtual device migrationSame as with virtual device migration– Can be combined with virtual device migrationCan be combined with virtual device migration
To complete both operations in one shotTo complete both operations in one shot
Netbus’ prototype implemented in Xen and Netbus’ prototype implemented in Xen and works for block and USB devicesworks for block and USB devices
Netbus EvaluationNetbus Evaluation
Testbed descriptionTestbed description– Two Dell Two Dell
Poweredge 2650s Poweredge 2650s connected through connected through a gigabit switcha gigabit switch
– Each machine has Each machine has 2 2.8 GHz, 2-wat 2 2.8 GHz, 2-wat HT Xeon CPUs and HT Xeon CPUs and 2 GB RAM2 GB RAM
– Iozone file I/O Iozone file I/O benchmarks usedbenchmarks used
Write throughput of block devices
Write throughput of block devices without buffer caching
Netbus LatencyNetbus Latency
Latency in accessing block devices
02468
10121416
Stages of access path
Late
ncy
(ms) Local VBD
RVBD (Netbus)
NBD
Latency incurred by various components in accessing IO devices
Latency in accessing USB devices
0
2
4
6
8
10
Stages of access pathLa
tenc
y (m
s)
Local USB
RUSB (Netbus)
Virtual Device MigrationVirtual Device Migration
MySQL server migration in RuBIS online auction benchmark
RuBIS Database server VM migrationRuBIS Database server VM migration
Effect on RUBiS Throughput due to MySQL Server Migration
Effect on Iozone throughput due to VM migration and disk hot-swapping
Questions??Questions??
Remote Device SharingRemote Device Sharing
Host1
Host3 Host4
Host2device1
device3
device2
device4
vdevice4
vdevice2vdevice4