Date post: | 20-Dec-2015 |
Category: |
Documents |
View: | 215 times |
Download: | 0 times |
Virtuoso: Distributed Computing Using Virtual Machines
Peter A. DindaPrescience Lab
Department of Computer Science
Northwestern University
http://plab.cs.northwestern.edu
2
People and Acknowledgements
• Students– Ashish Gupta, Ananth Sundararaj,
Dong Lu, Bin Lin, Jason Skicewicz, Billy Davidson, Andrew Weinrich, Jack Lange, Alex Shoykhet
• Collaborators– In-Vigo project at University of Florida
• Renato Figueiredo, Jose Fortes• http://invigo.acis.ufl.edu
• Funder– NSF through several awards
3
Outline
• Motivation
• Virtuoso Model
• Virtual networking and remote devices
• Information services
• Resource measurement and prediction
• Resource control
• Related work
• ConclusionsR. Figueiredo, P. Dinda, J. Fortes, A Case For Grid Computing on Virtual Machines, ICDCS 2003
4
How do we deliver arbitrary amounts of computational power to ordinary people?
5
How do we deliver arbitrary amounts of computational power to ordinary people?
Distributed and Parallel Computing
Interactive Applications
6
How do we deliver arbitrary amounts of computational power to ordinary people?
Distributed and Parallel Computing
Interactive Applications
7
IBM xSeriesvirtual cluster(64 CPUs),1 TB RAID
Northwestern
Internet
Sun Enterprise servers(E450, E250; 6 CPUs)
Development cluster(5 PowerEdge, 10 CPUs)
IBM xSeriesVirtual cluster(64 CPUs)
GbE switch
IBM zSeries mainframe(1-way, 3.36TB storage)
RAID array(1.2TB)
UFL
10/100 switch
IBM xSeriesDev. cluster(8 CPUs)
InteractivityEnvironmentCluster, CAVE(~90 CPUs),8 TB RAID
2 DistributedOptical TestbedClustersIBM xSeries (14-28 CPUs),1 TB RAID
Nortel OpteraMetro EdgeOptical Router
Distributed Optical Testbed(DOT) Private Optical Network
DOT clusters with opticalconnectivityIBM xSeries (14-28 CPUs),1 TB RAID: Argonne, U.Chicago, IIT, NCSA, others
8
Grid Computing
• “Flexible, secure, coordinated resource sharing among dynamic collections of individuals, institutions, and resources”
• I. Foster, C. Kesselman, S. Tuecke, The Anatomy of the Grid: Enabling Scalable Virtual Organizations, International J. Supercomputer Applications, 15(3), 2001
• Globus, Condor/G, Avaki, EU DataGrid SW, …
9
Complexity from User’s Perspective
• Process or job model– Lots of complex state: connections, special
shared libraries, licenses, file descriptors
• Operating system specificity– Perhaps even version-specific– Symbolic supercomputer example
• Need to buy into some “Grid API”
• Install and learn complex Grid software
10
Users already know how to deal with this complexity at another level
11
Complexity from Resource Owner’s Perspective
• Install and learn complex Grid software
• Deal with local accounts and privileges– Associated with global accounts or
certificates
• Protection
• Support users with different OS, library, license, etc, needs.
12
Virtual Machines• Language-oriented VMs
– Abstract interpreted machine, JIT Compiler, large library– Examples: UCSD p-system, Java VM, .NET VM
• Application-oriented VMs– Redirect library calls to appropriate place– Examples: Entropia VM
• Virtual servers– Kernel makes it appear that a group of processes are running on a
separate instance of the kernel– Examples: Ensim, Virtuozzo, SODA, …
• Virtual machine monitors (VMMs)– Raw machine is the abstraction– VM represented by a single image– Examples: IBM’s VM, VMWare, Virtual PC/Server, Plex/86, SIMICS,
Hypervisor, DesQView/TaskView. VM/386
13
VMWare GSX VM
14
Isn’t It Going to Be Too Slow?Application Resource ExecTime
(10^3 s)
Overhead
SpecHPC Seismic
(serial, medium)
Physical 16.4 N/A
VM, local 16.6 1.2%
VM, Grid virtual FS
16.8 2.0%
SpecHPC
Climate
(serial, medium)
Physical 9.31 N/A
VM, local 9.68 4.0%
VM, Grid virtual FS
9.70 4.2%
Experimental setup: physical: dual Pentium III 933MHz, 512MB memory, RedHat 7.1,30GB disk; virtual: Vmware Workstation 3.0a, 128MB memory, 2GB virtual disk, RedHat 2.0NFS-based grid virtual file system between UFL (client) and NWU (server)
Small relativevirtualizationoverhead;compute-intensive
15
Isn’t It Going To Be Too Slow?
0
0.5
1
1.5
2
2.5
3
0
0.5
1
1.5
2
2.5
3
No Load Light Load Heavy Load
Tasks onPhysicalMachine
Tasks onVirtual
Machine
Tasks onPhysicalMachine
Tasks onVirtual
Machine
Tasks onPhysicalMachine
Tasks onVirtual
Machine
Synthetic benchmark: exponentially arrivals of compute bound tasks, background load provided by playback of traces from PSCRelative overheads < 10%
16
Isn’t It Going To Be Too Slow?
• Virtualized NICs have very similar bandwidth, slightly higher latencies
– J. Sugerman, G. Venkitachalam, B-H Lim, “Virtualizing I/O Devices on VMware Workstation’s Hosted Virtual Machine Monitor”, USENIX 2001
• Disk-intensive workloads (kernel build, web service): 30% slowdown– S. King, G. Dunlap, P. Chen, “OS support for Virtual Machines”,
USENIX 2003
17
Virtuoso
• Approach: Lower level of abstraction– Raw machines, not processes
• Mechanism: Virtual machine monitors• Our Focus: Middleware support to hide complexity
– Ordering, instantiation, migration of machines– Virtual networking and remote devices– Connectivity to remote files, machines– Information services– Monitoring and prediction– Resource control
18
The Virtuoso Model1. User orders raw machine(s)
• Specifies hardware and performance• Basic software installation available
• OS, libraries, licenses, etc.
2. Virtuoso creates raw image and returns reference• Image contains disk, memory, configuration, etc.
3. User “powers up” machine4. Virtuoso chooses provider
• Information service5. Virtuoso migrates image to provider
• Efficient network transfer • rsync, demand paging, versioned filesystems
19
The Virtuoso Model6. Provider instantiates machine
• Virtual networking ties machine back to user’s home network
• Remote device support makes user’s desktop’s devices available on remote VM
• Remote display support gives user the console of the machine (VNC)
• Resource control to give user expected performance
7. User goes to his network admin to get address, routing for his new machine
8. User customizes machine• Feeds in CDs, floppies, ftp, up2date, etc.
20
The Virtuoso Model9. User uses machine
• Shutdown, hibernate, power-off, throw away 10. Virtuoso continuously monitors and adapts
• Various mechanisms, all invisible to user• Migrating the machine• Routing traffic between machines• Virtual network topology• Predictive scheduling versus reservations
• Various goals• Price• Interactivity
• Information service• Resource monitoring and prediction
21
Outline
• Motivation
• Virtuoso Model
• Virtual networking and remote devices
• Information services
• Resource measurement and prediction
• Resource control
• Related work
• ConclusionsR. Figueiredo, P. Dinda, J. Fortes, A Case For Grid Computing on Virtual Machines, ICDCS 2003
22
Why Virtual Networking?
• A machine is suddenly plugged into your network. What happens?– Does it get an IP address?– Is it a routeable address?– Does firewall let its traffic through?– To any port?
How do we make virtual machine hostileenvironments as friendly as the user’s LAN?
23
A Layer 2 Virtual Network (VLAN) for the User’s Virtual Machines
• Why Layer 2?– Protocol agnostic– Mobility– Simple to understand – Ubiquity of Ethernet on end-systems
• What about scaling?– Number of VMs limited– Hierarchical routing possible because MAC
addresses can be assigned hierarchically
24
A Simple Layer 2 Virtual Network
Client Server
Remote VM
PhysicalNIC
VM monitor
VirtualNIC
PhysicalNIC
SSH
Hostile Remote NetworkFriendly Local Network
25
A Simple Layer 2 Virtual Network
Client Server
Remote VM
PhysicalNIC
VM monitor
VirtualNIC
PhysicalNIC
SSH
Hostile Remote NetworkFriendly Local Network
26
A Simple Layer 2 Virtual Network
Client Server
Remote VM
PhysicalNIC
VM monitorBridged Bridged
VirtualNIC
PhysicalNIC
SSH TunnelOr SSL TCP
Hostile Remote NetworkFriendly Local Network
27
An Overlay Network
• Bridgeds and connections form an overlay network for routing traffic among virtual machines and the user’s home network
• Links can trivially be added or removed
28
Bootstrapping the Virtual Network
• Star topology always possible• TCP session from client must have been possible
• Better topology may be possible• Depends on security at each site
• Topology may change• Virtual machines can migrate
• Bootstrap to higher layers• Virtual filesystems
29
Remote Devices
Client Server
Remote VM
VM monitornbd-server nbd-client
VirtualCDROM
SSH TunnelOr SSL TCP
Linux Network Block Device Driver/dev/cdrom <-> /dev/nb0 <-> VMWare CD Image
Physical CDROM
30
Extending a Grid Information Service (GIS) to Support Virtual Machines
• A GIS contains information about the available resources in a grid– Hosts, routers, switches, software, etc.
• URGIS project at Northwestern– GIS based on the relational data model– Compositional queries (joins) to find collections of
resources. • “Find physical machines which can instantiate a virtual machine
with 1 GB of memory”• “Find sets of four different virtual machines on the same network
with a total memory between 512 MB and 1 GB”
– Nondeterministic query extension for scalability
• http://www.cs.northwestern.edu/~urgis
31
The RGIS Design (Per Site)
Oracle 9i Back EndWindows, Linux, Parallel Server, etc
Oracle 9i Front Endtransactional inserts and updates
using stored procedures, queries using select statements
(uses database’s access control)
UpdateManager
Web Interface
Content Delivery Network Interface
For loose consistency
Query Managerand Rewriter
Users
Schema, type hierarchy, indices,PL/SQL stored procedures
for each object
Applications
RDBMSUse of Oracle
is not a requirement of approach
site-to-site (tentative)
Updates encrypted using asymmetric cryptography on network. Only those with appropriate keys have access
Authenticated Direct Interface
SOAP Interface
32
Motivation for Non-deterministic Queries
• Queries for compositions of resources easily expressed in SQL:
• But such queries can be very expensive to execute• However, we typically don’t need the entire result set, just
some rows, and not always the same ones• And we need them in a bounded amount of time• Approach: return random sample of result set
“Find 2 hosts with Linux that together have 3 GB of RAM”
select h1.insertid, h2.insertid from hosts h1, hosts h2 where h1.os=‘LINUX’ and h2.os=‘LINUX’ and h1.mem_mb+h2.mem_mb>=3072
33
Implementing non-deterministic queriesselect nondeterministically h1.insertid, h2.insertid from hosts h1, hosts h2 where h1.os=‘LINUX’ and h2.os=‘LINUX’ and h1.mem_mb+h2.mem_mb>=3072 within 2 seconds
SELECT H1.INSERTID, H2.INSERTID FROM HOSTS H1, HOSTS H2 , INSERTIDS TEMP_H1 , INSERTIDS TEMP_H2 WHERE (H1.OS='LINUX' AND H2.OS='LINUX' AND H1.MEM_MB+H2.MEM_MB>=3072) AND (H1.INSERTID=TEMP_H1.INSERTID AND TEMP_H1.rand > 982663452.975047 AND TEMP_H1.rand <= 1025613125.93505) AND (H2.INSERTID=TEMP_H2.INSERTID AND TEMP_H2.rand > 1877769069.94039 AND TEMP_H2.rand <= 1920718742.90039)
Query Managerand Rewriter
Random sample ofinput tables
Probability of inclusiondetermined by time constraint
and server load
34
Deadlines
Climbing Climbing+Hard Limiting Estimation Estimation+Hard Limiting0
0.5
1
1.5
2
2.5
Mechanism
Target Deadline
35
P. Dinda, D. Lu, Nondeterministic Queries in a Relational Grid Information Service, SC 2003
D. Lu, P. Dinda, Synthesizing Realistic Computational Grids, SC 2003
D. Lu, P. Dinda, J. Skicewicz, Scoped and Approximate Queries in a Relational Grid Information Service, Grid 2003
36
Extending a Grid Information Service (GIS) to Support Virtual Machines
• Virtual indirection– Each RGIS object has a unique id– Virtualization table associates unique id of virtual
resources with unique ids of their constituent physical resources
– Virtual nature of resource is hidden unless query explicitly requests it
• Futures– An RGIS object that does not exist yet– Futures table of unique ids– Future nature of resource hidden unless query
explicitly requests it
37
Extending a Resource Monitoring and Prediction System to Support Virtual Machines
• Measuring and predicting dynamic resource availability to support adaptation– Virtual machine migration– Routing on the virtual network– Application-level adaptation
• RPS System at Northwestern– Host and network measurements for Unix and Windows– Emphasis on prediction (wide range of linear and nonlinear
models) and communication (wide range of transports)
P. Dinda, Online Prediction of the Running Time of Tasks, Journal of Cluster Computing, 2002
P. Dinda, A Prediction-based Real-time Scheduling Advisor, IPDPS 2002
J. Skicewicz, P. Dinda, J. Schopf, Multiresolution Resource Behavior Queries using Wavelets, HPDC 2001
38
RPS Toolkit• Extensible toolkit for implementing resource
signal prediction systems [CMU-CS-99-138]• Growing: RTA, RTSA, Wavelets, GUI, etc
• Easy “buy-in” for users• C++ and sockets (no threads)• Prebuilt prediction components• Libraries (sensors, time series, communication)
• http://www.cs.northwestern.edu/~RPS
39
Example: Multiscale Network Prediction
• Large, recent study of predictability• Hundreds of NLANR and other traces
– Mostly WANs
• Different resolutions– Binning and low-pass via wavelets
• Sweet Spot– Predictability often
maximized at particular resolution
Y. Qiao, J. Skicewicz, P. Dinda, Multiscale Predictability of Network Traffic, NWU-CS-02-13
40
Multiresolution Network Prediction
0
0.05
0.1
0.15
0.2
0.25
0.3
0.1 1 10 100 1000Bin Size (Seconds)
last
bm(8)
ma(8)
ar(8)
ar(32)
arma(4,4)
arima(4,1,4)
arima(4,2,4)
arfima(4,-1,4)
41
• Goal: monitor physical machine and infer behavior inside of virtual machine
• Current approach: /proc on physical machine to slowdown on resource rate in virtual machine– ARX models– Causality problem
Extending a Resource Prediction System to Support Virtual Machines
42
Resource Control
• Owner has an interest in controlling how much and when compute time is given to a virtual machine
• Our approach: A language for expressing these constraints, and compilation to real-time schedules, proportional share, etc.
• Very early stages. Trying to avoid kernel modifications.
43
How to Control: User Irritation Project
• Measure interactive user tolerance to resource stealing
• Conversely, what service must be provided to interactive users?
• “Irritation@Home”
44
Outline
• Motivation
• Virtuoso Model
• Virtual networking and remote devices
• Information services
• Resource measurement and prediction
• Resource control
• Related work
• ConclusionsR. Figueiredo, P. Dinda, J. Fortes, A Case For Grid Computing on Virtual Machines, ICDCS 2003
45
Related Work• Collective / Capsule Computing (Stanford)
– VMM, Migration/caching, Hierarchical image files• Denali (U. Washington)
– Highly scalable VMMs (1000s of VMMs per node)• CoVirt (U. Michigan)• Xenoserver (Cambridge)• SODA (Purdue)
– Virtual Server, fast deployment of services• Internet Suspend/Resume (Intel Labs Pittsburgh)• Ensim
– Virtual Server, widely used for web site hosting– WFQ-based resource control released into open-source Linux kernel
• Virtouzzo (SWSoft)– Ensim competitor
• Available VMMs: IBM’s VM, VMWare, Virtual PC/Server, Plex/86, SIMICS, Hypervisor, DesQView/TaskView. VM/386
46
Current Status (At Northwestern)
• Bridged components done– Mechanism for virtual networking– No policy yet
• Very preliminary system for acquiring and instantiating VMs done
• RGIS schema extensions done• Work In Progress
– Remote devices (management)– Virtual networking (policy + adaptation)– VM Monitoring using RPS– User Irritation
47
For MoreInformation
• Prescience Lab (Northwestern University)– http://plab.cs.northwestern.edu
• ACIS (University of Florida)– http://acis.ufl.edu
R. Figueiredo, P. Dinda, J. Fortes, A Case For Grid Computing on Virtual Machines, ICDCS 2003
48
Nondeterministic query performance
0.1
1
10
100
1
10
100
1000
10000
100000
1000000
0.0001 0.001 0.002Selection Probability
Query Time
Number ofResults
Select two hosts that together have >3GB of RAM500,000 host grid generated by GridGMemory distribution according to Smith study of MDS contentsDual Xeon 1 GHz, 2 GB, 240 GB RAID, RGIS2, Oracle 9i EnterpriseAverage of five trials
Meaningful tradeoff between query processing time and result set size is possible
49
Nondeterministic query performance
Select n hosts that together have >3GB of RAM500,000 host grid generated by GridGMemory distribution according to Smith study of MDS contentsDual Xeon 1 GHz, 2 GB, 240 GB RAID, RGIS2, Oracle 9i EnterpriseAverage of five trials
0.1
1
0
200
400
600
800
1000
1200
1400
1600
1800
2000
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16Number of Hosts
Query Time
Number ofResults
p=0.0001
p=0.00001
p=0.000005
p=0.00000001
Can use tradeoff to controlquery time independent of query complexity