Cloud Computing with Nimbus April 2009

Kate Keahey

(keahey@mcs.anl.gov)

University of Chicago

Argonne National Laboratory

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

Cloud Computing

IaaSInfrastructure-as-a-Service

PaaSPlatform-as-a-Service

SaaSSoftware-as-a-Service

elasticitycomputing on demand

capital expense

operational expense

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

Cloud Computing for Science

Environment Resource control

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

“Workspaces”

Dynamically provisioned environments Environment control Resource control

Implementations Via leasing hardware platforms: reimaging,

configuration management, dynamic accounts…

Via virtualization: VM deployment

Isolation

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

A Brief History of Nimbus

Research on agreement-based

services

Xen released

First WSRFWorkspace Service

release

EC2 gatewayavailable

Support for EC2 interfaces

2003 20092006

EC2 goes online

First STAR productionrun on EC2

Nimbus Cloudcomes online

Context Brokerrelease

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

Nimbus Goals

Allow providers to build clouds Private clouds (privacy, expense considerations) Workspace Service: open source EC2 implementation

Allow users to use cloud computing Do whatever it takes to enable scientists to use IaaS Context Broker: turnkey virtual clusters IaaS Gateway: interoperability

Allow developers to experiment with Nimbus For research or usability/performance

improvements Community extensions and contributions

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

VWSService

The Workspace Service

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

The Workspace Service

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

The workspace service publishesinformation about each workspace

Users can find outinformation about theirworkspace (e.g. what IP

the workspace was bound to)

Users can interact directly with their

workspaces the same way the would with a

physical machine.

VWSService

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

Workspace Service: Interfaces and Clients

Web Services based Web Service Resource Framework (WSRF)

WS + state management (WS-Notification) Elastic Computing Cloud (EC2)

Compatible with EC2 clients Supported: ec2-describe-images, ec2-run-instances, ec2-

describe-instances, ec2-terminate-instances, ec2-reboot-instances, ec2-add-keypair, ec2-delete-keypair

Unsupported: availability zones, security groups, elastic IP assignment, REST

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

Workspace Service: Security

GSI authentication and authorization PKI-based VOMS, Shibboleth (via GridShib), custom PDPs

Secure access to VMs EC2 key generation or accessed from .ssh

Validating images and image data Extensions from Vienna University of Technology Paper: Descher et al., Retaining Data Control in

Infrastructure Clouds, ARES (the International Dependability Conference), 2009.

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

Workspace Service: Networking

Network configuration External: public IPs or private IPs (via VPN) Internal: private network via a local cluster

network Each VM can specify multiple NICs mixing

private and public networks (WSRF only) E.g., cluster worker nodes on a private

network, headnode on both public and private network

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

Workspace Components

workspacecontrol

workspaceresourcemanager

workspacepilot

workspaceclient

workspaceservice

EC2

WSRF

OpenNebulaProject

See papers at: http://workspace.globus.org/papers/index.html• “Simple Leases with Workspace Pilot” (EuroPar08) • “Combining Batch Execution and Leasing Using Virtual Machines” (HPDC08),

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

Cloud Capabilities

workspacecontrol

workspaceresourcemanager

workspacepilot

workspaceclient

cloudclient

storageservice

workspaceservice

EC2

WSRF

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

The IaaS Gateway

IaaSgateway

EC2potentially other providers

workspacecontrol

workspaceresourcemanager

workspacepilot

workspaceclient

cloudclient

storageservice

workspaceservice

EC2

WSRF

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

User Environments

Cloud Computing Ecosystem

Appliance ProvidersMarketplaces, commercial providers,

Virtual OrganizationsAppliance management software

Deployment Orchestrator

VMM/DataCenter/IaaS User EnvironmentsVMM/DataCenter/IaaS

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

MPIMPI

Turnkey Virtual Clusters

Turnkey, tightly-coupled cluster Shared trust/security context Shared configuration/context information

IP1IP1 HK1HK1

IP1IP1

IP2IP2

IP3IP3

HK1HK1

HK2HK2

HK3HK3

Context Context BrokerBroker

IP2IP2 HK2HK2

IP1IP1

IP2IP2

IP3IP3

HK1HK1

HK2HK2

HK3HK3

IP3IP3 HK3HK3

IP1IP1

IP2IP2

IP3IP3

HK1HK1

HK2HK2

HK3HK3

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

Context Broker Goals

Can work with every appliance Appliance schema, can be implemented in

terms of many configuration systems Can work with every cloud provider

Simple and minimal conditions on generic context delivery

Can work across multiple cloud providers, in a distributed environment

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

Context Broker Status

Releases In alpha since 08/07, first release 06/08, update

01/09 Used to contextualize cluster composed of 100s of

virtual nodes for multiple production apps Contextualized images on workspace marketplace Working with rPath to make contextualization

easier for the user Discussing OVF extensions

Paper: Keahey&Freeman, Contextualization: Providing One-Click Virtual Clusters, eScience 2008

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

End of Nimbus Tour

workspacecontrol

workspaceresourcemanager

workspacepilot

workspaceservice

workspaceclient

cloudclient

IaaSgateway

cont

ext

brok

er

contextclient

EC2potentially other providers

storageservice

EC2

WSRF

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

Science Clouds

Goals Enable scientific projects to experiment with IaaS clouds Evolve software in response to the needs of scientific projects A laboratory for exploration of cloud interoperability issues

Participants University of Chicago (since 03/08, 16 nodes), University of Florida

(05/08, 16-32 nodes, access via VPN), Masaryk University, Brno, Czech Republic (08/08), Wispy @ Purdue (09/08)

In progress: Grid5K, Vrije, others Using EC2 for large runs

Simple governance model, access given to any scientific project http://workspace.globus.org/clouds

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

Who Runs on Nimbus at UC?

HadoopAliEnGT-scalabilitySTARMontage workflowsGridFTP testingworkspace-teamTestingOSGgeofestbioinformaticsOther

100+ DNs

projects ranging across Science, CS, education, build&test…

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

STAR

STAR: a nuclear physics experiment studies fundamental properties of nuclear matter

Computations require complex and consistently configured environments

Requirements A virtual OSG STAR cluster: OSG headnode (gridmapfiles,

host certificates, NFS, Torque), worker nodes: SL4 + STAR From Science Clouds to EC2 runs One-click virtual cluster deployment: Context Broker

Producing just-in-time results for Quark Matter conference: http://www.isgtw.org/?pid=1001735

Work by Jerome Lauret, Doug Olson, Leve Hajdu, Lidia Didenko at BNL

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

Alice HEP Experiment at CERN

Collaboration with CERNVM project HPCwire article

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

Sky Computing

U of FloridaU of Chicago

ViNErouter

ViNErouter

ViNErouter

Purdue

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

Sky Computing

Papers: “Sky Computing”, by K. Keahey, A. Matsunaga, M. Tsugawa, J.

Fortes. Submitted to IEEE Internet Computing. “CloudBLAST: Combining MapReduce and Virtualization on

Distributed Resources for Bioinformatics Applications” by A. Matsunaga, M. Tsugawa and J. Fortes. eScience 2008.

U of FloridaU of Chicago

Purdue

Hadoop cloud

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

IaaS Clouds vs Grids Grid computing

Assumption: site retains control over resources Remote interfaces to local site mechanisms Tradeoff: difficult to provide the right environments and control but

easy to deploy Cloud computing

Assumption: a user gets a “lease” on a remote resource that it gets to control

Enabled by virtauliaztion (Xen) Tradeoff: eanbles a larger class of applications but hard to deploy Raises issues: e.g., site licenses? Configuration support?

Towards “sky computing” I can now trust a remote resource: I configured it myself Cloud computing + virtual networks Local distributed environment

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

Friends and Family

Committers: Kate Keahey & Tim Freeman (ANL/UC), Ian Gable (UVIC)

A lot of help from the community, see: http://workspace.globus.org/people.html

Collaborations: Cumulus: S3 implementation (Globus team) EBS: IU project Appliance management: rPath, Bcfg2 project, CohesiveFT Virtual network overlays: University of Florida Security (research): Vienna University of Technology

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

Open Source IaaS Implementations

OpenNebula Open source datacenter implementation University of Madrid, I. Llorente & team, 03/2008

Eucalyptus Open source implementation of EC2 UCSB, R. Wolski & team, 06/2008

Cloud-enabled Nimrod-G Open source implementation of EC2 Monash University, MeSsAGE Lab, 01/2009

Industry efforts openQRM, Enomalism

04/09/23 The Nimbus Toolkit: http//workspace.globus.org

Parting Thoughts

Science-driven cloud computing Importance of open source

Drive requirements into the infrastructure, customize Drive the development of standards

Cloud computing for the user Combine with what we have (grid computing) Explore new potential

Future directions Creating the ecosystem, working out the issues, e.g.

licensing, appliance support Interoperability and standards Service Levels