ImageManagement Slides

7/27/2019 ImageManagement Slides

1/24

Abstract Image Management and

Universal Image Registration for

Cloud and HPC Infrastructures

https://portal.futuregrid.org

Javier Diaz, Gregor von Laszewski,

Fugang Wang and Geoffrey Fox

Community Grids Lab

Pervasive Technology Institute

Indiana University


2/24

Motivation

FutureGrid (FG) is a testbed providing users withgrid, cloud, and high performance computingresources

One of the goals of FutureGrid is to provide atestbed to perform experiments in a reproducibleway among different infrastructures

We need mechanism to ease the use of theseinfrastructures

FG Image Management framework allows users toeasily create customized environments by placingsuitable images onto the FG resources



3/24

Introduction I

Image management is a key component in any modern

compute infrastructure (virtualized or non-virtualized)

Processes part of the image management life-cycle:

http://futuregrid.org

Creating andCustomizing

Images

StoringImages

RegisteringImages

InstantiatingImages

User selectsproperties

and software stackfeatures

meeting his/herrequirements

Nimbus

Eucalyptus

OpenStack

OpenNebula

Bare Metal

Abstract

Image

Repository

Ad

aptingtheImages

(a) (b) (c) (d)


4/24

Introduction II

Targeting multiple infrastructures amplifies theneed for mechanisms to ease these imagemanagement processes

We have identified two mechanisms Introduce standards and best practices to interface with

the infrastructure (OVF, OCCI, Amazon EC2)

Provide tools that interface with these standards andexpose the functionality to the users while hiding the

underlying complexities Otherwise, only the most experienced users will be able

to manage images for multiple infrastructures under greatinvestment of time



5/24

FutureGrid Image Management

Framework Framework provides users with the tools needed to

ease image management across infrastructures

Users choose the software stacks of their images andthe infrastructure/s

Targets end-to-end workflow of the image life-cycle Create, store, register and deploy images for both

virtualized and non-virtualized resources in atransparent way

Allows users to have access to bare-metalprovisioning (departure from typical HPC centers) Users are not locked into a specific computational

environment offered typically by HPC centers



6/24

Architectural Overview


Image

Management

Client

IaaS and Bare-Metal HPC

Infrastructures

HPC Clusters

Image

Management

Server

API

Cloud IaaS

Frameworks

Bare Metal

Image

Generation

External Services:

Chef, Security tools

FG Shell

Portal

Image

Instantiation

Nimbus

Eucalyptus

AWS

OpenNebula

OpenStack

Image

Repository

Image

Registration


7/24

Image Generation

Creates images according to

users specifications:

OS type and version

Architecture

Software Packages

Software installation may be

aided by Chef

Images are not aimed to anyspecific infrastructure

Image stored in Repository or

returned to user


Command Line Tools

Update Image

User's ImageStore in Image

Repository

FG Software

Cloud Software

User Software

Requirements:

OS, version, hadrware,...

Base Image

Matching Base

Image in the

Repository?

Base OS

Base SoftwareGenerate Image

Yes

NoRetrieve

Image from

Repository

Install Software

OpenNebula

VMCentOS 5X86_64

VMUbuntu 12

X86

VMCentOS 6X86_64

Image Gen. Server


8/24

Image Repository

Service to query, store, and update images

Unique interface to store various kind of images for

different systems

Images are augmented with some metadata which is

maintained in a searchable catalog

Keep data related with the usage to assist performance

monitoring and accounting

Independent from the storage back-end. It supports a

variety of them and new plugins can be easily created



9/24

Image Metadata

Field Name Description

imgId Images unique identifier

owner owner

os Operating system

description Description of the image

tag Images keywords

vmType Virtual machine type

imgType Aim of the image

permission Access permission

imgStatus Status of the image

createdDate Upload datelastAccess Last time the image was accessed

accessCount # times the image has been

accessed

size Size of the image

User Metadata

Field

Name

Description

userId Users unique

identifier

fsCap Disk max usage (quota)

fsUsed Disk space used

lastLogin Last time user usedthe framework

status Active, pending,

disable

role Admin, User

ownedimg # of owned images


10/24

Image Registration I

Adapts and registers images into specific

infrastructures

Two main infrastructures types are considered

to adapt the image:

HPC: Create network bootable images that can

run in bare-metal machines (xCAT/Moab)

Cloud: Convert the images in VM disks and

enable VMs contextualization for the selected

cloud



11/24

Image Registration II

User specifies where to

register the image

Optionally, user can select

kernel from a catalog Decides if an image is

secure enough to be

registered

The process of registeringan image only needs to be

done once per infrastructure


Customize Image for:

OpenStack

Eucalyptus

Nimbus

OpenNebula

Amazon

Command Line Tools

Retrieve from

Image Repository

User's Image

Register Image in the

Infrastructure

HPC

Image Customized for the selected

Infrastructure

Image is Ready

for Instantiation in

the Infrastructure

Upload Image to the Infrastructure

Security Check

Requirements: Image,

Kernel, Infrastructure


12/24

Tests Results obtained from the

Analysis of the ImageManagement Framework



13/24

Methodology

Software deployed on the FutureGrid India cluster Intel Xeon X5570 servers with 24GB of memory

Single drive 500GB with 7200RPMm 3Gb/s

Interconnection network of 1Gb Ethernet

Software Client is in Indias login node Image Generation supported by OpenNebula

Image Repository supported by Cumulus (storeimages) and MongoDB (store metadata)

HPC supported by xCAT, Moab and Torque Performed different tests to evaluate the Image

Generation and the Image Registration tools



14/24

Scalability of Image Generation I

Concurrent requests to create CentOS images from scratch

Increasing number of OpenNebula compute nodes to scale


0

200

400

600

800

1000

1200

1 2 4 8

Time(s)

Number of Concurrent Requests

1 Compute Node

2 Compute Nodes

4 Compute Nodes


15/24

Scalability of Image Generation II

Analyze how the time is spent within theimage creation process

Only one OpenNebula compute node to better

analyze the behavior of each step of theprocess

Concurrent requests to create CentOS and

Ubuntu images Image creation performed from scratch and

reusing a base image from the repository



16/24

Create Image from Scratch


0

200

400

600

8001000

1200

1400

1 2 4 8

Time(s)


(4) Upload It to the Repository

(3) Compress Image

(2) Generate Image(1) Boot VM

0

200

400

600

800

1000

1200

1400

1 2 4 8

Time(

s)


(4) Upload It to the Repository(3) Compress Image(2) Generate Image(1) Boot VM

CentOS

Ubuntu


17/24

Create Image from Base Image


CentOS

Ubuntu

0

200

400

600

8001000

1200

1400

1 2 4 8

Time(s)


(4) Upload it to the Repository

(3) Compress Image

(2) Generate Image(1) Retrieve/Uncompress base image from Repository

0

200

400

600800

1000

1200

1400

1 2 4 8

Time(s)


(4) Upload it to the Repository

(3) Compress Image

(2) Generate Image

(1) Retrieve/Uncompress base image from Repository


18/24

Scalability of Image Registration

Register the same CentOS image in differentinfrastructures: OpenStack (Cactus version configured with KVM

hypervisor)

Eucalyptus (2.03 version configured with XENhypervisor)

HPC (netboot image using xCAT and Moab)

Concurrent registrations in Eucalytpus and

Openstack Only one request at a time is allowed for HPC

registration (modifies important parts of the HPCsystem)



19/24

Register Images on Cloud


0

100

200

300

400

500

600

700

800

900

1 2 4 8

Time(s)


(3) Upload/Register Image into Cloud Infrastructure

(2) Retrieve Image from Server Side

(1) Customize Image

0

100

200

300

400

500600

700

800

900

1 2 4 8

Time(s)


(3) Upload/Register Image into Cloud Infrastructure

(2) Retrieve Image from Server Side

(1) Customize Image

Eucalyptus

OpenStack


20/24

Register Image on HPC

0

20

40

60

80

100

120

140

1

Time(s)


(4) Packimage (xCAT)

(3) Retrieve Kernels and Update

xCAT Tables

(2) Uncompress Image

(1) Retrieve Image from

Repository



21/24

Conclusions I

We have introduced the FG user-controlledimage management framework to handleimages for different infrastructures

Framework abstracts the details of eachunderlying system

Users can easily create and manage

customized environments within FG Replicate software stack on the supported

cloud and bare-metal infrastructures



22/24

Conclusions II

Image management results show a linear increasein response to concurrent requests

Image Generation Create image from scratch in only 6 min and using a

base image in less than 2 min Scale by adding more nodes to the cloud

Support different OS and arch due to virtualization

Image Registration registers images in any

supported infrastructure in less than 3 min Image Repository supports perfectly the rest of the

framework with a negligible overhead



23/24

Ongoing Work

Integrate a messaging queue system (like

RabbitMQ or ZeroMQ) to process users

requests in an asynchronous way

Develop a portal interface

On-demand resource re-allocation between

infrastructures (usage, users requests)



24/24

Thank for your attention!!

Contact info:

Javier Diaz: [email protected] Laszewski:[email protected]

http://futuregrid.github.com/rain/



Date post:	02-Apr-2018
Category:	Documents
Upload:	saravanaraajaa
View:	218 times
Download:	0 times

ImageManagement Slides

Documents