A"Distributed"Application"Execution"System"for"an"Infrastructure"with"Dynamically"Configured"Networks�

NetCloud)2012)Workshop,)Dec.)4)2012,)Taipei�

Ryousei)Takano,)Hidemoto)Nakada,))Atsuko)Takefusa,)and)Tomohiro)Kudoh)

)Information*Technology*Research*Institute,**

National*Institute*of*Advanced*Industrial*Science*and*Technology*(AIST),*Japan�

Background�

•  Intercloud)(a.k.a.)cloud)of)clouds))–  Virtual)Infrastructure)over)multiple)region)and)domain)clouds)

–  LargeLscale)data)intensive)scientific)computing)platform)•  e.g.,)high)energy)physics,)bio)science,)and)geo)science)•  largeLscale)data)and)special)instruments)are)geographically)distributed))

2

container�

container�

container�

container�

Physical Infrastructure�

Virtual Infrastructure�

site A� site B�

Challenges�

•  To)seamlessly)deploy)existing)applications)that)run)on)a)conventional)cluster)computer)system)into)a)VI,)it)is)important)to)quickly)setup)a)tailored)virtualized)cluster)environment)and)execute)an)application)with)minimum)overhead)of)virtualization.)

•  The)fusion)of)computer)and)network)virtualization)technologies)may)help)us)to)realize)such)an)application)execution)environment.)–  Virtual)machine)and)OS)container)technologies)

–  Software)defined)network:)OpenFlow,)OGF)NSI,)etc)

�3

Contributions�

•  Automatic)construction)of)distributed)application)execution)einvironment)–  OneLstop)service)to)execute)and)monitor)user)applications)over)isolated)Intercloud)resources)

–  SliceLaware)contextualization)

•  Implementation)as)a)part)of)GridARS)middleware)suite)–  OSLlevel)virtualization)(container))and)dynamic)network)path)provisioning)

•  Demonstration)of)the)feasibility)–  Quick)slice)construction)and)low)overheads)of)virtualization)

Agenda�

•  GridARS)and)the)Application)Execution)System)

•  SliceLaware)contextualization)

•  Evaluation)•  Conclusion)and)Future)Work�

5

GridARS"and""the"application"execution"system�

GridARS:"Grid"Advanced"Resource"management"System�

•  GridARS)is)a)reference)implementation)of)GNSLWSI)–  Defined)by)the)GLlambda)project)

•  Collaboration)between))KDDI)R&D)Labs.,)NTT,)NICT,))and)AIST,)started)in)2004)

–  Web)services)I/F)to)reserve,))modify)and)release))various)resources)

–  PollingLbased))2Lphase)commit)protocol)

•  GridARS)supports)OGF)NSI)version)2.�

7

GRC$ DMS/A$�

NRM�

CRM�SRM�

CRM�NRM�

SRM�CRM�

GRC$ GRC$ DMS/A�DMS/A�

Domain 1� Domain 3�

User�

GRS$

Domain 0�

CRM�Domain 2�

DMC/C�DMC/C�

DMC/C�DMC/C�

DMC/C�

DMC/A�

Aggregator�

Collector�

GRC: Global Resource Coordinator RM: Resource Manger�

Allocated�

Application"Execution"System�

•  The)goal)is)to)provide)users)with)a)slice)that)looks)like))a)single)isolated)cluster)computer)system.)–  A)slice)consists)of)containers)and)dynamically)configured)network)paths.)

–  All)containers)belong)to)the)same)IP)network)segment.)

–  An)application)is)automatically)executed)at)the)reservation)time.)

–  The)user)can)monitor)the)resource)utilization)of)their)slice.)

8

�����

container�

container�

container�

container�

Physical Infrastructure�

Virtual Infrastructure�

site A� site B�

Requirements�

•  A)slice)is)constructed)at)the)start)of)reservation)time,)an)application)is)automatically)executed)on)it,)and)it)is)released)at)the)end)of)reservation)time.))

•  A)conventional)parallel)application)(e.g.,)an)MPI)program))requires)remote)login)and)process)execution)via)SSH.)–  SSH)public)keys)should)be)generated)and)exchanged)among)containers)in)advance.)

–  A)host)list)file,)which)includes)IP)addresses)of)all)participating)containers,)should)be)prepared)in)advance.)•  IP)addresses)are)dynamically)assigned.)

9

SliceFaware"contextualization�

•  The)key)for)automated)slice)construction)is))contextualization,)which)dynamically)adjusts)each)container)setting,)including)the)IP)address,)the)hostname,)and)SSH)keys,)at)deployment)time.)–  VM)image)contextualization:)Nimbus)Context)Broker,)OpenNebula)

•  However,)the)existing)techniques)assumed)to)be)used)within)a)single)site.)

•  We)propose)sliceLaware)contextualization,)which)contextualizes)a)slice)based)on)information,)exchanged)among)several)sites)in)a)hierarchical)manner.)

10

Design"and"Implementation�

Design"Overview�

•  Use)“Pilot)Job”)to)contextualize)and)monitor)a)slice)

•  Use)OSLlevel)virtualization)(Linux)container))for)isolation)

Resource$Management$Service�

Resource$Alloca:on$Planning$Service�

Distributed$Monitoring$Service�

Applica:on$Execu:on$System�

Monitoring�SliceBaware$contextualiza:on� File$system$weaving�

GridARS�

�%&"�&�$%��� �%&"�&�$%���

�%$(�!$�&� �%$(�!$�&�

Node"Manager�•  Local)job)scheduler)invokes)“pilot)jobs”)called)Node)Manager)

(NM))instead)of)user)jobs.)

•  The)NMs)set)up)a)virtual)cluster)and)execute)the)user)jobs.�

13

�����$%���

������ �

�%��'� ��)#�&�+!( ���*�$���&�'�&*�(!%$���&!���$�!$��+���#)���

��)$� �&� ��)$� �&�

�%�����$���&�

�%�����$���&���

����� ��

����� �

�'�&��%�� �'�&��%����������������������� ������

GRC

SliceFaware"Contextualization�

CRM CRM

NM NM NM NM NM NM

GRC: Global Resource Coordinator CRM: Compute Resource Manager NM: Node Manager

�Available address range: 192.168.0.0/24, 192.168.1.0/24

�Available address range: 192.168.3.0/24, 192.168.1.0/24

�Reserve 3node, 192.168.3.0/24 �Reserve 3node, 192.168.1.0/24

�SSH keys Address

�SSH keys Addresses �Hosts authorized_keys known_hosts

�Hosts authorized_keys known_hosts

Container Container Container Container Container Container

File"System"Weaving�

•  Setting)up)a)container)file)system)may)be)time)consuming.)

•  The)most)of)files)could)be)shared)with)the)host)OS.�

•  File)system)weaving)helps))to)quickly)set)up)a)container)file)system)and)isolate)from)the)host)OS)file)system.)–  aufs2)stackable)file)system)

–  bind)mount)option)

15

/ etc usr opt

home

usr opt alice

.ssh

/ etc usr opt

home alice

.ssh

A

B

C

D

E

host OS file system (read only)�

container file system�

writable layer�

Slice"Monitoring"Service"(1/2)�•  AEMD)gathers)monitoring)information)in)each)site)via)NMs.)

•  GridARS)DMS)aggregates)information)per)slice.�

16

�����$%���

�� �

�%&"�&�$%��� �%&"�&�$%���

�%�����$���&�

�%�����$���&�

�%$(�!$�&��'�&��%��

�'�&��%��

�%$(�!$�&�

����

�����

��

�����

���� ���

���������

���������

Slice"Monitoring"Service"(2/2)�

17

User’s view�Administrator’s view�

Another user status�

Network status�

Computer status�

Reservation Status �

Ganglia

Disclose resource information only to the users who made reservation on the resource

Evaluation�

Experimental"Setting�•  Slice)start)up)time:)

–  container)start)up)time�

–  contextualization)information)exchange)time)

–  barrier)synchronization)

19

Site A�

GRC

G� G� G�

Site B�

Site C�

Site D�

G: GtrcNET-1 - latency injection: 0 – 300 ms - per-VLAN traffic monitoring

PC spec.�

CPU� Intel Core 2 Q9550/2.83GHz�

Memory� 4 GB�

Ethernet� Intel PRO/1000�

OS Rocks 5.2 (kernel 2.6.30)�

Slice"Start"Up"Time�

0

1

2

3

4

5

0ms100ms

200ms300ms

0ms100ms

200ms300ms

0ms100ms

200ms300ms

0ms100ms

200ms300ms

Elap

sed

Tim

e [s

]

NM start - info gatheringinfo gathering - info distribution

info distribution - execution

Site ABCDSite A C Site A B Site A

20

Site A�

GRC

G� G� G�

Site B�

Site C�

Site D�

4�2�

2�

2� 2�1�

1�1�

1�

Container Setup�

Contextualization Information Exchange�

Barrier Synch.�

•  The)contextualization)process)depends)on)the)latencies)injected.)•  The)number)of)sites)does)not)affect)the)elapsed)times)very)much.)

Container"Setup"Time�

Elapsed"Time"(seconds)�

File)system)construction� 0.02)

Key)pair)generation� 0.44�

Guest)OS)start)up� 0.59�

Total� 1.05�

21

•  The)container)setup)is)quite)fast.)•  40)%)of)the)setup)time)is)consumed)by)generation)of)SSH)keys.)This)could)be)eliminated)by)inLadvance)generation)of)them.)

Conclusion"and"Future"Work�

Conclusion�

•  We)have)proposed)a)distributed)application)execution)system,)and)developed)an)implementation)of)it)as)a)part)of)the)GridARS)middleware)suite.)

•  The)key)for)automated)slice)construction)is)sliceFaware"contextualization.)

•  We)confirmed)that)a)slice)could)be)established)in)one)second,)leveraging)OSLlevel)virtualization)and)file)system)weaving.)

•  We)also)confirmed)that)the)overhead)for)propagating)contextualization)information)is)small)enough.)

23

Future"Work�•  Hardware)as)a)Service)(HaaS))over)Intercloud)

–  An)IaaS)provider)can)extend)their)hardware)resources)on)demand.)

–  HaaS)divides)resources)into)a)slice)and)provides)L2)network)connectivity)between)the)slice)and)the)IaaS’s)data)center.)

24

Data center A�

Data center B�

slice for DC A�

slice for DC B�

GridARS�

Data center C�CloudStack�

OpenStack�

Thanks)for)your)attention!�

25

This)work)was)partly)supported)by)the)National)Institute)of)Information)and)Communications)Technology)(NICT),)Japan.))

http://www.gLlambda.net/gridars/�