PRAGMA19, Sep. 15Resources breakout

Migration from Globus-based Grid to Cloud

Mason Katz, Yoshio Tanaka

Let’s review the past 8 years• Good things

– It was exciting to build a real Grid infrastructure across pacific rim.

– Experiences on building PRAGMA Grid gave us a lot of insights.

– Experiments on PRAGMA Grid gave us a lot of valuable input.

• Bad things – Who used, who are using PRAGMA Grid?– Globus-based Grid is still troublesome to use

• Need to check application run on every site.– Resources (e.g. network, computing resources) are not

enough to motivate users to use PRAGMA Grid.

Proposed direction- migration to Cloud -

• Build once, run everywhere!• What a user should do?

– Build and test applications on his local machine.– Create a VM image which is able to run on any

cloud resource.• What a resource provider should do?

– Build VM hosting servers.

Biomedical CLOUD

Resource Manager(edu.sdsc.nbcr.opal.manager.CSFJobManager)

Service Manager

Scheduling: Workflow Job

Array Job

AutoDock NAMD

OPAL2

CSF4

User Interface

Grid Sites

MetaScheduler

generate RSL files

Grid Resources

Input/Ouput Files: StageIn and StageOut

VM Replication Experimenthttp://goc.pragma-grid.net/wiki/index.php/VC-replication-2

SDSC VM hosting server AIST VM hosting server

AFG VM(original) AFG VM

(copy)

• VM hosting server: • Rocks 5.3 Xen roll

• Avian Flu Grid VM• Rocks VM• Globus/SGE• Autodock

• Replication updates • hostname and IP • Compute nodes• Network configurations• Globus configuration• SGE configuration

NBCR VM hosting server

AFG VM(copy)

VM replication

         

5

Data Service zone

GEO Grid System Zones

Laboratory zonesoftware develop

application develop

Cloud Service zone

homearchive

applications, librariesDEM, GRASS, GDAL,

WMS, WCS, WFS, CS-W?

WPS

Google Service map

Network SW

InternetCS-W?

Frontend serviceportal, workflow,

Agenda of breakouts• Have consensus of migration• Figure out possible problems on migrating to Cloud

– Does anybody need Globus-based Grid?, i.e. is Globus still mandatory?• Figure out missing pieces. What do we need to make PRAGMA

Cloud a practical infrastructure?– How make it easy to create VM images?

• Start with creating baseline VM images

– Interaction with data cloud– Meta scheduling / resource brokering– Security– Interoperation between VM hosting servers– …

• Rough schedule and milestones• Detailed discussions will be in this afternoon

Steps1. Identify sites

1. AIST, UCSD, NCHC, JILIN, Osaka, Indiana

2. Identify what baseline images should be published1. Publish a baseline Condor image (Phil UCSD by PRAGMA20)2. Gfarm client in baseline image3. Independent test is necessary

3. Automatic replication (Solve re-configuration / re-forming problems) (Mason by PRAGMA 20)

1. Hopefully by PRAGMA20@HK

4. Consider how to share VM images1. Build a stable Gfarm filesystem. (UCSD, AIST, Tsukuba)2. Build and use Gfarm image on Virtual clusters at SDSC and AIST. (by PRAGMA20)

1. physical clusters provided by SDSC/AIST. Tatebe will lead deployment)

5. Interoperability testing1. Future issues

6. Other research issues to be done (by each institution)1. Security, meta-scheduling, data, …