Date post: | 26-Mar-2015 |
Category: |
Documents |
Upload: | molly-hodges |
View: | 223 times |
Download: | 5 times |
Recent Advances in Grid Computing and Business
Models: A Gridbus Perspective
Rajkumar BuyyaGrid and Distributed Systems (GRIDS) LaboratoryDept. of Computer Science and Software EngineeringThe University of MelbourneMelbourne, Australia
www.gridbus.org
WW GridGrid Business Symposium 2005, Seoul, Korea
2
Outline
Introduction Utility Networks and Grid Computing
Global Grids and Challenges Grid Initiatives
World-wide with Australia and India Perspective Introduction to Gridbus Project and Grid Economy Grid Service Broker
Architecture, Design and Implementation Performance Evaluation: Experiments in Creation
and Deployment of Applications on Global Grids A Case Study in High Energy Physics Economy-based Scheduling in Data Grids
Summary
3
4 Essential Utilities and Delivery Networks
(1) Water
(2) Electricity
(3) Gas
(4) Telephone
4
(5) IT services as the fifth utility (water, electricity, gas, telephone, IT)
eScienceeBusiness
eGovernmenteHealth
MultilingualeEducation
…
5
A Bird Eye View of World-Wide Grid Environment
Grid Resource Broker
Resource Broker
Application
Grid Information Service
Grid Resource Broker
databaseR2R3
RN
R1
R4
R5
R6
Grid Information Service
7
Grid Challenges
Security
Resource Allocation & Scheduling
Data locality
Network Management
System Management
Resource Discovery
Uniform Access
Computational Economy
Application Construction
8
Some Grid Initiatives Worldwide
Australia Nimrod-G Gridbus DISCWorld GrangeNet. APACGrid ARC eResearch
Brazil OurGrid, EasyGrid LNCC-Grid + many others
China ChinaGrid – Education CNGrid - application
Europe UK eScience EU Grids.. and many more...
India I-Grid
Japan NAGERI
Korea...N*Grid
SingaporeNGP
USA Globus NASA IPG AccessGrid TeraGrid Cyberinfrasture
Industry Initiatives IBM On Demand
Computing HP Adaptive Computing Sun N1 Microsoft - .NET Oracle 10g Satyam – Grid Practice Infosys, Wipro, TCS StorageTek –Grid..
Public Forums Global Grid Forum Australian Grid Forum
Conferences: CCGrid Grid HPDC E-Science
http://www.gridcomputing.com
1.3 billion – 3 yrs
1 billion – 5 yrs
450million – 5 yrs
486million – 5 yrs
1.3 billion (Rs)
27 million
2? billion
120million – 5 yrs
10
Grid Computing in Australia(Courtesy: Jihyoun Park, SNU Visitor to
Melbourne)
AcademiaGovernment
Collaboration
Industry
11
Academic activities
1 University laboratories for Grid computing- Uni. of Melbourne(GRIDS lab): Gridbus (GridSim, GMD, GridBank, Alchemi, ..), Master of Engineering in Distributed Computing
- Monash Uni.: GriddlsS (Legacy SW to the computational grid), Nimrod-G - Australian national Uni. (Internet Futures Group)- Sydney Uni.(ViSLAB): high performance visualization &computing - Uni. of Adelaide (DHPC Group): DISCWorld - Queensland Uni. of Technology (PLAS): G2 (.NET based)
2 Grid Infrastructure ProjectsAPACGrid, National Neurosciece Facility, Australian Virtual Observatory, several state level facilities (VPAC, TPAC, SAPAC, QPSF, IVEC)
3 Grid Applications * Asia Pacific Bioinformatics Network/ Virtual Drug Design: Molecular
Modeling for Drug Design on P2P Grid/ HEPGrid: High Energy Physics and the Grid Network/ Access Grid/Australian Computational Earth Systems Simulator/.
* Recently 30 more applications are funded as part of ARC e-Research * Govt. has formed “National e-Research Coordination Committee”.
12
Grid Computing in India
AcademiaGovernment Collabor
ation
Industry(majority focus onGrid integration)
13
Grid Computing in India: Academic and Industrial Activities
Academic and Government Initiatives: TIFR, IITM, Anna University, IITD, UoH, etc. C-DAC’s Garuda – Ministry of IT
Software Companies in India: Top 4 Indian IT Companies: Satyam, Infosys, TCS (Tata
Consultancy Service), and Wipro. Oracle 10g, IBM, HP, Sun ertc. have a large Grid
development centers in Bangalore, India. Satyam is leading the pack in Grid Business push:
Grid Practice Centre with top management support. Singned MoU with Melbourne University and
extensively using Gridbus in powering applications. Also contributing the development of Gridbus
technologies (e.g., Alchemi) – SEI CMM Level 5 principles.
Application Verticals: Manufacturing, Security, Life Sciences, Finance
14
15
Australian and Indian Grid Efforts Compared
AcademiaGovernment
Collaboration
Industry
Australia
AcademiaGovernment Collabor
ation
Industry(majority focus onGrid integration)
AcademiaGovernment Collabor
ation
Industry(majority focus onGrid integration)
IndiaKorea: Is it like Australia or India?
16
The Gridbus Project @ Melbourne:Enable Leasing of ICT Services on
Demand
WWG
World Wide Grid!On Demand Utility
Computing
Gridbus
Distributed Data
17
The Gridbus Project: http://www.gridbus.org
A multi-institutional “Open Source” R&D Project with focus on: Architecture, Specification, and Open Source Reference Implementation. Service-Oriented Grid, Utility Computing & Distributed Data and Computation
Economy Scaling from Desktops, Clusters, Cluster Federation, Enterprise Grids to Global Grids.
Alchemi: Harnessing .NET/Windows-based Resources Grid Market Directory and Web Services Grid Bank: Accounting and Transaction Management Visual Tools for Creation of Distributed Applications Workflow Composition and Deployment Services Data Grid Brokering and Grid Economy Services Data Replication Strategies GridSim Toolkit: Enhanced to support Data Grid, Reservation, etc. Libra: SLA-based Allocation of Cluster Resources Coupling of Clusters and Computational Economy WWG: Global Data Intensive Grid Testbed Application Enabler Projects:
High-Energy Physics , Astronomy, Brain Activity Analysis – Osaka U., Natural Language Processing, Portfolio Analysis – Spain, BioGrid - WEHI (via APACGrid), SensorGrid (NICTA), Medical Imaging (HFI)
Supported by:
18
Grid Economy: Methodology for Sustained Resourced Sharing and Managing Supply-and-Demand for Resources
20
Grid Entities and Architecture
GSP site scheduler
accounting
Grid consumer
MarketMaker
GSP global scheduler
broker
Resource owners
End usersPrivate enterprises
National providers
GSP site scheduler
Resource owners
21
Grid Node N
A Reference Service-Oriented Architecture for Utility Grids
Grid Consumer
Pro
gra
mm
ing
En
viro
nm
ents
Grid Resource Broker
Grid Service Providers
Grid Explorer
Schedule Advisor
Trade Manager
Job ControlAgent
Deployment Agent
Trade Server
Resource Allocation
ResourceReservation
R1
Misc. services
Information Service
R2 Rm…
Pricing Algorithms
Accounting
Grid Node1
…
Grid Middleware Services
…
…
HealthMonitor
Grid Market Services
JobExec
Info ?
Secure
Trading
QoS
Storage
Sign-on
Grid Bank
Ap
pli
cati
on
s
Data Catalogue
22
Gridbus and Complementary Technologies – realizing Utility
Grid
AIXSolarisWindows Linux
.NET GridFabricSoftware
GridApplications
Core GridMiddleware
User-LevelMiddleware(Grid Tools)
GridBank
Grid Exchange & Federation
JVM
Grid Brokers:
X-Parameter Sweep Lang.
Gridbus Data Broker
MPI
Condor SGE TomcatPBS
Alchemi
Workflow
IRIX OSF1 Mac
Libra
Globus Unicore ……Grid
MarketDirectory
PDB
CDB
Worldwide Grid
GridFabricHardware
……
PortalsScience Commerce Engineering ……Collaboratories
……
Workflow Engine
Grid Storage Economy
Gri
d E
con
om
y NorduGrid XGrid
ExcellGrid
Nimrod-G
GRIDSIM
Gridscape
23
Alchemi: .NET-based Enterprise Grid Platform & Web Services
InternetInternet
InternetInternet
Alchemi Worker Agents
Alchemi Manager
Alchemi Users
Web Services
Web Services
•SETI@Home like Model•General Purpose•Dedicated/Non-dedicate workers•Role-based Security•.NET and Web Services•C# Implementation•GridThread and Job Model Programming•Easy to setup and use• Widely in use!
24
Some Users of Alchemi
Tier Technologies, USALarge scale document processing using Alchemi framework
CSIRO, AustraliaNatural Resource Modeling
The Friedrich Miescher Institute (FMI) for Biomedical Research, SwitzerlandPatterns of transcription factors in mammalian genes
Satyam Computers Applied Research Laboratory, IndiaMicro-array data processing using Alchemi framework
The University of Sao Paulo, BrazilThe Alchemi Executor as a Windows Service
stochastix GmbH, GermanyAsynchronous Excel Tasks using ManagedXLL and Alchemi .Net Grid Computing framework.
Many users in Universities: See next for an example.
25
On Demand Assembly of Services: Putting Them All Together
Data Source
(Instruments/distributed sources)
Data Replicator(GDMP) ASP Catalogue
Grid Info Service
Grid Market Directory
GSP(Accounting Service)
GridbusGridBank
Data
GSP(e.g., UofM)
PEGSP
(e.g., VPAC)
PE
GSP(e.g., IBM)
CPUorPE
Grid Service (GS)
(Globus)
Alchemi
GS
GTS
Cluster Scheduler
Grid Service Provider (GSP)
(e.g., CERN)
PECluster Scheduler
Job
8
GridResource Broker
2
Visual Application Composer
Application CodeExplore
data1
36
45
Resu
lts9 7
Results+
Cost Info
10
11
Bill
12Data Catalogue
The Gridbus Grid Service Broker for Data Grid
Applications
Builds on the Nimrod-G Computational Grid Broker and
Computational Economy [Buyya, Abramson, Giddy, Monash
University, 1999-2001]And
Extends its notion for Data and Service Grids
27
Gridbus Broker Architecture
Grid Middleware
Gridbus Client Gridbus ClientGribus Client
Grid Info Server
Schedule Advisor
Trading Manager
Gridbus Farming Engine
RecordKeeper
Grid Explorer
GE GIS, NWSTM TS
RM & TS
Grid Dispatcher
RM: Local Resource Manager, TS: Trade Server
G
G
CU
Globus enabled node.A
L
Alchemi enabled node.
(Data Grid Scheduler)
DataCatalog
DataNode
Unicore enabled node.
$
$
$
App, T, $, Opt
(Bag of Tasks Applications)
29
Gridbus Services for eScience applications
Application Development Environment: XML-based language for composition of task farming
(legacy) applications as parameter sweep applications. Task Farming APIs for new applications. Web APIs (e.g., Portlets) for Grid portal development. Threads-based Programming Interface Workflow interface and Gridbus-enabled workflow
engine. Resource Allocation and Scheduling
Dynamic discovery of optional computational and data nodes that meet user QoS requirements.
Hide Low-Level Grid Middleware interfaces Globus, Alchemi, Unicore, NorduGrid, XGrid, etc.
30
Figure 3 : Logging into the portal.
Drug DesignMade Easy!
Click Here for Demo
Economy-based Data Grid Scheduling
High Energy Physics as eScience Application Case
Study
CLICK HERE TO SKIP IF RUNNING OUT of TIME
32
Australian Belle Data Grid Testbed
Grid Service Broker
Replica Catalog
AARNET
NWS NameServer
VirtualOrganization
Analysis Request
Analysis Results
CertificateAuthority
NWSSensor
GridFTPGRIS
GlobusGatekeeper
Dual Intel Xeon 2.8 Ghz, 2 GB RAM
NWSSensor
GridFTPGRIS
GlobusGatekeeper
Dual Intel Xeon 2.8 Ghz, 2 GB RAM
NWSSensor
GridFTPGRIS
GlobusGatekeeper
Dual Intel Xeon 2.8 Ghz, 2 GB RAM
GRIDS Lab, University of Melbourne
Dept. of Physics,University of Sydney
ANU, Canberra
Dept. of Computer Science, University of Adelaide
NWSSensor
GridFTPGRIS
GlobusGatekeeper
Intel Pentium 2.0 Ghz, 512 MB RAM
Dept. of Physics,University of Melbourne
NWSSensor
GridFTPGRIS
GlobusGatekeeper
Dual Intel Xeon 2.8 Ghz, 2 GB RAM
33
Case Study: Event Simulation and Analysis
B0->D*+D*-Ks
• Simulation and Analysis Package - Belle Analysis Software Framework (BASF)• Experiment in 2 parts – Generation of Simulated Data and Analysis of the distributed data
Analyzed 100 data files (30MB each) were distributed among the five nodes
34
Resources Used and their Service Price
Organization
Node details Role Cost (in G$/CPU-sec)
CS,UniMelb belle.cs.mu.oz.au4 CPU, 2GB RAM, 40 GB HD, Linux
Broker host, Data host, NWS server
N.A. (Not used as a compute resource)
Physics, UniMelb fleagle.ph.unimelb.edu.au1 CPU, 512 MB RAM, 40 GB HD, Linux
Replica Catalog host, Data host, Compute resource, NWS sensor
2
CS, University of Adelaide
belle.cs.adelaide.edu.au4 CPU (only 1 available) , 2GB RAM, 40 GB HD, Linux
Data host, NWS sensor
N.A. (Not used as a compute resource)
ANU, Canberra belle.anu.edu.au4 CPU, 2GB RAM, 40 GB HD, Linux
Data host, Compute resource, NWS sensor
4
Dept of Physics, USyd
belle.physics.usyd.edu.au4 CPU (only 1 available), 2GB RAM, 40 GB HD, Linux
Data host, Compute resource, NWS sensor
4
VPAC, Melbourne
brecca-2.vpac.org180 node cluster (only head node used), Linux
Compute resource,NWS sensor
6
35
Network Cost (in Grid $/Currency!)
NETWORK COSTS BETWEEN THE DATA HOSTS AND THE COMPUTE RESOURCES
(IN G$ PER MB) Data Node
Compute Node ANU UniMelb
Physics Sydney Physics
VPAC
ANU 0 34.0 31.0 38.0 Adelaide CS 34.0 36.0 31.0 33.0 UniMelb Physics 40.0 0 32.0 39.0 UniMelb CS 36.0 30.0 33.0 37.0 Sydney Physics 35.0 33.0 0 37.0
36
Deploying Application Scenario
A data grid scenario with 100 jobs and each accessing remote data of ~30MB
Deadline: 3hrs. Budget: G$ 60K Scheduling Optimisation Scenario:
Minimise Time Minimise Cost
Results:
SUMMARY OF EVALUATION RESULTS
Scheduling strategy Total Time Taken (mins.)
Compute Cost (G$)
Data Cost (G$)
Total Cost (G$)
Cost Minimization 71.07 26865 7560 34425 Time Minimization 48.5 50938 7452 58390
37
Grid and Gridbus Technologies for Various Grid (Market) Types
commercialscientific
free trading
regulation
Publiccomputin
g(Alchemi)
National provider(Globus, Gridbus,..)
Private enterprises
(Libra, Gridbus, Globus)
Application Category
SharingModel
39
Summary and Conclusion
Grids exploit synergies that result from cooperation of autonomous entities:
Resource sharing, dynamic provisioning, and aggregation at global level.
Grid Economy provides incentive needed for sustained cooperation.
Grid Network has potential to serve as Cyberinfrastructure for Utility Computing
Grids offer enormous opportunities for realizing eScience and eBusiness at global level.
40
Any Questions ?
Gridbus Project - http://www.gridbus.org
41
Thanks for your attention!
The Gridbus Cooperation!http://www.gridbus.com
Backup Slides
45
What do Grids aim for and how to support them.
Grids aim at exploiting synergies that result from cooperation of autonomous distributed entities. Synergies include:
Resource sharing “On-demand” Virtual Enterprises creation Aggregation of resources on demand.
For this cooperation to be sustainable, participants needs to have (economic) incentive.
Therefore, “incentive” mechanisms should be considered as one of key design parameters of Grid computing.
46
Grid Market (Participant) Types and Application Category
commercialscientific
free trading
regulation
Publiccomputin
g
National provider
Private enterprises
Application Category
SharingModel
47
Appropriate Market Model for different market types
strongweak
high
low
Variable price
auction
Posted price
oligopoly
Commodity market
Demand elasticity
Willingness to Pay
49
Deadline (D) and Budget (B) Constrained Scheduling Algorithms
Algorithm
Execution Time (D)
Execution Cost (B)
Compute Grid
Data Grid
Cost Opt Limited by D
Minimize Yes Yes
Cost-Time Opt
Minimize if possible
Minimize Yes
Time Opt Minimize Limited by B
Yes Yes
Conservative-Time Opt
Minimize Limited by B, jobs have guaranteed minimum budget
Yes