ADAPTING THE DEVELOPMENT MODEL OF THE GRID
ANATOMY TO MEET THE NEEDS OF VARIOUS
APPLICATION DOMAINS
Soha Maad, Brian Coghlan, Gabriel Pierantoni,
Eamonn Kenny, John Ryan, Watson Ronan
Trinity College Dublin, Dublin, Ireland
Cracow Grid Workshop CGW'05 (Cracow, Poland)
Outline
● The grid in various application domains● Application domain and grid middleware development● Software Engineering perspective on grid middleware
development● Software Engineering Practice Proposed● A concrete implementation example
Cracow Grid Workshop CGW'05 (Cracow, Poland)
The grid in various application domains
Cracow Grid Workshop CGW'05 (Cracow, Poland)
Che
mis
try
Gov
ernm
ent
Phy
sics
Med
icin
e
Ast
rono
my
Env
iron
men
t
Eng
inee
ring
Med
ia
Fin
ance
Challenge
Challenge
Limitation
Limitation
Grid Solution
Grid Solution
The grid in various application domains
Cracow Grid Workshop CGW'05 (Cracow, Poland)
Che
mis
try
Gov
ernm
ent
Phy
sics
Med
icin
e
Ast
rono
my
Env
iron
men
t
Eng
inee
ring
Med
ia
Fin
ance
meeting the deadline of the start date for conducting the physics experiments
Limitation
Limitation
conducting experiments that need to process tera-bytes scale of data (e.g. LHC, ALICE, ATLAS).
Challenge
Challenge
the EGEE middleware is developpedGrid Solution
Grid Solution
The grid in various application domains
Cracow Grid Workshop CGW'05 (Cracow, Poland)
Che
mis
try
Gov
ernm
ent
Phy
sics
Med
icin
e
Ast
rono
my
Env
iron
men
t
Eng
inee
ring
Med
ia
Fin
ance
1) producing interactive medical simulations (e.g. heart simulation).2) analysing and managing medical images.3) supporting virtual collaboration in e-hospital.
Challenge
Challenge
Grids provide computing power but lack capabilities needed for communication and knowledge sharing. Limitation
Limitation - Special purpose middleware (e.g. innerGrid Nitya developed by Grid Systems, also CrossGrid)- Resource brokers for the discovery of suitable clusters for the execution of parallel image reconstruction algorithms. - Metagrid Engine that provides a superset of functionalities across different Grid Engines with privacy and QoS.
Grid Solution
Grid Solution
The grid in various application domains
Cracow Grid Workshop CGW'05 (Cracow, Poland)
Che
mis
try
Gov
ernm
ent
Phy
sics
Med
icin
e
Ast
rono
my
Env
iron
men
t
Eng
inee
ring
Med
ia
Fin
ance
Challenge
Challenge
Analysis of tera-bytes of astronomical image data from telescopes. This necessitates: data intensive computation; scalable file I/O in GB/s; replica management; and parallel/distributed processing of files.
Grid Solution
Grid Solution
GFARM Grid file system offered a special purpose Grid middleware for data intensive computation
Limitation
Limitation
No direct connection between the GFARM system and the telescope devices.
The grid in various application domains
Cracow Grid Workshop CGW'05 (Cracow, Poland)
Che
mis
try
Gov
ernm
ent
Phy
sics
Med
icin
e
Ast
rono
my
Env
iron
men
t
Eng
inee
ring
Med
ia
Fin
ance
Challenge
Challenge parallel running of hundreds of programs corresponding to large scale air pollution, nuclear waste storage, pollution and weathering models
Grid Solution
Grid Solution
For large scale air pollution models:National (UK): 7 institutions using GT3International: 21 institutions using CrossGrid
Limitation
Limitation
Future research work aims at deploying a WSRF compliant middleware.
The grid in various application domains
Cracow Grid Workshop CGW'05 (Cracow, Poland)
Che
mis
try
Gov
ernm
ent
Phy
sics
Med
icin
e
Ast
rono
my
Env
iron
men
t
Eng
inee
ring
Med
ia
Fin
ance
Challenge
ChallengeVirtual collaboration on design, production and maintenance of products that are described in complex structured product model databases.
Limitation
Limitation
- the need for the grid to be aware of the business concepts - to make typical server side applications grid-computing compatible, - to allow CAD applications to interface with the grid.
Grid Solution
Grid Solution
The VO concept should be enhanced to a robust framework with flexibility, adaptability, and security.Proposed generic business-object-aware middleware extensions: 1) grid-enabling existing applications using toolkits; 2) extending the grid architecture with semantics and ontologies.
The grid in various application domains
Cracow Grid Workshop CGW'05 (Cracow, Poland)
Che
mis
try
Gov
ernm
ent
Phy
sics
Med
icin
e
Ast
rono
my
Env
iron
men
t
Eng
inee
ring
Med
ia
Fin
ance
Challenge
ChallengeThe production, broadcasting, delivery and playout, of interactive media content (audio, video, image) in real time.
Limitation
Limitation
Difficulties:- To split media production pipeline distribute it to grid resources;- To guarantee reliable and secure interaction with the pipeline. - To provide reliable and secure interactivity with the grid
(CrossGrid is only ‘near-interactive)
Grid Solution
Grid SolutionCrossGrid:- The development of the Grid Visualisation Kernel (GVK) which allows the visualisation pipeline to be ported on grid resources. - The development of G-Vid which is based on GVK for production of real time interactive video.
The grid in various application domains
Cracow Grid Workshop CGW'05 (Cracow, Poland)
Che
mis
try
Gov
ernm
ent
Phy
sics
Med
icin
e
Ast
rono
my
Env
iron
men
t
Eng
inee
ring
Med
ia
Fin
ance
Challenge
Challenge1) Solving real world large scale investment problems 2) Realistic financial modelling 3) Regulatory constraints (Basle II) force big finance houses to create their own grid apps.
Grid Solution
Grid Solution
Implementation of the Open Grid Service Environment (model large scale computational financial problems as abstract workflows).
Limitation
Limitation
Need to converge to web service implementation
The grid in various application domains
Cracow Grid Workshop CGW'05 (Cracow, Poland)
Che
mis
try
Gov
ernm
ent
Phy
sics
Med
icin
e
Ast
rono
my
Env
iron
men
t
Eng
inee
ring
Med
ia
Fin
ance
Challenge
ChallengeMolecular design and engineering (e.g. QSAR/QSPR methodology involves 3D structure generation, semi-empirical calculations, descriptor calculation, and model building.
Grid Solution
Grid Solution
OpenMolGRID system, based on UNICORE grid middleware.
The grid in various application domains
Cracow Grid Workshop CGW'05 (Cracow, Poland)
Che
mis
try
Gov
ernm
ent
Phy
sics
Med
icin
e
Ast
rono
my
Env
iron
men
t
Eng
inee
ring
Med
ia
Fin
ance
Challenge
Challenge
Limitation
Limitation
Grid Solution
Grid Solution
DiscussionImpact of the Application Domain on Grid Middleware
Cracow Grid Workshop CGW'05 (Cracow, Poland)
Che
mis
try
Gov
ernm
ent
Phy
sics
Med
icin
e
Ast
rono
my
Env
iron
men
t
Eng
inee
ring
Med
ia
Fin
ance
deve
lopi
ng n
ew m
iddl
ewar
e ke
rnel
s
deve
lopi
ng n
ew m
iddl
ewar
e ke
rnel
s
deve
lopi
ng r
esou
rce
brok
ers
deve
lopi
ng r
esou
rce
brok
ers
intr
oduc
ing
new
met
agri
d en
gine
s
intr
oduc
ing
new
met
agri
d en
gine
s
Dev
elop
ing
grid
ser
vice
API
s
Dev
elop
ing
grid
ser
vice
API
s
addi
ng n
ew m
iddl
ewar
e se
rvic
es
addi
ng n
ew m
iddl
ewar
e se
rvic
es
deve
lopi
ng g
rid
appl
icat
ion
tool
kits
deve
lopi
ng g
rid
appl
icat
ion
tool
kits
Rai
sing
gri
d aw
aren
ess
of b
usin
ess
conc
epts
Rai
sing
gri
d aw
aren
ess
of b
usin
ess
conc
epts
advi
sing
sem
antic
-aw
are
sol
utio
ns
advi
sing
sem
antic
-aw
are
sol
utio
ns
Dev
elop
ing
spec
ial p
urpo
se g
rid
mid
dlew
are
Dev
elop
ing
spec
ial p
urpo
se g
rid
mid
dlew
are
Key Issue Addressed
Grid Middleware Software Development Model
Bottom-up approach
Top-down approach
Cracow Grid Workshop CGW'05 (Cracow, Poland)
Bottom-up Approach
“Ecosystem” of Grid Components (Globus Alliance)
Individual parts of the system are specified in detail and the parts are then composed to form larger components, which are in turn composed until a complete system is formed
The exercise of building a Grid system or application is framed as a software integration problem, hopefully, leveraging existing grid components to reduce the development cost .
Cracow Grid Workshop CGW'05 (Cracow, Poland)
Top-Down Approach
In WebComG, applications are specified as Condensed Graphs in a manner which is independent of the execution architecture, thus separating the application and execution environments.
Cracow Grid Workshop CGW'05 (Cracow, Poland)
However …..
However, our survey reveals that the development cost of grid applications using a bottom-up approach is considerable.
By adopting a bottom-up approach, most of grid middleware development has evolved in isolation of application domain needs.
Cracow Grid Workshop CGW'05 (Cracow, Poland)
However …..
A top-down approach for grid middleware development may lead to narrow solutions and tend to be less generic
Cracow Grid Workshop CGW'05 (Cracow, Poland)
Adapting the development model of the grid anatomy
Cracow Grid Workshop CGW'05 (Cracow, Poland)
We propose marrying a top-down approach with a bottom-up approach for middleware development:
From a software engineering perspective, we need bridge bridge infrastructuresinfrastructures capable of merging a top down with a bottom-up approach. This relies on:
● A Unified Computational Model capable of expressing the workflow of operations
● Stateful Web Services for the implementation of the Bridge Infrastructures.
A Concrete Example
Cracow Grid Workshop CGW'05 (Cracow, Poland)
Border servicesBorder services involve :
Guardian services:Guardian services: have the duty to check whether all the conditions to cross the border are met
Translation services:Translation services: have the duty to translate information between the different protocols and languages used in WebCom-G and the various Grid Middlewares.
Proxy servicesProxy services have the duty to act in one middleware on behalf of a service residing in the other.
A Concrete Example
Cracow Grid Workshop CGW'05 (Cracow, Poland)
Border Region
Border Region
LCG2 RegionLCG2 Region
GT4 RegionGT4 Region
WebComRegion
WebComRegion
At most two technologies overlap
WebCom-G Interoperability with existing Grid MiddlewareWebCom-G Interoperability with existing Grid Middleware
Cracow Grid Workshop CGW'05 (Cracow, Poland)