22
21 Sep 2005 LCG's R-GMA Applications R-GMA and LCG Steve Fisher & Antony Wilson
| Date post: | 28-Mar-2015 |
| Category: |
Documents |
| Upload: | jose-nolan |
| View: | 216 times |
| Download: | 1 times |
21 Sep 2005 LCG's R-GMA Applications
R-GMA and LCG
Steve Fisher & Antony Wilson
21 Sep 2005 LCG's R-GMA Applications
A unified approach to I & M
• Provides information on both:– The Grid itself
• Mainly for the middleware packages, but the user may also query it to understand the status of the Grid
– Grid applications• For users
• Flexible infrastructure – All data has a timestamp enabling its use for
monitoring– Dynamic addition and deletion of information
producers– Dynamic creation and deletion of tables in schema– Able to cope with failures in a distributed environment
21 Sep 2005 LCG's R-GMA Applications
R-GMA – one cloud
• A relational implementation of GMA (from GGF)– Powerful data model and
query language• All data modelled as tables• SQL can express complex
queries in one expression
• Creates impression that you have one RDBMS per VO
Producer
Consumer
R-GMA
ApplicationsApplications
21 Sep 2005 LCG's R-GMA Applications
Job Monitoring• Requirements
– Allow grid users to monitor the progress of their jobs– Allow VO administrators to get an overview of what is
happening on the grid
• The Problem– The location where a grid job will run is not known– Worker nodes are behind firewall so are not accessible
• The Solution– R-GMA provides transport across firewalls
• The wrapper approach• Instrumenting the code
– Using a logging service– Calling R-GMA directly
21 Sep 2005 LCG's R-GMA Applications
21 Sep 2005 LCG's R-GMA Applications
LCG Job Monitoring
Deployed by LCG
No changes needed to user code
R-GMA Primary Producer
SP R-GMA Secondary Producer
PP
Job wrapper
PPUser code
Job wrapper
PPUser code
SPDatabase
SP
21 Sep 2005 LCG's R-GMA Applications
LCG Job Monitoring
R-GMA Primary Producer
SP R-GMA Secondary Producer
PP
Deployed by LCG
Job wrapper
PPUser code
Job wrapper
PPUser code
SPDatabase
SP
L&B Server
PP
21 Sep 2005 LCG's R-GMA Applications
Producers
PP
PP
PP
SP
PP - Primary ProducersInitial source of dataData published by user code stored by Producer Service
SP- Secondary ProducersUsed to republish data in order to:
co-locate information to speed up queries reduce network traffic use different producer properties
21 Sep 2005 LCG's R-GMA Applications
Producer Properties
• Producers may use:– Memory
• Gives best performance for continuous queries– Database
• Best performance for joins
• Primary and Secondary Producers must support – Continuous queries
• as soon as new data becomes available it is broadcast to all interested parties
• Primary and Secondary Producers may support– History Queries
• over time sequenced data– Latest Queries
• correspond to intuitive idea of current information
SEE DEMOSEE DEMO
Job Monitoring Web Interface
21 Sep 2005 LCG's R-GMA Applications
Application Monitoring
Users can define their own tables
C, C++, Java and Python apis available log4j appender available
PPUser code
PPUser code
R-GMA Primary Producer
SP R-GMA Secondary Producer
PP
SPDatabase
SP
21 Sep 2005 LCG's R-GMA Applications
Network Monitoring
Deployed by EGEE
Data obtained from network monitoring sensors are published via R-GMA
R-GMA Primary Producer
SP R-GMA Secondary Producer
PP
PP
Network sensor
PP
Network sensor
SPDatabase
SP
21 Sep 2005 LCG's R-GMA Applications
21 Sep 2005 LCG's R-GMA Applications
GridFTP Monitoring
Deployed by LCG
Data are extracted from GridFTP logs and published via R-GMA
R-GMA Primary Producer
C R-GMA Consumer
PP
C Oracle
RDBMS
PP
GridFTP log monitor
PP
GridFTP log monitor
21 Sep 2005 LCG's R-GMA Applications
Consumer Service
• Query types:– History Queries
• return time sequenced data
– Latest Queries• correspond to intuitive idea of current
information
– Continuous queries• as soon as new data becomes available it is
broadcast to all interested parties
SEE DEMOSEE DEMO
Grid ViewGridFTP Monitor
21 Sep 2005 LCG's R-GMA Applications
Network Monitoring
Deployed by Grid Ireland
TCP logs are queried in response to queries and the results are returned via R-GMA
R-GMA Consumer
ODP R-GMA On-Demand Producer
C
CUser code
CUser code
SPTCP Log
ODP
21 Sep 2005 LCG's R-GMA Applications
On Demand Producers
• Provides a Producer interface to non relational data sources
• User has to write code to handle SQL• Has been used to provide access to
TCPdump data
21 Sep 2005 LCG's R-GMA Applications
R-GMA plans
• Will be taken into EGEE-II– Change from Servlets to Web Services– Add multi-VDB support– Support more than MySQL– Add authorization
• Conform to GGF/INFOD specification
21 Sep 2005 LCG's R-GMA Applications
Further Information
• R-GMA web site– http://www.r-gma.org/
• JRA1-UK web site– http://hepunx.rl.ac.uk/egee/jra1-uk/
• E-mail– [email protected]
