mardi 30 mars 2010
Lavoisier : a way to integrate heteregeneous monitoring systems.
Cyril L’Orphelin
IN2P3/CNRS Computing Centre, Lyon, France
HEPiX Spring 2010 Workshop 2
OutlinesOutlines
- Motivations
- Main benefits of using Lavoisier
- Concrete example : the Operations Dashboard
- Future developments
HEPiX Spring 2010 Workshop 330/03/10 3
EGEE / LCG information providersEGEE / LCG information providers
CIC-DB- VO Information - VO Contacts- User Contacts
SQLSQLVO repository (static)
GOC-DB - Site Information- Site Contacts- Downtimes
SQLSQLSite repository (static)
GGUS - Create ticket- Update ticket- View ticket
SOAPTicketing System
SAM- Test results- Status per nodes
X-SQLJob Submission Framework
BDII LDAPDynamic information per nodes , per sites ...
LDAP Database for sites
VOMS HTTP + Script - VOMS certificates.- VOMS users
VO User repository (dynamic)
- GIIS status per site- Storage Information- Jobs Information
GSTAT HTTPGIIS browser
HTTP
Monitoring Framework Nagios Notifications about problem on services
AMQ
HEPiX Spring 2010 Workshop 4
Multiple heterogeneous sources and technologies :
No common language
No direct cross queries
• For a web application , it means lot of computations and treatment and a dependency to the technologies proposed
The information might be : Not usable Not available Too long to be queried
Motivations - ContextMotivations - Context
HEPiX Spring 2010 Workshop 530/03/10 5
Our solution : LavoisierOur solution : Lavoisier
An answer developped at CC-IN2P3: the Lavoisier service
A good mechanism to build unified views from heterogeneous data sources
http://grid.in2p3.fr/lavoisier
"Nothing is lost, nothing is created, all is transformed" Antoine Laurent Lavoisier (1743 - 1794)
HEPiX Spring 2010 Workshop 6
Access data views through SOAP, REST/JSON or REST/XMLo get data data viewo cross-query data views using standard languages: XSLT (and soon: XPath, XQuery)
Accesses to dataAccesses to data
HEPiX Spring 2010 Workshop 7
Separation of roles (changes related to a role have no impact on other roles)o the plug-in developer adds support for new technologies by developing adapters.o the service administrator configures the adaters and cache management according to the
characteristics of the data views and data sources, and according to the constraints of the use-cases.
o the user queries the data views.
Separation of rolesSeparation of roles
HEPiX Spring 2010 Workshop 8
Data cache management o select cache type
in-memory (simple or hierarchical, parsed once) on-disk (simple or hierarchical) no-cache (directly invoke the data source)
o combine cache refresh triggering rules: whenCreated, whenNotified, whenTime, whenMaxUnrefreshedTimeElapsed, whenMaxUnrefreshedTimeElapsedAndDataAccessed, whenUpdated, whenRefreshed, whenRefreshedAndPublished
Extensibility
o Reusable adapters are provided for commonly used technologies: HTTP, LDAP, RDBMS, XSLT...
o Support for new technlogies can be added by developing adapters
Validation of generated XML data o check XML syntax only o check XML syntax and conformity to provided XML Schema
Administration o get data views index and properties o get data views status o on-the-fly reconfiguration (detect views to be reconfigured)
Other main featuresOther main features
HEPiX Spring 2010 Workshop 9
Lavoisier Console
HEPiX Spring 2010 Workshop 10
The Operations dashboard
Heterogeneous data XML HTML
HEPiX Spring 2010 Workshop 11
The Synoptic view
https://operations-portal.in2p3.fr/
HEPiX Spring 2010 Workshop 12
Access to details
HEPiX Spring 2010 Workshop 13
Future developments (Lavoisier 1.x)
RDBMS adaptor (on-going development)user see the entire database as a huge XML document
structure described by Hibernate or Doctrine ORM mapping file
query with XPath / response as XML
but of course this huge XML document is never builtXPath query is translated to SQL query
only the result set is converted to XML data
XQuery and XPath as alternatives to XSLTavailable through SOAP and REST operations
available as adapters (=> cacheable processing result)
HEPiX Spring 2010 Workshop 14
Future developments (Lavoisier 2.0)
Refactoring of core enginebased on JMS to increase flexibility for notifications
within a Lavoisier instance (through in-memory messages)
between Lavoisier instances (though OpenWire protocol)
improve extensibility with adapter interfaces for…
Data protocol connectors
Data format transformers
Cache content managers 4 hard-coded managers
Cache refresh triggers 8 hard-coded triggers
Cache refresh scheduler No scheduler (overload risk)
Adapters do both (less reusable)
Lavoisier 2.0Lavoisier 2.0 Lavoisier 1.xLavoisier 1.x
HEPiX Spring 2010 Workshop 15
Links
Lavoisier : • http://grid.in2p3.fr/lavoisier/
Dashboard :• https://operations-portal.in2p3.fr (you need to be
registered in GOC DB )
• https://forge.in2p3.fr/projects/show/opsportaluser