Date post: | 20-Jan-2018 |
Category: |
Documents |
Upload: | suzan-carter |
View: | 234 times |
Download: | 0 times |
www.egi.euEGI-InSPIRE RI-261323
EGI-InSPIRE
www.egi.euEGI-InSPIRE RI-2613231
EGI-InSPIRE Review:Work Package 6 (SA3)
Jamie ShiersCERN
SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
www.egi.euEGI-InSPIRE RI-2613232
Activity Overview
• This slide will be provided by the PO• It will summarise the activity in tables by:
– The # partners, # people, # countries– The # PM and #FTE per country
• It will summarise the activity in graphics by:– The % effort of the activity within the project– The geographical spread across Europe
SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
www.egi.euEGI-InSPIRE RI-2613233
Partner Breakdown
Taken from the Description of Work
Participant number Participant Name Person-months per participant
10 KIT-6 27
12 CSIC 45
13 CSC 18
14 CNRS 83
19 TCD 21
21 INFN 126
28 CYFRONET 6
31 ARNES 3
32 UI SAV 18
35 CERN 341
37 EMBL 44
Total 732
www.egi.euEGI-InSPIRE RI-2613234
Task Breakdown
SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
Task ActivitiesTSA3.1 Activity ManagementTSA3.2 Shared services and toolsTSA3.3 Services for High Energy Physics (HEP)TSA3.4 Services for Life Sciences (LS)TSA3.5 Services for Astronomy and Astrophysics (A&A)TSA3.6 Services for Earth Sciences (ES)
Other HUCS: • Computational Chemistry and Materials Sciences and Technologies (CCMST),• Fusion (F)
www.egi.euEGI-InSPIRE RI-2613235
Work Package Objectives
• Transition to sustainable support:
– Identify tools of benefit to multiple communities
– Migrate these as part of the core infrastructure
– Establish support models for those relevant to individual communities
SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
www.egi.euEGI-InSPIRE RI-2613237
Communities & Activities
Maria Girone, IEEE MSST
High Energy Physics The four LHC experiments use grid computing for data distribution,
processing and analysis. Strong focus on common tools and solutions. Areas supported include: Data Management, Data Analysis and Monitoring. Main VOs: ALICE, ATLAS, CMS, LHCb
Life Sciences
Covers a variety of projects including the European Extremely Large Telescope (E-ELT), the Square Kilometre Array (SKA) and Cerenkov Telescope Array (CTA). Activities focus on visualisation tools and database/catalog access from the grid.Covers seismology, atmospheric modelling, meteorological forecasting, flood forecasting and climate change. Provides access from the grid to resources within the Ground European Network for Earth Science Interoperations - Digital Repositories (GENESI-DR). Also assists scientists working on climate change via the Climate-G testbed.
Astronomy & Astrophysics
Earth Sciences
Life Sciences
Focuses on medical, biomedical and bioinformatics sectors to connect worldwide laboratories, share resources and ease access to data in a secure and confidential way. Supports 4 VOs (biomed, lsgri, vlemed and pneumogrid) across 6 NGIs via the Life Science Grid Community.
www.egi.euEGI-InSPIRE RI-2613238
Communities & Activities
Maria Girone, IEEE MSST
High Energy Physics The four LHC experiments use grid computing for data distribution,
processing and analysis. Strong focus on common tools and solutions. Areas supported include: Data Management, Data Analysis and Monitoring. Main VOs: ALICE, ATLAS, CMS, LHCb
Life Sciences
Covers a variety of projects including the European Extremely Large Telescope (E-ELT), the Square Kilometre Array (SKA) and Cerenkov Telescope Array (CTA). Activities focus on visualisation tools and database/catalog access from the grid.Covers seismology, atmospheric modelling, meteorological forecasting, flood forecasting and climate change. Provides access from the grid to resources within the Ground European Network for Earth Science Interoperations - Digital Repositories (GENESI-DR). Also assists scientists working on climate change via the Climate-G testbed.
Astronomy & Astrophysics
Earth Sciences
Life Sciences
Focuses on medical, biomedical and bioinformatics sectors to connect worldwide laboratories, share resources and ease access to data in a secure and confidential way. Supports 4 VOs (biomed, lsgri, vlemed and pneumogrid) across 6 NGIs via the Life Science Grid Community.
These and other communities / projects supported by shared tools & services
www.egi.euEGI-InSPIRE RI-2613239
Services for HUCs
1. Successfully supported major production computing at an unprecedented scale – both quantitatively and qualitatively
2. Successfully delivered common solutions in a variety of areas – with other activities in progress
3. Actively participated in EGI Technical & User Forum via presentations, tutorials and demos
4. Broadened the use of grid technology and HUC services to related projects within the HUC domain (such as unfunded – by EGI-InSPIRE – LS / ES projects)
5. Completed first round of Milestones & Deliverables together with associated technical work
6. Identified – across all HUC communities – areas of common technology investigation for the future
7. Developed a S.W.O.T. analysis of each main discipline and made significant steps on the road to sustainability
SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
10
Service Quantity & Quality
• The first year of EGI-InSPIRE saw the use of grid computing at an unprecedented scale!– More than 100 CPU-millennia delivered;– More than 50 PB of data stored;– Data transfer rates of 200TB/day
• At the same time, its power in turning scientific data into publications at record speed was publically acknowledged (Economist, July 2010)
• And service delivery was typically smooth with a small number of problems requiring in-depth investigation
• Quality plots in backup slides
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
11
Common Solutions – ExamplesGanga Extensively used as a “gridification tool” by many projects /
disciplines. This includes not only communities within EGI-InSPIRE but also others in many fields
Numerous non-HEP and non-HUC projects
Mini-Dashboard
To be used with Ganga to monitor Ganga-based activity on the grid. Used by EnviroGRIDS and offered to NA3
EnviroGRIDS, NA3
Experiment Dashboards
Common schema and code base for all job monitoring applications: job summary, historical view & task monitoring – implemented from July 2010
ATLAS + CMS
GRelC Exploited by the Earth Science (ES) community (Climate-G testbed) and by other projects/disciplines related to Environment and Bioinformatics
LS, A&A, ES
MPI High impact on multiple user communities CCMST, A&A, F
Frameworks Use of LHCb’s DIRAC framework by LCD/ILC and Belle collaborations. Investigation of DIRAC by ES + others
HEP (beyond LHC), ES
Data Management
Data popularity (dynamic data placement / caching), consistency of catalogs / storage
LHC VOs
Site Stress Testing
HammerCloud service used at ATLAS, CMS and LHCb. Fully applicable to other VOs / communities
HEP
www.egi.euEGI-InSPIRE RI-26132312
Common Solutions
Maria Girone, IEEE MSST
Life Sciences
Astronomy & Astrophysics
Earth Sciences
Life SciencesConnect various tools with
multiple communities:the main disciplines and also
Fusion, Computational Chemistry
Fusion
HEP
Comp Chem
www.egi.euEGI-InSPIRE RI-26132313
Common Solutions
Maria Girone, IEEE MSST
Life Sciences
Astronomy & Astrophysics
Earth Sciences
Life Sciences
Ganga: job definition and management in HEP
(ATLAS and LHCb, Compass, Harp), Fusion, L&E Sciences.
Fusion
HEP
Comp Chem
www.egi.euEGI-InSPIRE RI-26132314
Common Solutions
Maria Girone, IEEE MSST
Life Sciences
Astronomy & Astrophysics
Earth Sciences
Life SciencesHammercloud: a site stress
testing system to validate site usability. Used by ATLAS, CMS and
LHCb.
Fusion
HEP
Comp Chem
www.egi.euEGI-InSPIRE RI-26132315
Common Solutions
Maria Girone, IEEE MSST
Life Sciences
Astronomy & Astrophysics
Earth Sciences
Life SciencesDashboards: single entry point to
monitoring of all 4 LHC experiment activities on the grid. (Mini)
Dashboard also used by E&L Sciences.
Fusion
HEP
Comp Chem
www.egi.euEGI-InSPIRE RI-26132316
Common Solutions
Maria Girone, IEEE MSST
Life Sciences
Astronomy & Astrophysics
Earth Sciences
Life SciencesDIRAC: workload and data
management used by LHCb plusother HEP experiments withinterest from Earth Sciences.
Fusion
HEP
Comp Chem
www.egi.euEGI-InSPIRE RI-26132317
Common Solutions
Maria Girone, IEEE MSST
Life Sciences
Astronomy & Astrophysics
Earth Sciences
Life Sciences
MPI: used by A&A, Fusion, Computational Chemistry to handle
parallel execution in grid environments
Fusion
HEP
Comp Chem
www.egi.euEGI-InSPIRE RI-26132318
Common Solutions
Maria Girone, IEEE MSST
Life Sciences
Astronomy & Astrophysics
Earth Sciences
Life SciencesGRelC: set of advanced data grid
services to manage Databases on the Grid.
Used by L&E Sciences and A&A.
Fusion
HEP
Comp Chem
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
19
EGI TF & UF• Actively participated in both EGI Technical Forum (September
2010, Amsterdam) and User Forum (April 2011 Vilnius)
– TF: Two sessions covering overview of tasks and sub-tasks; One session dedicated to discussion of Common Requirements
– UF: Numerous presentations, tutorials, demonstrations covering all areas of activity
– All sessions well attended with good feedback
• EGI TF 2011: sessions planned to highlight main achievements of WP in meeting goals plus a mini-workshop on sustainability
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
20
Broadening Use of the Grid
Concrete examples:
– Use of HUC solutions and expertise for “3 generations” of HEP experiments – LEP to ILC
– Bi-directional sharing of solutions and techniques not only for those activities explicitly supported through EGI-InSPIRE but also other projects, such as PARTNER, ULICE and EnviroGRIDS
– Other possible Earth Science projects in the pipeline
www.egi.euEGI-InSPIRE RI-261323
EGI-InSPIRE
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
21
Domain-Specific Work
A Summary of the Main Sub-Tasks: Details in Backup Slides
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
22
Services for HEP
• Distributed Analysis Support– Support for the tools used by
the experiments; common error-logger for ATLAS/CMS/LHCb
• Data Management Support– Dynamic caching / data popularity:
common solutions deployed• Persistency Framework
– Event and detector data: common to three LHC experiments
• Monitoring– All aspects of production and analysis:
additional common solutions deployed
www.egi.euEGI-InSPIRE RI-261323NAx - <<name>> - EGI-InSPIRE EC Review 2011
23
Services for Life Sciences
• TSA3.4 - Virtual Research Community building– Life Sciences Grid Community (LSGC)
• Services development and provision– TSA3.2.1 - Dashboard: design phase– TSA3.2.3 - Data encryption service: prototype
deployed– TSA3.2.3 - Database interface: GRelC deployed– TSA3.2.4 - Workflows: work on Taverna to start in
year 2– TSA3.4 - CoreBio services: work to start in year 2
www.egi.euEGI-InSPIRE RI-261323
Services for A&A• Preparatory studies:
– Visualization tools: VisIVO integration (TSA3.5.2)– Parallel (MPI/OpenMP) and GPU computing using
CUDA (TSA3.5.3)– Database services and integration with Virtual
Observatory (TSA3.5.4)• Achieved results:
– ms608: Gridification of VisIVO and VisIVO Service (TSA3.5.2)
– Parallel programming: testing activities with cosmological simulations codes (FLY, Gadget,Flash) (TSA3.5.3)
– Database services analysis (TSA3.5.4)
SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 24
www.egi.euEGI-InSPIRE RI-261323
Services for ES
• TSA 3.6 Support ES Activities in ES communities and projects, and carried out by researchers & students in Universities
• Main activity by proposal: access to GENESI-DEC– Status : Webservice available and validated with application– Extensions dependent on GENESI-DEC progress– Further developments in Task: Integration with available GEOSS services to access
Genesi and CLimate Data from ESG
• Since Jan 2011 common developments with the climate Earth System Grid (ESG)
– Access to ESG data from EGI-Infrastructure and vice versa– Main problem to solve: A & A due to different federations– Institute IPSL/CNRS, IPGP now unfunded partner in TSA3.6
SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 25
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
26
Shared Tools & ServicesTool / Service Description Communities
Dashboards Experiment Dashboards used by LHC VOs – single schema and application. Mini-dashboard – to be used with Ganga – also by ES
HEP, ES
Ganga / DIANE Both distributed analysis frameworks (ATLAS, LHCb) and general gridification tools
Wide range of diverse communities and applications
Hydra / GRelC Encryption service for LS; Grid-access to DBs and catalogues
LS, A&A, ES
SOMA2 / Taverna / Kepler
MPI Handles parallel execution in grid environments
CCMST, A&A, F
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
27
S.W.O.T. Analyses – D6.2Discipline Strengths Weaknesses Opportunities Threats
HEP 24 x 7 petascale computing
Time to resolve some incidents
Increased commonality
Need to adapt to new technologies
Life Sciences Ramp-up of LSGC
Issues with use of multi-grids
Community growth
No long-term funding
Astronomy & Astrophysics
Successful use of DCIs
Coordination of A&A activities
Wider use of DCIs
Funding
Earth Sciences Existing user community
Diversity of demands and technologies
Community growth
Funding
GRelC Community based approach, cross - discipline
Little feedback about the SA3 DB questionnaire
Community growth
Lack of use of registry
MPI Widely regarded as an important tool
External factors Use of large SMPs / GPGPUs
Need for standardisation
Notable overlap in these independent analyses
www.egi.euEGI-InSPIRE RI-261323
EGI-InSPIRE
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
28
Use of Resources
Breakdown by Task & Partner
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
29
Effort Breakdown (1/2)Task Partner Worked PM Funded Committed PM Achieved PM %
TSA3.1 35-CERN 8.8 6.0 147%
TSA3.2.1 14A-CNRS 0 0.7 0%
TSA3.2.1 14C-HealthGrid 0.3 3.3 8%
TSA3.2.1 35-CERN 22.4 20.0 112%
TSA3.2.2 31B-JSI 0.5 1.0 47%
TSA3.2.2 32-UI SAV 2.6 6.0 43%
TSA3.2.2 35-CERN 17.1 20.0 86%
TSA3.2.3 14A-CNRS 7.5 3.3 224%
TSA3.2.3 14B-CEA 0 2.7 0%
TSA3.2.3 21E-SPACI 4.0 9.0 44%
TSA3.2.4 12C-CIEMAT 6.9 6.0 115%
TSA3.2.4 13-CSC 5.5 6.0 91%
TSA3.2.4 28C-ICBP 0.7 2.0 33%
TSA3.2.4 37-EMBL 0 6.0 0%
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
30
Effort Breakdown (1/2)Task Partner Worked PM Funded Committed PM Achieved PM %
TSA3.1 35-CERN 8.8 6.0 147%
TSA3.2.1 14A-CNRS 0 0.7 0%
TSA3.2.1 14C-HealthGrid 0.3 3.3 8%
TSA3.2.1 35-CERN 22.4 20.0 112%
TSA3.2.2 31B-JSI 0.5 1.0 47%
TSA3.2.2 32-UI SAV 2.6 6.0 43%
TSA3.2.2 35-CERN 17.1 20.0 86%
TSA3.2.3 14A-CNRS 7.5 3.3 224%
TSA3.2.3 14B-CEA 0 2.7 0%
TSA3.2.3 21E-SPACI 4.0 9.0 44%
TSA3.2.4 12C-CIEMAT 6.9 6.0 115%
TSA3.2.4 13-CSC 5.5 6.0 91%
TSA3.2.4 28C-ICBP 0.7 2.0 33%
TSA3.2.4 37-EMBL 0 6.0 0%
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
31
Effort Breakdown (1/2)Task Partner Worked PM Funded Committed PM Achieved PM %
TSA3.1 35-CERN 8.8 6.0 147%
TSA3.2.1 14A-CNRS 0 0.7 0%
TSA3.2.1 14C-HealthGrid 0.3 3.3 8%
TSA3.2.1 35-CERN 22.4 20.0 112%
TSA3.2.2 31B-JSI 0.5 1.0 47%
TSA3.2.2 32-UI SAV 2.6 6.0 43%
TSA3.2.2 35-CERN 17.1 20.0 86%
TSA3.2.3 14A-CNRS 7.5 3.3 224%
TSA3.2.3 14B-CEA 0 2.7 0%
TSA3.2.3 21E-SPACI 4.0 9.0 44%
TSA3.2.4 12C-CIEMAT 6.9 6.0 115%
TSA3.2.4 13-CSC 5.5 6.0 91%
TSA3.2.4 28C-ICBP 0.7 2.0 33%
TSA3.2.4 37-EMBL 0 6.0 0%
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
32
Effort Breakdown (1/2)Task Partner Worked PM Funded Committed PM Achieved PM %
TSA3.1 35-CERN 8.8 6.0 147%
TSA3.2.1 14A-CNRS 0 0.7 0%
TSA3.2.1 14C-HealthGrid 0.3 3.3 8%
TSA3.2.1 35-CERN 22.4 20.0 112%
TSA3.2.2 31B-JSI 0.5 1.0 47%
TSA3.2.2 32-UI SAV 2.6 6.0 43%
TSA3.2.2 35-CERN 17.1 20.0 86%
TSA3.2.3 14A-CNRS 7.5 3.3 224%
TSA3.2.3 14B-CEA 0 2.7 0%
TSA3.2.3 21E-SPACI 4.0 9.0 44%
TSA3.2.4 12C-CIEMAT 6.9 6.0 115%
TSA3.2.4 13-CSC 5.5 6.0 91%
TSA3.2.4 28C-ICBP 0.7 2.0 33%
TSA3.2.4 37-EMBL 0 6.0 0%
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
33
Effort Breakdown (2/2)Task Partner Worked PM Funded Committed PM Achieved PM %
TSA3.2.5 12A-CSIC 5.6 9.0 62%
TSA3.2.5 19-TCD 7.0 7.0 100%
TSA3.2.5 21D-UNIPG 11.5 3.0 385%
TSA3.3 21A-INFN 0 20.0 0%
TSA3.3 35-CERN 65.6 67.7 97%
TSA3.4 14A-CNRS 2.6 3.3 77%
TSA3.4 14C-HealthGrid 2.9 6.3 46%
TSA3.4 37-EMBL 0 8.7 0%
TSA3.5 21C-INAF 8.3 10.0 83%
TSA3.6 10G-FRAUNHOFER 2.1 9.0 23%
TSA3.6 14A-CNRS 4.5 8.0 57%
Total: 186.3 244.0 76%
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
34
Effort Breakdown (2/2)Task Partner Worked PM Funded Committed PM Achieved PM %
TSA3.2.5 12A-CSIC 5.6 9.0 62%
TSA3.2.5 19-TCD 7.0 7.0 100%
TSA3.2.5 21D-UNIPG 11.5 3.0 385%
TSA3.3 21A-INFN 0 20.0 0%
TSA3.3 35-CERN 65.6 67.7 97%
TSA3.4 14A-CNRS 2.6 3.3 77%
TSA3.4 14C-HealthGrid 2.9 6.3 46%
TSA3.4 37-EMBL 0 8.7 0%
TSA3.5 21C-INAF 8.3 10.0 83%
TSA3.6 10G-FRAUNHOFER 2.1 9.0 23%
TSA3.6 14A-CNRS 4.5 8.0 57%
Total: 186.3 244.0 76%
If the Grid Observatory effort is removed, the underspend in PY1 is > 25%. Hopefully, this effort will be made up early in PY2.
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
35
Plans for Next Year
• In PY2 we will continue to identify and deliver common solutions within and across existing communities, further the work on sustainability and address key areas of weakness / concern identified in PY1
• This includes the evaluation – and eventual deployment – of new solutions / techniques
• No modifications to the DoW are foreseen• Carry-over of PM to later in the project?
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
36
Review of ObjectivesObjective Status
Supporting the tools, services and capabilities required by different HUCs
Achieved – work continues in PY2 / PY3
Identifying the tools, services and capabilities currently used by the HUCs that can benefit all user communities and to promote their adoption
Several additional items identified and shared across other communities – work will expand and continue in PY2 / PY3
Migrating the tools, services and capabilities that could benefit all user communities into a sustainable support model as part of the core EGI infrastructure
Not started – needs further work and discussion
Establishing a sustainable support model for the tools, services and capabilities that will remain relevant to single HUCs
Collaborative support is the basic model for sustainability that does not depend on individual partners nor specific project funding. A workshop on sustainability at the EGI TF is proposed – work continues and expands in PY2 and PY3. See also D6.2.
www.egi.euEGI-InSPIRE RI-26132337
Summary Successfully supported major production computing at an unprecedented scale
– both quantitatively and qualitatively Successfully delivered common solutions in a variety of areas – with other
activities in progress Actively participated in EGI Technical & User Forum via presentations, tutorials
and demos Broadened the use of grid technology and HUC services to related projects
within the HUC domain (such as unfunded – by EGI-InSPIRE – LS / ES projects) Completed first round of Milestones & Deliverables together with associated
technical work Identified – across all HUC communities – areas of common technology
investigation for the future Developed a S.W.O.T. analysis of each main discipline and made significant
steps on the road to sustainability
SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
www.egi.euEGI-InSPIRE RI-261323
EGI-InSPIRE
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
38
BACKUP SLIDES
www.egi.euEGI-InSPIRE RI-26132339
Milestones & Deliverables
SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
Milestone / Deliverable
Due Date Lead Partner (#) Title
MS601 PM1 CSC (13) HUC Contact points and the support model
MS602 PM4 INFN (21) HUC Software Roadmap
MS603 PM4 CERN (35) Services for High Energy Physics
D6.1 PM4 CERN (35) Capabilities offered by the HUCs to other communities
MS604 PM4 CNRS (14) Services for the Life Science Community
MS605 PM8 TCD (19) Training and dissemination event
D6.2 PM9 CERN (35) Sustainability plans for the HUC activities
MS606 PM10 INFN (21) HUC Software Roadmap
D6.3 PM11 CERN (35) Annual Report on the Tools and Services of the HUCs
MS607 PM12 CNRS (14) Hydra service deployment
MS608 PM12 INFN (21) Integration of the VisIVO server with the production infrastructure
www.egi.euEGI-InSPIRE RI-261323
EGI-InSPIRE
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
40
HEP
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
41
WLCG Service Incidents
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
42
SIRs – Time to Resolution
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
43
GGUS Tickets – LHC VOs
13-Ja
n23
-Feb5-A
pr
16-M
ay26
-Jun6-A
ug16
-Sep27
-Oct7-D
ec17
-Jan27
-Feb9-A
pr
20-M
ay30
-Jun
10-Aug
20-Se
p31
-Oct
11-Dec
21-Ja
n3-M
ar13
-Apr0
20
40
60
80
100
120
Total ALICETotal ATLASTotal CMSTotal LHCb
www.egi.euEGI-InSPIRE RI-26132344
Data Management
• Crucial area for LHC and other HEP experiments– Data volumes: tens of PB/year, rates: up to 200TB/day between
sites, several hundred active analysis users / experiment, 1M analysis jobs / day
• Experience from first data taking has shown that some assumptions on data placement are no longer optimal– based on decade+ old model “MONARC” which assumed network was
scarce and expensive resource
• Working to adapt computing models to reflect such changes
Maria Girone, IEEE MSST
www.egi.euEGI-InSPIRE RI-26132345
Data Placement & Dynamic Caching
• Based on MONARC, the initial phase of LHC data distribution was based on static pre-placement – Significant fraction of such data never read!
• Computing models now driving towards dynamic data placement– Replication is based on usage (“popularity”) – this results in
better network and storage utilization
• Implemented first for ATLAS, now for CMS and LHCb
Maria Girone, IEEE MSST
www.egi.euEGI-InSPIRE RI-26132346
Catalogue Consistency
• With 50PB of data storage across a large number of sites worldwide, inconsistencies can easily arise!
– Data that resides on Storage Elements but not in various catalogs (grid, experiment) referred to as “Dark Data”
• One site recently reported 70TB dark data!
• Using a messaging-based system, various catalogs and SEs can talk to each other and implement lazy synchronization
Maria Girone, IEEE MSST
www.egi.euEGI-InSPIRE RI-26132347
Data Analysis Support
• Covers the final stage of data processing leading on to publication– Large number of users/month (~1000) and analysis jobs/day (~1M) running across
(~100) Tier2 and other sites – “chaotic” data access– All frameworks support heterogeneous back-ends
• Ganga (ATLAS, LHCb) used by 10 other communities and 500 – 600 users
• Common site stress testing system (Hammercloud) used by ATLAS, CMS and LHCb
• Areas of commonality and optimization– Move to “community support” model– Simplify data access and improve monitoring– Use common components and frameworks, such as for job submission and file
transfer (built on gLite /EMI FTS)• An area of potential future common work and simplification
Maria Girone, IEEE MSST
www.egi.euEGI-InSPIRE RI-261323
EGI-InSPIRE
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
48
LS
www.egi.euEGI-InSPIRE RI-26132349
Life Sciences Grid Community
• LSGC: http://wiki.healthgrid.org/LSVRC:Index – 4 VOs, 6 NGIs, HG association, 2 EU projects– Communication channels: wiki, mailing lists,
monthly phone conferences• Technical team (shifters)
– Infrastructure monitoring and troubleshooting (Nagios server with LS resource probes)
• User management tools – early design phase– User registration and management DB– To be integrated in LS Dashboard
SA3 – Johan Montagnat – EGI-InSPIRE EC Review 2011
www.egi.euEGI-InSPIRE RI-261323NAx - <<name>> - EGI-InSPIRE EC Review 2011
50
LS Dashboard• Need to integrate
– Nagios monitoring dedicated interface (possibly based on GOC dashboard)
– VRC-level accounting information– User management tools (possibly based on
VOMRS)
www.egi.euEGI-InSPIRE RI-26132351
Data encryption service• Hydra server
– Encryption keystore server based on Shamir’s secret sharing algorithm
– Software packages available for gLite 3.1– Installation procedure documented for gLite 3.0
• Current status (MS607)– Working on gLite 3.1, not yet on gLite 3.2– Client CLIs to be installed on all LS supporting sites
• Perspectives– Deployment of a three-heads hydra server on three
sitesSA3 – Johan Montagnat – EGI-
InSPIRE EC Review 2011
www.egi.euEGI-InSPIRE RI-26132352
Database interface
• GRelC (Grid Relational Catalog) service provision to support LS use cases
• Discussion about new LS use cases and data resources analysis
• Identification of a couple of biological databases to be ported in grid (relational DBs)
• Contribution to the SA3 questionnaire related to “grid-databases”
SA3 – Johan Montagnat – EGI-InSPIRE EC Review 2011
www.egi.euEGI-InSPIRE RI-26132353
LS issues, mitigation & perspectives
• VOMS and LFC servers are single point of failures– Replication procedures being set up
• Infrastructure monitoring is time consuming– Scheduled downtimes should be better reflected in BDII– Dedicated view of Nagios results in LS dashboard– Nagios probes improvements
• LSGC is a multi-VOs / multi-grids community– No tooling available to manage VRCs– Multi-grids hardly addressed in the context of EGI
• Few feedback on the SA3 questionnaire– More dissemination is needed in PY2
SA3 – Johan Montagnat – EGI-InSPIRE EC Review 2011
www.egi.euEGI-InSPIRE RI-261323
EGI-InSPIRE
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
54
A&A
www.egi.euEGI-InSPIRE RI-261323
A&A Grid Community
• 23 VOs and 7 NGIs– The most part of active A&A VOs relate to the
astroparticle physics community– Communication channels:
• wiki (under construction)• mailing lists (a general mailing list currently in place,
others more specialized could follow) • phone conferences (organized on a monthly basis)
– A&A VRC meetings and workshops• The first one of the EGI era is under organization in
Paris (7 November 2011, ADASS Conference)
SA3 – Johan Montagnat – EGI-InSPIRE EC Review 2011 55
www.egi.euEGI-InSPIRE RI-261323
VisIVO• Porting of VisIVO server in Grid.
– Preparatory activity: enabling the usage of VisIVO directly within a code during the production phase.
• A software layer has been developed using the internal arrays and without the need of producing intermediate files.
• A library of VisIVO was designed and implemented– The first issue of the gridified version of VisIVO
server has been released in April 2011 (milestone MS608).
SA3 – Johan Montagnat – EGI-InSPIRE EC Review 2011 56
www.egi.euEGI-InSPIRE RI-261323
A&A issues, mitigation & perspectives
• A&A is a complex community: its coordination is quite challenging.
• No EU A&A projects currently funded shortage of funds coordination activity tricky no easy deployment of tools and services
• Our willingness is to continue the coordination and the efforts to strengthen the community– We rely on EGI.eu and NGIs support– We exploit as much as possible collaborative
tools and services to establish robust and stable communications
SA3 – Johan Montagnat – EGI-InSPIRE EC Review 2011 57
www.egi.euEGI-InSPIRE RI-261323
EGI-InSPIRE
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
58
ES
www.egi.euEGI-InSPIRE RI-261323
ES HUC
• ES HUC VOs:– “ESR VO” „catch all ES people“ provides resources and support for ES
projects and anorganized ES users – EU project VERCE (seismology) will start with this VO and will
work with SA3.6• VOs associated to ES HUC
– See http://www.euearthsciencegrid.org/content/egee-virtual-organisations-earth-science
• ES HUC Activities: – Yearly session at European Geoscience Union, General Assembly
2011 about “ES and E-infrastructures
SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 59
www.egi.euEGI-InSPIRE RI-261323
EGI-InSPIRE
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
60
Other Tools & Services
www.egi.euEGI-InSPIRE RI-261323
GRelC: Activity (I)
NAx - <<name>> - EGI-InSPIRE EC Review 2011 61
• User support through the implementation of a grid-database “registry” (EGI Database of Databases)– Easy search and discovery (cross-VO) of grid-DB
resources distributed across the EGI grid. – Community-based approach to attract new users and
address sustainability• SA3 Questionnaire – A census about database
resources, related needs and future plan– Distributed among the HUCs (end of Q3)– Few feedback; more dissemination during Y2
www.egi.euEGI-InSPIRE RI-261323
GRelC: Activity (II)
NAx - <<name>> - EGI-InSPIRE EC Review 2011 62
• The back-end modules of the “registry” finalized during Y1:– A MySQL catalog for the registry– Several Java classes to manage charts, grid-DB, VOs,
GRelC services, community-oriented aspects• Front-end modules related to the “registry”
finalized in Y1 (now tested, on line during Q5)– Registry view completed– Grid-Database view completed – Integration of Web2.0 (Mash-up, Google Maps,
permalinks) and community-oriented aspects (comments, scores, discussions groups, etc.)
www.egi.euEGI-InSPIRE RI-261323
GRelC: Registry Snapshots
NAx - <<name>> - EGI-InSPIRE EC Review 2011 63
www.egi.euEGI-InSPIRE RI-261323
GRelC: User support
NAx - <<name>> - EGI-InSPIRE EC Review 2011 64
• HUC support in terms of:– grid-metadata management for the Earth Science and
Environmental context (e.g. Climate-G, CMCC)– setup and hosting of a new GRelC service to implement
and run LS use cases (e.g. biological)– grid-DBs census and requirements collection for the HUC
through the SA3 Questionnaire (end of Q3)– Training and documentation resources
• Issues that arose:– Few feedback (from end of Q3) regarding the SA3
Questionnaire (further dissemination is needed during Y2, EGI-UF, EGI-TF, etc.)
www.egi.euEGI-InSPIRE RI-261323
GRelC: Plans for next year• The registry will be available online in PQ5• It will be the core part of the DashboardDB
(available during Y2)• The DashboardDB will provide specialized
views, charts and statistics about the GRelC instances deployed across the EGI grid
• The SA3 Questionnaire will be refined and distributed among the HUCs – The feedback will be reported into the registry
• No changes to the current plan are foreseen
NAx - <<name>> - EGI-InSPIRE EC Review 2011 65
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
66
Workflows / Schedulers– TSA3.2.4 – Kepler:
Kepler actors for gLite and Unicore fully operational. Tutorials provided, material available online. Use cases designed, created, and deployed. More use cases in year 2.
– TSA3.2.4 – GridWay: Available and used as standalone metascheduler. Integration GridWay - Kepler, to start in year 2.
www.egi.euEGI-InSPIRE RI-261323
Workflows / Schedulers
• Kepler actors for gLite and Unicore fully operational
• First workflows with different use cases have been developed
• Still work required for GridWay• Tutorials provided, online material• Interest of communities reached• SOMA2… (next)
www.egi.euEGI-InSPIRE RI-261323
SOMA2• SOMA2 in EGI-InSPIRE
– WP6: Services for the Heavy User Community (SA3)• TSA3.2 Shared Services and Tools• TSA3.2.4 Workflows and Schedulers
– 1st Project Year Goals• DCI integration
• Support for use of Grid middleware.• Users’ X509 certificate handling.
• Grid enabled services’ setup• Autodock 4 integration
• SOMA2 1.4 release• Includes grid support + more
– SOMA2 as a Service• Currently provided for Finnish academic researchers• In our roadmap we plan to offer the service to EGI as well (2nd year)
http://www.csc.fi/soma
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
69
MPI/Parallel Computing (I)
• Cross disciplinary activity and support• PY1 core objective achieved
– Centralized documentation in the wiki:• Admin Manual: https://wiki.egi.eu/wiki/MAN03• User Guide: https://wiki.egi.eu/wiki/MPI_User_Guide
– Multiple application models and algorithms– Outreach, training and dissemination
• Training material centralized at user guide wiki: https://wiki.egi.eu/wiki/MPI_User_Guide#Application_Execution
• Modest increase in #production sites– Improvements in monitoring => better service
www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011
70
MPI/Parallel Computing (II)
• User defined processes per node allocation– implemented in EMI-1 WMS, CREAM and MPI-Start– However, not yet in production– Extensive exploitation/testing by CCMST/ INFN (See
Laganà et al talk @ EGI UF 2011)– Hybrid OpenMP/MPI testing by TheoMPI (see Alfieri
et al. talk @ EGI UF 2011)• GP-GPU integration (with KVM)
– Some open issues • batch support, Lack of standardisation, Accounting etc
www.egi.euEGI-InSPIRE RI-261323
CCMST (volunt. serv.)• Virtual Research Community building
– Computational Chemistry & Material Sciences and Technology (CCMST) members certification and user support
• Services development and provision– GriF - a user friendly tool for job distribution on the
grid providing QoU and QoS information– GCres - a credit award system based on GriF – Workflows - evolution of Kepler, Pgrade and others– Packages and programs - Gaussian, GEMS,
Chimere, DL_Poly, Gromacs, CMAST– Virtual laboratory - Insilico Lab
NAx - <<name>> - EGI-InSPIRE EC Review 2011 71