Tony [email protected]
“GridPP2 – Project Specification”, GridPP9 Collaboration Meeting, Edinburgh, 4 February 2004
Tony Doyle - University of Glasgow
OutlineOutline
• Background… A Year in
the Life
• ARDA first thoughts
• Patchwork Quilting
• PMB High Level View
• EGEE Grid Operations
• GridPP2 Deployment Board
• GridPP2 User Board
• Who reports where?
• ARDA second thoughts
• We are legion..
• Reporting: General Points
• Proposed Structure
• Where does the work get done?
• Areas on Green Alert
• Areas on Amber Alert• Outcome?
Tony Doyle - University of Glasgow
4. Background…4. Background…GridPP2 steps towards project spec. GridPP2 steps towards project spec.
0. PPARC Call (February 2003)1.GridPP2 Proposal (30/5/03)2. Feedback from Reviewers3. Responses from the PMB See http://www.gridpp.ac.uk/docs/gridpp2/
4. Projects Peer Review Panel (14-15/7/03)5. Grid Steering Committee (28-29/7/03)See http://www.gridpp.ac.uk/talks/GridPP_GSC_030728.ppt
6. Tender Document (11/11/03)See http://www.gridpp.ac.uk/cb/doc/GridPP2_Tender.doc7. Science Committee (17-18/11/03)8. PPARC (25/11/03)9. CB (4/12/03) 10.PRSC (16/1/04). PRSC summary11.PMB (26/1/04). Work area effort tables12. GridPP9 Collaboration Meeting (4-5/2/04)13.OC (18/2/04)… GridPP2 presentation. (unlucky for some..)
2003 2004 2005 2006 2007 2008
Middleware Deployment Operations
ResourceProvider
UserCommunity
EDG
LCG
LCG
EGEE
EGEE
HEP/BIOMED/…
Prototype-1EDG 1.x
(GT2, Condor-G)
Prototype-2LCG-1 (EDG-2)
Prototype-3LCG-2 Releases
Production-1LCG-3
Production-2 Production-3
Migrate to OGSA/F. Follow GGF Standards
Tony Doyle - University of Glasgow
3. Background…3. Background…A Year in the Life of GlobusA Year in the Life of Globus
Tony Doyle - University of Glasgow
2. Background… 2. Background… UK e-Science DevelopmentUK e-Science Development
1. The UK e-Science Production Grid will have significant overlap with the GridPP2 Production Grid – Mutual benefit to engagement– e.g. convergence talks organised prior to
Christmas– GT2-based approach
2. OMII developments will be based upon the Web Services Resource Framework– The OMII delivery timescales are in 2005 onwards– Incorporates e.g. OGSA-DAI – Limited overlap with our proposed programme
Tony Doyle - University of Glasgow
1. Background… 1. Background… LCG DevelopmentLCG Development
1. AliEn provided a pre-Grid implementation
2. ARDA provides a framework for PP application middleware – We can and should (and do)
participate here– Identify your development within
this structure
Tony Doyle - University of Glasgow
ARDA thoughts..ARDA thoughts.. Ian Stokes-Rees Ian Stokes-Rees
• “ARDA” is the LCG middleware – framework to deliver a distributed analysis environment to the experiments.
• We need to look at the decomposition and contribute coherently to this
• There are existing products which need to be used as building blocks here, e.g.– “GANGA”=Grid Job
Submission Wizard– “DIRAC”=Distributed MC
Production Service– “Spitfire” = Metadata Service– “R-GMA” = Information Service
http://www-pnp.physics.ox.ac.uk/~stokes/arda/ARDA-views-ijstokes-20040122.pdf
Tony Doyle - University of Glasgow
Patchwork Quilting Patchwork Quilting the GridPP2 HourGlassthe GridPP2 HourGlass
Tender Document Outcome
Tier-1 Tier-1
Applications Applications
Middle/Sec/NetMiddle/Sec/Net
Tier-2 Tier-2
+ 1 Portal+ 1 PhenoGrid
-0.5 BaBar
-1 LHC Deployment
+1 LHC Deployment-1 Network-0.5 Workload
-1 Data Management
+1 Data Management
III. Grid Middleware
e.g. Information Services
I. Experiment Layer
e.g. Portals
II. Application Middleware
e.g. Metadata
IV. Facilities and Fabrics
e.g. Storage Services
Tony Doyle - University of Glasgow
SysMan Experiments
Developers
Deployment
Board
User
Board
Dissemination
Team
Developer
Team
GridPP2 PMB GridPP2 PMB High Level ViewHigh Level View
Project Management
Board
(PMB)
Collaboration Board
(CB)
Production Grid Deployment
Board
(DB)
User Board
(UB)
Project Management
Board
(PMB)
Collaboration Board
(CB)
Production Grid Deployment
Board
(DB)
User Board
(UB)
“Opinion
Formers”
PMB
Tony Doyle - University of Glasgow
PMB
User
Board
Developer
Team
GridPP2GridPP2Organisational View? (~75 FTEs)Organisational View? (~75 FTEs)
Deployment
Board
Dissemination
Team
Tony Doyle - University of Glasgow
Grid and Regional Grid and Regional OperationsOperations
• Summary Equation:GOC > ROC (= ROC+CIC)– UK has a leading role.. – Grid Operations Centre
a.k.a. "International Rescue”
– But if its that exciting by the end of GridPP2, then we will have failed..
• Regional Operations Centre– Core Operational Tasks – Monitor infrastructure, components and
services– Troubleshooting– Verification of new sites joining Grid– Acceptance tests of new middleware
releases– Verify suppliers are meeting SLA– Performance tuning and optimisation– Publishing use figures and accounts– Grid information services – Monitoring services – Resource brokering – Allocation and scheduling services – Replica data catalogues – Authorisation services – Accounting services
Tony Doyle - University of Glasgow
EGEE Grid OperationsEGEE Grid Operations
EGEEGridPPJISCCore UK
Production Manager
Grid Support Centre5 FTE
Core Grid Coordinator1 FTE
MiddleWare Specialist Support6 FTE
Deployment Team8 FTE
Grid Operations Centre3 FTE
Security OfficerRAL
Data and Storage ManagementGlasgow, Bristol, Edinburgh
VO Management and ServicesNorth (0.5 FTE)
Workload Management ServicesLondon
Network ManagementLondon (0.5 FTE)
2 Tier1 DeploymentRAL
4 Tier 2 UK CoordinatorsLondonGrid,NorthGridScotGrid, SouthGrid
1 Tier2 CoordinatorIreland
Applications ExpertRAL
+1
+1
+
Technical Editor
~ 23 FTE (EU + non-EU funded)
The characters are the same whichever “OS”
Tony Doyle - University of Glasgow
SysMan
GridPP2 Deployment Board GridPP2 Deployment Board Foundation ViewpointFoundation Viewpoint
Production ManagerGrid Support Centre
5 FTECore Grid Coordinator
1 FTEMiddleWare Specialist Support
6 FTEDeployment Team
8 FTEGrid Operations Centre
3 FTE
• Summary:The Production Manager & DB have a
major role in (re-)defining and implementing Production Planning. Refs: GridPP-PMB-24-Production
GridPP-PMB-20-Management
III. Grid Middleware
e.g. Information Services
IV. Facilities and Fabrics
e.g. Storage Services
Deployment
Board
Tony Doyle - University of Glasgow
SysMan
GridPP2 Deployment Board: GridPP2 Deployment Board: Two RolesTwo Roles
The role of Production Manager is to:1. lead the Production Grid Team;
2. sit on the Deployment Board;
3. ensure the formulation and implementation of procedures necessary to ensure timely deployment;
4. report to the Deployment Board on progress and issues;
5. liaise with all the Tier centres on the technical implementation of the production grid.
The role of the Deployment Board Chair is to:1. convene and chair Deployment Board meetings;
2. sit on the Project Management Board;
3. ensure that the Deployment Board are kept fully aware of the objectives of the project and the priorities identified by the PMB;
4. ensure that the Project Management Board is full aware of technical progress and any issues that arise;
5. ensure that progress toward deliverables is monitored against the project schedule on a quarterly basis;
6. ensure that the use of GridPP resources is monitored and reported;
7. report to the Project Manager on matters relating to any project deliverables and milestones.
Deployment
Board
Tony Doyle - University of Glasgow
SysMan
GridPP2 Deployment Board GridPP2 Deployment Board Terms of ReferenceTerms of Reference
• The Deployment Board is the body at which technical developments and problems wider than a single Working Group or Tier Centre are discussed and solved.
• The Deployment Board meets approximately quarterly to:1. determine and execute the technical project plan necessary to implement the strategy of
the Project Management Board on the deployment of a production quality grid for Particle Physics;
2. report to the Project Management Board, through its chair, on progress towards this goal;
3. ensure that problems and ideas with implications beyond individual working groups are discussed and solved in the most efficient manner;
4. provide a forum in which all constituent parts of the project can exchange information on technical issues of mutual interest;
5. manage the deployment of GridPP2 hardware resources in the UK, ensuring they are efficiently and effectively used to deliver the Particle Physics Grid with the minimum of duplication;
6. deal with matters of coordination and liaison between UK Tier1 and Tier2 centres;
7. ensure that technical developments elsewhere, particularly at CERN, are efficiently integrated into the project.
Deployment
Board
Tony Doyle - University of Glasgow
User
Board
Experiments
• The User Board ensures that the requirements for the major Particle Physics experiments that will use the Grid are properly taken into account in the strategy of the PMB.
• The User Board meets approximately quarterly to:1. ensure that the GridPP2 project remains science driven and that the
project develops to meet the needs of the UK Particle Physics community;
2. keep an overview of both technical and experimental software developments, ensuring these developments are compatible;
3. facilitate close working relationships between the UK experimental collaborations to identify common areas for the development of Grid-enabled experimental software;
4. overview the major hardware purchases within GridPP2 to ensure compatibility with the needs of the experimental collaborations and between collaborations;
5. promote the use of the developing Grid structures for a wide variety of computing needs in the whole of the UK Particle Physics community.
GridPP2 User Board Terms of ReferenceGridPP2 User Board Terms of Reference
I. Experiment Layer
e.g. Portals
II. Application Middleware
e.g. Metadata
Tony Doyle - University of Glasgow
• The PMB is the central part of the management structure. • The Project Management Board meets approximately
weekly to:1. determine the detailed objectives of the project;
2. approve the allocation of resources to meet these objectives;
3. ensure timely progress is made towards meeting the objectives;
4. report on progress to the Oversight Committee;
5. inform the community of progress, formally via the Collaboration Board;
6. establish an identity for GridPP2 reflecting the set of GridPP deliverables;
7. ensure that GridPP2 is visible externally and is recognised
as a flagship project.
GridPP2 Project Management BoardGridPP2 Project Management Board Terms of Reference Terms of Reference
III. Grid Middleware
e.g. Information Services
I. Experiment Layer
e.g. Portals
II. Application Middleware
e.g. Metadata
IV. Facilities and Fabrics
e.g. Storage Services
PMB
UserBoard
DeploymentBoard
PMB
Tony Doyle - University of Glasgow
Who reports where?Who reports where?
Developers
Developer
Team
Dissemination
Officer“Opinion
Formers”
Project Management
Board
(PMB)
Collaboration Board
(CB)
Production Grid Deployment
Board
(DB)
User Board
(UB)
Project Management
Board
(PMB)
Collaboration Board
(CB)
Production Grid Deployment
Board
(DB)
User Board
(UB)
Representatives
• Formal roles are (pre-)defined in the proposal
• GridPP-PMB-20-Management
Tony Doyle - University of Glasgow
GAG
LCG
PEB
Alice ATLAS CMS LHCb
Generic Middleware
Resource providers
RequirementsGuidelines
Service specs
Integration: workers’ forum
ARDA – “A Realisation of ARDA – “A Realisation of Distributed Analysis” Distributed Analysis”
LCG AAprojects
Tony Doyle - University of Glasgow
ARDA: many facetsARDA: many facets
Tony Doyle - University of Glasgow
Application InterfacesApplication Interfaces [FTE] [FTE]
• Applications– 18.5 FTEs: ongoing
programme of work can continue
– Difficult to involve experiment activity not already engaged within GridPP
• Project needs to build on cross-experiment collaboration
•Encourage new joint developments across experiments
•New areas are PhenoGrid and generic portal work
GridPP2 Work Area FTE
ATLAS/LHCb (GANGA) 2.0
ATLAS 2.5
BaBar 2.0
CDF/D0 (SAM) 2.0
CDF 1.0
CMS 3.0
D0 1.0
LHCb 2.0
Portal 1.0
UKQCD 1.0
PhenoGrid 1.0
Total 18.5
Tony Doyle - University of Glasgow
Middleware, Security and Network Middleware, Security and Network [FTE][FTE]
• Information Services [4+4 FTE] and Networking [2+1.5+1.5 FTE]: strategic roles within EGEE• Security expands to meet reqts.• Storage and Workload Management continue• No further configuration management development
• programme defined by – mission criticality (experiment requirements
driven) – International/UK-wide lead – leverage of EGEE, UK core and LCG developments
• Redefine (EDG-based) UK Work Packages into Development Groups
• Metadata area requires group defined in association with applications area
• General emphasis on deployment/support rather than development per se
SecurityMiddleware
Networking
GridPP2 Work Area
PPARC funding Other funding
Metadata 1.0 0.0
Storage Management 2.0 0.0
Workload Management 1.0 4.0*
Security 3.5 2.0*
Information & Monitoring 4.0 4.0
Network Sector 2.0 3.0
LHC Applications 1.0 0.0
Totals 14.5 9.0 -13.0
Tony Doyle - University of Glasgow
Tier-2 Services [FTE]Tier-2 Services [FTE]
• Four Regional Tier-2 Centres • London: Brunel, Imperial College, QMUL, RHUL, UCL.• SouthGrid: Birmingham, Bristol, Cambridge, Oxford, RAL PPD.• NorthGrid: CCLRC Daresbury, Lancaster, Liverpool, Manchester, Sheffield.• ScotGrid: Durham, Edinburgh, Glasgow.
• Hardware provided by Institutes
• GridPP provides added manpower GridPP2 Work Area FTE
Security 1.0
Resource Broker 1.0
Network 0.5
Data Management 1.5
Storage Management 1.5
VO Management 0.5
Operations 8.5
Total 14.5 [+4.0]
Tony Doyle - University of Glasgow
Tier-1/A Services [FTE]Tier-1/A Services [FTE]
• High quality data services
• National and International Role• UK Focus for International Grid development
• Highest single priority within GridPP2
Total Manpower
CPU 2.0
Disk 1.5
Tape 2.5
Core Services 2.0
Operations 2.5
Networking 0.5
Deployment 2.0
Experiments 2.0
Management 1.5
Total 16.5
Tony Doyle - University of Glasgow
In LCG In LCG ContextContext
Proposed Proposed StructureStructure
ARDA
Ex
pm
tsEG
EE LCG
Deployment Board
Tier1/Tier2,Testbeds,
Rollout
Servicespecification& provision
User Board
Requirements
ApplicationDevelopment
Userfeedback
Metadata
Workload
Network
Security
Info. Mon.
PMB
CB
Storage
Tony Doyle - University of Glasgow
GridPP2 User Board: GridPP2 User Board: Two Roles??Two Roles??
The role of Applications Manager(?) is to:1. lead the Application Development Team;
2. sit on the User Board;
3. ensure the formulation and implementation of application middleware necessary to ensure timely deployment;
4. report to the User Board on progress and issues;
5. liaise with all the application experts on the technical implementation of the production grid.
The role of the User Board Chair is to:1. The role of the User Board Chair is to:
2. convene and chair User Board meetings;
3. sit on the Project Management Board;
4. inform the User board of issues from the PMB;
5. solicit input from users and potential users;
6. inform the PMB and DB of user requirements and issues;
7. ensure that progress towards deliverables in the application interface area is monitored on a quarterly basis;
8. report to the Project Manager on matters relating to any project deliverables and milestones defined within his or her purview.
User
Board
Experiments
Tony Doyle - University of Glasgow
GeneralGeneral Points Points
1. All post-holders report back at 3-monthly intervals (using similar reporting mechanisms to those used for the majority of GridPP posts). Reporting through either the:1. User Board; or
2. Deployment Board
2. All GridPP2 posts will end on Aug 31st 2007. Existing posts that are renewed will commence at the end-date of the present grant.
3. A review of posts will be made prior to this end point, in order to determine their effectiveness within the GridPP2 programme.
Tony Doyle - University of Glasgow
Where does the work get done? Where does the work get done? Areas on Green AlertAreas on Green Alert
• “Approved subject to minor/moderate changes”: timeline described by Steve…
• Groups bridge across the applications/middleware/support boundaries
• Active subgroups linking to EGEE:1. Information Services
– RAL• LCG developments (GridPP2/LCG/EGEE joint programmes?)
1. “Security” inc. VO management Group• Manchester, RAL [DK, AM]
2. “Workload Management” • IC [DC]
• Deployment groups1. “Data Replication Group” (support, small, but necessary)
• RAL, Glasgow, Bristol…2. Others..
• These can also be task-force like areas addressing specific
issues.. Workshop organised by RM
Tony Doyle - University of Glasgow
Areas on Amber AlertAreas on Amber Alert
• First discussions on “not yet approved” and “approved subject to major changes” areas
• We need to develop a detailed plan taking on board PRSC input not defined a priori 1. “Metadata Group”
• Glasgow, Oxford, IC/Bristol? ++? [TD]
2. “Storage Management Group”• RAL, Edinburgh, Bristol [JG]
3. “Networking Group”• UCL, DL, M’cr [PC]
• New deliverables in “1-2 months”…
Tony Doyle - University of Glasgow
Outcome?Outcome?
• We need to specify management/technical roles in advance of GridPP2 to ensure we are successful in developing/deploying the Production Grid
• By the end of this GridPP2 specification process we will be able to:
– Continue the work started in GridPP1 (by many people)– Define the GridPP2 Project Map tasks and metrics– Appoint additional people in recognised areas to contribute to agreed
tasks/roles– Contribute fully to EGEE/LCG– Start to go from Prototype to Production
• Anticipate new management/technical posts and holders are defined before GridPP10 meeting at CERN in June
• Beyond this, we anticipate a transition after GridPP2 in the PPARC Road Map where these areas are supported within the Particle Physics Programme
• We are masters of our own destiny..