LCG Applications Area

LCG Applications Area

Torre Wenaus, BNL/CERN

LCG Applications Area Manager

http://cern.ch/lcg/peb/applications

DOE/NSF Review of US LHC Physics and Computing Projects

January 14, 2003

Agency Review, January 14, 2003 Slide 2


The LHC Computing Grid Project Structure

Project Overview Board

ProjectExecution

Board (PEB)

Software andComputingCommittee

(SC2)

Requirements,Work plan,Monitoring

WP

RTAG

WP WP WP WP

Project Leader

GridProjects

Project Work Packages



LCG Areas of Work

Fabric (Computing System) Physics Data Management Fabric Management Physics Data Storage LAN Management Wide-area Networking Security Internet Services

Grid Technology Grid middleware Standard application

services layer Inter-project

coherence/compatibility

Physics Applications Software Application Software

Infrastructure – libraries, tools Object persistency, data

management tools Common Frameworks –

Simulation, Analysis, .. Adaptation of Physics

Applications to Grid environment Grid tools, Portals

Grid Deployment Data Challenges Grid Operations Network Planning Regional Centre Coordination Security & access policy



Applications Area Organization

Project

WP WP WP

Project

WP

Project

WP WPWP

Overall management, coordination, architectureApps AreaLeader

ProjectLeaders

Work PackageLeaders

ArchitectsForum

…

Direct technical collaboration between experiment participants,IT, EP, ROOT, LCG personnel



Focus on Experiment Need

Project structured and managed to ensure a focus on real experiment needs

SC2/RTAG process to identify, define (need-driven requirements), initiate and monitor common project activities in a way guided by the experiments themselves

Architects Forum to involve experiment architects in day to day project management and execution

Open-ness of information flow and decision making Direct participation of experiment developers in the projects Tight, iterative feedback loop to gather user feedback from

frequent releases Early deployment and evaluation of LCG software in experiment

contexts Success defined by experiment adoption and production

deployment



Applications Area Projects

Software Process and Infrastructure (SPI) (operating – A.Aimar) Librarian, QA, testing, developer tools, documentation, training, …

Persistency Framework (POOL) (operating – D.Duellmann) POOL hybrid ROOT/relational data store

Mathematical libraries (operating – F.James) Math and statistics libraries; GSL etc. as NAGC replacement Group in India will work on this (workplan in development)

Core Tools and Services (SEAL) (operating – P.Mato) Foundation and utility libraries, basic framework services, system

services, object dictionary and whiteboard, grid enabled services Physics Interfaces (PI) (launched – V.Innocente)

Interfaces and tools by which physicists directly use the software. Interactive (distributed) analysis, visualization, grid portals

Simulation (launch planning in progress) Geant4, FLUKA, simulation framework, geometry model, …

Generator Services (launch as part of simu) Generator librarian, support, tool development

Bold: Recent developments (last 3 months)



Project Relationships

Sof

twa

re P

roce

ss &

Inf

rast

ruct

ure

(S

PI)

Core Libraries & Services (SEAL)

Persistency(POOL)

PhysicistsInterface

(PI)

MathLibraries…

LCG Applications Area

Other LCG Projects in other Areas

LHC

Exp

erim

ents



02Q1 02Q2 02Q3 02Q4 03Q1 03Q2 03Q3 03Q4 04Q1 04Q2Simulation tools XDetector description & model XConditions database XData dictionary XInteractive framew orks XStatistical analysis XDetector & event visualization XPhysics packages XFramew ork services XC++ class libraries XEvent processing framew ork XDistributed analysis interfaces XDistributed production systems XSmall scale persistency XSoftw are testing XSoftw are distribution XOO language usage XLCG benchmarking suite XOnline notebooks X

Candidate RTAG timeline from March

Blue: RTAG/activity launched or (light blue) imminent



LCG Applications Area Timeline Highlights

2002 200520042003

Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4Q1 Q2 Q3 Q4Q1 Q2 Q3 Q4

Hybrid Event Store available for general users

Distributed production using grid services

First Global Grid Service (LCG-1) available

Distributed end-user interactive analysis

Full Persistency Framework

LCG-1 reliability and performance targets

“50% prototype” (LCG-3)

LCG TDR

Applications

LCG

POOL V0.1 internal releaseArchitectural blueprint complete

LCG launch week



Architecture Blueprint

Executive summary Response of the RTAG to the mandate Blueprint scope Requirements Use of ROOT Blueprint architecture design precepts

High level architectural issues, approaches Blueprint architectural elements

Specific architectural elements, suggested patterns, examples Domain decomposition Schedule and resources Recommendations

RTAG established in June

After 14 meetings,much email...

A 36-page final reportAccepted by SC2 October 11

http://lcgapp.cern.ch/project/blueprint/



Component Model

Granularity driven by component replacement criteria; development team organization; dependency minimization

Communication via public interfaces Plug-ins

Logical module encapsulating a service that can be loaded, activated and unloaded at run time

APIs targeted not only to end-users but to embedding frameworks and internal plug-ins



Basic Framework

Foundation Libraries

Simulation Framework

Reconstruction Framework

Visualization Framework

Applications

. . .

Optional Libraries

OtherFrameworks

Software Structure

Implementation-neutral services

STL,ROOT libs,

CLHEP,Boost, …

Grid middleware, …

ROOT, Qt, …



Distributed Operation

Architecture should enable but not require the use of distributed resources via the Grid

Configuration and control of Grid-based operation via dedicated services

Making use of optional grid middleware services at the foundation level of the software structure

Insulating higher level software from the middleware Supporting replaceability

Apart from these services, Grid-based operation should be largely transparent

Services should gracefully adapt to ‘unplugged’ environments

Transition to ‘local operation’ modes, or fail informatively



Managing Objects

Object Dictionary To query a class about its internal structure Essential for persistency, data browsing, etc. The ROOT team and LCG plan to develop and converge on a

common dictionary (common interface and implementation) with an interface anticipating a C++ standard (XTI) (Timescale ~1yr?)

Will contact Stroustrup, who has started implementation Object Whiteboard

Uniform access to application-defined transient objects, including in the ROOT environment

What this will be (how similar to Gaudi, StoreGate?) not yet defined Object definition based on C++ header files

Now that ATLAS as well as CMS will use this approach, it is being addressed in a common way via the LCG AA



Dictionary: Reflection / Population / Conversion

ROOT I / O

LCGDictionaryCI NT

DictStreamer

.h

in

out

ROOTCI NT

CI NT generatedcode Dict generated

code

.adl.xml

ADL/ GOD

OtherClients

LCG

to

CIN

TD

ict

gate

way

(2)

(1)

Population

Conversion

Reflection

.h

GCC-XMLIn progress

New in POOL 0.3



Other Architectural Elements

Python-based Component Bus Plug-in integration of components providing a wide

variety of functionality Component interfaces to bus derived from their C++

interfaces Scripting Languages

Python and CINT (ROOT) to both be available Access to objects via object whiteboard in these

environments Interface to the Grid

Must support convenient, efficient configuration of computing elements with all needed components



EventGeneration

Core Services

Dictionary

Whiteboard

Foundation and Utility Libraries

DetectorSimulation

Engine

Persistency

StoreMgr

Reconstruction

Algorithms

Geometry Event Model

GridServices

I nteractiveServices

Modeler

GUIAnalysis

EvtGen

Calibration

Scheduler

Fitter

PluginMgr

Monitor

NTuple

Scripting

FileCatalog

ROOT GEANT4 DataGrid Python Qt

Monitor

. . .MySQLFLUKA

EventGeneration

Core Services

Dictionary

Whiteboard

Foundation and Utility Libraries

DetectorSimulation

Engine

Persistency

StoreMgr

Reconstruction

Algorithms

Geometry Event Model

GridServices

I nteractiveServices

Modeler

GUIAnalysis

EvtGen

Calibration

Scheduler

Fitter

PluginMgr

Monitor

NTuple

Scripting

FileCatalog

ROOT GEANT4 DataGrid Python Qt

Monitor

. . .MySQLFLUKA

Domain Decomposition

Products mentioned are examples; not a comprehensive list

Grey: not in common project scope(also event processing framework, TDAQ)



Use of ROOT in LCG Software

Among the LHC experiments ALICE has based its applications directly on ROOT The 3 others base their applications on components with

implementation-independent interfaces Look for software that can be encapsulated into these components

All experiments agree that ROOT is an important element of LHC software

Leverage existing software effectively and do not unnecessarily reinvent wheels

Therefore the blueprint establishes a user/provider relationship between the LCG applications area and ROOT

LCG AA software will make use of ROOT as an external product Draws on a great ROOT strength: users are listened to very carefully!

So far so good: the ROOT team has been very responsive to needs for new and extended functionality coming from POOL



Blueprint RTAG Outcomes

SC2 decided in October…

Blueprint is accepted RTAG recommendations accepted to

Start common project on core tools and services Start common project on physics interfaces



Applications Area Personnel Status

18 LCG apps hires in place and working; +2 in Jan, Feb Manpower ramp is on target (expected to reach 20-23) Contributions from UK, Spain, Switzerland, Germany, Sweden,

Israel, Portugal, US

~10 FTEs from IT (DB and API groups) also participating ~8 FTEs from experiments (CERN EP and outside CERN) also

participating in (mainly) POOL, SEAL, SPI CERN established a new software group as the EP home of the

LCG applications area (EP/SFT) Led by John Harvey. Taking shape well. Localized in B.32

Fraction of experiment contribution which is US-supported (CERN or US resident) is currently ~30%

US fraction of total effort is <10%



LHC Manpower needs for Core Software

2000Have (miss)

2001 2002 2003 2004 2005

ALICE 12(5) 17.5 16.5 17 17.5 16.5

ATLAS 23(8) 36 35 30 28 29

CMS 15(10) 27 31 33 33 33

LHCb 14(5) 25 24 23 22 21

Total 64(28) 105.5 106.5 103 100.5 99.5

Only computing professionals counted

From LHC Computing (‘Hoffman’) Review (FTEs)



Personnel Resources – Required and Available

Estimate of Required Effort

FTEs today: 18 LCG, 10 CERN IT, 8 CERN EP + experiments

0

10

20

30

40

50

60S

ep-0

2

Dec

-02

Mar

-03

Jun-

03

Sep

-03

Dec

-03

Mar

-04

Jun-

04

Sep

-04

Dec

-04

Mar

-05

Quarter ending

FT

Es

SPI

Math libraries

Physics interfaces

Generator services

Simulation

CoreToolsS&Services

POOL

Blue = Available effort:

Future estimate: 20-23 LCG, 13 IT, 28 EP + experiments

Now



Current Personnel Distribution

FTE assignments

POOL

SPI

CLS

PI

Simu

Gen

Math

ROOT

Grid

Arch

Mgmt

Summary of LCG-funded Resources Used - Estimate to end 2002

Experience-Weighted FTEsCERN Special FundingApplications 8.9

Persistency 1.8Software Process Support 2.0Simulation 1.8Root 1.4Architecture 0.0Grid Interfacing 1.1Training 0.7

Fabric 5.4System Management & Operations 1.6Development (e.g. Monitoring) 1.8Data Storage Management 0.8Grid Security 1.1

Grid Technology 3.0Data Management 2.3Grid Gatekeeper 0.7

Grid Deployment 4.6Int 2.1OPS 2.5

Management 2.1LCG 2.1

EU FundingDataGrid 6.9

Total Weighted FTEs in calendar 2002 30.8



U.S. Leadership

Direct leadership and financial contribution: T.Wenaus as AA manager

In addition to contributions via ATLAS and CMS A .75FTE job requiring CERN residence Salary support from the BNL base program (is this fair?) CERN residency and US travel costs borne by CERN Together with the strong U.S. presence in CMS and ATLAS

computing leadership, this role gives the U.S. a strong voice in the LCG applications area

Not a dominating influence of course; e.g. at this point all the applications area project leaders are Europeans

Presence at CERN is very important, like it or not Importance is increased because of the utterly deplorable state of

the CERN infrastructure for both audio and video conferencing The U.S. should put up the money to fix this, if no one else will;

it is in our own vital interest



Schedule and Resource Tracking (example)



Apps area planning materials

Planning page linked from applications area page Applications area plan spreadsheet: overall project plan

http://lcgapp.cern.ch/project/mgmt/AppPlan.xls High level schedule, personnel resource requirements

Applications area plan document: overall project plan http://lcgapp.cern.ch/project/mgmt/AppPlan.doc Incomplete draft

Personnel spreadsheet http://lcgapp.cern.ch/project/mgmt/AppManpower.xls Currently active/foreseen apps area personnel, activities

WBS, milestones, assigned personnel resources http://atlassw1.phy.bnl.gov/Planning/lcgPlanning.html

Follow Applications Area planning link on the review web page



Core Libraries and Services (SEAL) Project

Launched in Oct, led by Pere Mato (CERN/LHCb) 6-member (~3 FTE) team initially; M.Marino from ATLAS

Scope: Foundation, utility libraries Basic framework services Object dictionary Grid enabled services

Many areas of immediate relevance to POOL; these are given priority Users of this project are software developers in other projects and the

experiments Establishing initial plan, reviewing existing libraries and services

Process for adopting third party code will be addressed in this project Initial workplan will be presented to SC2 on Jan 10 2003/3/31: SEAL V1 essentials in alpha



SEAL Work Packages

Foundation and utility libraries Boost, CLHEP, …, complementary in-house development

Component model and plug-in manager The core expression in code of the component architecture described in

the blueprint. Mainly in-house development. LCG object dictionary

Already active project in POOL; being moved to SEAL (wider scope than persistency). Will include filling dictionary from C++ header files.

Basic framework services Object whiteboard, message reporting, component configuration, ‘event’

management Scripting services Grid services: common interface to middleware Education and documentation

Assisting experiments with integration



Physicist Interface (PI) Project

Led by Vincenzo Innocente (CERN/CMS) Covers the interfaces and tools by which physicists will directly use

the software Planned scope:

Interactive environment: physicist’s desktop Analysis tools Visualization Distributed analysis, grid portals

Very poorly defined and understood Currently surveying experiments on their needs and interests In more of an ‘RTAG mode’ than project mode initially, to flesh out

plans and try to clarify the grid area Will present initial plans (and possibly an analysis RTAG proposal) to

SC2 on Jan 29



Software Process and Infrastructure (SPI)

Code documentation, browsing Doxygen, LXR, ViewCVS Testing Framework CppUnit, Oval Memory Leaks Valgrind Automatic Builds Probably the ATLAS system Coding and design guidelines RuleChecker CVS organization Configuration/release mgmt SCRAM Software documentation templates

http://spi.cern.ch

Components available:



SPI Services

CVS repositories One repository per project Standard repository structure and #include conventions being finalized this

week Will eventually move to IT CVS service when it is proven

AFS delivery area, Software Library /afs/cern.ch/sw/lcg Installations of LCG-developed and external software Installation kits for offsite installation LCG Software Library ‘toolsmith’ started in December

Build servers Machines with various Linux, Solaris configurations available for use

Project portal (similar to SourceForge) http://lcgappdev.cern.ch Very nice new system using Savannah (savannah.gnu.org) Used by CMS as well as LCG; ATLAS will probably be using it soon Bug tracking, project news, FAQ, mailing lists, download area, CVS

access, …





POOL

Pool of persistent objects for LHC, currently in prototype Targeted at event data but not excluding other data Hybrid technology approach

Object level data storage using file-based object store (ROOT) RDBMS for meta data: file catalogs, object collections, etc (MySQL)

Leverages existing ROOT I/O technology and adds value Transparent cross-file and cross-technology object navigation RDBMS integration

Integration with Grid technology (eg EDG/Globus replica catalog) network and grid decoupled working modes

Follows and exemplifies the LCG blueprint approach Components with well defined responsibilities Communicating via public component interfaces Implementation technology neutral



Pool Release Schedule

End September - V0.1 (Released Oct 2) All core components for navigation exist and interoperate Assumes ROOT object (TObject) on read and write

End October - V0.2 (Released Nov 15) First collection implementation

End November - V0.3 (Released Dec 18) First public release EDG/Globus FileCatalog integrated Persistency for general C++ classes (not instrumented by ROOT),

but very limited: elementary types only Event metadata annotation and query

June 2003 – Production release



POOL Milestones



Simulation Project

Mandated by SC2 to initiate simulation project following the RTAG Project being organized now Expected to cover

generic simulation framework Multiple simulation engine support, geometry model, generator

interface, MC truth, user actions, user interfaces, average tracking, utilities

ALICE virtual MC as starting point if it meets requirements Geant4 development and integration FLUKA (development and) integration physics validation simulation test and benchmark suite fast (shower) parameterisation generator services



Comment on Grid Technology Area (GTA)

Quote from slide of Les:

LCG expects to obtain Grid Technology from projects funded by national and regional e-science initiatives -- and from industry

concentrating ourselves on deploying a global grid service

All true, but there is a real role for the GTA, not just deployment, in LCG:

Ensuring that the needed middleware is/will be there, tested, selected and of production grade

(Re)organization in progress to create an active GTA along these lines

Important for the Applications Area: AA distributed software will be robust and usable only if the grid middleware it uses is so



Concluding Remarks

Essentially the full expected AA scope is covered by the anticipated activities of the projects now defined

Manpower is in quite good shape Buy-in by the experiments, apart from ALICE, is good

Substantial direct participation in leadership, development, prompt testing and evaluation, RTAGs

U.S. CMS represented well because of strong presence in computing management and in CERN-based personnel

U.S. ATLAS representation will improve with D.Quarrie’s relocation to CERN as Software Leader; further increases in CERN presence being sought

Groups remote from CERN are contributing, but it isn’t always easy Have pushed to lower the barriers, but still it isn’t easy

New CERN EP/SFT group is taking shape well as a CERN hub for applications area activities

POOL and SPI are delivering, and the other projects are ramping up First persistency prototype released in 2002, as targeted in March 2002

Date post:	03-Jan-2016
Category:	Documents
Upload:	malcolm-holcomb
View:	23 times
Download:	0 times

LCG Applications Area

Documents