US Grid EffortsUS Grid Efforts

Lee Lueking

D0 Remote Analysis Workshop

February 12, 2002

February 12, 2002 Lee Lueking - D0 RACE 2

All of these projects are working towards the common goal of providing transparent access to the massively distributed computing infrastructure that is needed to meet the challenges of modern experiments … (From the EU DataTAG proposal)

February 12, 2002 Lee Lueking - D0 RACE 3

Grid Projects TimelineGrid Projects Timeline

Q3 00

Q4 00

Q4 01

Q3 01

Q2 01

Q1 01

Q1 02

GriPhyN: $11.9M+$1.6M

PPDG:$9.5M

iVDGL:$13.65M

EU DataGrid: $9.3M

EU DataTAG:4M Euros

GridPP:

February 12, 2002 Lee Lueking - D0 RACE 4

PPDG PPDG

Develop, acquire and deliver vitally needed Grid-enabled tools for data-intensive requirements of particle and nuclear physics.

Collaboration of computer scientists with a strong record in distributed computing and Grid technology, and physicists with leading roles in the software and network infrastructures for major high-energy and nuclear experiments.

Goals and plans are ultimately guided by the immediate, medium-term and longer-term needs and perspectives of the physics experiments.

February 12, 2002 Lee Lueking - D0 RACE 5

GriPhyN: Grid Physics NetworkGriPhyN: Grid Physics Network

Virtual data technologies. Advances are required in information models and in new methods of cataloging, characterizing, validating, and archiving software components to implement virtual data manipulations

Policy-driven request planning and scheduling of networked data and computational resources. We require mechanisms for representing and enforcing both local and global policy constraints and new policy-aware resource discovery techniques.

Management of transactions and task-execution across national-scale and worldwide virtual organizations. New mechanisms are needed to meet user requirements for performance, reliability, and cost.

February 12, 2002 Lee Lueking - D0 RACE 6

iVDGL:International Virtual iVDGL:International Virtual Data Grid LaboratoryData Grid Laboratory

The iVDGL will provide a global computing resource for several leading international experiments in physics and astronomy,

Global services and centralized monitoring, management, and support functions functions will be coordinated by the Grid Operations Center (GOC) located at Indiana University, with technical effort provided by GOC staff, iVDGL site staff, and the CS support teams.

GriPhyN and Particle Physics Data Grid will provide the basic R&D and software toolkits needed for the laboratory.

The European Union DataGrid is also a major participant and will contribute basic technologies and tools.

The iVDGL will be based on the open Grid infrastructure provided by the Globus Toolkit and will also build on other technologies such as Condor resource management tools.

February 12, 2002 Lee Lueking - D0 RACE 7

Comparison of PPDG and iVDGLComparison of PPDG and iVDGLPPDG iVDGL

Funding US DOE approved

1/1/3/3/3 $M, 99 – 03

US NSF proposed

3/3/3/3/3 $M, 02 – 06

Computer Science

Globus (Foster), Condor (Livny), SDM (Shoshani), SRB (Moore)

Globus (Foster, Kesselman), Condor (Livny)

Physics BaBar, Dzero, STAR, JLAB, ATLAS, CMS

ATLAS, CMS, LIGO,

SDSS, NVO

National Laboratories

BNL, Fermilab, JLAB, SLAC, ANL, LBNL

ANL,BNL, Fermilab (all unfunded collaborators)

Universities Caltech, SDSS, UCSD, Wisconsin

Florida, Chicago, Caltech, UCSD, Indiana, Boston, Wisconsin at Milwaukee, Pennsylvania State, Johns Hopkins, Wisconsin at Madison, Northwestern, USC, UT Brownsville, Hampton, Salish Kootenai College

Hardware None ~20% of funding(Tier-2 Centers)

Network No funding requested No funding requestedDataTAG complementary

February 12, 2002 Lee Lueking - D0 RACE 8

PPDG CollaboratorsPPDG Collaborators

February 12, 2002 Lee Lueking - D0 RACE 9

PPDG Computer Science GroupsPPDG Computer Science GroupsCondor – develop, implement, deploy, and evaluate mechanisms and policies

that support High Throughput Computing on large collections of computing resources with distributed ownership.

http://www.cs.wisc.edu/condor/

Globus - developing fundamental technologies needed to build persistent environments that enable software applications to integrate instruments, displays, computational and information resources that are managed by diverse organizations in widespread locations

http://www.globus.org/

SDM - Scientific Data Management Research Group – optimized and standardized access to storage systems

http://gizmo.lbl.gov/DM.html

Storage Resource Broker - client-server middleware that provides a uniform interface for connecting to heterogeneous data resources over a network and cataloging/accessing replicated data sets.

http://www.npaci.edu/DICE/SRB/index.html

February 12, 2002 Lee Lueking - D0 RACE 10

Delivery of End-to-End ApplicationsDelivery of End-to-End Applications& Integrated Production Systems & Integrated Production Systems

to allow thousands of physicists to share data & computing resources for scientific processing and analyses

Operators & Users

Resources: Computers, Storage, Networks

PPDG Focus:

- Robust Data Replication

- Intelligent Job Placement and Scheduling

- Management of Storage Resources

- Monitoring and Information of Global Services

Relies on Grid infrastructure:- Security & Policy- High Speed Data Transfer- Network management

February 12, 2002 Lee Lueking - D0 RACE 11

Common ServicesCommon Services Job Description Language Scheduling and Management of Processing and Data

Placement Activities Monitoring and Status Reporting Storage Resource Management Reliable Replica Management Services File Transfer Services Collect and Document Current Experimental Practices R & D, Evaluation Authentication, Authorization, and Security End-to-End Applications and Testbeds

February 12, 2002 Lee Lueking - D0 RACE 12

Project Activities,Project Activities,End-to-End ApplicationsEnd-to-End Applications

and Cross-Cut Pilotsand Cross-Cut Pilots

Project Activities are focused Experiment – Computer Science Collaborative developments.

Replicated data sets for science analysis – BaBar, CMS, STARDistributed Monte Carlo production services – ATLAS, D0, CMSCommon storage management and interfaces – STAR, JLAB

End-to-End Applications used in Experiment data handling systems to give real-world requirements, testing and feedback.

Error reporting and responseFault tolerant integration of complex components

Cross-Cut Pilots for common services and policies Certificate Authority policy and authenticationFile transfer standards and protocolsResource Monitoring – networks, computers, storage.

February 12, 2002 Lee Lueking - D0 RACE 13

Super Computing 2001 in Denver

February 12, 2002 Lee Lueking - D0 RACE 14

February 12, 2002 Lee Lueking - D0 RACE 15

PPDG activities as part of the PPDG activities as part of the Global Grid CommunityGlobal Grid Community

Coordination with other Grid Projects in our field:GriPhyN – Grid for Physics NetworkEuropean DataGridStorage Resource Management collaboratoryHENP Data Grid Coordination Committee

Participation in Experiment and Grid deployments in our field:ATLAS, BaBar, CMS, D0, Star, JLAB experiment data handling systemsiVDGL/DataTAG – International Virtual Data Grid LaboratoryUse DTF computational facilities?

Active in Standards Committees:Internet2 HENP Working Group Global Grid Forum

February 12, 2002 Lee Lueking - D0 RACE 16

PPDG and GridPP ProjectsPPDG and GridPP Projects

Use of Standard Middleware to Promote Interoperability Move to Globus infrastructure: GSI, GridFTP Use of Condor as a supported system for job

submission Publish availability of resources and file catalog

Additional Grid Functionality for Job Specification, Submission, and Tracking Use Condor for migration and check pointing Enhanced job specification language and services

Enhanced Monitoring and Diagnostic Capabilities Fabric Management

February 12, 2002 Lee Lueking - D0 RACE 17

PPDG Management and CoordinationPPDG Management and CoordinationPIs

Livny, Newman, Mount

Steering CommitteeRuth Pordes, Chair

Doug Olson, Physics Deputy ChairMiron Livny, Computer Science Deputy Chair

Computer Science Group RepresentativesPhysics Experiment Representatives

PIs (ex officio)

STAR SDM BaBar SRB JLAB ATLAS Globus CMS Condor DZero

Executive Team (>1.0 FTE on PPDG)Steering Committee Chair

Steering Committee Physics and CS Deputy Chairs

February 12, 2002 Lee Lueking - D0 RACE 18

iVDGLiVDGL

International Virtual-Data Grid Laboratory A global Grid laboratory with participation from US, EU, Asia,

etc. A place to conduct Data Grid tests “at scale” A mechanism to create common Grid infrastructure A facility to perform production exercises for LHC experiments A laboratory for other disciplines to perform Data Grid tests

“We propose to create, operate and evaluate, over asustained period of time, an international researchlaboratory for data-intensive science.”

From NSF proposal, 2001

February 12, 2002 Lee Lueking - D0 RACE 19

iVDGL Summary InformationiVDGL Summary Information Principal components (as seen by USA)

Tier1 sites (laboratories) Tier2 sites (universities and other institutes) Selected Tier3 sites (universities) Fast networks: US, Europe, transatlantic International Grid Operations Center (iGOC) Computer Science support teams Coordination, management

Proposed international partners Initially US, EU, Japan, Australia Other world regions later Discussions w/ Russia, China, Pakistan, India, South America

Complementary EU project: DataTAG Transatlantic network from CERN to STAR-TAP (+ people) Initially 2.5 Gb/s

February 12, 2002 Lee Lueking - D0 RACE 20

US Proposal to NSFUS Proposal to NSF US proposal approved by NSF Sept. 25, 2001

“Part 2” of GriPhyN project Much more application oriented than first GriPhyN proposal $15M, 5 years @ $3M per year (huge constraint) CMS + ATLAS + LIGO + SDSS/NVO + Computer Science

Scope of US proposal Deploy Grid laboratory with international partners Acquire Tier2 hardware, Tier2 support personnel Integrate of Grid software into applications CS support teams (+ 6 UK Fellows) to harden tools Establish International Grid Operations Center (iGOC) Deploy hardware at 3 minority institutions (Tier3)

February 12, 2002 Lee Lueking - D0 RACE 21

US iVDGL Proposal ParticipantsUS iVDGL Proposal Participants

T2/Software

CS support

T3/Outreach

T1/Labs

U Florida CMS Caltech CMS, LIGO UC San Diego CMS, CS Indiana U ATLAS, iGOC Boston U ATLAS U Wisconsin, Milwaukee LIGO Penn State LIGO Johns Hopkins SDSS, NVO U Chicago CS U Southern California CS U Wisconsin, Madison CS Salish Kootenai Outreach, LIGO Hampton U Outreach, ATLAS U Texas, Brownsville Outreach, LIGO Fermilab CMS, SDSS, NVO Brookhaven ATLAS Argonne Lab ATLAS, CS

February 12, 2002 Lee Lueking - D0 RACE 22

iVDGL Partners iVDGL Partners

National partners PPDG (Particle Physics Data Grid ) DTF: Distributed Terascale Facility CAL-IT2 (new California Grid initiative)

Current international partners EU-DataGrid UK PPARC funding agency UK Core e-Science Program 6 UK Fellowships INFN (Italy) 2 Japanese institutes 1 Australian institute (APAC)

February 12, 2002 Lee Lueking - D0 RACE 23

iVDGL Map Circa 2002-2003iVDGL Map Circa 2002-2003

Tier0/1 facility

Tier2 facility

10 Gbps link

2.5 Gbps link

622 Mbps link

Other link

Tier3 facility

February 12, 2002 Lee Lueking - D0 RACE 24

iVDGL RequirementsiVDGL Requirements

Realistic scale In number, diversity, distribution, network connectivity

Delegated management and local autonomy Management needed to operate as large, single facility Autonomy needed for sites and experiments

Support large-scale experimentation To provide useful information for building real Data Grids

Robust operation For long running applications in complex environment

Instrumentation and monitoring Required for an experimental facility

Integration with international “cyberinfrastructure” Extensibility

February 12, 2002 Lee Lueking - D0 RACE 25

ApproachApproach

Define a laboratory architecture Define expected laboratory functions Build in scalability, extensibility, reproducibility Define instrumentation, monitoring Establish CS support teams (develop/harden tools, support

users) Define working relationship, coordination with partners

Create and operate global-scale laboratory Deploy hardware, software, personnel at Tier2, Tier3 sites Establish iGOC, single point of contact for monitoring,

support, … Help international partners establish sites

Evaluate and improve iVDGL through experimentation CS support teams will work with experiments Extend results to partners

Engage underrepresented groups Integrate minority institutions as Tier3 sites

February 12, 2002 Lee Lueking - D0 RACE 26

iVDGL as a LaboratoryiVDGL as a Laboratory

Grid Exercises “Easy”, intra-experiment tests first (10-30%, national,

transatlantic) “Harder” wide-scale tests later (30-100% of all

resources) CMS is already conducting transcontinental simulation

productions Operation as a facility

Common software, central installation to ensure compatibility

CS teams to “harden” tools, support applications iGOC to monitor performance, handle problems

February 12, 2002 Lee Lueking - D0 RACE 27

Emphasize Simple OperationEmphasize Simple Operation

“Local” control of resources vitally important (Site level or national level) Experiments, politics demand it

Operate mostly as a “partitioned” testbed (Experiment, nation, etc.) Avoids excessive coordination Allows software tests in different partitions

Hierarchy of operation must be defined E.g., (1) National + experiment, (2) inter-expt., (3)

global tests

February 12, 2002 Lee Lueking - D0 RACE 28

Other DisciplinesOther Disciplines

Use by other disciplines Expected to be at the 10% level Other HENP experiments Virtual Observatory (VO) community in Europe/US Gravity wave community in

Europe/US/Australia/Japan Earthquake engineering Bioinformatics Our CS colleagues (wide scale tests)

February 12, 2002 Lee Lueking - D0 RACE 29

US iVDGL Management and CoordinationUS iVDGL Management and Coordination

Project DirectorsAvery, Foster

Project Coordination GroupProject Coordinator

Project DirectorsCoordinators of Systems Integration, Education/Outreach

Physics Experiment RepresentativesUniversity Research Center or Group Representatives

PACI Representatives

iVDGL Design and Deployment

Integration with Applications

University Research Centers / Groups

International Grid Operations Center

Collaboration Board (Advisory)

External Advisory Board

February 12, 2002 Lee Lueking - D0 RACE 30

ConclusionConclusion

PPDG, and iVDGL are complementary in their approach and deliverables.

These efforts, along with our European partners will provide exciting new ways to share data and computing resources.

Dzero Grid involvement offers many challenges, but even more opportunities.

Acknowledgements: Richard Mount (SLAC), Paul Avery (University of Florida), Ruth Pordes (FNAL).