Date post: | 02-Jan-2016 |
Category: |
Documents |
Upload: | ivan-santana |
View: | 39 times |
Download: | 0 times |
Atlas Grid Status - part 1
Jennifer SchopfJennifer Schopf
ANLANL
U.S. ATLAS Physics and Computing Advisory Panel ReviewU.S. ATLAS Physics and Computing Advisory Panel Review
Argonne National LaboratoryArgonne National LaboratoryOct 30, 2001Oct 30, 2001
Oct 2001Oct 2001Jennifer Schopf, ANLJennifer Schopf, ANL 2
Globus/Atlas Interactions
GRAPPA/Gardner
Grid DataAccess/Malon
Magda/Wenaus
GridView and other MonitoringDe, Schopf, Yu
Condor (G)
GRAM
GSI
MDS/GIIS/GRIS
GridFTP
Replica Cat
Replica MgrPacMan/YoussefPackaging
Oct 2001Oct 2001Jennifer Schopf, ANLJennifer Schopf, ANL 3
Globus ToolkitTM
Core protocols and servicesCore protocols and services Grid Security Infrastructure
Grid Resource Access & Management
MDS information service & monitoring
GridFTP data access & transfer
Data Grid technologiesData Grid technologies Replica catalog, replica management service
Reliable file transfer
Defacto-standard for Grid projects:Defacto-standard for Grid projects: GriPhyN, PPDG, NEES, EU DataGrid, ESG, Fusion Collaboratory, DISCOM,
NASA IPG, NSF TeraGrid, DOE Science Grid, EU DataGrid, UK Grid Center, U.S.
GRIDS Center, Access Grid, GridPort, MPICH-G2, Condor-G, GrADS, and others
Oct 2001Oct 2001Jennifer Schopf, ANLJennifer Schopf, ANL 4
Globus Status
New releaseNew release In alpha4 now, beta before SC ‘01 (Nov 15), 4Q01 release
New packaging: enables modular binary and source distributions
GRAM 1.5 (job submission): enhanced robustness
MDS-2.1 (information service): security, better performance, etc.
GridFTP- secure large file transfer
Replica Mgt- Data management, catalogs for replicas
Continuing workContinuing work Community Authorization Services
Reliable file transfer
Java, other “Commodity Grid” toolkits
Oct 2001Oct 2001Jennifer Schopf, ANLJennifer Schopf, ANL 5
ATLAS GriPhyN/iVDGL
GriPhyN funded Fall 2000, $11.9M/5 yearsGriPhyN funded Fall 2000, $11.9M/5 years
iVDGL funded Fall 2001, 5 years, $13.65M/5 yeariVDGL funded Fall 2001, 5 years, $13.65M/5 year
Both involve ATLAS, CMS, Ligo and SDSSBoth involve ATLAS, CMS, Ligo and SDSS
ATLAS support: 2.5 FTE IU, 1.5 FTE BU, $331K/5yr IU HWATLAS support: 2.5 FTE IU, 1.5 FTE BU, $331K/5yr IU HW
R. Gardner is the ATLAS lead, J. Schopf is the CS liaisonR. Gardner is the ATLAS lead, J. Schopf is the CS liaison
GriPhyN proposal emphasizes virtual data requirements, GriPhyN proposal emphasizes virtual data requirements,
collaboration between experimentscollaboration between experiments
iVDGL proposal emphasizes testbed and infrastructure iVDGL proposal emphasizes testbed and infrastructure
issues across experimentissues across experiment
Oct 2001Oct 2001Jennifer Schopf, ANLJennifer Schopf, ANL 6
GriPhyN
Principal ATLAS GriPhyN/iVDGL deliverables:Principal ATLAS GriPhyN/iVDGL deliverables: 2001: Testbed with GriPhyN VDT 1.0, packaged with PacMan
2002: Serving DC1 data
2003: Dataset re-creation/Data signature
Additional efforts:Additional efforts: Monitoring: Dantong Yu, BNL, and J. Schopf,ANL, joint with PPDG
GRAPPA: Rob Gardner and Randy Bramley, IU
Oct 2001Oct 2001Jennifer Schopf, ANLJennifer Schopf, ANL 7
GriPhyN Testbed Issues
GriPhyN is defining VDT 1.oGriPhyN is defining VDT 1.o SW install for GriPhyN/PPDG, compatible with EDG as well
Globus 2.0 beta when it’s released (10/30) GSI, GridFTP, MDS, repl, cat stuff, etc (Gram 1.5)
GDMP 2.0 (supports flat files)
Condor 6.3.1 (also Condor-G, Dagman)
Extra tools for ATLASExtra tools for ATLAS objectivity 6.1
Magda
Still need to resolve CA issue between EDG and US test Still need to resolve CA issue between EDG and US test
sitessites
Oct 2001Oct 2001Jennifer Schopf, ANLJennifer Schopf, ANL 8
GriPhyN ATLAS Goal 1Serving DC1 Data (July-Dec 2002)
Limited reconstruction analysis job using grid job Limited reconstruction analysis job using grid job
submission interfacesubmission interface
Serving the data results form DC1Serving the data results form DC1 As part of DC1, data must be tagged with meta data for ease of
access
Minimal keywords would be sufficient
Magda already implements portions of this
Job submission with minimal smartsJob submission with minimal smarts Extend GRAPPA work
Move compute resources to data sites
Oct 2001Oct 2001Jennifer Schopf, ANLJennifer Schopf, ANL 9
GriPhyN ATLAS Goal 2Dataset Re-creation (Jan-Sept 2003)
Goal: be able to re-create a data fileGoal: be able to re-create a data file
Need to evaluate what parameters need to be kept track ofNeed to evaluate what parameters need to be kept track of
Need to evaluate data needed for full data signatureNeed to evaluate data needed for full data signature
Need to develop a metric for evaluating success - what is Need to develop a metric for evaluating success - what is
good enough?good enough?
Oct 2001Oct 2001Jennifer Schopf, ANLJennifer Schopf, ANL 10
GRAPPA: Grid Access Portal for Physics Applications
Provide a point of access to ATLAS grid resourcesProvide a point of access to ATLAS grid resources IU (Physics, CS), Northwestern (ECE), ANL (CS), BU (Physics),
Provide a simple interface for physicists to submit and Provide a simple interface for physicists to submit and
monitor jobs on the Gridmonitor jobs on the Grid Web-based as well as script-based
Ability to “replay”
Compatible with both ATLSim and Athena architectureCompatible with both ATLSim and Athena architecture
Adaptable and/or “extensible” to new developments in Grid Adaptable and/or “extensible” to new developments in Grid
software, Athena, etcsoftware, Athena, etc
Oct 2001Oct 2001Jennifer Schopf, ANLJennifer Schopf, ANL 11
GRAPPA Components
User InterfaceUser Interface
Job submissionJob submission
MonitoringMonitoring
BookkeepingBookkeeping
Resource selectionResource selection
Oct 2001Oct 2001Jennifer Schopf, ANLJennifer Schopf, ANL 12
Current Status
Simple prototype that allows users to submit an Athena job Simple prototype that allows users to submit an Athena job
from a web interface to the Condor pool on the Atlas IU from a web interface to the Condor pool on the Atlas IU
cluster (via Globus)cluster (via Globus)
Next stepsNext steps Adding more Athena functionality to the interface (e.g., user's
defined libraries)
Experiment with other job launch mechanisms Condor-G and DAGMAN description language Web Services Flow Language as a more general workflow description Explore interfaces for multiple metadata and replica catalog systems.
http://lexus.physics.indiana.edu/~griphyn/grappa/http://lexus.physics.indiana.edu/~griphyn/grappa/
Oct 2001Oct 2001Jennifer Schopf, ANLJennifer Schopf, ANL 13
Grid-Enabled Data Access in Athena
David Malon, ANLDavid Malon, ANL
Integrate Grid Data Access techniques (Globus replica Integrate Grid Data Access techniques (Globus replica
catalog and/or GDMP) into the Athena event selection catalog and/or GDMP) into the Athena event selection
modulemodule
When a file is needed, a check is done to see if the file is When a file is needed, a check is done to see if the file is
locallocal if not, use Grid Data Access tools to make it local
Oct 2001Oct 2001Jennifer Schopf, ANLJennifer Schopf, ANL 14
Grid-Enabled Data Access in Athena
Athena and GlobusAthena and Globus Search Globus replica catalog and select
Transfer file using the protocol associated with the location object in
the catalog (gsiftp, https/globus-url-copy from a remote
gass_server, ...)
Athena and GDMPAthena and GDMP Work to date uses GDMP 1.2.2, which had not yet incorporated the
Globus replica catalog
Uses the GDMP import/export catalogs
Supports certain subscription-based approaches
Automatically updates Objectivity/DB internal catalog when
Objectivity database files are transferred
Oct 2001Oct 2001Jennifer Schopf, ANLJennifer Schopf, ANL 15
Status
Paper presented at CHEPPaper presented at CHEP
Prototype up and running on ANL systemsPrototype up and running on ANL systems GDMP-based prototype work done between CERN and Milan
Next Steps:Next Steps: GDMP will use Globus data replica work
Use GDMP to copy files
Requirements and design work for interfaces between Athena data
producers and metadata catalogs and replica management services
(e.g., Magda)
Metadata work – extending the Athena Event Selector properties to
allow for data signature/virtual data
Oct 2001Oct 2001Jennifer Schopf, ANLJennifer Schopf, ANL 16
Monitoring
Joint working group set up between PPDG and GriPhyN to Joint working group set up between PPDG and GriPhyN to
investigate monitoring issuesinvestigate monitoring issues
Led by J. Schopf and D. YuLed by J. Schopf and D. Yu
Monitoring currently defined very broadly:Monitoring currently defined very broadly: Is this router configured correctly?
Has the application finished using that file yet?
What information do I need to determine where to run my
application?
Oct 2001Oct 2001Jennifer Schopf, ANLJennifer Schopf, ANL 17
GridView (Kaushik De)
Tool to show the status of the 8 test bed machines on the Tool to show the status of the 8 test bed machines on the
webweb
Uses Globus GRAM to query sites every 30 minsUses Globus GRAM to query sites every 30 mins
Hostname, Uptime, Idletime, # users, and Load averageHostname, Uptime, Idletime, # users, and Load average
Next steps includeNext steps include Integration with Globus information service (MDS)
Visualization as part of cross PPDG/GriPhyN monitoring work
Oct 2001Oct 2001Jennifer Schopf, ANLJennifer Schopf, ANL 18
Oct 2001Oct 2001Jennifer Schopf, ANLJennifer Schopf, ANL 19
Application Level Monitoring (Taylor)
Initial testbed to incorporate GRID monitoring capabilities Initial testbed to incorporate GRID monitoring capabilities
into Athenainto Athena Collaboration with Valerie Taylor, David Quarrie, and others
Very long ramp-up due to difficulty of “outsiders” running an Very long ramp-up due to difficulty of “outsiders” running an
Athena applicationAthena application
Working on developing an Auditor for AthenaWorking on developing an Auditor for Athena
http://www.ece.nwu.edu/EXTERNAL/vtaylorlab/http://www.ece.nwu.edu/EXTERNAL/vtaylorlab/
prophesy.htmlprophesy.html
Oct 2001Oct 2001Jennifer Schopf, ANLJennifer Schopf, ANL 20
Current Status
Mailing list has been set upMailing list has been set up [email protected]
Defining usage casesDefining usage cases Sensors
Predictors
Archiving
Will gather requirements and look at extending Globus Will gather requirements and look at extending Globus
MDS as a common framework to meet these requirementsMDS as a common framework to meet these requirements
Oct 2001Oct 2001Jennifer Schopf, ANLJennifer Schopf, ANL 21
Summary
Globus Globus
GriPhyN/iVDGLGriPhyN/iVDGL
GrappaGrappa
Grid-Enabled Data AccessGrid-Enabled Data Access
Monitoring and VisualizationMonitoring and Visualization