WLCG Grid Deployment Board, CERN 9 July 2008
Proposal for a Dynamic Megatable
Flavia DonnoCERN/IT
Tier-2 Sites & CapacitiesTier-2 Sites & Capacities A little difficult to determine exact resources installed in the Tier2s; must be improved
(Ian Bird, March LCG OB)
WLCG Grid Deployment Board, CERN 9 July 2008 3
The requirementsThe requirements
The goal Provide the management with information about installed capacity (per VO). Provide the management with information about resource usage (per VO).
This info can also be used by VO operations people in order to "monitor" their usage of the resources
Main focus Storage resources Computing resources
Dynamic view Information should be retrieved as dynamically as possible in order to be reliable The information will be made available through the WLCG accounting system Use the information system as source
WLCG Grid Deployment Board, CERN 9 July 2008 4
The documentationThe documentation
The technical specifications are availablein the CCRC twiki in the SSWG section:https://twiki.cern.ch/twiki/bin/view/LCG/WLCGCommonComputingReadinessChallenges#Storage_Solution_Working_Group_S
Storage only: The documentation available reports on the conclusions reached during
focused meetings with developers and information providers Specific solutions found to cover dCache internal specialized buffers and
avoid double counting
The document has been agreed by storage developers, storage information providers, data management developers, …
WLCG Grid Deployment Board, CERN 9 July 2008 5
The Storage ResourcesThe Storage Resources Current accounting harvests data from BDII
… but many errors in data. combination of Glue1.2 and 1.3
Need to improve information providers and configuration. Existing Reports
Current status (last hour) Monthly Report (like manual Tier1 reports).
WLCG Grid Deployment Board, CERN 9 July 2008 6
The Storage ResourcesThe Storage Resources Use GlueSA class
It describes a Storage Area: logical view of a portion of space (can include disks and tapes) Online resources only for the moment GlueSAs do not overlap. Shared spaces can be represented with one GlueSA with multiple
GlueSAAccessControlBaseRule
A GlueSA can be published for unreserved space as well In this case Total=Reserved Size (see later)
GlueSA can have one or more VOInfo associated objects
VOInfoTag is optional The VOInfo object can be used to publish the path associated to a VO
using the associated GlueSA (in case of shared SAs)
WLCG Grid Deployment Board, CERN 9 July 2008 7
The Storage ResourcesThe Storage Resources GlueSA Sizes
GlueSAReservedOnlineSize Space physically allocated to a VO (or a set of VOs) – Installed capacity
GlueSATotalOnlineSize (in GB = 109 bytes) Total Online Space available at a given moment (it does not account for
broken disk servers, draining pools, etc.) GlueSAUsedOnlineSize (in GB = 109 bytes)
Size occupied by files that are not candidates for garbage collection – Resources usage
For Castor this is equal to GlueSATotalOnlineSize for T1D0 (they differ for T0D1)
GlueSAFreeOnlineSize (in GB = 109 bytes) Free = Total - Used
GlueSACapability Key/value pair to publish agreed characteristics of Storage Area, i.e.
access=read, network=WAN-LAN, etc. Experiments can use this field to select dCache pools
Dynamic information providers for CASTOR by J. Jensen Corrections needed to comply with agreement Packaging and distribution effort through the CASTOR CVS First installation foreseen at CERN Availability: end of July 2008?
Dynamic information providers for dCache available by R. Trompert Need to verify with dCache developers that pinned files usage info is available, and
other details 2-3 weeks to implement the proposal The changes will be reflected also in the new official dCache information providers
by P. Millar
WLCG Grid Deployment Board, CERN 9 July 2008 8
Storage: Status and plansStorage: Status and plans
Dynamic information providers for DPM by Michel Jouvin The proposal has been already implemented Installed at Edinburgh – Testing phase started
Information already available for StoRM by developers Probable minor additions for VOInfoPath
Need to work with OSG and NorduGrid
WLCG Grid Deployment Board, CERN 9 July 2008 9
Storage: Status and plansStorage: Status and plans
WLCG Grid Deployment Board, CERN 9 July 2008 10
The Computing ResourcesThe Computing Resources Pledges expressed in terms of KSI2000 per federation in current Megatable Use available information published in BDII about installed capacity
Red=deprecated
WLCG Grid Deployment Board, CERN 9 July 2008 11
The Computing ResourcesThe Computing Resources Ongoing work to make published information about installed capacity more reliable (see presentation of Steve Traylen at
GDB) Current tools publish always 1 GlueCluster and 1 GlueSubCluster
when a GlueCE is published (problematic CPU counting) Difference between SubCluster PhysicalCPUs and
GlueCETotalCPUs Is Subcluster PhysicalCPUs coherent with SubCluster Host
BenchmarkSI00? When SubCluster PhysicalCPUs=0 consider GlueCETotalCPUs:
Number of slots per queue. Compatible with Subcluster Host BenchmarkSI00? Is this number correct (extracted from batch system)?
VOView TotalCPUs deprecated. Difficult to guess VO shares.
WLCG Grid Deployment Board, CERN 9 July 2008 12
Computing Resources ExampleComputing Resources Example
Need to interpret results Refinement and cleaning (discrepancy between SubClusters
and GlueCE, CPUs vs. cores, HEP only, etc.) Improve information providers (Glue 2.0 helps)
Double-countedResources
Computing resources First procedures in place. Looking for better technologies to make
results available. We are currently in the process of understanding and refining
information
Work with sites Check that needs are covered by Glue 2.0 Help automate the publishing process Work with OSG and NorduGrid Provide a working prototype for APEL by the end of August 2008 Give feedback to GDB and MB Help producing monthly reports
WLCG Grid Deployment Board, CERN 9 July 2008 13
Computing: Status and plansComputing: Status and plans
WLCG Grid Deployment Board, CERN 9 July 2008
Thank You