Date post: | 02-Jan-2016 |
Category: |
Documents |
Upload: | angelica-dalton |
View: | 215 times |
Download: | 0 times |
1New developments in the HENP-GC HENP-GC Collaboration
New Capabilities in the New Capabilities in the HENP Grand Challenge Storage Access System HENP Grand Challenge Storage Access System
and its Application at RHICand its Application at RHIC
David Malon, Argonne presenting for the HENP-GC collaboration*
(http:/www-rnc.lbl.gov/GC/)
8 Feb 2000
CHEP*slides thanks to L. Bernardo, D. Olson, A. Shoshani, S. Vanyashin
2New developments in the HENP-GC HENP-GC Collaboration
OutlineOutline
• Overview of HENP-GC
• What’s new since CHEP’98
• STACS
• Experiment interface
• Scalability testing
• Conclusion
3New developments in the HENP-GC HENP-GC Collaboration
High-Energy & Nuclear Physics Grand ChallengeHigh-Energy & Nuclear Physics Grand Challenge
• 3 year project, mid-1997 to mid-2000
• Funded by DOE/MICS with contributed effort from DOE/HENP
• Participants:– NERSC/Berkeley Lab
• L. Bernardo, A. Mueller, H. Nordberg, A. Shoshani, A. Sim, J. Wu
– Argonne• D. Malon, E. May, G. Pandola
– Brookhaven Lab• B. Gibbard, S. Johnson, J. Porter, T. Wenaus
– Nuclear Science/Berkeley Lab• D. Olson, A. Vaniachine, J. Yang, D. Zimmerman
4New developments in the HENP-GC HENP-GC Collaboration
What is the Grand Challenge architecture?What is the Grand Challenge architecture?
• An order-optimized prefetch architecture for data retrieval from multilevel storage in a multiuser environment
• Queries select events and specific event components based upon tag attribute ranges– query estimates are provided prior to execution– collections as queries are also supported
• Because event components are distributed over several files, processing an event requires delivery of a “bundle” of files
• Events are delivered in an order that takes advantage of what is already on disk, and multiuser policy-based prefetching of further data from tertiary storage
• GCA intercomponent communication is CORBA-based, but physicists are shielded from this layer
5New developments in the HENP-GC HENP-GC Collaboration
System OverviewSystem Overview
Client
GCASTACS
Stagedeventfiles
EventTags (Other)
disk-residentevent data
Index
HPSSpftp
File Catalog
ClientClient
Client
Client
6New developments in the HENP-GC HENP-GC Collaboration
STorage Access Coordination System STorage Access Coordination System (STACS)(STACS)
QueryEstimator
QueryMonitor
CacheManager
PolicyModule
Bit-SlicedIndex
FileCatalog
QueryStatus,CacheMap
Query
Estimate
File Bundles,Event lists
Pftp andfile purgecommands
List offile bundles and events
Requests forfile cachingand purging
7New developments in the HENP-GC HENP-GC Collaboration
What is new since CHEP’98What is new since CHEP’98
• Multi-component multi-file event model– Event is composed of separate components– Components of a single event are stored in separate
files
• Removal of Objectivity/DB-specific dependencies– eventID is an experiment-specific typedef– CORBA file catalog interface
• User-accessible file bundle information for user-code- dependent file I/O
• CORBA interface to tag database
• Scalability tests– 10M events, 7 components, 100-250 queries
8New developments in the HENP-GC HENP-GC Collaboration
Multiple-Component EventsMultiple-Component Events
• Event Components– partition each event into 5-10 pieces– tracks, hits, vertices …
• Queries can request one or more components– all components of an event must be in disk cache at the
same time
• Problem: how to manage multiple component files to minimize re-caching of files
• Pseudo query language SELECT tracks, hits
FROM Run17WHERE glb_trk_tot>0 & glb_trk_tot<10 & n_vert_total<3
9New developments in the HENP-GC HENP-GC Collaboration
Example of multiple componentsExample of multiple components
e1
e2
e3
e4
e5
e6
e7
e8
e9
Files ofComponent A
Files ofComponent B
Component Aof event e1
Component Bof event e1
File 4File 3
File 1 File 2
File Bundles: (F1,F2: e1,e2,e3,e5), (F3,F2: e4,e7), (F3,F4: e6,e8,e9)
10New developments in the HENP-GC HENP-GC Collaboration
Multicomponent Event DeliveryMulticomponent Event Delivery
• Grand Challenge software– partitions the collection of qualifying events
according to which file bundles must be cached on disk to permit their processing
– attempts to optimize the order of bundle delivery in a multiuser environment
– supports prefetching of bundles
11New developments in the HENP-GC HENP-GC Collaboration
File Weight Policy for Multi-Component EventsFile Weight Policy for Multi-Component Events
• File weight (bundle) = 1 if it appears in a bundle, = 0 otherwise
• Initial file weight = SUM (all bundles for each query) over all queries
• Dynamic file weight: the file weight for a file in a bundle that was processed is decremented by 1
queries bundles
bundleperweightfileIFW ___
queries
bundlesprocessed
bundleperweightfile ___
queries
bundlesall
bundleperweightfileDFW ___
12New developments in the HENP-GC HENP-GC Collaboration
Caching Policy (1)Caching Policy (1)
• Query service policy– round robin– query is skipped if no bundle fits available
cache– when query is skipped, a skip_service counter
is incremented.– if counter is above preset limit, all activity
stops till this query is serviced
13New developments in the HENP-GC HENP-GC Collaboration
Caching Policy (2)Caching Policy (2)
• Bundle caching policy
– select bundle with most files in cache– if a tie, select bundle with highest weight– if not enough space in cache, select next
bundle that fits in cache– if none fit in cache, select next bundle with
one less file in cache, etc.– if no bundles found, skip query, and increase
skip_service counter
bundlethein
filesall
weightfiledynamicweightBundle ___
14New developments in the HENP-GC HENP-GC Collaboration
Caching Policy (3)Caching Policy (3)
• File purging policy– files are in 2 categories:
• file currently in use• file not currently in use
– purge file with lowest dynamic_file_weight– if a tie, purge largest file
• Pre-fetching policy– Initially: unlimited– this parameter can be assigned dynamically
15New developments in the HENP-GC HENP-GC Collaboration
File Tracking LogFile Tracking Log
Query 1starts here
Query 2starts here
Bundle wasfound in cache
Bundle sharedby two queries
Bundle (3 files)formed, then
passed to query
16New developments in the HENP-GC HENP-GC Collaboration
STAR event modelSTAR event model
T. Ullrich, Jan. 2000
17New developments in the HENP-GC HENP-GC Collaboration
database
Interfacing GCA to experimentInterfacing GCA to experiment
GC System
StIOMaker
fileCatalog
tagDB
QueryMonitor
CacheManager
QueryEstimatorgcaClient
FileCatalog
IndexFeeder
GCA Interface STAR Components
IndexBuilder
18New developments in the HENP-GC HENP-GC Collaboration
Experiment-specific Experiment-specific ImplementationsImplementations
• IndexFeeder utility– Experiment provides “tag database” that has
for each event• attributes used for event selection (“tags”)• fileID for each event component
– IndexFeeder reads the experiment’s “tag database” so that GC-provided “index builder” can create index
• FileCatalog server– FileCatalog queries the “file catalog” database
of the experiment to translate fileID to HPSS & disk path
19New developments in the HENP-GC HENP-GC Collaboration
Client-side implementationClient-side implementation
• gcaClient interface– gcaResources interface: initialization,
configuration, establishment of contact with remote STACS components
– QueryObject: query definition, estimation, execution
– Order-Optimized Iterator: delivery of event ids (and optional file information) as bundles are cached
20New developments in the HENP-GC HENP-GC Collaboration
Iterator ExtensionsIterator Extensions
• // iterator is initialized with this query's token, and a // pointer to GCA_Resources for access to remote STACS //components and configuration parameters:
OrderOptIter GCIter(query->token(), &GCA_Resources);
while (GCIter.next(eventID& myEvent){ usercode(myEvent); // process an event
}
• In order to allow for experiment-specific code to handle file I/O an optional flag that signals a new file bundle is used along with a method to retrieve the file name for each event component:
while(GCIter.next(eventID& myEvent, bool& thisIsANewBundle)) { ...}
• string getComponentFileName(const string componentName)
21New developments in the HENP-GC HENP-GC Collaboration
STAR uses the fileCatalog & instance tables in STAR uses the fileCatalog & instance tables in MySQL to satisfy the fcFileCatalog CORBA interfaceMySQL to satisfy the fcFileCatalog CORBA interface
// fileCat.idl// Luis Bernardo <[email protected]>, Alex Sim <[email protected]>// Lawrence Berkeley National Laboratory// May 99// Purpose: defines interface between File Catalog (server) and Cache// Manager, Query Estimator and Query Monitor (clients).
#include "smDefs.idl"
struct FileInfo { FID_T fid; double fileSize; string localFileName; string remoteFileName; string tapeID;};
typedef FileInfo FILEINFO_T;
interface fcFileCatalog { double getFileSize(in FID_T fid); double getSumFileSizes(in FIDSET_T fset); FSIZE_SET_T getListFileSizes(in FIDSET_T fset); FILEINFO_T getFileInfo(in FID_T fid); void updateFileCatalog(in string asciifilecat);};
22New developments in the HENP-GC HENP-GC Collaboration
Schema of fileCatalog tableSchema of fileCatalog table
mysql> desc fileCatalog;+--------------+-----------------------------------+------+-----+---------------------+----------------+| Field | Type | Null | Key | Default | Extra |+--------------+-----------------------------------+------+-----+---------------------+----------------+| prodType | enum('unknown','daq','sim','job') | | MUL | unknown | || prodName | varchar(80) | | | | || prodSerie | int(11) | | | 0 | || prodInstance | int(11) | | | 0 | || fileSequence | int(11) | | | 0 | || dbServer | enum('unknown','bnl','lbl') | | | unknown | || eventType | int(11) | | | 0 | || path | varchar(64) | | | | || fileName | varchar(20) | | | | || dataset | varchar(64) | | | | || size | int(11) | | | 0 | || createTime | datetime | | | 0000-00-00 00:00:00 | || insertTime | timestamp(10) | YES | | NULL | || Nevents | mediumint(9) | | | 0 | || NevLo | mediumint(9) | | | 0 | || NevHi | mediumint(9) | | | 0 | || owner | varchar(20) | | | | || grp | varchar(20) | | | star | || permit | varchar(10) | | | -rw-r----- | || type | varchar(20) | | | | || component | varchar(20) | | | | || format | varchar(10) | | | | || site | varchar(10) | | | | || hpss | enum('Y','N') | | | Y | || status | smallint(6) | | | 0 | || comment | blob | | | NULL | || tape | smallint(6) | | | 0 | || generation | smallint(6) | YES | | 0 | || ID | mediumint(9) | | PRI | 0 | auto_increment |+--------------+-----------------------------------+------+-----+---------------------+----------------+29 rows in set (0.08 sec)
Path to primary instancein HPSS
23New developments in the HENP-GC HENP-GC Collaboration
Schema of instance tableSchema of instance table
mysql> desc instances;+--------------+---------------+------+-----+---------------------+-------+| Field | Type | Null | Key | Default | Extra |+--------------+---------------+------+-----+---------------------+-------+| fID | int(9) | | PRI | 0 | || instance | tinyint(4) | | PRI | 0 | || created | datetime | | | 0000-00-00 00:00:00 | || volume | varchar(30) | | | n/a | || path | varchar(128) | | | n/a | || permit | varchar(10) | YES | | NULL | || owner | varchar(20) | YES | | NULL | || grp | varchar(20) | YES | | NULL | || hpss | enum('Y','N') | | | Y | || lastAccessed | datetime | | | 0000-00-00 00:00:00 | || site | varchar(10) | | | BNL | || location | varchar(10) | | | rcf | || tape | smallint(6) | | | 0 | || comment | varchar(255) | YES | | NULL | || enteredDB | timestamp(10) | YES | | NULL | |+--------------+---------------+------+-----+---------------------+-------+15 rows in set (0.10 sec)
Disk path, location wherecache manager puts file
24New developments in the HENP-GC HENP-GC Collaboration
Scalability testingScalability testing
• Test Dataset– 10M events– 7 event components– 1.6 TB– 4700 files
• QE tested up to 100 concurrent queries
• QM tested up to 250 concurrent queries
• 24 hour runs
• Bugs were found & fixed, system ran OK
25New developments in the HENP-GC HENP-GC Collaboration
File processing by 100 queriesFile processing by 100 queries
26New developments in the HENP-GC HENP-GC Collaboration
File stage requestsFile stage requests
27New developments in the HENP-GC HENP-GC Collaboration
QE estimation timesQE estimation times
28New developments in the HENP-GC HENP-GC Collaboration
ConclusionConclusion
• HENP-GC has developed a system for optimized access to multi-component event data files stored in HPSS.
• General CORBA interfaces are defined for interfacing with the experiment.
• A client component encapsulates interaction with the servers and provides an ODMG-style iterator.
• Has been tested up to 10M events, 7 event components, 250 concurrent queries.
• Is currently being integrated with the STAR experiment ROOT-based I/O analysis system.