STACS: Storage Access Coordination of Tertiary Storage for High Energy Physics Applications Arie Shoshani, Alex Sim, John Wu, Luis Bernardo*, Henrik Nordberg*, Doron Rotem* Scientific Data Management Group Computing Science Directorate Lawrence Berkeley National Laboratory. - PowerPoint PPT Presentation
48
STACS TACS: Storage Access Coordination TACS: Storage Access Coordination of Tertiary Storage of Tertiary Storage or High Energy Physics Application or High Energy Physics Application Arie Shoshani, Alex Sim, John Wu, Arie Shoshani, Alex Sim, John Wu, Luis Bernardo*, Henrik Nordberg*, Luis Bernardo*, Henrik Nordberg*, Doron Rotem* Doron Rotem* Scientific Data Management Group Scientific Data Management Group Computing Science Directorate Computing Science Directorate Lawrence Berkeley National Laboratory Lawrence Berkeley National Laboratory * no longer at LBNL
Transcript
STACS
STACS: Storage Access Coordination STACS: Storage Access Coordination of Tertiary Storage of Tertiary Storage
for High Energy Physics Applicationsfor High Energy Physics Applications
Arie Shoshani, Alex Sim, John Wu,Arie Shoshani, Alex Sim, John Wu,
Luis Bernardo*, Henrik Nordberg*,Luis Bernardo*, Henrik Nordberg*,
Doron Rotem*Doron Rotem*
Scientific Data Management GroupScientific Data Management GroupComputing Science DirectorateComputing Science Directorate
Lawrence Berkeley National LaboratoryLawrence Berkeley National Laboratory
* no longer at LBNL
STACS
Outline
• Short High Energy Physics overview (of data handling problem)
• Description of the Storage Coordination System• File tracking• The Query Estimator (QE)
— Details of the bit-sliced index• The Query Monitor
— coordination of “file bundles”• The Cache Manager
— tertiary storage queuing and tape coordination— transfer time for query estimation
STACS
Optimizing Storage Management forOptimizing Storage Management for High Energy Physics Applications High Energy Physics Applications
Collaboration# members/institutions
Date offirst data # events/year
total datavolume/year-TB
STAR 350/35 2000 107-10
8 300
PHENIX 350/35 2000 109 600
BABAR 300/30 1999 109 80
CLAS 200/40 1997 1010 300
ATLAS 1200/140 2004 109 2000
Data Volumes for planned HENP experiments
STAR: Solenoidal Tracker At RHICRHIC: Relativistic Heavy Ion Collider
STACS
Particle Detection Systems
STAR detector at RHIC Phenix at RHIC
STACS
Result of Particle Collision (event)
STACS
Typical Scientific Exploration Process
• Generate large amounts of raw data— large simulations— collect from experiments
• Post-processing of data— analyze data (find particles produced, tracks)— generate summary data
• e.g. momentum, no. of pions, transverse energy
• Number of properties is large (50-100)
• Analyze data— use summary data as guide— extract subsets from the large dataset
• Need to access events based on partialproperties specification (range queries)
• e.g. ((0.1 < AVpT < 0.2) ^ (10 < Np < 20)) v (N > 6000)
— apply analysis code
STACS
• STAR experiment
— 108 events over 3 years
— 1-10 MB per event: reconstructed data
— events organized into 0.1 - 1 GB files
— 1015 total size
— 106 files, ~30,000 tapes (30 GB tapes)
• Access patterns
— Subsets of events are selected by region in high-dimensional property space for analysis
-- The index will generate a set of bundles {[F7, F16: E4,E17,E44], [F13, F16: E6,E8,E32], …}
that the query needs
-- The bundles can be returned to the application in any order
-- Bundle: the set of files that need to be in cache at the same time
STACS
File Weight Policy for ManagingFile Bundles
• File weight (bundle) = 1 if it appears in a bundle, = 0 otherwise
• Initial file weight = SUM (all bundles for each query) over all queries
• Example:— query 1: file FK appears is 5 bundles
— query 2: file FK appears is 3 bundles
Then, IFW (FK) = 8
queriesinbundlesall
jbundleinisFifotherwiseij
queriesall
i j
kkk FWFIFW 1
0{)()(
STACS
File Weight Policy for ManagingFile Bundles (cont’d)
•Dynamic file weight: the file weight for a file in a bundle that was processed is decremented by 1
• Dynamic Bundle Weight
queriesin
bundlesprocessed
jbundleinisFifotherwiseij
queriesall
i j
kkkk FWFIFWFDFW 1
0{)()()(
ibundlein
filesall
kkFDFWBiDBW )()(
STACS
How file weights are usedfor caching and purging
• Bundle caching policy— For each query, in turn, cache the bundle
with the most files in cache— In case of a tie, select the bundle with the
highest weight— Ensures that a bundle that include files
needed by other bundles/queries have priority• File purging policy
— No file purging occurs till space is needed— Purge file not in use with smallest weight— Ensures that files needed in other bundles
stay in cache
STACS
Other policies
• Pre-fetching policy— queries can request pre-fetching of bundles
subject to a limit— Currently, limit set to two bundles— multiple pre-fetching useful for parallel
processing• Query service policy
— queries serviced in Round Robin fashion— queries that have all their bundles cached
and are still processing are skipped
STACS
Managing the queues
.
.
....
.
.
.
.
.
.
.
.
.
QueryQueue
BundleSet
FileSet
FilesBeing
Processed
Filesin Cache
STACS
File Tracking of Bundles
Query 1starts here
Query 2starts here
Bundle wasfound in cache
Bundle sharedby two queries
Bundle (3 files)formed, then
passed to query
STACS
Summary
• The key to managing bundle caching and purging policies is weight assignment— caching - based on bundle weight— purging - based on file weight
• Other file weight policies are possible— e.g. based on bundle size— e.g. based on tape sharing
• Proving which policy is best - a hard problem— can test in real system - expensive, need stand alone— simulation - too many parameters in query profile can
vary: processing time, inter-arrival time, number of drive, size of cache, etc.
— model with a system of queues - hard to model policies— we are working on last two methods
STACS
The Storage Access Coordination System (STACS)
QueryEstimator
(QE)
CacheManager
(CM)
QueryMonitor
(QM)
Query estimation /execution requests
file cachingrequest
CachingPolicy
Module
FileCatalog
(FC)
Bit-Slicedindex
file purging
Disk Cache
file caching
User’sApplication
open,read,close
STACS
Queuing File Transfers
• Number of PFTPs to HPSS are limited— limit set by a parameter - NoPFTP— parameter can be changed dynamically
• CM is multi-threaded— issues and monitors multiple PFTPs in parallel
• All requests beyond PFTP limit are queued• File Catalog used to provide for each file
— HPSS path/file_name— Disk cache path/file_name— File size— tape ID
STACS
File Queue Management
• Goal— minimize tape mounts— still respect the order of requests— do not postpone unpopular tapes forever
• File clustering parameter - FCP
— If the file at top of queue is in Tapei
and FCP > 1 (e.g. 5) then up to 4 files from Tapei will be selected to be transferred next
— then, go back to file at top of queue• Parameter can be set dynamically F(Ti)
F(Ti)
F(Ti)
F(Ti)
1
2
3
4
5
STACS
File Queue Management
• Goal— minimize tape mounts— still respect the order of requests— do not postpone unpopular tapes forever
• File clustering parameter - FCP
— If the file at top of queue is in Tapei
and FCP > 1 (e.g. 4) then up to 4 files from Tapei will be selected to be transferred next
— then, go back to file at top of queue• Parameter can be set dynamically
F1(Ti)
F3(Ti)
F2(Ti)
F4(Ti)
1
2
3
4
5
Orderof fileservice
STACS
File Caching Order fordifferent File Clustering Parameters
— HPSS error (I/O, device busy)• remove part of file from cache, re-queue• try n times (e.g. 3), then return error
“transfer_failed”
— HPSS down• re-queue request, try repeatedly till successful• respond to File_status request with “HPSS_down”
STACS
Summary
• HPSS Hierarchical Resource Manager (HRM)— insulates applications from transient
HPSS and network errors— limits concurrent PFTPs to HPSS— manages queue to minimize tape mounts— provides file/query time estimates— handles errors in a generic way
• Same API can be used for any MSS, suchas Unitree, Enstore, etc.
