Page 1
Unified QualityofService and DataLifecycle Definitions for Data Storage
and Access
… or how to managing expectations
Paul Millar
RDA Plenary 6 BoF (20150925)
Page 2
Why are we here?
In INDIGODataCloud ...● We've identified a problem,
(well, two actually),● We want to fix this problem,
(we hope you do too!)● We want your help in fixing it
(we hope you do too!)
Page 3
Storage software: Free, OpenSource
https://github.com/dCache/dCachemailto:[email protected]
Page 4
Software running throughout the world
Page 5
dCache and INDIGODataCloud
Page 6
The problem...
Quality of Serviceand
Data LifeCycle
Page 7
Quality of Service
Page 8
Store data on disk or tape?
Page 9
Now we have more media options
Page 10
Replicating data
How many copies? Where are they located?
Page 11
Motivation: budgets
Page 12
How to make this a possibility
What are my options? How do I choose?
Page 14
Attributes and islands
Page 15
Combining QoS attributes
Independent Dependent
Continuous
Discrete
Discrete
Discrete “Islands of QoS”“Fee selection of QoS”
Page 16
Figureofmerit: allowing decisions
Best available QoS
Best available QoS
Page 18
DLC usecases: the story of a file
Time
CreatedMain
analysis complete
Publicembargo
ends
Anticipated end of
interestEnd of life
Change QoS
Allow public access
Change QoS
Delete data
Change QoS
Accept/Reject
Deadline
(not to scale)
Page 19
Format for DLC rules
<trigger> <action>
(e.g., <after 6 months> <add publicaccess ACE>)
Page 20
The plan:Definition of terms
Protocol definition
Implementations
Photos: greeblie@flickr, Steve Jurvetson, Gorazd Božič
Page 21
Proposal: RDA WG “dictionary of terms”
Photo: greeblie@flickr
Page 22
“Speed” accesslatency or bandwidth?
Photo: John Holm
Page 23
How fast is “High”?
Photo: lungstruck@flickr
Page 24
Would you be able to work this?
Photo: David Pursehouse
Page 25
Thanks for listening
Page 26
Used to search for the Higgs boson
Page 27
Feed data for HPC applications
HPC jobs on supercomputer
HPC jobs get access to dCache storage.
Page 28
Research: pushing frontiersPower supply
HGSTDisk
Clip Yves
Page 29
Software that scales up to tens of PiB
Page 30
Pool
NFS 4.1/pNFS
HTTP/WebDAV
PoolManager
gPlazma
1 TB
700 MHz ARM512 MB Memory2 * USB 2100 MB Ethernet
… and down to a single Raspberry Pi