ScienceDataandtheNDNparadigmInderMongaCTO,ESnetDivisionDeputyofTechnology,Scien?ficNetworkingDivisionLawrenceBerkeleyNa?onalLab NDNComm2015
Experimental and observational science deals with big and small
instruments, and a lot of data!
2
● DatavolumesareincreasingfasterthanMoore’sLaw
● Newalgorithmsandmethodsforanalyzingdata
● Infeasibletoputasupercompu>ngcenterateveryexperimentalfacility
Compu?ngSciencesArea
Alltoocommonprocessofdiscovery
3
ESnet
NewMath
Real->meanalysis
HighperformanceSoFware
Novelcompute/dataplaHorms
Datamgmt.andsharing
Program-mablenetwork
Extreme Data Science Facility
(XDSF)
MS-DESI
ALS
LHC
JGI
APS
LCLS
Other data-producing sources
-4-
‘Superfacility’Vision:Anetworkofconnectedfacili>es,soFwareandexper>setoenablenewmodesofdiscovery
ESnetisadedicatedmissionnetworkengineeredtoaccelerateabroadrangeofscienceoutcomes.
Wedothisbyofferinguniquecapabili?es,andop?mizingthenetworkfordataacquisi?on,data
placement,datasharing,datamobility.
ESnetisdesignedfordifferentgoalsthan
generalInternet.
Feb-94 Apr-98 Jun-02 Aug-06 Oct-10 Dec-14
1017
1010
1011
1012
1013
1014
1015
1016
Month
Tota
l
Bytes of Science Data Transferred Each Monthby the Energy Sciences Network
August2015:29.13PB
Lotsofdatatomovearound
Feb-94 Apr-98 Jun-02 Aug-06 Oct-10 Dec-14
3 × 1016
0
5 × 1015
1 × 1016
1.5 × 1016
2 × 1016
2.5 × 1016
Month
Tota
l
Bytes of Science Data Transferred Each Month
by the Energy Sciences Network
August2015:29.13PB
Lotsofdatatomovearound(contd.)
High-levelobjec>vesforscien>ficdata:alignmentwithNDNapproach
10/25/159
• Radicallysimplifyhowscien?ficusersmanage,moveandmanipulatelarge,distributed,sciencedatarepositories,butwithhigh-throughputend2end
• Abstractthestorageandnetworkcapabilityandloca?ondependencefromtheuser-datainterac?on
• Enabletheabilityforuserstospecifyandretrievepor?onsofdatatheworkflowneeds
• Createasecure,scalableframeworkbasedonintegrateddatamanagementandnetworktransport
UseCase#1
10/25/1510
ResearchersfromBerkeleyLabandSLACconductedproteincrystallographyexperimentsatLCLStoinves>gatephotoexcited
statesofPSII,withnear-real->mecomputa>onalanalysisatNERSC.
“Takingsnapshotsofphotosynthe?cwateroxida?onusingfemtosecondX-raydiffrac?onandspectroscopy,”NatureCommunica.ons5,4371(9July2014)
50TBmovedanight
UseCase#2:LHCONEdata–mul>plereplicas,globalreach
UseCase#3:Interna>onalClimateData
10/25/1512
Percep>onoflimita>onsofNDNmo>va>ngresearchques>ons
1. IfIammoving50TBofdatathroughasinglepath,fromanexperimenttoastoragefacility,IreallydonotwanttocacheitateveryintermediateNDNnode
– Whatistherightstrategyforalloca?ngdiskresourcestocaching?Whatifonedatatransferconsumesallcacheresourcesorthereisnotenoughspace?
2. Whatistheperformanceoftheend-to-enddatatransfer?HowcanIgetlineratethroughput?
3. HowdoIleveragetheknowledgeofnetworkcapabilityinchoosingthetransferpath?HowdoIbuildintheknowledgeofunderlayintotheNDNoverlay?
4. HowdoIleveragenetworkprogrammabilitytodotheabove?
5. Andmanyotherques?ons….
10/25/1513
Whereareweat?
• Collabora?onwithChristosandColoradoState–high-poweredNDNdevicesbetweenthreerepresenta?veclimatesitesasatestbed– Susmitworkingonansweringsomeofthehigh-levelobjec?vesasdescribed
• HEPandASCRinterestinNDNfromaresearchperspec?ve–paperearlierthisyear@CHEP,andPhilwilltalkaboutnext-stepsrightaher
• Interestinexpandingafedera?onofhigh-poweredNDNdeviceswiththerightstrategyforcachinganddatamanagement
• CombiningNDNwithSDN–wehaveanext-genSDNtestbedacrossUSandEurope–canwecombinethattoprovidetherightprimi?vesforhigh-performanceNDN?– Letsdoitera?veexperimenta?onandimprovement!!!!!!!
10/25/1514
ALBQ
AMST
ANL
AOFA
ATLA
BNL
BOIS
BOST
CERN
CHICDENV
ELPA
FNAL
HOUS
KANS
LANL
LBL LLNL
LOND
NASHNERSC
NEWY
ORNL
PNNLPNWG
SACR
SAND
SLAC
STAR
SUNN
WASH
ESnet PE Router
(2+)x10GE
(n)x10GE
Testbed Host
Deployed SDN Testbed node locations Deployed SDN Testbed connectivity overlay (using OSCARS circuits)
ESnetSDNTestbed
AMST
CERN
AOFA
WASH
STAR
ATLA
DENV
LBL
August2015 iDiscovery2020InderMonga15
StatusUpdate:• Testbeddeployedatallloca?ons• QoSsupportverified,pressrelease
nextweek• ENOSdemoonTestbed@SC
Thankyou!
• Pleasefeelfreetoemailmewithques?ons,commentsorarrowsat
imongaatesdotnet
10/25/1516
10/25/1517