Presented at Computing in High Energy PhysicsPresented at Computing in High Energy PhysicsInterlaken, Switzerland, September 2004Interlaken, Switzerland, September 2004
ContentsContents GoalsGoals Java IO library for RootJava IO library for Root
– ImplementationImplementation– Test SuiteTest Suite– BenchmarksBenchmarks– Networked Root IO (rootd/xrootd)Networked Root IO (rootd/xrootd)– Demo toolsDemo tools– Root Plugin for AIDARoot Plugin for AIDA
Examples of UseExamples of Use– AIDATLD Web InterfaceAIDATLD Web Interface– JAS3JAS3– WIRED4WIRED4
Future plansFuture plans Links for more informationLinks for more information
Java Root IO: GoalsJava Root IO: Goals GoalsGoals
– Pure Java package for reading Root FilesPure Java package for reading Root Files Will be extended to Will be extended to writingwriting later later
– High PerformanceHigh Performance We want to do data analysis so we need very good We want to do data analysis so we need very good
performanceperformance– Should not need to know about objects ahead Should not need to know about objects ahead
of timeof time(no need for dll’s, .so files (no need for dll’s, .so files etc.etc.))
But should provide full support for reading user-But should provide full support for reading user-defined objectsdefined objects
– Easy to useEasy to use– Provide access to Root Provide access to Root datadata, not C++ code, not C++ code
ImplementationImplementation
Use “Use “TStreamerInfoTStreamerInfo” objects inside Root files ” objects inside Root files (since Root 3.0) to decode contents of file(since Root 3.0) to decode contents of file
Dynamically generate Java proxies for each Root Dynamically generate Java proxies for each Root object readobject read
Each proxy has custom “streamer” Each proxy has custom “streamer” – Build as Java Build as Java bytecodebytecode using BCEL ( using BCEL (http://http://
jakarta.apache.org/bceljakarta.apache.org/bcel) ) – Converted at runtime to machine code by HotSpot VMConverted at runtime to machine code by HotSpot VM
Uses java.nio package (new since Java 1.4)Uses java.nio package (new since Java 1.4)– Standard Java package for high-performance binary IOStandard Java package for high-performance binary IO
Implementation DetailsImplementation Details
TStreamerInfo
Java InterfaceAccessor method for each data member .java file
Java ProxyCustom version based on how object is stored in file
Implem
ents
InterfaceGenerator
Example of UseExample of Use
Test SuiteTest Suite Most classes are dynamically generated from info Most classes are dynamically generated from info
in Root file, so are fairly robust against changesin Root file, so are fairly robust against changes Need bootstrap process, in particularly need Need bootstrap process, in particularly need
enough custom Java classes to read the streamer enough custom Java classes to read the streamer info, and interpret itinfo, and interpret it– Relatively few classes, butRelatively few classes, but
This tends to change from one Root release to anotherThis tends to change from one Root release to another Typically have to make small changes in the bootstrap code Typically have to make small changes in the bootstrap code
for each new Root releasefor each new Root release We have test suite that creates a set of test files We have test suite that creates a set of test files
from each new root release, and verifies that our from each new root release, and verifies that our library can read the files.library can read the files.– Currently test 3.00.06, 3.01.06, 3.02.07, 3.03.07, 3.05.07, Currently test 3.00.06, 3.01.06, 3.02.07, 3.03.07, 3.05.07,
3.10.02, 4.00.083.10.02, 4.00.08 In addition we have a set of Root files collected In addition we have a set of Root files collected
from users, and the test suite verifies that we can from users, and the test suite verifies that we can (continue) to read these files with each release.(continue) to read these files with each release.
Java Root IO - BenchmarkJava Root IO - Benchmark
1.4GHz AMD Athlon, Windows 1.4GHz AMD Athlon, Windows XP, 512MByte DDR, ATA100 XP, 512MByte DDR, ATA100 disk..disk..
SS= Split level = 0,1,2= Split level = 0,1,2 CC= Compression = 0,1= Compression = 0,1 Each test run 5 times in Each test run 5 times in
succession, lines show range in succession, lines show range in results results – in later runs file is typically in later runs file is typically
cached in memory == faster if cached in memory == faster if task IO limitedtask IO limited
Task: Loop over all events, loop over all tracks, sum Task: Loop over all events, loop over all tracks, sum px for all tracks. 5000 event files created with “event” px for all tracks. 5000 event files created with “event” test program distributed with Root (200-400 MByte).test program distributed with Root (200-400 MByte).
C S ms/Event
0 0 12.3 - 4.23
0 1 10.7 -- 2.75
0 2 1.70 -- 0.20
1 0 10.5 -- 5.12
1 1 9.6 -- 4.77
1 2 2.7 -- 0.25
Network Root IO (x)rootdNetwork Root IO (x)rootd Recently added ability for Java library to Recently added ability for Java library to
talk directly to rootd or xrootd servertalk directly to rootd or xrootd server– Defined “root:” Defined “root:” protocol handlerprotocol handler for URL class. for URL class.– Defined Defined AuthenticatorAuthenticator for authenticating to for authenticating to
rootdrootd– Extended Extended RootFileReaderRootFileReader to accept (any) URL to accept (any) URL
argument to constructorargument to constructor If URL corresponds to If URL corresponds to file:file: read the file directly read the file directly If URL corresponds to If URL corresponds to root:root: protocol use xrootd/rootd protocol use xrootd/rootd
(random access)(random access) If URL corresponds to If URL corresponds to http: http: and web server supports and web server supports
http 1.1 “ranges”, use extended httphttp 1.1 “ranges”, use extended http Otherwise buffer URL contents into memory and read Otherwise buffer URL contents into memory and read
from there (suitable for small files only).from there (suitable for small files only).
Authentification IssuesAuthentification Issues Authentification can be set programmatically, or via Authentification can be set programmatically, or via
GUI AuthenticatorGUI Authenticator Rootd supports many authentification schemesRootd supports many authentification schemes
• Currently FreeHep library supports only 1 and 2Currently FreeHep library supports only 1 and 2• Work underway on 3 (but some compatibility problems Work underway on 3 (but some compatibility problems
between Root and Java RSA implementations)between Root and Java RSA implementations)• Java libraries exist for all other methods, so can be Java libraries exist for all other methods, so can be
supported in future.supported in future.
Network Root IO: Example of UseNetwork Root IO: Example of Use
Root Plug-in for JAIDARoot Plug-in for JAIDA JAIDAJAIDA = Java implementation of AIDA = Java implementation of AIDA AIDAAIDA = Abstract Interfaces for Data Analysis = = Abstract Interfaces for Data Analysis =
Histograms + Tuples + Fitting + Plotting + IO data Histograms + Tuples + Fitting + Plotting + IO data analysis package analysis package – (See Monday’s talk by Victor Serbo)(See Monday’s talk by Victor Serbo)
Root plugin for JAIDA makes it possible to read root Root plugin for JAIDA makes it possible to read root files via the AIDA interfaces and treat the objects files via the AIDA interfaces and treat the objects contained in the root file as AIDA objectscontained in the root file as AIDA objects– Useful if:Useful if:
You like AIDA interfaces better than root interfacesYou like AIDA interfaces better than root interfaces You want to use language not supported by RootYou want to use language not supported by Root You want to analyze data in mixed formatsYou want to analyze data in mixed formats You want to work in J2EE, J2ME, Applet You want to work in J2EE, J2ME, Applet etc.etc.
Root Plug-in for JAIDA: ExampleRoot Plug-in for JAIDA: Example
Root Java IO: Root Java IO: Examples of UseExamples of Use
AIDATLD Web InterfaceAIDATLD Web Interface
JAS3JAS3
WIRED4WIRED4
AIDATLD + RootAIDATLD + Root AIDATLD = AIDA Tag AIDATLD = AIDA Tag
Library for use in JSP Library for use in JSP
pagespages – Trivial to put “live” Trivial to put “live”
histograms on webhistograms on web– Directly supports Root Directly supports Root
filesfiles
AIDATLD Example: GLAST System AIDATLD Example: GLAST System Tests Web InterfaceTests Web Interface
JAS3 + RootJAS3 + Root
JAS3 = Modular, Extensible, Data Format JAS3 = Modular, Extensible, Data Format Independent data analysis system.Independent data analysis system.
Among many other formats JAS3 supports Among many other formats JAS3 supports RootRoot– Contents of Root files can be accessed directly, Contents of Root files can be accessed directly,
or via Root->AIDA interfaceor via Root->AIDA interface Full access to AIDA functionality for fitting, projections, Full access to AIDA functionality for fitting, projections,
plotting etc.plotting etc.
– Can operate on Root data Can operate on Root data using GUIusing GUI using scripting (Python, Pnuts) using scripting (Python, Pnuts) using compiled Java codeusing compiled Java code
JAS3 Access to Root FilesJAS3 Access to Root Files
JAS3 + Root: ScriptingJAS3 + Root: Scripting
Root + WIRED4Root + WIRED4 Build yourself a super cool event display for your custom Build yourself a super cool event display for your custom
Root file in under 1 hour!Root file in under 1 hour!
Future PlansFuture Plans Continue to track new Root releasesContinue to track new Root releases
– Root network protocolRoot network protocol More work on authentificationMore work on authentification
– Root file writingRoot file writing Most of work already done by Petr Vokac/Julius Most of work already done by Petr Vokac/Julius
HrivnacHrivnac Easier than reading because only need to support Easier than reading because only need to support
one root protocol versionone root protocol version Support Root Support Root XML formatXML format
– Will be much slower, but perhaps more robustWill be much slower, but perhaps more robust Use Use LCGDICT systemLCGDICT system to allow Java access to allow Java access
to C++ methods as well as data?to C++ methods as well as data?– Probably also slower, but more flexibleProbably also slower, but more flexible
