- ADL Scripting (06mar01 - ARC Final CERN) Athena Data Dictionary and Athena Scripting Craig E....

[email protected] - ADL & Scripting (06mar01 - ARC Final Meeting @ CERN)

Athena Data Dictionaryand

Athena Scripting

Craig E. TullHCG/NERSC/LBNL

Architecture Review Committee CERN - March 6, 2001


Outline

• Data Dictionary—Who: C.Tull, LAPP - A.Bazan, P.Ghez, T.Le Flour,

S.Lieunard—What, Where, How, Why—January 2001 MileStones

• White Paper, ADL Language, ADL Tools, April Prototype Functionality (CBEs)

• Scripting—Who: C.Tull, C.Day, D.Quarrie—Python

• Current Status• Simplicity vs Performance

• Conclusion


Short-Mid Term Milestones

• January 2001— Evaluation of ADL Candidates/Tools Complete— Athena Dictionary Language (ADL) Chosen — ADL Compiler Front End (CFE) Chosen — Subset of ADL CBEs Defined

• April 2001 — Prototype DD Implementation — Standalone ADL CFE Functional — One ADL CBE Implemented & Used in Athena — Athena Dictionary Language (ADL) Frozen

• September 2001 — Data Dictionary Deployed — ADL CFE Integrated with ATLAS Release Tools — ADL Reflection API Available in Athena — Mult. CBEs Implemented & Available & Integrated in Athena


January 2001 Milestones

• Evaluation of ADL Candidates/Tools Complete— Complete: See White Paper

• Athena Dictionary Language (ADL) Chosen— Proposal: Extended Proper Subset of IDL 2.0

• Keywords added for persistency, bidirectional relationships, & opaque objects

• ADL Compiler Front End (CFE) Chosen — Proposal: JavaCC

• Subset of ADL CBEs Defined— Proposal: 2/3 Possibilities

• Converter Generator• DataObject Scripting• Code Template Gen.


Evaluation of ADL Candidates/Tools

• Complete: See White Paper


White Paper

• Athena Data Dictionary White Paper - Vers. 1.0.1—Craig E. Tull, Alain Bazan, Philippe Ghez,

Thierry Le Flour, Sophie Lieunard• ATLAS Computing Architecture Data

Dictionary Review Official Documents—http://atlas.web.cern.ch/Atlas/GROUPS/

SOFTWARE/OO/architecture/DataDictionary/Review/Official_Documents/WP_20-01-2001.pdf

• Initially intended to capture technical consid-erations, arguments, decisions, implementations

• Other, non-technical issues to be addressed later• Please read and send comments, corrections to

[email protected] (backlog of comments to add)


Athena Dictionary Language (ADL) Chosen

• Proposal: Extended Proper Subset of IDL 2.0—Keywords added for persistency, bidirectional

relationships, and opaque objects—We believe the concerns expressed in the

November interim ARC Report are being addressed.• Interface vs. Implementation

—keywords: persistent, private• Interesting Object Relationships

—keywords: bidirectional, relationship• Constructors

—default patterns/object factories/unneeded• Complex Data Types

—predefined types


Example: Java vs XML vs XMI

• PROCESS• XML initially seemed

attractive, but...

• Java method declarationpublic void

writeXML(XMLWriteContext p1)

• XML (from the sublime)<ADL.method>public

void writeXML(XMLWriteContext p1)</ADL.method>

• XMI (to the ridiculous)<Foundation.Core.Operation xmi.id="_146"> <Foundation.Core.ModelElement.name> writeXml </Foundation.Core.ModelElement.name> <Foundation.Core.ModelElement.visibility xmi.value="public" /> <Foundation.Core.Feature.ownerScope xmi.value="instance" /> <Foundation.Core.BehavioralFeature.parameter> <Foundation.Core.Parameter xmi.id="_147"> <Foundation.Core.ModelElement.name> p1 </Foundation.Core.ModelElement.name> <Foundation.Core.Parameter.kind xmi.value="in" /> <Foundation.Core.Parameter.type> <Foundation.Core.Classifier xmllink="simple" href="com.sun.xml.tree.XmlWriteContext" title="XmlWriteContext" /> </Foundation.Core.Parameter.type> </Foundation.Core.Parameter> </Foundation.Core.BehavioralFeature.parameter></Foundation.Core.Operation>


Athena Dictionary Language (ADL)

• Proposal: Extended Proper Subset of IDL 2.0— attribute, boolean, char,

const, double, enum, float, in, inout, interface, long, module, octet, out, readonly, sequence, short, string, struct, unsigned, void

—ODL Keywords added for bidirectional relationships• inverse, relationship

—Keywords added for persistency• persistent, private

—Keyword added for opaque objects• external

keyword IDL2 ODL Usagemodule X X Scope (C++ name space, J ava pac kage)interface X X Class declarationclass X « abstract X Class QualifierValueBase X Class ? ?valuetype Xpublic X Attribute qualifier scope ValueBaseprivate X « attribute X X Data memberreadonly X X attribute qualifierin X X Op param qualifier out X X « inout X X « oneway X X Op qualifiervoid X Xraise X X For opcontext X Xboolean X X Predefined typeschar X Xdouble X Xfloat X Xlong X Xoctet X Xshort X Xstring X Xunsigned X Xconst X Xany X X Predefined structured typesenum X Xsequence X Xstruct X Xunion X Xswitch X X For unioncase X Xdefault X Xabstract Xcustom X For valuetype


Introspection w/ ATLAS data

atlsim > ls geomDUI: Listing = ... Name * Type * Used * Alloc'd * Size D mfld_sys * * 3 * -1 * -1D atls_sys * * 1 * -1 * -1...D amdb_sys * * 1 * -1 * -1D much_sys * * 9 * -1 * -1atlsim > table/show geom/pixb_sys/pixb_pbcaTDMTABLE: Table = ...struct pixb_pbca { float system[2]; long layer, ncables; float dxcable, dycable, dzcable, xcable, zcable;} atlsim > table/print geom/pixb_sys/pixb_pbca ROW # system[0] system[1] layer ncables dxcable dycable

dzcablexcable zcable 0: 9.8113e+05 0 3 6 -0.083965 1

5.9 0.4 3.1


ADL/IDL vs StAF/IDL

• Extended Proper Subset of IDL 2.0—attribute, boolean, char, const, double, enum,

external, float, in, inout, interface, inverse, long, module, octet, out, persistent, private, readonly, relationship, sequence, short, string, struct, unsigned, void

• StAF Tables use a small subset of the proposed IDL keywords -- Already available in ATLSIM.


LAr Objects in IDL w/ extentions

LAr Classes Already Implemented in ADL


ADL Compiler Front End (CFE) Chosen

• Proposal: JavaCC


ADL Compiler Front End (CFE)

• Proposal: JavaCC - Java Compiler Compiler—http://www.metamata.com/JavaCC/

• Large number of Language Grammars—34 Grammars (from Ada to XML)

• http://www.cobase.cs.ucla.edu/pub/javacc/—Includes most ADL Candidates considered

• C++, Java, IDL, ODL, XML• Easily adapted to ATLAS needs

—ATLAS keywords already added—CFE/CBE design, Standard AST, grammar, etc.

• Widely/Actively used, supported, and developed• Some non-ADL Grammars may be(come) useful


Subset of ADL CBEs Defined

• Proposal: 2/3 Possibilities for April—1 Converter Generator 3—2 DataObject Scripting 2—3 Code Template Gen. 1—Priority/Importance/Urgency is inversely

correlated with ease of implementation*• Other possible, future CBEs include:

—Data Object Browser, Inter-Language Data Interchange, Reverse Engineering, XML Emitter, StoreGate, PLO, CLIDs etc.

—N.B. some may use introspection, not CBE• This will prob'ly be an ongoing, growing list. If done

properly, the introduction of new CBEs can be done in an adiabatic manner.


Code Generation

• Code generation tools can be used to —eliminate tedious and error-prone rote programming,

ease migration between technologies, generate templates & skeletons

• Std. 2-tier design robustness & functionality—Single Compiler Front End (CFE) AST—Multiple Compiler Back Ends (CBE) - Emitters

• Schema evolution should not require significant re-writes of user-written code using generated code.


Subset of ADL CBEs

• Proposal: 3 Possibilities—Converter Generator

• Automatically Generate Converters for DataObjects—DataObject Scripting

• Access and manipulate DataObjects in TES—Code Template Generation

• May prove more fruitful than "ATLAS C++ Coding Standard" + Template/Examples

• http//atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/qc/C++_Coding_Standard.pdf

• Focus on Helping write the code to standard, rather than Enforcing standard on written code.

—April Prototype will likely not support full ADL keyword list. Will provide at least 2 of 3 CBEs.


N.B. - Code Generation Mock Ex.

• Descriptioninterface foo{

persistent attribute float x;};

• Not as simple as:class foo{

public: float x;};

• Determined by ATLAS coding standards, practices, & binding choices.

#include <iostream.h>

class foo {private:

float m_x;

public:

//=== Constructors definition area

foo(){this->m_x=0;};foo(const foo& afoo){this->m_x = afoo.get_x();};foo(const float x){this->m_x = x;};~foo(){};

//=== End of Constructors definition area

//=== public method : data member access definition area(Accessor/Mutator)

void set_x(const float x) {this->m_x = x;}float get_x() {return this->m_x;}

//=== public method : end of data member access definition area

//=== public method

void print_data(ostream &astream){astream << this << endl << "x = " << this->m_x << endl;};

};


Opaque & Pre-defined ADL

• Opaque classes are those not defined in ADL—external/legacy classes, fast prototyping—existence can be deduced from ADL, but not

shape, behavior, or content—code generation can generate hooks for

persistency, etc (eg. ala strategy pattern)• Pre-defined ADL are set of core-provided general-

purpose ADLs for use within user-written ADLs.—eg. Some STL types map naturally onto

available IDL constructs.—Some thought still needed on others.


Athena Scripting


Scripting Interface

• Athena scripting support is available in prototype form.The functionality is likely to change rapidly, so users should check with the latest release notes for changes or new functionality.

• In keeping with the design philosophy of Athena and the underlying GAUDI architecture, scripting is defined by an abstract scripting service interface, with the possibility of there being several different implementations.

• Implementation is available based upon the Python scripting language.

• Availability:—1.2.1 - Separate Build, 1.2.2 - Available—1.3.0 - Deployed


How to enable Python scripting

• Python scripting is enabled by adding the following lines to the job options file.

ApplicationMgr.DLLs += { "SI_Python" }; [1]ApplicationMgr.ExtSvc += { "PythonScriptingSvc/ScriptingSvc" }; [2]

• Notes:—1. This entry specifies the component library that

implements Python scripting. Care should be taken to use the "+=" syntax in order not to overwrite other component libraries that might be specified elsewhere.

—2. This entry specifies the Python scripting implementation of the abstract Scripting service. As with the previous line, care should be taken to use the "+=" syntax in order not to override other services that might be specified elsewhere.


Property Access Model

• There are Python Classes for ApplicationMgr, Algorithm and Service

• There is a “static” Python Class attribute, names, that lists the Athena Objects of that class

• Use the Class names as constructor with Athena Object name string as argument to make a Python Object that shadows the Athena Properties—theApp is predefined Object on Python side to

map the single ApplicationMgr in Athena


Property Access Model, cont.

• Each Python Object has a properties attribute that lists the names and values of the associated Athena Properties.

• Type checking and verification are done on the Athena side.


Scripting Functionality

• The deployed functionality is currently limited to the following capabilities.—1. The ability to read and store basic

Properties for framework components (Algorithms, Services, Auditors) and the main ApplicationMgr that controls the application. Basic properties are basic type data members (int, float, etc.) or SimpleProperties of the components that are declared as Properties via the declareProperty() function.

—2. The ability to store individual elements of array properties.

—3. theApp.Go - to execute event loop


Property manipulation from Python

>>>Algorithm.names[1][2]('TopSequence', 'Sequence1', 'Sequence2')>>> Service.names

[3]('MessageSvc', 'JobOptionsSvc', 'EventDataSvc', 'EventPersistencySvc','DetectorDataSvc', 'DetectorPersistencySvc', 'HistogramDataSvc','NTupleSvc', 'IncidentSvc', 'ToolSvc', 'HistogramPersistencySvc','ParticlePropertySvc', 'ChronoStatSvc', 'RndmGenSvc', 'AuditorSvc','ScriptingSvc', 'RndmGenSvc.Engine')>>> TopSequence.properties

[4]{'ErrorCount': 0, 'OutputLevel': 0, 'BranchMembers': [],'AuditExecute': 1, 'AuditInitialize': 0, 'Members':['Sequencer/Sequence1', 'Sequencer/Sequence2'], 'StopOverride': 1,'Enable': 1, 'AuditFinalize': 0, 'ErrorMax': 1}>>> TopSequence.OutputLevel=1

[5]>>> TopSequence.Members=['Sequencer/NewSeq1', 'Sequencer/NewSeq1']

[6]


Property Manipulation (Cont)

>>> TopSequence.properties{'ErrorCount': 0, 'OutputLevel': 1, 'BranchMembers': [],'AuditExecute': 1, 'AuditInitialize': 0, 'Members':['Sequencer/NewSeq1', 'Sequencer/NewSeq1'], 'StopOverride': 1,'Enable': 1, 'AuditFinalize': 0, 'ErrorMax': 1}>>> theApp.properties

[7]{'JobOptionsType': 'FILE', 'EvtMax': 100, 'DetDbLocation':

'empty','Dlls': ['HbookCnv', 'SI_Python'], 'DetDbRootName': 'empty','JobOptionsPath': 'jobOptions.txt', 'OutStream': [],'HistogramPersistency': 'HBOOK', 'EvtSel': 'NONE', 'ExtSvc':['PythonScriptingSvc/ScriptingSvc'], 'DetStorageType': 0,

'TopAlg':['Sequencer/TopSequence']}>>>


Property Manipulation: NOTES

• 1. The ">>>" is the Python shell prompt • 2. The set of existing Algorithms is given by the Algorithm.names

command.• 3. The set of existing Services is given by the Service.names

command.• 4. The values of the properties for an Algorithm or Service may be

displayed using the <name>.properties command, where <name> is the name of the desired Algorithm or Service.

• 5. Single valued properties (e.g. IntegerProperty) may be set using an assignment statement. Boolean properties use integer values of 0 (false) and -1 (true). Strings are enclosed in "'" characters (single-quotes).

• 6. Multi-valued properties (e.g. StringArrayProperty) are set using "[...]" as the array delimiters.

• 7. The theApp object corresponds to the ApplicationMgr and may be used to access its properties.


Property Access, cont.

• Trying to be consistent with Python-style access—Can use individual ArrayProperty items

>>> myAlg.myProp[1] = 3—Can use slices of ArrayProperties

>>> myAlg.myReals[1:3] = [ 1.3, -6.3, .05 ]—Cannot del any Athena Properties

>>> del myAlg.myPropCannot delete Athena Property


Control Access

• Simple form—Python prompt triggered by JobOptions

entries• ApplicationMgr.DLLs+={"SI_Python"};• ApplicationMgr.ExtSvc+={"PythonScriptingSvc/ScriptingSvc"};

—Prompt is just before event loop— >>> theApp.Go returns to event loop—Python prompt again after loop— <ctrl-D> to exit Athena


Python Scripting Language

• Object Oriented, Clean Lang., Easy to Use, Well Documented, Many publicly available modules


OpenSource Python Modules

891 (from 885 2 wks ago)


Language Comparison

• An empirical comparison of C, C++, Java, Perl, Python, Rexx, and Tclhttp://wwwipd.ira.uka.de/~prechelt/Biblio/jccpprt_computer2000.pdf

• 80 implementations of the same set of requirements are compared for several properties, such as run time, memory consumption, source text length, comment density, program structure, reliability, and the amount of effort required for writing them. The results indicate that, for the given programming problem, which regards string manipulation and search in a dictionary, “scripting languages” (Perl, Python, Rexx, Tcl) are more productive than “conventional languages” (C, C++, Java). In terms of run time and memory consumption, they often turn out better than Java and not much worse than C or C++. In general, the differences between languages tend to be smaller than the typical differences due to different programmers within the same language.


Scripting vs Programming


Scripting: C++ vs Python

• Python

>>> myAlg.String 'yeyeyeyeye'

>>> myAlg.String = 'Now is the time for Python'

>>> myAlg.x = yourAlg.y

>>> for i in range(0,1000):>>> myAlg.id[i]

• C++

>>> cout << myAlg->get_String() << endl;

'yeyeyeyeye'

>>> myAlg->set_String("Now is the time for Python");

>>> myAlg->set_x(yourAlg->get_y());

>>> do (i=0;i<1000;i++){>>> cout << myAlg->get_id(i) << endl;>>> }

• For IProperty interface, Python is different, but simpler. • NB. C++ interface here assumes conventional user accessor/mutators. Raw IProperty is less natural.

• Same principle will hold for most/all other interfaces.• C++ invocation syntax not self contained nor constant.


Conclusions

• ADL - April Prototype—JavaCC + IDL (extended proper subset)

• Works today—CBEs - Skeleton, Scripting, Persistency

• Meeting last week w/ DataBase & StoreGate folks• Scripting—Poor-man's reflection (IProperty) used—Simple (too) theApp.Go—Do we need JobOptions.txt instead of

JobOptions.py?—Extensions to scripting interface

needed/coming.





And Then There's CINT

• "One tool for everything" or "The right tool for the job".—Partly good OOP practice/Partly a matter of taste

• Inappropriate tight coupling between Scripting, Persistency, Introspection(?), Programming Language (1 possible), Implementation, Interface—No ability to pick & choose functionality or evolve

• However...

Date post:	18-Jan-2018
Category:	Documents
Upload:	donald-francis
View:	225 times
Download:	0 times