Metadata and COMA

Elizabeth Gallas

Oxford.

ATLAS Weekly / Open EB MeetingOctober 18, 2011

Outline Overview of ATLAS Metadata Interfaces What is COMA ? Principles Tour

VERY quick tour today…hope you will try yourself! Summary and Plans Conclusions

Backup Slides: Links

Thanks to many experts, developers and also many of you who post questions / use cases to e-groups !!!

June 2011 Elizabeth Gallas - COMA 2

Subsystem specific metadata applications... driven by use cases/workflow Trigger: TriggerTool … Web: trigconf and trigger timelines Geometry DB: Detector Description Browser Conditions DB:

runQuery (Run information from Conditions DB) ATLAS WEB DQ COOL Tag Browser

Data Summary Reports (Luminosity, Beam) Beam Spot Summary GANGA and PAthena Panda / monitor DQ2 Client Tag Collector – software releases ...

Dedicated Metadata Catalogs TAGs – event level (file and relational formats)

iELSSI and Suite of TAG Services TAG Catalog AMI – Dataset, processing … other metadata

And the AMI Suite of services COMA – Conditions and configuration

ATLAS Metadata Application Overview

June 2011 Elizabeth Gallas - COMA 3

Connected in many ways (and with links to many above applications) …Some mentioned today in the course of “COMA” topic discussed today

Medicine: deep sleep … a state of extreme unresponsiveness

? coma ? Many definitions:

June 2011 Elizabeth Gallas - COMA 4

Optics: aberration inherent to certain optical designs or due to imperfection in the lens or other components…

Astronomy: luminous cloud surrounding the frozen solid nucleus in the head of a comet …

ATLAS: COMA is an ~ acronym for a metadata catalog

and its related services:

Conditions / Configuration Metadata for ATLAS

June 2011 Elizabeth Gallas - COMA 5

What is COMA ? COMA: “Conditions/Configuration Metadata for ATLAS”

Originally: built to support dynamic queries of TAG DB Evolved: into a standalone system with interfaces Now part of general effort to consolidate/relate ATLAS Metadata

COMA Components: Relational Database

Contains extracted, refined, reduced, and derived information from system specific data sources (e.g. Conditions DB, …) plus information from non-database sources.

Interfaces A set of Browsers and Reports

Main Documentationhttps://twiki.cern.ch/twiki/bin/viewauth/Atlas/ConditionsMetadata

COMA Portal (grid certificate in browser is required)https://atlas-tagservices.cern.ch/tagservices/RunBrowser/index.html

June 2011 Elizabeth Gallas - COMA 6

COMA Interfaces: General Principles Report or Browse the COMA catalogue:

Originally developed to check COMA loading and do cross checks Evolved to current form: on request from experts and users

Logical division of information into separate but cross linked reports Include links to more details (COMA links and External links)

runQuery, Luminosity, AMI, Trigger, DQ …

Provide reports [distinctive from, complimentary to] other systems (Avoid duplication of effort) Display COMA ‘derived’ information

Not available in other systems

Note: All the COMA Reports use GET methods: Inputs are not complex, user can modify URL to get report they want,

put links into TWiki or other documentation

https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?runs=161562

June 2011 Elizabeth Gallas - COMA 7

COMA Interfaces Portalhttps://atlas-tagservices.cern.ch/tagservices/RunBrowser/index.html

Tour

stops

today

DB mtgWeds

June 2011 Elizabeth Gallas - COMA 8

Introduction: ATLAS Data Periods A Data Period is a set of ATLAS Runs grouped for a purpose

Defined by Data Preparation Coordinators Used in ATLAS data processing, assessment, and selection … Each Period uniquely defined with a combination of

Project name (i.e. ‘data10_7TeV’) Period name (i.e. ‘C1’, ‘C2’, ‘C’, ‘AllYear’ …)

Before 2011, Data Periods were Described on TWiki page

https://twiki.cern.ch/twiki/bin/view/AtlasProtected/DataPeriods Stored in a file based system

Edited by hand by Data Prep Coordination (experts) Structure evolved over last year with experience

This experience valuable to decide/define long term solution Early in 2011: Data Periods stored in the COMA DB

Thanks: Beate (DataPrep Coordinator), AMI team, me.

June 2011 Elizabeth Gallas - COMA 9https://atlas-tagservices.cern.ch/RBR/rBR_Period_Report.php

Menu Purpose: Show DataPrep defined Periods giving links to reports of general info about their Runs.

Choose the Period of interest: By Year

E.G. all ‘2010’ By Project

e.g. ‘data10_7TeV’ By specific Period or Group

Click on the project and then Period of interest

Wildcards can be entered in this optional section, then click on Submit button

“highlight”expanding

sections

June 2011 Elizabeth Gallas - COMA 10

Period Description Report Example for data10_7TeV

-/+ highlighted links:

These sections expand

to show period members

Members of

data11_7TeV.M

are M1, M2, M3, M4

Links: to COMA & runQuery

multi-Run Reports

for chosen Period

(COMA link: next slide)

Header:

Input criteria;

Links: Contact,

Documentation

Period “Status”: M1 is ‘frozen’,

M2-M4 are ‘locked’ … M ‘unlocked’ (M5 is on-going)

June 2011 Elizabeth Gallas - COMA 11

COMA Multi-Run Report:Purpose: show basic information for all runs matching the input criteria

Run links: COMA, runQuery, and

LumiDataSummary Single Run report

(COMA link: next slide)

SMK links:

COMA Trigger Super Master Key Report

Links in Table column headers:

Short description of column

Hover over any link:

Indicates what

will happen

https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?fnt=data10_7TeV&pn=VdM

June 2011 Elizabeth Gallas - COMA 12

COMA Single Run 180776 Overviewhttps://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?runs=180776

General Run info: Project, LB range, Date … Events, HLT Release … Links to COMA reports

COMA Master Key report (no time to show today)

Links to external reports HLT release, AMI, LumiDS,

runQuery, Trigger, PSEvol

Yellow sections expand: Data Periods Active Streams AMI tags (TAG production)

Shortcut find runs w/AMI tag Prescale Evolution

PS Key changes Trigger section

All HLT signaturesAll LVL1 items

June 2011 Elizabeth Gallas - COMA 13

COMA Single Run -/+AMITag SectionRecent data11_7TeV M4 Period Run 190933

Shows: All physics streams

Including debug ! “First Pass” – first processing

New columns appear for reprocessing passes

AMI tag used to process data Link: COMA multi-Run Report

All runs processed w/this tag “?”: link to AMI

Description of AMI tag

Launch iELSSI ! For this Run/Stream/AMItag Take a quick look at this run with

the ELSSI TAG browser !!! See if events pass your trigger in

the debug stream …

June 2011 Elizabeth Gallas - COMA 14

COMA Single Run -/+Trigger SectionStart to see here some COMA “derived” info

Open this section in next slide…

COMA divides HLT trigger signatures: “Physics”: complete EF-L2-L1 chains “Commissioning”: has missing level(s)COMA looks at each LB range:1. Decides if each chain is “active” based

on prescales, passthrough, rerun in every LB range of Prescale evolution

2. For “active” chains via prescale, calculates an “Aggregate prescale” (EF*L2*L1 prescale) in each LB range

3. Determines Min, Max of AggPS over the Run for each chain

4. Determines Run and LB-wise PS,PT,RR Flags

June 2011 Elizabeth Gallas - COMA 15

COMA single-Run Report: “Physics” Trigger SectionChoose (using jQueryColumnFilters)

Chains: EF*JPSI and

Stream: egamma

COMA Chain Report:

All Runs w/ this chain/item

COMA Prescale Report:

LB-wise PS info for

this Run/Signature

(next slide)

Run-wise Aggregate prescale Range

and Flags(PT,PS,RR)

June 2011 Elizabeth Gallas - COMA 16

COMA Prescale Report (Run 180776,EF_e9_tight_e5_tight_Jpsi)

LB Range Summary table: LB-wise Prescale Evolution Link:Trigger DB Prescale

Reports (useful cross check !) Prescales:

L1,L2,EF and Aggregate PS Chain Inactive from

LB 1-99 and 114-134(LB ranges in COMA always inclusive)

Next slide:COMA Chain Report forEF_e9_tight_e5_tight_Jpsi

Run-wise Summary: Includes links to COMA reports:

Run, Master Key, Chain and Prescale Reports (each level)

Associated L2, L1 names Aggregate PS Range: 4 to 5 Flags:

Partially active via PS Never passthrough (L2 or EF) Never Rerun

June 2011 Elizabeth Gallas - COMA 17

RBR Chain Report (EF_e9_tight_e5_tight_Jpsi)

Activation Summary:Shows the Runs in which this element is

“active” separated by: “Physics” EF-L2-L1 signatures

Via prescale Via pass through Via rerun

“Commissioning” signatures Via prescale Via pass through Via rerun

Configuration Summary:Shows where this element is configured:

Super Master Key(s), Runs (Summary)

Period Evolution: Shows chain/item bit, version evolution

for “Physics” chains during Period Runs

Open these sections in next slides…

June 2011 Elizabeth Gallas - COMA 18

COMA Chain Report (EF_e9_tight_e5_tight_Jpsi)Expand Run-wise Activation … “Physics” EF-L2-L1 signatures

Active via Prescale Runs in Data Periods

Table Shows (Run Count): Periods, Link: Run, SMK Reports Level bit assignments Link to: Chain/Item Reports (3) Range of Aggregate Prescale while

chain is active via prescale in Run Links: COMA Prescale Report (3)

70 Period Runs where this chain is “active”

June 2011 Elizabeth Gallas - COMA 19

COMA Chain Report (EF_e9_tight_e5_tight_Jpsi)Period Evolution Section … For “Physics” chains in Period Runs Separate table for each EF-L2-L1

signature at each beam energy

Each Table Shows: Row-wise:Distinct set of bit and

chain/item versions

Columns: Bit assignments Chain version (links to Trig diff) Chain Report Links Range of AggPS, SMK, Run,

Period, Date, HLTRelease

Thanks to Tomasz, Joerg

for many useful discussions

June 2011 Elizabeth Gallas - COMA 20

COMA Chain Wildcard Report (input: “EF_g10%”)Purpose: See all the names matching a pattern or

Find exact name from part of the nameReport: Displays chain/item names matching the input string …

text size proportional to occurrence in SMK In Period Runs and in All Runs

June 2011 Elizabeth Gallas - COMA 21

Summary and Plans COMA – an integral part of ATLAS Metadata infrastructure

Essential to ATLAS event-level metadata decoding Ideally placed to provide links and interface to other metadata

Special relationship to AMI (and TAG catalog) Launch iELSSI to take a quick look at any Run

Primary source for “ATLAS Data Periods” Periods in Lum, DQ, Run Summary, AMI reports comes from COMA

Reports feature “derived” information not available elsewhere Trigger experts recommend COMA Trigger/Prescale

Report usage: from ~200 to over 5000 pages viewed/month. Peaked in July as users did final preparations for summer conferences

Current efforts: COMA Database content growing

Watch use cases to identify new areas to focus growth COMA Report and Browser development

keep pace with content, improve functionality and usability Beyond COMA: Interface development

Connecting COMA to other parts of the infrastructure

June 2011 Elizabeth Gallas - COMA 22

COMA Conclusions

This is an evolving system … information in the system is growing based on information available and use cases Adding more dimensions to the Conditions data

With suitable relationships to facilitate queries Making that criteria available in dynamic useable interfaces

We want to insure the Metadata is complete enough to satisfy use cases while reflecting accurately its limitations

Interfaces are being constructed to use selection syntax, criteria, and communication in common use in ATLAS This facilitates cross checks with other systems

Continuous process: talking with various experts to ensure data integrity, completeness, compatibility w/other systems

… Very positive feedback so far … more always welcome …

hn-atlas-physicsMetadata@cern.ch

June 2011 Elizabeth Gallas - COMA 23

Backup

Link SummaryMain TWiki Documentation:https://twiki.cern.ch/twiki/bin/viewauth/Atlas/ConditionsMetadata

COMA Report Documentation:https://atlas-tagservices.cern.ch/RBR/rBR_Documentation.html

COMA Portal (grid certificate in browser is required)https://atlas-tagservices.cern.ch/tagservices/RunBrowser/index.html

Trigger General Meeting (June 8, 2011):“User Guide to Trigger/Prescale information in COMA Reports”https://indico.cern.ch/materialDisplay.py?contribId=0&materialId=slides&confId=125723Data sources, Business rules and Caveats; Form input; Example links; details of Data Period & derived quantities;

Data Periods TWiki (revised):https://twiki.cern.ch/twiki/bin/view/AtlasProtected/DataPeriods

June 2011 Elizabeth Gallas - COMA 24

June 2011 Elizabeth Gallas - COMA 25

COMA Report(s) Example LinksCOMA single-Run Report (Run 161562):https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?runs=161562

General Run information, AMI tags, DQ, Prescale Evolution, Triggers Trigger tables show derived Run-integrated prescales, passthrough

COMA Trigger Super Master Key Report (Key 877):https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?smk=877

Runs using this SMK, physics/commissioning chains, Level 1 items

COMA Chain Report (Chain EF_g11_etcut):https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?cn=EF_g11_etcut

Summary: SMK, Runs, Signature appearances of this chain Runs with this chain ‘active’

COMA Chain Wildcard Report (chain name matching string EF_g1%):https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?cn=EF_g1%

Summary: SMK, Runs with this chain: Configured, Active

COMA Prescale Report (Run 162882 Prescales for chain EF_g11_etcut):https://atlas-tagservices.cern.ch/RBR/runBrowserReport.php?runs=162882&cn=EF_g11_etcut

Derived chain activity over the Run and LB-wise prescales

June 2011 Elizabeth Gallas - COMA 26

Data Periods: Links to Reports and Services

Revised TWiki page:https://twiki.cern.ch/twiki/bin/view/AtlasProtected/DataPeriods

Interactive USERS COMA Data Period Documentation Interface https://atlas-tagservices.cern.ch/RBR/rBR_Period_Report.php

Comments: hn-atlas-physicsMetadata@cern.ch.

Programmatic USERSFor systems needing period info: runQuery, beamspot, Data Quality, …, “Data Period Services” provided via pyAMI: http://ami.in2p3.fr/opencms/opencms/AMI/www/Client/DataPeriods_pyAMI.pdf

Comments: AMI / Tag_Collector Team.

Data Preparation EXPERTS: Entry Interface: https://ami.in2p3.fr/AMI/servlet/net.hep.atlas.Database.Bookkeeping.AMI.Servlet.Command?linkId=1479

Comments: AMI / Tag_Collector Team.

June 2011 Elizabeth Gallas - COMA 27

COMA runBrowserReport Index page INDEX: https://atlas-tagservices.cern.ch/RBR/runBrowserReport.html GUIDE: https://atlas-tagservices.cern.ch/RBR/rBR_Documentation.html

“Index” page: Enter one/more input quantity,

IE: a Run number IE: a Chain Name

Click on submit (or return key)

Links at bottom of every page:

Reports use “GET” methods:Modify the URL to change input

Reports contain linksNo need to use index page for

every queryLinks lead you to

other COMA reports other external reports

AMI, runQuery, TriggerDB…