Post on 19-Jan-2016
description
transcript
Lesson 1
Lesson 2:
Association
Lesson 1:
dfos - vision and architecture
Directory structure
Installation
Tips and tricks
Lesson 3:
Operations
www.eso.org/~qc/dfos/ dfos: tutorial
Lesson 1
Complexity ...
... comes by itself:
Simplification ...
... needs efforts
The answer: dfos = Data Flow Operations System
www.eso.org/~qc/dfos/
Vision
Lesson 1
To obtain …•common understanding of tools and workflows •contributions from everyone possible•easier multi-tasking•cross-instruments training possible
•Principle:•all workflow steps in common dfos tools•all instr-specific info in config files
dfos regime
user regime
config files
dfos tools
Vision
Lesson 1
ArchitectureTools for two workflows:
– daily workflow
– packing workflow
Level 1 tools: called by user
Level 2 tools: called by level 1 tools
Interaction:
– command line
– graphical user interfaces: DFO monitor; SM monitor (self-managing HTML pages)
Lesson 1
Architecture
Inter-tool “communication”: status flags
– DFO_status; SM_status etc. (read by monitors)
– possible to be read by wrapper tools
$DFO_MON_DIR/DFO_STATUS:
Lesson 1
Architecture
central element of dfos:
the Association Block AB (--> Lesson 2)
Lesson 1
Architecture
daily workflow sketch:
Lesson 1
Directory structure
• dfos needs well-defined directory structure
• environment variables
• many have DATE substructure, with DATE as 2005-01-14:
– e.g. $DFO_RAW_DIR
• others are flat
– e.g. $DFO_MON_DIR
• one is more complex:
– $DFO_SRV_DIR/<run_id>/<DATE>/raw etc.
– $DFO_SRV_DIR/<run_id>/<obs_id>/sci_raw etc.
Lesson 1
InstallationGet a complete installation of dfos (daily workflow) in four steps:
1. Getting started
2. Initial tool set
3. Creating ABs
4. Finish installation
this is where you can get without ABs
Lesson 1
Step 1: Getting startedObtain a dfos installation:
• prepare the initialization file .dfosrc
• prepare the dfos directory structure
• download dfos tools
• prepare the configuration files
1. Prepare your account and install the initialization file .dfosrc
download .dfosrc into $HOME
edit as appropriate
create directory structure as defined
make sure .dfosrc is sourced properly
4x : ok, continue
Lesson 1
Step 1: Getting started
2. Install dfosExplorer (by hand):
download and untar
execute
2x : ok, continue
3. tune your browser to become more interactive (tune your browser)
4. Read the documentation (‘help’ and ‘dfos’)
Lesson 1
dfosExplorer
use action button to download edit config file
Lesson 1
Step 2: Initial tool set5. Install dfosInstall (using dfosExplorer)
6. Install:
– dfosLog
– dvdMonitor and dfosCron
• dfosInstall: new installation/upgrade
– new installation: installs everything, incl. template config file (old one saved!)
– upgrade: config file is installed in $DFO_CONFIG_DIR/template (no overwrite!)
• config files: always come as template; some need no editing at all, others need heavy editing. Always read documentation, all configurable parameters are documented! Remember: config files are your contribution and your responsibility.
Lesson 1
Step 2: Initial tool set7. Install: dataclient, fitsreport, createReport, checkDownloads
– these tools define the pre-processing part of dfos
– they don’t need ABs
– dataclient: the tool for header/fits file downloads
implicit downloads (specify type, date, mode):
• dataclient -t h -d 2005-01-31: database headers
• dataclient -t r -m all -d 2005-01-31: all raw fits files
explicit downloads (specify ARCFILE names):
• dataclient -t r -m l -l my_list.txt -o /data03/data/my_outpath/
– fitsreport: the workhorse for fits key reading
• used by various different tools
• flexible (formatted/unformatted), performant
– createReport: general reporting tool (dfo reports, user reports)
Lesson 1
Step 3: Creating ABsWe are here:
Next: AB creation, processing workflow
Today: no details about association, just assume you can do it ...
ABs: text file with all information about a set of raw frames necessary to:
•process and/or
•pack
--> Even without a pipeline, you want to work with ABs.
Lesson 1
Step 3: Creating ABs8. Install: createAB (comes with: helpAB, listMcalib)
– it uses fitsreport to create lists: raw_list, vcalib_list, mcalib_list (all under $DFO_MON_DIR)
9., 10. create the config files (Lesson 2)
– prepare:
• know your raw data types (DPR keys) and product types (PRO.CATG)
• know the match keys (‘setups’)
• know your pipeline (recipe names)
• know your DRS
• sketch your calibration map (piece of paper is fine)
Lesson 1
Step 4: Finish installation
11. Install: createJob, processAB; getStatusAB
– supported DRS: esorex, RBS, REI
12. Install: measureQuality, certifyProducts
– prepare your QC procedures (MIDAS prog is attached as example)
– QC procedures are as instr-specific as pipelines, no attempt made to include them into dfos
– measureQ is a wrapper to call these procedures
– certifyP is also a wrapper around a user-provided procedure to call the QC1 plots (example provided)
– if you have no procedures yet: skip that part
Lesson 1
Step 4: Finish installation13. Install: moveProducts, renameProducts, updateAB
– moveProducts calls renameProducts, updateAB, listMcalib
– listMcalib updates the mcalib list, using fitsreport
– cdbIngest is not yet called by moveProducts, TBD
14. Install: updateDP, finishNight
– updateDP reads ABs for packing
– supports: DATE and OB scheme
– has a graphical monitor
– also needed if pipeline support is lacking
– finishNight: cleans up (almost) everything
Lesson 1
Step 4: Finish installation
updateDP:
Lesson 1
Tips and tricksTune your browser
– follow the description to make your browser an interactive GUI:
• launch shell scripts by clicking on a button
• open an editor to edit your config files with the browser
• used by the dfos monitor tools: dvdMonitor, dfosExplorer, dfoMonitor, updateDP, smMonitor, relMonitor
– after tuning, mozilla will execute any script with extension .ash or .esh just by clicking
Lesson 1
Tips and tricksCronjobs
– delegate some work to cronjobs: dvdMonitor, header download, createReport, dfosExplorer
– use dfosCron to call these tools
– test the cronjob and the way it sources .dfosrc!
FAQ
– read that web page before asking FAQs ...