NEON School
NEON Archive Observing School
Alberto Micol
ESA
Space Telescope Operations Division15 July 04
ESO & HST ArchivesESO & HST Archives
NEON School
NEON School 15 July 2004 NEON School 15 July 2004 LayoutLayout
What’s the role of the Data Provider
Three levels of Archive complexity– Raw Data (ESO Archive)– Reduced Products (The importance of Service
Observing)– Enhanced Products (HST/WFPC2 Associations)
Enhanced Interfaces (Querator & VO) Publications and Observing Programs
NEON School
Operations Technical Support
The ESO Data Flow Model and its Supporting Organizational Structure
Data Archive & Supporting Databasesstorage and re-distributionPI packages distribution
Phase Iproposal submissionprogram selectiontime allocation
Phase IIproposal submissionobserving blocksmedium term schedule
Observationshort term schedulearchival of data
Quality ControlMaster Cal files prod.Instrument trendingPI products preparation
User S
uppo
rt Gro
up
Par
anal
/La
Sil
la O
bser
vato
ries
Data Flow Operations Group
Data Flow, what you were toldData Flow, what you were told
NEON School
Data Flow, the joyless TruthData Flow, the joyless Truth
NEON School
Space-based observatories:•The archive is a core activity from the start.
Ground-based observatories:•Priority given to Observers, not to CommunityWith the difference that Observers can go observer themselves.
ESO has changed such paradigm, by introducing Service Mode.But it takes time for the community to accept it,
and effort within the Observatory to implement it
The Archive: last in the chain?The Archive: last in the chain?
NEON School
http://www.eso.org/org/dmd/usg/philosched.htmlService ModeService Mode
NEON School
Real Archive vs Virtual ObservatoryReal Archive vs Virtual Observatory
Real ArchivesThree types of Archives
– Level 0: Data Repository– Level 1: Ability to offer reduced products– Level 2: Ability to offer enhanced products
1 Instrumental signature-free data
2 Enhanced Products (e.g., mosaics)
0 Data Repository
“Real” Archives
BoringHandy
NEON School
Level 0: Data RepositoryLevel 0: Data Repository
Data Repository:1. Acquisition2. Data Management3. Metadata4. User Interface
NEON School
Level 0: Data Repository (1)Level 0: Data Repository (1)
Acquisition:· Observing run · The Control S/W acquires the Data· Stored in the on-line data archive· Transmitted to Science Archive
NEON School
Level 0: Data Repository (2a)Level 0: Data Repository (2a)
Data management:Integrity checks
· Did I get all I should have gotten?· Check for corrupted data· Take action if corrupted
NEON School
Level 0: Data Repository (2b)Level 0: Data Repository (2b)
Data management:Storing on medium of choice Optical disks, CDs, DVDs, Magnetic Disks
Bookkeeping Which file was stored where?When? In which format?Which compression?Store all this info in a Database (RDBMS)
NEON School
Level 0: Data RepositoryLevel 0: Data Repository
Data Repository:1. Acquisition2. Data Management3. Metadata4. User Interface
NEON School
Level 0: Data Repository ( 3 )Level 0: Data Repository ( 3 )
Metadata:· Where was the telescope pointing? (RA, Dec)· Which telescope/instrument was used? (telescope, instrument)· When did the observation occur? (start time, end time, exptime)· In which band? (filter or dispersive element used)· By whom? (Principal Investigator [PI])· For which reason? (Observing Program Identifier)· Plus many other details about the observing mode, etc.
· All this information is usually extracted from the Header of the FITS files containing the Data.· Metadata are typically stored in a Database
NEON School
Level 0: Data RepositoryLevel 0: Data Repository
Data Repository:1. Acquisition2. Data Management3. Metadata4. User Interface
NEON School
Level 0: Data Repository ( 4a )Level 0: Data Repository ( 4a )
User Interface
Finally, an astronomer wants to seewhether some information about photons regarding a certain celestial object are already stored in our archive.
Metadata (Database) -> Internet (Web)
NEON School
Level 0: Data Repository (4 Archive)Level 0: Data Repository (4 Archive)
User Interface
NEON School
Level 0: Data Repository (4 Form)Level 0: Data Repository (4 Form)
User Interface
Finally, an astronomer wants to seewhether some information about photons regarding a certain celestial object are already stored in our archive.
Metadata (Database) -> Internet (Web)
NEON School
Level 0: Data Repository ( 4b )Level 0: Data Repository ( 4b )
User Interface
•Name Resolvers: SIMBAD (CDS), NED (IPAC)•Search by Box/Cone (RA, Dec, size/radius)
•Search by Night (Observing run, calibration obs.)•Observing Blocks, Data Product Types: very ESO specific •Exptime, Filter, Grism, Grating, Slit: very instrument specific
•Release Date: Are the data public?Proprietary period implements Observatory policy
NEON School
Level 0: Data Repository ( 4 Query )Level 0: Data Repository ( 4 Query )
User Interface
Finally, an astronomer wants to seewhether some information about photons regarding a certain celestial object are already stored in our archive.
Metadata (Database) -> Internet (Web)
NEON School
Level 0: Data Repository ( 4 Results )Level 0: Data Repository ( 4 Results )
User Interface
Finally, an astronomer wants to seewhether some information about photons regarding a certain celestial object are already stored in our archive.
Metadata (Database) -> Internet (Web)
NEON School
Level 0: Data Repository ( 4 Submission )Level 0: Data Repository ( 4 Submission )
User Interface
Finally, an astronomer wants to seewhether some information about photons regarding a certain celestial object are already stored in our archive.
Metadata (Database) -> Internet (Web)
NEON School
Level 0: Data Repository ( 5 )Level 0: Data Repository ( 5 )
Request HandlerA system which takes good care of the submitted archive
requests. The RH consists of a set of procedures, interfaces, C
programs and libraries, and relational tables; It allows to: • submit archive requests, • keep track of their handling, • inform (via email or via a web interface) the operators of
any intervening problem,• notify the archive users about the status of their requests • up to successful closure.
NEON School
Level 0: Users’ point of viewLevel 0: Users’ point of view
A Level 0 archive only offers RAW Data.Removing the instrumental signature from the
RAW files is left to the astronomer:
• Identify suitable BIAS, DARK and FLAT frames• Produce Master frames (e.g., mean averaging, etc.)
• Run reduction software (e.g., bias&dark, flat fielding)
Couldn’t the Observatory provide that service?
NEON School
Level 1: Ability to offer reduced products Level 1: Ability to offer reduced products (1)(1)
Data Repository & Processing:1. Actual Storage2. Data Management3. Metadata4. User Interface5. Calibration DB + Rules6. Reduction Pipeline
NEON School
Level 1: Ability to offer reduced products (2) Level 1: Ability to offer reduced products (2)
Calibration DB + Rules· Instrument Scientists (ESO: Quality Control group [QC])
build the “Master Calibration Files” (or “Reference Files” in HST terminology)
· Reference files are stored in the Archive
· They associate to each observation a set of reference files via a set of rules.
· Rules are applied and results stored in the Calibration Database
NEON School
Level 1: Ability to offer reduced products Level 1: Ability to offer reduced products (3)(3)
Calibration DB + Rules
Simple-minded example· Rules and recipe to build a master DARK
file:
· Retrieve 5 DARK images · the ones closer to the date of the observation, · With the same read-out mode and exposure time; · De-bias each DARK image; · Average them while removing cosmic rays;· Normalise to “dark current per second”; · Flag dubious pixels (Data Quality flag)
NEON School
Level 1: Ability to offer reduced products Level 1: Ability to offer reduced products (4)(4)Reduction Software
· Instrument Scientists and Software Group must implement the Reduction Software:
a collection of processes necessary to remove the instrumental signature from a given observation.
If such software can run unattended, we call it Reduction Pipeline.
NEON School
Level 1: Ability to offer reduced products Level 1: Ability to offer reduced products (5)(5)
Reduction Products· Once the Calibration DB and the
Reduction Pipeline are in place,
the Science Archive Request Handler can make use of it
to offer Reduced Products to the relieved community.
NEON School
Level 1: HST reduced products (form)Level 1: HST reduced products (form)
HST
NEON School
Level 1: HST reduced products (results)Level 1: HST reduced products (results)
HST
NEON School
PreviewPreview
PREVIEW
•A highly compressed view of an observation, to provide a quick-look product for quality assessment.
It is generated via a specialised pipeline takes in input the reduced (or otherwise raw) observation and processes it (resizing, cropping, reducing dynamical range, applying wavelets, etc.).
Look at it, do not use it for your science!
NEON School
Level 1: HST reduced products (preview)Level 1: HST reduced products (preview)
HST
NEON School
Level 1: HST reduced products Level 1: HST reduced products (submission)(submission)
HST
NEON School
Level 2: Enhanced Products & InterfacesLevel 2: Enhanced Products & Interfaces
How to improve the archive beyond the on-the-fly recalibration?
1. Deeper and cleaner products2. Better User Interfaces
NEON School
Level 2: Deeper and Cleaner: AssociationsLevel 2: Deeper and Cleaner: Associations
HST/WFPC2 (Wide Field Planetary Camera 2)
An observation in space is affected by Cosmic RaysNo detector is perfect: bad columns, hot pixels, etc.
Special observing strategy to cope with these facts:
CR-SPLIT: Splitting one single long exposure into many short ones;
POS-TARG: Dithering is the technique of displacing the telescope between observations either on integral pixel scales (to assist in removing chip blemishes such as hot pixels) or on sub-pixel scales (to improve sampling and thus produce a higher-quality final image).
NEON School
Level 2: Deeper and Cleaner: AssociationsLevel 2: Deeper and Cleaner: Associations
HST/WFPC2 (Wide Field Planetary Camera 2)
CR-SPLIT and POS-TARG have to do with the Data Flow System of the HST observatory, not with the scientific exploitation of the data.
The associations, i.e., grouping together those observations which cover the same sky region, provide a way to isolate the archive userfrom those technical details.
NEON School
Level 2: WFPC2 Associations (form)Level 2: WFPC2 Associations (form)
NEON School
Level 2: WFPC2 Associations (asn)Level 2: WFPC2 Associations (asn)
NEON School
Level 2: WFPC2 Associations (members)Level 2: WFPC2 Associations (members)
NEON School
Level 2: WFPC2 Associations (comparison)Level 2: WFPC2 Associations (comparison)
NEON School
Level 2: Advanced Interfaces: QueratorLevel 2: Advanced Interfaces: Querator
Astronomers requires Scientific interfacesThey don’t want to know Observatory detailsHence, not only instrumental signature but also Data Flow signature must be removed
Querator goes in this direction.Querator in two words:
A multi-archive search engine specialised on multi-colour imaging data.
NEON School
Querator (form)Querator (form)
NEON School
Querator (results)Querator (results)
NEON School
Querator: (Filters)Querator: (Filters)
Filters
NEON School
Last but not list: PublicationsLast but not list: Publications
NEON School
Last but not list: PublicationsLast but not list: Publications
NEON School
Last but not list: PublicationsLast but not list: Publications
NEON School
Last but not list: PublicationsLast but not list: Publications
NEON School
Astrophysical Virtual ObservatoryAstrophysical Virtual Observatory
NEON School
RecapRecapYou have seen the exciting day to day life ofYou have seen the exciting day to day life ofa Data Provider.a Data Provider.
URLs:URLs:
Service mode:Service mode:http://www.eso.org/org/dmd/usg/philosched.html
Archive Home Page:Archive Home Page:http://archive.eso.org/http://archive.eso.org/
Querator:Querator:http://archive.eso.org/querator/http://archive.eso.org/querator/