raw data recipes SGDP
Science Grade Data ProductsWolfram Freudling
DPD / SDP Group
Science grade data products (SGDPs) are data products which can be used as-is to extract scientific conclusions, or to carry out quantitative measurements.
Instrument signature removed physical units signal-to-noise ratio close to the optimum error estimates all aspect of data taking and processing fully documented
Data Product Survey Nov 2008
AMBER: 4
CRIRES: 11
FORS1: 3
FORS2: 3
FORS12: 16
GIRAFFE: 9HAWK-I: 3ISAAC: 12
MIDI: 6
NACO: 2
SINFONI: 4
UVES: 6
VIMOS: 5
VISIR: 10
About 50% overlap with DFS tickets
Proposed Projects
Science Ready Pipelinesas reported by Survey
• CRIRES LS Spectroscopy • GIRAFFE/MEDUSA• Hawk-I• UVES• VISIR
address issues which apply to several instruments
• Multi-OB image combination • Telluric lines tool• illumination correction • distortions • background subtraction• detector “monitoring”
Austrian contribution
Austrian contribution
Looking for science-ready instrument to develop Reflex
workflow• Hawk-I:– Major problem with background subtraction
=> fixed– Distortions:
• not enough calibration data• needed recipe to analyze distortions• distortions not stable
=> Austrian in-kind
Looking for science-ready instrument to develop Reflex
workflow
• UVES:– error-bars– Combination of calibrations– CR-like artefacts
=> most issues fixed draft of report by Daniel Bramich
• Extrapolation: Every instrument needs substantial work before data products can be considered science-grade
• Many easy to spot problems• Problems often involve both calibration data
and pipeline• Bottle-neck is the resources available for
validation and verification• Only 2.5 astronomers work in SDP group => need IOT involvement
Conclusions so far
New process to set priorities
http://eso.org/observing/dfo/sdp/
Why?• Tickets driven development not good way to set
priorities• Rush to submit something to pipeline priority
meeting• PSD resources are spread too thin:
– do fewer things with more resources
• Projects are too software-centric– full support of projects by IOT/SDP
• Goals of projects are ill-defined– start with good specifications, test plan
“Proposals should be articulated, comprehensive, and set some specific measurable goals to improve data quality. “
Proposals will be assessed by DData PProducts BBoard.
Instrument upgrades should use the same route.
SDP Proposals
The Board
• The Data Product Board is chaired by the head of SDP
• meets twice per year, monthly progress meetings
• Members: IOT Coordinator, head of LPO/PSO, the head of DMO/DPD, head of SDD/PSD, representative from INS/IPD
• First meeting: Feb 2011
but...
• IOT/IS does not have the time, resources, motivation ... to write a SDP proposals
• It is de-motivating if IOT proposal needs DPB approval
• We will not be able to process some data if operational critical tickets are not addressed right away
• I prefer to talk directly to software developer
We should work only on projects which we can complete.
Approved projects get lots of resources.
We need to use our resources efficiently.
This is understood and accepted.
What should be in a proposal / data products requirements
• Specific requirements for data quality• Proposal is not compilation of tickets• Separate requirements from
implementation suggestions• Prototypes for algorithms are great, but...• Detail resources
Effect on Pipeline Development
• All pipeline development frozen be default• Tickets should be submitted but will not be
attended immediately.• Exceptions have to be approved on a case-
by-case basis• “new” pipelines under development not
affected.
Pipelines not frozen• VIMOS: support for upgrade• XSHOOTER: still in development for several
modes• VISIR: basic support for the new detector
only; additional development must go through an enhancement proposal
• OMEGACAM: support for commissioning• VIRCAM: pipeline still in development for
several modes• PRIMA: still in development
Questions?
Comments?