WEAVE Operational Rehearsals - Onderzoekweaveahm2017/talks/WAHM... · WEAVE Operational Rehearsals...

WEAVE Operational Rehearsals

David MurphyCambridge Astronomical Survey Unit*

(CASU)

*also implicated:Clare Worley, Jim Lewis, Nic Walton

Mike Irwin, Luis Peralta de ArribaWEAVE AHM: 1st – 3rd November 2017,

Groningen

WEAVE Operational Rehearsals

David MurphyCambridge Astronomical Survey Unit

(CASU)See also:

Nic Walton – 14:45 today (Springezaal)SPA breakout session – Thursday AM (Room 10a)

Sky subtraction breakout session: Thursday PM (Room 3)Me – Coffee breaks!

WEAVE AHM: 1st – 3rd November 2017, Groningen


Quick talk outline

l Why are we doing Operational Rehearsals (OpRs)?l What did we do in OpR1?l What did we do in OpR2?l What did we improve in OpR2.5l What we’ve done since OpR2.5 (”OpR2.5++”)l Plans for OpR3

* “We” = The WEAVE SPA and others involved in the operational dataflow.


OpR0

l What is the point? Why are we doing this?l Implementation of dataflow plans à tangible, testable system

infrastructure


2. Node interaction matrix

1. Node dataflow diagram

3. Detailed operational schema

OpR1 – how to begin?

Start


OpR1 outputs and products• Two nights of simulated data, comprising:

• 38 FITS files per night• 1.5GB per night

• After several test-runs, the data were stored in the correct places, and flags correctly written

• Real-time monitoring of progress via web page:

How are OpR1 and OpR2 different?

• OpR1 main objectives:• Testing of core-DMS inter-node

communications• Performance & quality of data reduction /

analysis• OpR2 focus more on reduction quality

aspect part• OpR2 – include this part of the data flow

diagram: survey team-driven input OpR1

OpR2


OpR1 à OpR2: step up in complexity



OpR1 à OpR2: step up in complexity


OpR2Three principal phases:

• [SWG] Catalogue compilation and submission: up to 3rd May 2017• [SIM/CPS/APS] Observation + proc. of targets: 31st May – 7th June 2017• [WAS/QAG] Archiving, analysis of raw & pipeline products: July 2017

Why do it?• More inclusive dataflow – incorporates catalogue submissions• Realistic actions on simulated data: pipeline processing, QL reductions…• Survey team engagement – start thinking about WEAVE!


OpR2 SummaryMigrated performance-critical nodes from VM à metal

Version-controlled repository of WEAVE catalogue data (WASP)

• Quick turnarounds for verification of submitted catalogue files

• Creation of CNAME’d catalogue tarballs

• Regular automated email status reports on submission progress to survey leads and SWG coordinators

Simulation of 790 frames, incl. fields & calibration files (all Gavin!)

• 61 OBs observed

• 117 Low-Res fields, 81 High-Res fields (Blue arm: 75, Green arm: 6)

Realistic real-time observing and processing of 8 nights of simulated data:

• 714 raw FITS files, 31GB

• 4664 L1 fits files (inc. intermediates; 104 stacks), 777GB

• 39 L2 fits files, 7GB

Searchable WAS archive of all important WEAVE data for access by QAG

QA analysis of data, and performance of OpR2 nodes exposed some niggles in:

• The source data• Input catalogues• L1 pipeline• Communications platform• Archive functionality


OpR2 data analysis

• Basic noise model was a problem

• CNAMEs mangled

• Wobbly spectra

• Overexposed spectra

• Spectra in MOSB where there should be none.

• Spectral range extended too far into red

• Extra bright region in reddest part of HR spectra

• Magnitudes and errors that don't exist were flagged as 30 or 0: should be set to NULL…

• ….similarly for proper motion and parallax.


OpR2 data analysis

• Basic noise model was a problem

• CNAMEs mangled

• Wobbly spectra

• Overexposed spectra

• Spectra in MOSB where there should be none.

• Spectral range extended too far into red

• Extra bright region in reddest part of HR spectra

• Magnitudes and errors that don't exist were flagged as 30 or 0: should be set to NULL…

• ….similarly for proper motion and parallax.


OpR2 data analysis

This was the point – OpR2 was a success because of this!

How do we learn from OpR2, what are the next steps?

Fixed most of these issues, with changes to:• raw data

• L1 pipeline• Bulk re-processing of the fibinfo tablesKept the scope almost exactly the same (no extra features)

Not a full “simulation” / repeat of OpR2• reprocessing of new data of already-submitted targets

No new /revised / targets were submitted

Date: 9th August 2017, 9 nights generated in total• 56GB GB of raw data (+25GB), 1.4TB GB of L1 data (+623GB)• Final ”bonus” night with dark moon for some fields (+26 OBs, +54 stacks)


OpR2.5 – fixes to input data & pipeline

See following talks from science teams on the results from OpR2.5…• …many extremely useful reports from SWG and QAG members on the

analysis of pipeline products.

• Further analysis from both QAG and CASU post-OpR2.5 revealed some ongoing problems…

• …the majority of these have been addressed with fixes to either the pipelines or simulated data

• In some cases, the OpR2.5 data have been reprocessed again


OpR2.5 – fixes to input data & pipeline

“OpR2.5++”: post-QA bugfixes

Issues relating to sky subtraction:• Overscan region: 999.5 à randomly output 999 or 1000 for

each quadrant, each detector

• Vignetting correction factor applied to spectra, but not calibs• Results in poor residuals in sky-subtracted sky• Fixed in COSMOS & Lockman, some Cluster fields, RSG fields

– some L1 data now in the wild


OpR2.5++ bugfixes

• Want to know more? à Sky subtraction breakout session


Histogram:1. Calculate median sky

(in one OB)2. Sky – median Sky

3. Bin

OpR2.5++ bugfixes



OpR2.5++ bugfixes



OpR2.5++ bugfixes



OpR2.5++ bugfixes



StePS COSMOSz=0.6 ETGMag_r = 21Deep stack (7 OBs)

OpR2.5++ bugfixes



~250 ADU continuum

0.50 ADU

Continuumresidual

<< 1% sky

StePS COSMOSz=0.6 ETGMag_r = 21Deep stack (7 OBs)

OpR2.5++ bugfixes & current status• Sky subtraction in HR: issues with curvature at bluest end of blue

arm. Causing noisy flat field corrections

• Wavelength solution: ongoing LR(HR)±2km/s(~same) aim:1km/s

• RSG over-exposure / saturation: fixed with new magnitudes


(Monguio & SCIP QAG)

BLUEARM

OpR2.5++ bugfixes & current status• Some fibres assigned to targets, skies, calibs have no signal –

fixed, using RSG as a test

• Sawtooth profile seems to have been fixed (unsure how, but after missing spectra bug was fixed!)

• Low signal in wings of fibre PSF à poor crosstalk calculation. Longer fibre flats fix this by better sampling the wings (tested)


(Monguio & SCIP QAG)

OpR3Context: By end of OpR2.5: 9 nights of observations from a set of submitted catalogues. What do we do next?

• Surveys want to submit more targets…• …want more data• SPA need to expose system to more faults, build resilience

Imagine a new submission round, based on updated list of targets already observed..

• Are these cumulative? Diffs? Do they include already-observed? Etcetc…

• This is a process we need to understand

OpR3: will be a cyclic multi-semester operationWEAVE AHM: 1st – 3rd November 2017,

Groningen

OpR3 – Main aims1. OpR3 will test readiness of the cyclic, multi-semester, and

evolutionary nature of WEAVE operations.

2. The SWG will be responsible for catalogue consolidation and fibre allocation.

3. An OCS and Scheduler managed by ING, with a prototype form of the final design.

4. More rigorous testing of fault-tolerances and procedures. These include (but are not limited to): failed/aborted OBs, data checksum errors, weather, broken fibres, data retraction.

Full OpR3 plan document now on BSCW & Confluence



OpR2/2.5

OpR3l Not a huge step up in complexityl QAG added (access ill-defined)

l SWG handles configure


OpR2/2.5

OpR3l Not a huge step up in complexityl QAG added (access ill-defined)

l SWG handles configurel Time domain is more complex…


OpR3

l Time domain is more complex…

“David’s firstGantt chart”:

Events should flow top à bottom

4/10 for effort

OpR3: Milestones• Present plan at AHM2017: October 2017

• Revisions to plan document, SPA-general meeting: Mid-November 2017

• Release of component interaction matrix: Mid-November 2017

• Input catalogue survey column changes deadline: Mid-January 2018

• Interaction matrix deadline: End-January 2018

• Input catalogue changes approval, new catalogue templates: End-January 2018

• Interaction matrix review: Mid-February 2018

• Slack workspace open: March 2018

• Implementation phase

• WASP pre-rehearsal submissions open: April 2018

• OpR3 S1 input catalogue WASP submission deadline: Start June 2018

• OpR3 S3 concludes: mid-July 2018

• Preliminary post-OpR3 QAG report: mid-July 2018

• Final OpR3 QAG report: mid-September 2018


Date post:	10-Apr-2018
Category:	Documents
Upload:	truongkhuong
View:	214 times
Download:	2 times

WEAVE Operational Rehearsals - Onderzoekweaveahm2017/talks/WAHM... · WEAVE Operational Rehearsals...

Documents