Post on 04-Jan-2016
transcript
December 16, 2002December 16, 2002 11NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR
NOAO Mosaic PipelineNOAO Mosaic Pipeline
Technical PresentationTechnical Presentation
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 22
Outline of Technical PresentationOutline of Technical Presentation
• Introduction
• Contexts
• Capabilities
• Architecture
• Implementation
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 33
Presentation GoalsPresentation Goals
Convince you that:• we understand the
– problem– requirements– resources– components
• and that the project– is feasible– has a solution for the primary application– has a flexible design for expansion and wider application
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 44
Guiding PrinciplesGuiding Principles
• Modest project
• Part of Data Products Program
• (NOAO) Mosaic Imaging Data
• Dedicated pipeline
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 55
Principles: Modest ProjectPrinciples: Modest Project
• Reuse as much software as possible
• Keep it simple software
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 66
Principles: DPPPrinciples: DPP
• MDHS: Mosaic Data Handling System
• IRAF: Image Reduction and Analysis Facility
• NSA: NOAO Science Archive
• DTS: Data Transport System
• OPUS: AURA sister institution (STScI)
• GONG: AURA sister institution (NSO)
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 77
Principles: (NOAO) Mosaic DataPrinciples: (NOAO) Mosaic Data
• Use experience of Mosaic Survey Teams
• Need to deal with specific peculiarities– Crosstalk, pupil reflections
• Allow for high performance per exposure (for real-time telescope context) by capitalizing on the inherent data parallel nature of mosaic imaging data
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 88
Principles: Dedicated PipelinePrinciples: Dedicated Pipeline
• Network of similar computers
• No competition with general users
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 99
What does this project encompass?What does this project encompass?
• Pipeline infrastructure• CCD mosaic data reduction• Data quality assessment• Image differencing• Catalog production• Database entry and querying• Source merging/classification
• Archive ingest and retrieval• Alerts• Monitoring• Data transport• High performance computing• Parallel computing• More …
Algorithms, interfaces, and software for:
December 16, 2002December 16, 2002 1010NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR
ContextsContexts
In what contexts will the pipeline run?In what contexts will the pipeline run?
Can we design a pipeline to satisfy multiple contexts?Can we design a pipeline to satisfy multiple contexts?
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 1111
ContextsContexts
• NOAO– Telescope/operational context– Archive/NVO context
• Community– NOAO Mosaic surveys and observers– Other mosaic instruments
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 1212
PrioritiesPriorities
1. NOAO Archive
2. NOAO Mosaic observers− telescope− downtown− home institution
3. NOAO Mosaic observers at home
4. Community
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 1313
NOAO ContextsNOAO Contexts
• Downtown center fed from telescope
• Mountain at telescope
• Archive on-the-fly reprocessing
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 1414
Pipeline LocationsPipeline Locations
La SerenaArchive
TucsonArchive
Kitt Peak Cerro Tololo
Pipeline
Pipeline Pipeline
Pipeline
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 1515
Context: Downtown PipelineContext: Downtown Pipeline
ObserverDCA
Data Spooland Transport
Pipeline
DSC
User @telescope,downtown,
home
Archive
DTS
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 1616
Context: Mountain PipelineContext: Mountain Pipeline
DCAData Spool
and TransportPipeline
User @telescope
Archive
DTS
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 1717
Context: Archive PipelineContext: Archive Pipeline
User @home
Pipeline
Archive
DTS
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 1818
Context: User PipelineContext: User Pipeline
User @home
Pipeline@ home
More
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 1919
Proposed ContextProposed Context
• Downtown pipeline for NOAO archive
• Observer may subscribe to data products– At telescope, downtown, home– Images, catalogs, alerts, …
• Observer may connect to DQ monitors• Pipeline software available at telescope with
minimal support• DQ task/monitors may run at telescope
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 2121
Data RequirementsData Requirements
The pipeline design is dependent on the information available about the input data.
• Basically we require data with the current NOAO Mosaic readout format that includes:– identification of exposure type (object, etc)– description of regions (data, overscan)– an approximate world coordinate system
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 2222
Data RequirementsData Requirements
There may be additional information that the pipeline will use if present.
• Associations: type, ID, total and index SEQUENCE = ‘zero2002-12-16T043244.20.3’
SEQUENCE = ‘dither2002-12-16T043244.5.2’
If not present heuristics will be used based on a requirement that data enters in time order
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 2727
CapabilitiesCapabilities
• Capabilities
• Major Features and Goals
• Data Products– Basic– Advanced
• Data Quality Assessment
• Instrumental Calibration
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 2828
CapabilitiesCapabilities
• Calibrate mosaic exposures• Update instrumental calibrations• Identify potential bad data (data quality assessment)• Monitor trends and maintain database• Stack dither sets• Catalog and classify objects and artifacts• Get and subtract reference image and detect sources• Identify interesting sources• Automatically provide data products to subscribers• Keep up with observing given sufficient CPU resources
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 2929
Major Features and GoalsMajor Features and Goals
• Data products for NOAO archive and NVO node• Data products for observers (by subscription)• Pipeline for NOAO and mosaic community• Basic CCD mosaic calibrations• Advanced time-domain data products• Real-time data quality assessment and monitoring• High performance, data parallel system• LSST testbed• Fairly generic pipeline infrastructure (NEWFIRM, …)• Automated operation• Thorough processing history and data documentation
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 3030
Data Products: BasicData Products: Basic
• Instrument calibrated mosaic exposures• Rough photometric zero point• Astrometric calibrations• Data quality evaluations• Updated calibrations• Bad pixel, saturated, bleed trail masks• Object catalogs• Object masks• Observing logs• Processing information
– logs– graphs
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 3131
Data Products: AdvancedData Products: Advanced
• Dither stacks• Exposure masks• Field Catalogs• Difference image detections
– Relative to dither stack– Relative to archive or catalog reference
• Light curves• Variable object detections• Unusual object alerts• Moving object trajectories
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 3232
Data Quality AssessmentData Quality Assessment
Instrument• Telemetry• Crosstalk• Overscan• Bias, flat• Noise• Focus / Distortions
Sky• Seeing (PSF)• Sky brightness• Approx. zero point• Twilight• Moon up / distance
Data quality measures are monitored against preset and user limits as well as adaptive time series limits. Some quantities include mean, sigma, and spatial variations.
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 3333
Instrumental CalibrationsInstrumental Calibrations
• Crosstalk [1]
• CCD defects [2,4,5]
• Saturated pixels [2,4,5]
• Bleed trails [2,4,5]
• Cosmic rays [2,4,5]
• WCS update [3]
1. Requires image data from full mosaic (non-parallel)
2. Each image element independent of others (parallel)
3. Global calculation on measurements images (parallel and non-parallel)
4. Interpolate in data
5. Flag in mask
• Overscan [2]
• Bias [2]
• Flat field [2]
• Pupil pattern [3]
• Fringing [3] • Approx. zero point [3]
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 3434
Instrumental CalibrationsInstrumental Calibrations
Two-pass calibration for telescope context:
1. Nighttime pass for immediate and nearly complete calibrated exposures
2. Daytime pass for calibration update from the full night’s data set
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 3535
Nighttime PassNighttime Pass
• Perform standard CCD calibrations:– Use afternoon master bias– Use most recent flat field
• Apply pupil and fringe correction– Use most recent pupil and fringe templates
• Apply global coordinate calibration
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 3636
Daytime PassDaytime Pass
• Determine if night’s data is suitable for deriving updates to library calibrations
• Derive new pupil, fringe, and sky flat calibrations• Evaluate changes and significance of new
calibrations• Update library calibrations for next night• Update night’s exposures with new calibrations• Combine afternoon biases into new master bias• Combine afternoon dome flats if no library flat
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 3737
Other ContextsOther Contexts
• For archive data will either already have best calibration from library or will be derived by requesting raw data for night
• At home or in the community raw data will be queued as at telescope
• Documentation and support (data ingest applications) will be provided
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 3838
Data Products SubscriptionData Products Subscription
• Capability of the DPP system– Not necessarily specific to the pipeline but
requires interfacing with DTS
• Allows external software to request notification of new data products
• Allows flexibility and broader access– Has implications for the pipeline context
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 3939
ArchitectureArchitecture
• What is a pipeline?
• Mosaic Pipeline Architecture Concept
• Pipeline Components– Controls and Monitors– Modules– Calibrations and Database (Rafael Hiriart)– Archive (Robyn Allsman)
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 4040
What is a Pipeline?What is a Pipeline?
System to transform input data to output data
• Automated
• Composed of processing steps (modules)
• Steps connected by rules (triggers)
• Provides monitoring and alerts
• Error tolerant (continue with next input data)
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 4141
Mosaic Pipeline Architecture ConceptMosaic Pipeline Architecture Concept
• Multiple CPUs but no dependency on N
• Multiple types of sub-pipelines by function– One for operations over all mosaic elements– One for operations on individual elements– One for cataloging– One for image differencing
• All types on all CPUs: no master!
• Sub-pipelines triggered by files
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 4242
• All CPUs with identical pipeline software, possibly on common NFS disk
• Assign work by minimum data backlog• Transfer data to local CPU disk: not NFS!
– Optimize by modules writing to next trigger directory
• Controls connected to operator console• Monitors viewed via network by multiple
parties
Mosaic Pipeline Architecture ConceptMosaic Pipeline Architecture Concept
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 4343
Network of Sub-pipelines and CPUsNetwork of Sub-pipelines and CPUs
Pipeline
CPUCPU
CPU
CPU
CPU
MEF
SIF
SIF
MEF
SIF
MEF
MEFCPU
SIF
MEF
SIF
MEF
SIF
MEF: pipeline for operations over all mosaic extensions; eg crosstalk, global WCS correction
SIF: pipeline for single CCD images; eg ccdproc, masking
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 4444
Data Flow ConceptData Flow Concept
Last module in one pipeline writes output directly to the data directories of the host for next pipeline, with the host selected by having the minimum number of waiting data files.
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 4545
Data Flow AlgorithmData Flow Algorithm
• Search list of potential hosts:– Check if host is up– Check number of trigger files– Assign output filename to data directory of
host with least number of data files– Network filenames are used: (eg. host!
directory/filename
• Module runs and writes output files
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 4646
Data Flow NetworkingData Flow Networking
• Use a daemon automatically spawned the first time data is transferred to a host
• Daemon provides portability across platforms; eg. Unix and VMS
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 4747
Data Flow Networking: ExampleData Flow Networking: Example
• Crosstalk input is Obj123.fits with 2 extensions• Output names are generated from Host.dat:
– Host1 has two waiting files, Host2 has one, Host3 is down, Host4 has none
– Host2!Obj123.1, Host4!Obj123.2
• Crosstalk module runs and writes output files directly to the hosts
• There are no extra network copy or splitting steps
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 4848
Data Flow Networking: ExampleData Flow Networking: Example
Host0:
Crosstalk
Host1:
Obj456.1
Obj321.2
Host2:
Obj567.2
Host3:
Obj123Obj123.2
Obj123.1Host3!Obj123.1
Host2!Obj123.2
Host4:
DOWN
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 4949
Pipeline ComponentsPipeline Components
Data Source(DTS, user)
Pipeline
Controls & Monitors
Calibrations &Databases
Data Sink(DTS, user)
raw
data
data
productsModule Module
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 5050
Pipeline ModulesPipeline Modules
Pipeline
Module Module Module
CLSH APICSH
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 5151
Data Parallel ModulesData Parallel Modules
Some algorithms may need to be (re-)implemented specifically for a data parallel pipeline.
One type is where measurements are made across the mosaic for a global calibration.
Rather than requiring all pieces to be in one pipeline arrange for measurements made in parallel to be collected for the global calibration and then apply the global calibration to the pieces in parallel.
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 5252
Data Parallel ModulesData Parallel ModulesWCS ExampleWCS Example
• Catalog objects in each CCD in parallel• Bring catalogs (not images) together
– Only need x/y coordinates of brighter stars
• Match sources to ref. catalog (eg. USNO)• Compute global correction (shift, scale, etc.)• Return correction coefficients to parallel
pipelines to be applied to each CCD• Cataloging and correction stages can be
separated and run asynchronously with other stages
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 5353
Data Parallel ModulesData Parallel ModulesFringe/Pupil ExampleFringe/Pupil Example
• Determine best global scaling of pupil and fringe templates to each exposure and then subtract scaled template– Compute statistics over each CCD in parallel– Combine statistics to get global scale factor– Subtract template with global scale from each
CCD in parallel
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 5454
Pipeline TriggersPipeline Triggers
• Files: trigger on appearance of files• Flags: trigger on particular set of flags• Timers: trigger at times or intervals• File contents: trigger on keywords, etc• Messages: trigger on messages• Resources: trigger on resources
May be more but one type can mimic others
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 5555
Pipeline TriggersPipeline Triggers
• File triggers useful for initiating a pipeline
• Flag triggers useful within a pipeline to communicate success of previous steps
• Flag triggers also useful for waiting for completion of parallel steps
• Timer triggers useful in telescope pipeline for performing different daytime/nighttime steps
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 5656
Pipeline
DataDirectory
TriggerDirectory
Module
obj123.fits obj123.trig GO
File TriggersFile Triggers
May containinformation
such asoutput path
More
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 5757
Pipeline
TriggerDirectory
DataDirectory
obj123a.trigobj123b.trigobj123c.trigobj123a.fitsobj123b.fitsobj123c.fits
--------
d------
dd----
ddd--
ddddobj123d.fits obj123d.trig
Module
GO
Flag Triggers and MergingFlag Triggers and Merging
Module
GO
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 5858
Timer Triggers and Two-PassesTimer Triggers and Two-Passes
• Nighttime pipeline runs and leaves data in starting directory for daytime pipeline
• Daytime pipeline is triggered at end of night by timer
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 5959
Controls & MonitorsControls & Monitors
Pipeline
Process Manager
Obs. Manager
Status Monitor
Keyword Monitor
Module
ToDatabase
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 6161
Data ManagerData Manager• Interacts with the pipeline, operator, and potentially other
parts of the system such as archives or external applications• Record
– New calibrations from pipeline or operator– New parameters from operator– Processing information from pipeline
• Responds to queries for – Calibrations– Parameters– Processing history– Documentation and reports for data products
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 6262
Data Manager ArchitectureData Manager Architecture
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 6363
What do we want to store in the What do we want to store in the database?database?
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 6464
Where is Data Manager?Where is Data Manager?
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 6565
CalibrationsCalibrations
• The Data Manager responds to requests from pipeline for current calibration for a particular date, filter, etc.
• Updates calibrations produced by pipeline (or externally) for a particular date, filter, etc. Calibration updates may require operator confirmation.
• Calibrations include– Biases and flat fields– Pupil and fringe templates– Standard star data– Astrometry coordinates
• Some queries are satisfied through secondary queries to other databases such as USNO, GSC2, Landolt, etc.
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 6666
ParametersParameters
• Responds to requests from pipeline for current parameters for– Pipeline module– Observation date, filter, exposure type, etc.– Position on sky
• Updates parameters supplied by operator
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 6767
Processing InformationProcessing Information
• All information produced by the pipeline is recorded (keyed by a data identifier). This includes all the information provided to the keyword monitor as well as other data processing sources (logs, graphics, etc.)
• Pipeline requests processing information for a pipeline execution packaged as an associated data product for the archive.
• The operator can query processing information for diagnostic purposes.
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 6868
ReportsReports
• Produces reports for a particular data product• Documentation is created from processing information
according to some template and desired format (eg xml, html)
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 6969
Pipeline/Archive Ingest InterfacePipeline/Archive Ingest Interface
• Desirable traits
– Independence of database semantics– Use of self-describing data description
standards– Hiding data’s physical location
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 7070
Archive IngestArchive Ingest
Who, What, WhereAuthority
Payload
DataReceiving
Agent
Data Store
Archive Ingest
Manager
December 16, 2002December 16, 2002 7171NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR
Strawman ImplementationStrawman Implementation
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 7272
Things We Looked At / Aware OfThings We Looked At / Aware Of
• Macho pipeline• SM/SN pipeline• Sloan pipeline• Pan-Starrs: IMCAT, Vista• IRAF: Core, IMRED pipelines, STSDAS, PYRAF, etc• MIDAS: Mosaic Imager Data Archive System• Linda and descendants• Elixar (CFH), Terapix (CFH), Subaru, ESO WFI, INT WFI• Condor / PVM / NOAO message bus• Opus pipelines: HST. MSSO, GONG• Databases: MySQL, Postgres
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 7373
Software and SystemsSoftware and Systems(Blue Ribbon)(Blue Ribbon)
• OPUS• IRAF System
– CLSH (enhanced), KI, OBM/GUI
• IRAF Tasks– MSCRED, ACE
• SM/SN Alard/Lupton Algorithm• POSTGRES• DTS• NSA
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 7474
Software and SystemsSoftware and Systems(Honorable Mention) (Honorable Mention)
• PVM
• Condor
• Other scripting languages and systems– PYRAF and Python– Perl– MLCL
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 7575
Pipeline ModulesPipeline Modules
Pipeline
Module Module Module
CLSH OAPICSHMSCRED, etc
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 7676
Controls & MonitorsControls & Monitors
Pipeline
Process Manager
Obs. Manager
Status Monitor
Keyword Monitor
Opus
Opus
IRAF GUI
IRAF GUI
OpusModule
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 7777
Switchboard ServerSwitchboard Server
CPU
Pipeline
ModuleModule
Pipeline
ModuleModule
CPU
Pipeline
ModuleModule
Pipeline
ModuleModule
Switchboard
Server
Backup
Keyword
Monitor
Status
Monitor
Database
Manager
Other types
Or instances
Switchboard address set by environment variable
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 7878
TriggersTriggers
OPUS provides:
• Files: trigger on appearance of files– Data entry pipeline initiation
• Flags: trigger on “blackboard” flags– Internal sequencing of modules– Parallel to Global sequencing
• Timers: trigger at certain times or intervals– Nighttime/Daytime Two-Pass Control
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 7979
Monitoring IRAF TasksMonitoring IRAF Tasks
• IRAF tasks, including scripts, will open a messaging connection and write status and monitor information
• Minimal changes will be required to tasks• If a server is not running or disappears the tasks
will continue to run with output spooled locally
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 8080
Monitoring IRAF TasksMonitoring IRAF Tasks
• Initially the broadcasting will be a socket connection with a server that multiple clients may connect to for rebroadcast
• The monitor tasks are IRAF GUI tasks which provide flexibility for changes to the GUI or functionality
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 8181
Monitoring IRAF TasksMonitoring IRAF Tasks
• The GUI monitors will include:– Adaptive alarms– Adaptive heartbeat monitoring– Advanced graphics
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 8282
IRAF Keyword Monitor PrototypeIRAF Keyword Monitor Prototype
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 8383
IRAF Keyword Monitor PrototypeIRAF Keyword Monitor Prototype
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 8484
IRAF Status Monitor PrototypeIRAF Status Monitor Prototype
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 8585
NOAO Mosaic PipelineNOAO Mosaic Pipeline
Development Plan
1. Basic Calibration Pipeline
2. Advanced Time-Domain Pipeline
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 8686
1. Basic Calibration Pipeline1. Basic Calibration Pipeline
• Basic single exposure calibrations• Data quality assessment and monitoring• High-performance pipeline infrastructure• Simple data transport system• Connection to the NOAO Science Archive
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 8787
2. Advanced Time-Domain Pipeline2. Advanced Time-Domain Pipeline
• Catalogs• Image difference detections• Multiple detection ident. and merging• Time series• Alerts• Archiving of new data products
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 8888
Timeline TargetsTimeline Targets
• Test version of basic calibration pipeline– July 2003
• Operational– September 2003
• Test version of time-domain pipeline– July 2004
• Operational– September 2004
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 8989
Work BreakdownWork Breakdown
• Pipeline
• Monitors
• Data Manager
• Input and Output
• Data Products
• Archive
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 9090
• Pipeline– Define methods for running IRAF tasks in OPUS
• Parameters• Error handling• I/O
– Define and verify data flow balancing method– Define, develop, and implement DQ methods– Develop data parallel algorithm steps for
• WCS• Fringe/pupil removal
– Develop data parallel OPUS architecture– Setup development system of at least two machines
Work Breakdown
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 9191
• Monitors– Develop status monitor
• Experiment with different GUI formats
– Develop keyword monitor• Experiment with different GUI formats
– Develop switchboard server
Work Breakdown
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 9292
Work Breakdown
• Data Manager– Define interfaces
• Pipeline• DBMS• NVO/web services• External clients
– Define database structures– Define archive data products– Design processing reports– Design calibration library storage and methods– Design and implement manager application
• Include GUI monitor and operator interface
– Install and configure DBMS
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 9393
• Input and Output Services– Contribute to DTS– Implement interim data transport, staging, and
queuing
Work Breakdown
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 9494
• Archive– Contribute to NSA development of automatic
ingest– Adjust data product specification to include
NSA requirements
Work Breakdown
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 9595
• Specify Data Products– File types– Headers– Documentation
Work Breakdown
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 9696
Implementation PlanImplementation Plan
• It is important to deliver core functionality quickly• Some technologies are new (to the development
team)• Delivery timeframe is short
This project will proceed using a Phased Delivery lifecycle development model –– i.e., phases of detailed design-develop-test as core functionality is built up. This is especially appropriate when:
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 9797
Management PlanManagement Plan
Key elements of the management plan are:• Management/staffing• Work Breakdown [covered by FV]• Schedule• Risk Management
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 9898
PersonnelPersonnelStaff
MemberRole
Allocation
Responsibilities
Dick Shaw Project Manager 5% Schedule development, resource planning
Frank Valdes Team Lead 40%Allocation of work, tracking technical progress, lead designer, documentation
Chris Smith Project Scientist 10%Definition of requirements, use cases, verification & validation, documentation
Rafael Hiriart S/W Engineer 25%Database & infrastructure design & development, use case development
Robyn Allsman S/W Sys. Eng. 5%Archive interface definition, archive system updates, data storage planning, consultant
F. Pierfederici Scientific Progr. 30% Implementation, testing
TBD (U. MD) S/W Engineer 50% Implementation, testing
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 9999
Staffing ProfileStaffing Profile
0%
20%
40%
60%
80%
100%
120%
140%
160%
180%
2002Q4
2003Q1
2003Q2
2003Q3
2003Q4
2004Q1
2004Q2
U.MD
Pierfederici
Allsman
Hiriart
Smith
Valdes
Shaw
December 16, 2002December 16, 2002 NOAO Mosaic Pipeline CoDRNOAO Mosaic Pipeline CoDR 101101
Risk ManagementRisk Management
• Heavily matrixed staff– New staff will also off-load other work from team lead
• Staff distributed across continents & institutions– Project leadership remains in Tucson– Extended visits by new remote staff– Weekly videoconferences
• New staff has limited experience in problem domain– Project leader to work closely with new staff
• Use of new/third-party software– Make effective use of expertise from external partners