WSDC Operations - California Institute of...

National Aeronautics and Space AdministrationJet Propulsion LaboratoryCalifornia Institute of Technology

1

WSDC Operations

R. Cutri, R. Beck, T. Conrow - IPAC

October 7-8, 2009

Operational Readiness Review – October 7-8, 2009


2

WISE Science Data Processing Flow



3

Operations Procedures

•  Documented in WSDC D-C001 through C005 •  All in mature draft stage •  Final pre-launch versions in the next month •  Update procedures as necessary post-launch in response to

on-orbit data needs



4 Operational Readiness Review – October 7-8, 2009

•  raw_copy script on Ingest computer (WSDCIN) checks filenames, sizes and creates a manifest

•  get_wsdcin script on the WSDS checks for new manifests and copies over new data.

•  write multiple archives to tape of MOS/HRP data.

•  ingestpipe for MOS/HRP copies sequence products, CK to archive location, loads HK. HRP ingest decommutates science telemetry, creates L0, QL and scans dirs and starts QL.

Data Transfer/Ingest Procedure

MOS data

HRP data

dmz

WSDS MOS

WSDS HRP

Ingest

L0 data

Quicklook

Ingest

dmz



Scan Frame pipeline

L1 images

Multi Frame pipeline

L3 source lists L3 images

L1 source lists

L0 data

•  run_scans script starts scan frame pipelines, monitors progress and creates summary when complete. •  showme_summary script shows same as operator console.

• wmcpipe creates wmfpipe commands to create coadds for completed scan frame areas. • monitor_coadds script does just that.

Scan Frame/Multi Frame Processing Procedures



IMGPrep IMGPrep IMGPrep IMGPrep

IRSA L1 metadata

IRSA L3 metadata

L1 images L3 images

IMGPrep IMGPrep IMGPrep DBPrep

IRSA L1 lists

L3 source lists

L1 source lists

IRSA L3 lists

•  dbprep script takes L1 or L3 source lists and creates pipe-delimited database load files

•  imgprep script takes L1 or L3 images and creates pipe-delimited database load files.

DB Load Preparation Procedure



NODES Maintenance Procedures

Monitors cluster performance and disk space usage and overall performance of the cluster.

- nodes script - allows single command to be executed on all nodes - good for checking space, number of processes running, starting background processes

- node_clean script - ensures nodes do not run out of work space on their local disks - creates logs per node for what gets deleted

- condor commands - starts, stops, delete running or queued processes - monitor running processes, queued processes, owner of processes


8

Daily Processing Operations

•  Scheduled time summary

–  Minimal required ops support Daily run: 4 hr In/QL (in parallel) + 4 hr SF + 4 hr backup Elapsed: 8 hrs

–  Normal ops Daily run: 4 hr In/QL (||) + 4 hr SF + 2 hr MS + 2 hr Coadd + 4 hr backup Elapsed: 12 hours

–  Catch up or rerun Daily run: 4 hr In/QL (||) + 4 hr SF + 2 hr MS + 2 hr coadd + 4 hr backup Catch up: 4 hr SF + 4 hr MS (or other rerun combinations) Elapsed: 20 hours



9

Daily Processing Operations


PST Activity

09

10

11

12

13

14

15

16

17

18

19

20 21

22

23

00

01

0 2

03 04

05

06

07

08

HRP Xfers (auto)

T1

T2

T3

T4

Ingest/QL (auto)

InQL 3

InQL 4

InQL 1

InQL 2

Scan/frame (T+0)

S/F 1,2

S/F 3,4

Backups Backups

Multi-scan (T+1 days)

MS 1-4

Coadds (T+2 days)

Coad 1-4

Reruns, Make-up

Idle Idle

Min

. Ops

Sup

port

N

orm

al O

ps

Staffed Unstaffed

Daily Ops Processing


10

Daily Processing Operations Personnel Deployment

•  2 FTE net Ops Engineers; 1 FTE + 0.5 FTE + 0.5 FTE; occasional back-up from 2 other WSDC engineers

•  6 day coverage (Mon.-Sat.), staggered 4-8 hr work shifts •  Daily duties and time allocations (M-F, reduced list on Sat.)

–  Monitor data arrival and automated transfer – 1 hour –  Confirm normal automated Ingest/Quicklook runs – 1 hour –  Start Scan/frame runs for recent data – 1 hour –  Start Multi-scan, coadds, WMOPS for completed scans – 2 hours

•  Including scheduling overnight runs –  Telem. and Ops DB backups – 2 hours –  Disk, run time monitoring and clean-up – 1 hour –  Handle special-purpose deliveries, reruns – 2 hours –  Archive loading – 1 hour –  Misc. reports – 1 hour –  Total: 12 hours



11

QA Ops Staff Deployment IOC


• Assumptions − Net 5 FTEs available − 7 day coverage − Staggered 9h shifts

• During critical events (boresight alignment, scan rate calib) stretch daily coverage as necessary


12

QA Ops Staff Deployment Survey Ops


• Assumptions − Net 5 FTEs available − 5day coverage − 9h workdays (8h + 1h

meal)


13

Visiting Team Member Support

•  Approximately 10 project team members at IPAC during IOC and 1-2 months of survey operations

•  Facilities –  Project “Situation Room” with work stations, dedicated meeting facility (MR214;

reconfiguration underway ) –  Temporary visitor offices for additional guests

•  Visitors will supply own computers –  IPAC ISG will provide user support –  Visitors will scp processed data to their computers for analysis

•  Connectivity –  Hardline network connection in Situation Room to WSDC computer to download

raw or processed data –  Wireless network for general Internet access –  Small volume data transfers via Ground Test Data Archive sftp site for SDL




Key Remaining Tasks Before Launch

•  Complete script to archive raw HRP and MOS TLM to tape − 3 copies generated: 1 off-site in deep archive

•  Finalize Multi-Scan pipeline procedure

•  Finalize Pipeline Operations interactions and reporting formats with MOS (following ingest) and internal WSDC subsystems (following standard processing)

•  Train new WSDC Ops Engineer


15

Backup Slides



16

Daily Processing Status

•  Most Pessimistic Week –  Max. load period; early ops

•  Full data load •  Scan/frame automation not ready yet •  5/40 coverage •  Weekend free period inaccessible to operators •  Full daily multi-scan pipelines and coadds generated

–  Will probably not really be generated for a few weeks –  Can run as few/many as we can afford



17


Most Pessimistic Week; No Weekend Coverage



18


•  Most Pessimistic Week; No Weekend Coverage

•  Transfers 98 hours Parallel 3.5 hr x 28/week •  Ingest/QL 42 hours Parallel 1.5 hr x 28/week •  Scan/frame 28 hours 4 hr x 7/week •  Multi-scan 14 hours 2 hr x 7/week •  Coadds (optional) 14 hours 2 hr x 7/week •  WMOPS 8 hours 2 hr x 4/week •  Backups/Maintenance 28 hours 4 hr x 7/week •  Total 232 hours •  Elapsed (clock time) 92 hours •  Free/pad 76 hours •  Non-weekend free/pad 28 hours (23%)


Date post:	28-Jan-2021
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

WSDC Operations - California Institute of...

Documents