Digital Packaging Processor - Overview

Gordon Hurford

Nov 7, 2011

EOVSA Technical Design Meeting - NJIT

Digital Packaging Processor

• Role of DPP

• Overall assumptions and priorities

• Interface Overview

• Tasks and algorithms

• Hardware/software Implementation

-----------------------------------------------------------

• DPP Interface Details

Data System Approach Fundamental Drivers

• Very limited software resources• Non-trivial data rate and volume • Automated analysis pipeline for efficient observing• Must have science-useable system in place by

September 2013• Data products to be readily useable by broader solar

community– Data products with preset parameters– Data products with user-selected parameters– Tools and support for experienced users

Data System Approach Implications

• Monomode observing• Calibrated data archived in application-specific databases• Reliance on existing software packages

– Miriad package for calibration & mapping– RHESSI SolarSoft package for user interface and data product

display– RHESSI database model

• Err on side of over-rejection of data • Limited initial support for ‘nice-to-have’ options• Limited initial support for calibration refinements• Limited support for non-solar applications

Data System Assumptions

• All information required for data analysis is written by the DPP to the Interim Data Base

• Engineering data acquisition, archiving and display is the responsibility of the ACC, and is “largely” decoupled from science data.

Nomenclature

• Data frame = Interval representing data from one correlator cycle (20 ms, ~4000 channels with 500 MHz range)

• Spectral frame = Data corresponding to a complete frequency-agile cycle (nominal 1 second, 10s to 100’s of ‘science channels, 18 GHz rang) Corresponds to a state frame.

• Scan: Observing interval within which target and frequency cycling pattern is unchanged

Role of Digital Packaging Processor

• To filter, average, partially calibrate and convert raw correlator output into a Miriad-compatible format that is written to Interim Data Base

• Real time, irreversible processing

DPP Interface Overview

DPP

State Frame

ACC

Correlator

Start / End Scan Commands

Scan-independent Calibration Parameters

Scan Parameters

Frame parameters

Frame status report

<P>, <P2>,

Correlations

Interim Data Base

Miriad

format

Internal RFI

Database1 s timing tick

0.02 s timing tick

RFI

results

DPP Task Timing

• Occasional – non operational– Accept, store and preprocess calibration parameters

• Scan initiation– Accept, store and preprocess scan-specific parameters

• Data frame (20 ms) – filter, and frequency-average correlator output

• Spectral frame (1 s) – Assemble, pre-calibrate, reformat and write data to Interim

database

• TBD – Format results and write to RFI database

DPP – Stage 1 ProcessingEvery data frame (20ms)

• Evaluate kurtosis data to identify RFI-affected subbands as a function of frequency only.

• Save RFI statistics• Combine with pre-flagged subbands to

generate a “destination vector” for each subband

• Apply complex gains at subband level ???• Average subband data into spectral channels • Save 1st 3 moments of averages ???

DPP Stage 2 ProcessingEvery spectral frame (1s)

– Convert antenna-based flags (e.g. slewing) from state frame to baseline-based, frequency-independent flags

– Apply time-independent complex gains if available– Apply baseline corrections– Apply non-linearity corrections– Correct for attenuator settings– Correct for spectral simultaneity

• Miriad format this is no longer optional

– Convert visibility, uv and analysis-relevant state-frame data to Miriad-compatible format

– Write spectral frame to IDB– Report DPP status to state frame

DPP - Implementation

• Original concept was to follow FASR plan for a cluster-based DPP

• Estimate processing requirements for EOVSA DPP at ~100 MIPS = 1/60 of FASR requirements

• Implementation will be based on a single multi-core machine

• Software organization will be compatible with migration to a cluster if necessary

DPP Software Architecture

ACC State Frame Correlator IDB

Coordination Task

I/O, data assembly, no processing per se

Header Processing

Stage 1 Processing

Stage 2 Processing

Pointers within shared memory

Conventional, time-independent processing tasks

C1

C2 C3, C4 C2

Cn = core within a quad core processor or nodes in a cluster

DPP

RFI database

ParameterProcessing

C2

DPP Status• Software architecture and tasks identified

• Detailed definition of interfaces is underway

• EOVSA to Miriad format conversion being tested with FST data– (Fortran 77 for Miriad compatibility)

• Next:1. Complete definition of interfaces2. Code Stage 1 tasks (GH)

• Evaluate timing requirements3 Code Coordination task (JM). 3. Detailed definition of processing algorithms 4. Code of Stage 2 tasks

5. Machine selection and purchase • Development platform?

• Goal: Functional DPP to support prototype testing