Trends in Data Processing for Space Science Missions

James L. Green and Rick Burley

Space Science Data Operations Office, GSFC

Science Data Processing WorkshopFebruary 27, 2002

Outline

• Characteristics of previous space science missions• Current trends in space science missions• New approaches for new missions• End-to-end data flow analysis • IMAGE data processing system• Summary

Characteristics of Previous Space Science Missions

• Space science data processing systems are intimately tied to flight missions

• Mission types:– Space physics: Exclusively Principal Investigator (PI) missions

– Astrophysics: PI plus a number of observatory class missions

• Projects did not have full life cycle budget responsibility– Mission operations (MO) performed by a service organization with

separate budget

– Ensured keeping the science operation systems separate from MO

– Development costs not MO&DA costs drove the mission design

• Project’s philosophy was to “fix problems on the ground”

• End-to-end mission data systems trade studies were rarely performed

Example #1: Previous Mission Data System

• Projects did not considered the full effect on the ground data system to all mission design decisions

• Astrophysics Mission launched in the early ‘90s– Low Earth orbit (~90 min. period)

– Observations of the faintest objects were as long as 4-5 hours

– Must make non-contiguous observations of same object over multiple orbits

• Result: extensive ground system problems and work arounds– Science scheduling system become complicated and labor intensive to

execute

– Science data processing system had to accommodate all types of observations

Lack of Ground Data System Trade Studies• Scientific Data Processing System Functions

– Group the observations by object for non-contiguous targets

– Calibrate the data and format the result (FITS)

• Analysis of the software system components:

– Science algorithms: 11%

– Data management: 78%

– Orbit/aspect + miscellaneous: ~11%

– 306K lines of code

– ~100 MY effort (estimated) over 7 years

• Expect a higher orbit vs. ground system software development trade study could have saved costs

– New Requirement: Plan all observations of targets to be contiguous making operations simpler

• Personnel and hardware/software systems development and maintenance costs would be significantly reduce

• Mission could operate for less for longer (plan on mission extensions!)

Example #2: A “Facilities-Oriented” Approach

DSNTelemetry & Command Support

IPDPACOR: Datacapture & pro-cessing. DDFdata distribution

MOCPOCC: s/ cops & controlCMS: CommandSupport

FDF

Orbit,attitudeanalysis

CDHF

Online data files

EOF

SOHO scienceplanning andoperations

GGS SPOF

GGS SciencePlanning

state vectors

TRK data

Files,pages

SelectedPB data

Telemetry

Commands

TLM, Tape (MDI(HR))

CMD coord

Files, pages

R/ T,PBdata

Files,online data

Files

NSSDC

Files

Coord.

Coord.Orbit & Att

data

ESTEC

REMOTE PIFACILITIES

Distribution Data

Onlinedata

CMD, coord

Archive

SOHO Ground System InterfaceNASA/ SOHO Flight Operations Plan Operations Issue November, 1995

Mission and Science Operations Manpower Estimate

Category FY 97 Staffing

Flight Dynamics Facility 22

Flight Operations Teams IMP 1Wind/Polar 17SOHO 22

Science Planning and Operations Facility 8

Experiment Operations Facility 5

Data Processing Centers Central Data Handling Facility 19Data Distr ibution Facility 10Packet Processor Facility 9Data Capture Facility 34

Maintainence 22

Management (includes QA, admin, configcontrol and test teams)

38

Total FTE’s 207

• These functions were spread out over 5 or 6 buildings• Separate staff for each function and no cross training• Dependent on a large number of missions before systems became cost effective

Current Trends In Space Science Missions

• Types of Space Science missions:– Observatory class: Hubble, Chandra, …

– PI class: Explorers (ie; IMAGE), Solar Terrestrial Probes, …

• Regardless of mission type the science community expects:– Rapid access to all mission data products and services

– Short or no proprietary periods

– Better data sets into the long term archive

• Cost Capped Mission must cover total mission costs

• Forced to find simpler and cost effective ways to satisfy more requirements

New Approaches for New Missions

• Use lessons learned from the previous missions

• Include ground data system into overall mission design and perform the appropriate trade studies– Consider the effect of operations on the end-to-end data system

– Assume success and plan on MO into extended mission phase

– Recognize that a lot of operations personnel over time is expensive

• Automation where possible reducing operations staff

• Co-locate facilities and functions provide cross training

• Recognize software development costs– Maximize the use of standard formats and software reuse

End-to-End Data Flow Analysis• Data requirements

– Define the users of the data

• Flight operators (housekeeping)

• Investigators (science data)

• Engineers (housekeeping)

• Other scientists (archived data)

• Public (science results)

– What data products are needed for which users?

– Time scale for delivery (real-time or later)

– Quality of the data and associated documentation

– Other issues (security, proprietary rights, …)

• Characterize the data flow to each user– Determine what processing is needed and where it is to be done

• Identify what associated data is required

• Form and format of all data produced

– Determine transport or delivery protocol

– Determine any derived requirements

End-to-End Flow Process (Cont’d.)

• Evaluate approach– Determine the cost and the schedule

– Identify driving requirements (most cost/most time)

• Develop alternative scenarios and refine– System improvements

• Standards (interface, transport, form/format)

• Software reuse

• Commercial off-the-shelf software

• Hardware technologies migration strategies

– Process improvements

• Co-locate functions (SOC/MOC, product generation, etc.)

• Distribute functions (government, university, industry, etc.)

– Multi-mission infrastructure

– Requirements reduction

– Other trades (i.e., orbits)

Science Objectives:• Identify the dominant mechanisms for injecting plasma into the magnetosphere on substorm and magnetic storm timescales,• Determine the directly driven response of the magnetosphere to changes in the solar wind• Discover how and where magnetospheric plasmas are energized, transported, and subsequently lost during substorms and magnetic storms.

IMAGE is the first of the MIDEX series which feature a capped MO&DA cost.

http://image.gsfc.nasa.gov/

IImager for MMagnetopause-to-AAurora GGlobal EExploration

Facts in Brief:Launched: March 25, 2000 Launch Vehicle: DeltaLaunch Site: Vandenberg AFBOrbit: 1000 km x 7 RE altitude polar orbitTelemetry: 2.2Mbps via DSN 34mMission Duration: Two years (prime)

3 more years extended

IMAGE REQUIREMENTS SUMMARY

Instruments: 3 Neutral Atom Imagers from 10ev to 200kev. FUV Wideband Imaging Camera, Spectrographic Imager and Geocorona photometers, EUV Imager, Radio Plasma Imager. Uplink of consolidated science commands once per week - nominally.

Data Processing: Level-0 processing done for each science instrument and housekeeping. Browse Products for each science instrument and housekeeping. Emphasis on quick data processing and distribution.

Data Distribution: All data is public - no proprietary period. Level-0 and BP data will be available via world-wide-web site within 48 hours after each DSN pass All data delivered to NSSDC for permanent archive, and public distribution.

IMAGE mission has IMAGE mission has been designed to be low maintainence.been designed to be low maintainence.

Cost Effective Elements of the IMAGE DS

• Put mission operations and science data processing into one facility with one staff to perform required functions– Science and Mission Operations Center - SMOC

• IMAGE plan for automation based on the requirements of minimizing mission operations

– Planned for lights out operations• Selective data retransmission was automated

– 8 x 5 shifts only – Large margin simplify operations

• Data volume, command storage, thermal, power, …– SERS (s/c emergency response system) - page operator when sustained limits

violated, or health & safety limits violated– Maximize the use of standards (including network protocols)

• Used the I&T based tools for Mission Ops (ASIST, FEDS)• Use existing science correlative tools (CDAWeb, SSCWeb)

– Science and ancillary mission data from the open archive– Engineering/housekeeping data to perform all standard trend analysis

IMAGE Ground Data System Overview• DS development budget less than 10%10% of cost of similar ground

systems• Schedule: 36 months development/test schedule • Spacecraft Control Team (SCT) approach within the SMOC

– Flight Operations• Command management, data acquisition, heath & safety of s/c

– Mission Planning– Ground data processing (level 0 & 1)– System Administration

• Operations staff: – All staff brought on board before spacecraft I&T (~ 1 year prior to launch)

– 4 person Spacecraft Control Team through launch & early orbit – 2.5 person Spacecraft Control Team through End-of-Mission – 0.3 person DSN scheduling

• Total Mission Operations cost $450K/year (excludes DSN costs)

IMAGE Science Data Process System • Use IDL scripts for automated data processing

– Allowed PIs to independently develop their algorithms and integrate into the SMOC software system

– Processed data files stored on FTP web server

• No commercial DB systems needed

– Engineering data also processed

• Twice daily processing (2 DSN passes) – Quick-look

– Daily processing forwarded onto the NSSDC

– NSSDC loads into CDAWeb daily

• SMOC maintains ~2 month of processed data online – No pushing of QL data to investigators: they look at it at their

convenience

– No project dependent archive needs to be maintained

IMAGE Data Flow

• Science data products were generated into standard formats– Universal Data Format (UDF) - for low level telemetry data– Common Data Format (CDF) - for higher level and “browse” products

• NSSDC provided software to IMAGE to bundle its data products for archiving

• Unique joining of data

– NASA: ACE, FAST, IMAGE, IMP-8, Polar, SAMPEX, Wind

– Other US: GOES, LANL

– International: Geotail, SOHO, Ulysses, Interball, Cluster

• Continuing growth in popularity

– An international community of users

– Service mirrored in Europe and Japan

• Large community of users in 2001

– ~330 distinct (heavy) users made >100 requests

– ~1080 distinct (regular) users made >10 requests

Coordinated Data Analysis Web (CDAWeb)

• CDAWeb is a publicly-accessible data “browsing” service (plots and listings) that provides simultaneous access to multi-mission data needed to understand global interactions, structure & dynamics

• Part of the NSSDC archival system

Scope of the CDAWeb

User Community in 2001

.GOV

7%

.NET

1%

.EDU

20%

.COM

15%

Other

44%

Other US

1%

CANADA

3%

OTHER

5%

RUSSIA

11%

JAPAN

10%

EUROPE

27%

US

TOTAL

41%

January-April 2001 Usage (% of requests)

US DOMAINS

The Growth in CDAWeb Usage

0

5000

10000

15000

20000

25000

30000

Jul-1996Jan-1997Jul-1997Jan-1998Jul-1998Jan-1999Jul-1999Jan-2000Jul-2000Jan-2001Jul-2001Jan-2002

Monthly Usage

CDF Files Transferred (*0.1)

Lists & ASCII Files Created

Plots Created

Web "Hits" (*0.1)

Growth in CDAWeb Database

• Now serves 420,000 files, 1300+ science parameters

• Contains ~6 years of ~6 years of simultaneous data from >100 datasets on every day

0

500

1000

1500

2000

2500

3000

3500

Number of Days

02/0210/0103/0111/005/00 11/99 7/99 1/99 12/98 3/98

The CDAWeb Database (July 1992 to the present): Growing Total Days of Data, Growing Datasets per Day

Days with 141-160 Datasets

Days with 121-140 Datasets

Days with 101-120 Datasets

Days with 81-100 Datasets

Days with 61-80 Datasets

Days with 41-60 Datasets

Days with 21-40 Datasets

Days with 1-20 Datasets

Avg Datasets/Day (46) (54) (60) (71) (77) (87) (90) (93) (100) (103)Date of Database

In February 2002, 2123 days (or 5.8 yrs) with >100 datasets every day !

Examples of IMAGE Data Products

Satellite Situation Center Web• SSCWeb finds & characterize times of interesting s/c configurations

– Multi-mission spacecraft positions both definitive and predictive, in a science framework, to answer science questions

• ~30 current space physics missions maintained in the database

• Geophysical coordinate systems, geophysical models, conjunctions among s/c & ground stations

• Products include graphics & listings– Science planning & data

understanding

– Web-accessible to the international science community

Current SSCWeb Usage

0100020003000

400050006000

Jul-1996Jan-1997Jul-1997Jan-1998Jul-1998Jan-1999Jul-1999Jan-2000Jul-2000Jan-2001Jul-2001Jan-2002

Monthly Usage

Plots

Listings & Queries

Web "Hits" *0.1

Geotail ,

IMP-8 , &

Interball

(Tail) in

Jan-Feb98FAST Magnetic-Mapped Orbit:

1998 Jan 1, near Tromso Norway

IMAGE Engineering Data• Over 150 instrument and s/c

parameters are written as CDF files and treated just like the science data

• Processed daily delivered to the NSSDC for archiving and loaded into CDAWeb

• Use CDAWeb to perform all standard trend analysis

• Sustaining engineering can be done anyway, only access to the internet is needed

• No specialized system needed!

Summary • Entire mission data system needs to be part of the mission

design and a part of any mission trade study

• Science data processing should be success oriented - automate it, test it, implement it, and assume it works – Scientists look at it daily and if it doesn’t work we know about it

immediately

– QC done by the scientists no operations personnel used

• Look at all aspects of consolidation, cross training, automation– Using current technology for automation based on a cost-benefit

analysis

• Every year the mission is extended saves money for doing science

Backup

Workshop Goals• Explore ways to deliver data to users in a more timely, cost effective

manner• Reduce the development time and costs associated with getting data

from missions to all users• Share methodologies across multiple projects• Activities:

– Explore new methodologies– Exchange new ideas for innovative approaches– Examine lessons learned – Identify areas for collaboration

• Topics:– Off the shelf science data processing sw tools – Improved and innovative cost-effective approaches to process & distribute – Common suites of ground data processing system designs and features – Improvements towards maximizing science data return– Challenges of on-board processing