AMSR-E SIPS Processing Status

Presented by Helen ConoverInformation Technology and Systems Center

at the University of Alabama in Huntsville

AMSR-E Joint Science Team MeetingSept 11-12, 2012

Portland, OR

hconover@itsc.uah.edu www.itsc.uah.edu ghrc.nsstc.nasa.gov 256.961.7807

2

Outline

• Introduction• Sea Ice and Sea Ice Drift

Updates• Software Updates for

Reprocessing– Streamlined Processing

Automation– Browse Image Updates– Provenance Collection

• End of Mission Plan– Reprocessing Plan– Data Preservation RequirementsAMSR-E Joint Science Team Meeting

Portland, OR Sept. 11-12, 2012

3

AMSR-E SIPS Personnel

• NASA MSFC: H. Michael Goodman, PI• UAHuntsville ITSC (Staff shared with GHRC DAAC):

Management: Dr. Sara Graves, Director, ITSCDevelopment: Helen Conover, Lead

Bruce Beaumont Mary Nair

Engineering: Kathryn Regner Sandra Harper Data Management: Lamar Hawkins Sherry HarrisonSystems Administration: Will Ellett, Lead Michele Garrett

AMSR-E Joint Science Team Meeting Portland, OR Sept. 11-12, 2012

AMSR-E SIPS Processing and Dependencies

4

Level-2A Brightness Temperatures(with Metadata and QA files)

L2B Ocean(1/2-orbit)

L2B Land(1/2-orbit)

L2B Rain(1/2-orbit)

L3 Ocean(Daily)

L3 Land(Daily)

L3 Snow(Daily)

L3 Sea Ice(Daily, 3 Res)

L3 Snow(Pentad)

L3 Ocean(Weekly)

L3 Ocean(Monthly)

L3 Rain(Monthly)

L3 Snow(Monthly)

Once per day

Once every

five days

Once per week

Once permonth

ProductGeneration

at SIPS-GHRC

-------------~ 29 times

per day

Level-2A Brightness Temperatures

L3 Sea IceDrift

~ 29 times per day at SIPS-RSS

AMSR-E Joint Science Team Meeting Portland, OR Sept. 11-12, 2012

5

Algorithm DAP Version Summary

Algorithm Short Name Current Version

L2A Tb V11

L2B Land V10

L2B Ocean V07

L2B Rain V11

L3 Land V07

L3 Ocean V05

L3 Rain V07

L3 Sea Ice V14

L3 Sea Ice Drift B03

L3 Snow V10

AMSR-E Joint Science Team Meeting Portland, OR Sept. 11-12, 2012

6

Sea Ice and Sea Ice Drift

• Reprocessing Sea Ice V14 from beginning of mission with new land mask.

• Processing Sea Ice Drift from beginning of mission with B03 algorithm, using Sea Ice V14 as input

• Full mission processing of all Sea Ice products should be complete Spring 2013

AMSR-E Joint Science Team Meeting Portland, OR Sept. 11-12, 2012

SIPS Software Updates

In place for SI/SID reprocessing:• Refactored full suite of processing automation

software to streamline system for reprocessing• Updated browse image process

– New 24-bit PNG images with embedded metadata (e.g., source data, parameter, date/time, projection)

– KMZ files suitable for display on map or globe– Traditional “framed” image with title, color bar and

annotations

To be in place for L2A and subsequent reprocessing:• Incorporating provenance capture utilities

Provenance logging added to processing automation; ready for integration and test

Lineage metadata and DOIs to be included in data files; design complete, development in progressAMSR-E Joint Science Team Meeting

Portland, OR Sept. 11-12, 2012 7

AMSR-E Approach to Provenance / Lineage

• Why worry about data provenance and lineage?– Important to understand how a data product was

generated– Data Preservation Requirements from ESDIS

• Specification document from Rama and John Moses includes algorithm information, processing history, input data

• AMSR-E approach: Incorporating results from NASA-sponsored provenance research into production– ISO-compliant lineage information to be included in data

files• Incorporating lineage information into the data files as

additional metadata means that information stays with the files 

• Current push toward ISO-compliant metadata for NASA ES data

– Provenance Browser will provide easy access to • Browse imagery • Basic algorithm information with pointers to full

documentation• Data lineage and processing history

AMSR-E Joint Science Team Meeting Portland, OR Sept. 11-12, 2012

8

AMSR-E Joint Science Team Meeting Portland, OR Sept. 11-12, 2012

9

AMSR-E Provenance BrowserModes of Exploration

• Browse list by product type, file name– Mid-winter Sea Ice data was incorrect after reprocessing. Look

at January data to determine what was different about this processing run.

• Browse images by product type, parameter/variable– Scan series of images to look for possible problems, e.g.,

incomplete data for 2011-10-04

• Search by file name– Question about a particular data file that doesn’t look right.

Search by file name; may discover that this was a problem file later replaced, and get new file.

• View general processing information– For information about how files from a particular product are

generated, view general processing graph and get pointers to more detailed documentation.

http://provenance.itsc.uah.edu/

AMSR-E Joint Science Team Meeting Portland, OR Sept. 11-12, 2012

10

AMSR-E Provenance Browser

Browse by Product Type

Browse by Product Images

Search by File Name

Explore Provenance

http://provenance.itsc.uah.edu/

AMSR-E Joint Science Team Meeting Portland, OR Sept. 11-12, 2012

11

http://provenance.itsc.uah.edu/

AMSR-E Provenance Browser (2)

Processing Graph

Algorithm Information

Granule Metadata

Parameters and Algorithms

AMSR-E Joint Science Team Meeting Portland, OR Sept. 11-12, 2012

12

AMSR-E Provenance Browser (3)

Algorithm Used

Parameter Selected

http://provenance.itsc.uah.edu/

13

Outline

• Introduction• Sea Ice and Sea Ice Drift

Updates• Software Updates for

Reprocessing– Streamlined Processing

Automation– Browse Image Updates– Provenance Collection

End of Mission Plan– Reprocessing Plan– Data Preservation RequirementsAMSR-E Joint Science Team Meeting

Portland, OR Sept. 11-12, 2012

14

SIPS Reprocessing Planning

AMSR-E Joint Science Team Meeting Portland, OR Sept. 11-12, 2012

15

Data Preservation Requirements

• Data Preservation Goals – No loss (or loss within scientifically

acceptable bounds) as bits move across systems as well as over time

– Readability over time– Long-term understandability – Repeatability of previously obtained

results • Without a standard, missions may not fully

plan for preservation of required content Content to be preserved is held by

various organizations during mission’s life. They must provide essential information for archiving before end of their involvement.

Diverse types of supporting information and data may be well understood by originator’s community, but gaps in a-priori knowledge are difficult or impossible to fill once the originator has moved on.

http://earthdata.nasa.gov/about-eosdis/requirements

AMSR-E Joint Science Team Meeting Portland, OR Sept. 11-12, 2012

Categories of Content to be Preserved

1. Preflight/Pre-Operations: Instrument/Sensor characteristics including pre-flight/pre-operations performance measurements; calibration method; radiometric and spectral response; noise characteristics; detector offsets

2. Science Data Products: Raw instrument data, Level 0 through Level 4 data products and associated metadata

3. Science Data Product Documentation: Structure and format with definitions of all parameters and metadata fields; algorithm theoretical basis; processing history and product version history; quality assessment info

4. Mission Data Calibration: Instrument/sensor calibration method (in operation) and data; calibration software used to generate lookup tables; instrument and platform events and maneuvers

5. Science Data Product Software: Product generation software and software documentation

6. Science Data Product Algorithm Input: Any ancillary data or other data sets used in generation or calibration of the data or derived product; ancillary data description and documentation

7. Science Data Product Validation: Records, publications and data sets

8. Science Data Software Tools: product access (reader) tools.WHAT ELSE IS IMPORTANT FOR LONG TERM PRESERVATION OF AMSR-E DATA???

17

Data Preservation Checklist

Preflight / Pre-Operations Calibration

Instrument Description High-level description on NSIDC web siteMore details available?

Preflight / Pre-Ops Calibration Data

Science Data Products

Raw and Derived Products At NSIDC DAAC: L0, L1 (JAXA), L2 & L3 (US) products

Metadata At NSIDC DAAC

Mission Calibration Data*

Calibration Method

Calibration Data

* Documentation of in-line changes to calibration or to instrument or platform operations or conditions that occur throughout the mission.

AMSR-E Joint Science Team Meeting Portland, OR Sept. 11-12, 2012

18

Data Preservation Checklist (2)

Science Data Product Documentation

Product Team Included on NSIDC product pages

Product Requirements and Designs

Original ATBD?

Processing and Algorithm Version History

• NSIDC “versions” web page• Provenance information

embedded in files at final reprocessing

Product Generation Algorithm Documentation

• Science software• Updated ATBD (all products?)• More detail needed?

Product Quality Documentation • NSIDC product web pages• Data quality flags

Product Application Published articles about use of data, user feedback

AMSR-E Joint Science Team Meeting Portland, OR Sept. 11-12, 2012

19

Data Preservation Checklist (3)

Other Data, Software and Documentation

Science Data Product Software and Documentation

Delivered Algorithm Package

Science Data Product Algorithm Inputs

• Ancillary data included in DAP• All inputs (AMSR-E and ancillary)

referenced in provenance metadata

Science Data Product Validation: Datasets and Documentation

• Validation data and metadata at NSIDC

• Cal/val plans• Detailed history of

activities• List of published papers

Science Data Software Tools HDF-EOS and related tools

Deliverables Checklist This is a start

AMSR-E Joint Science Team Meeting Portland, OR Sept. 11-12, 2012

20

Summary

• Ready to begin full mission reprocessing with– Same or new science algorithms– Updated browse imagery– ISO Lineage metadata and DOIs included in

data files

• AMSR-E SIPS will be one of the first to implement ESDIS Data Preservation requirements as part of end of mission activities

AMSR-E Joint Science Team Meeting Portland, OR Sept. 11-12, 2012

Backup

TLSCF ScienceSoftware I&T SIPS-GHCC

PO.DAAC

JAXA-EOCL1A Product Generation

SIPS-RSSL2A Product Generation

L2B and L3 Product Generation(science, metadata, qa, ph, subsets, browse)

L2 and L3algorithms

L2A algorithm

NSIDC DAAC Data Archive and Distribution

Level-1

Level-2 and Level-3

Level-1

Level-1

NASAEDOS

GBAD (Pre Ephem) & PDS

GroundStations

L2B and L3 Algorithms

Level-0

Product Delivery Record Server (PDRS)

Algorithm Teams

Ocean Sea Ice

Rain

Land

Snow Data

NASA Flt Dynamics

GBAD(Def Ephem)

AMSR-E Data Flow

AMSR-E Joint Science Team Meeting Portland, OR Sept. 11-12, 2012

23

SIPS-GHCC HardwareOperational Configuration (Dec

‘10)

Near line storage of all Level-2A and most recent Level-2B and Level-3

Restricted FTP Server

Multiple processing environments across three machines results in much greater flexibility for processing and

reprocessing.

Sun Storage 7310With J4400 Disk

Array10TB NAS

Pri

vate

Netw

ork

(1 G

B/s

ec)

Forward Processing

Reprocessing

Dev, Integration & Test

Sun Fire x4270 x642 x 2.93 GHz Sun Intel Xeon Processors

4 x 2GB DIMM , RHEL 5AMSR 1-3

AMSR-E Joint Science Team Meeting Portland, OR Sept. 11-12, 2012