1/9

Contribution of SAFE to XFDUContribution of SAFE to XFDUContribution of SAFE to XFDUContribution of SAFE to XFDUCCSDS/MOIMS/IPR Workshop

(11-15/04/2005 Athens)CCSDS/MOIMS/IPR Workshop

(11-15/04/2005 Athens)

Stéphane MBAYEGAEL Consultant

Stephane.Mbaye@gael.fr

2/9

RationaleRationale

SAFE Format Specifications 1.5 – [SAFE-FMT] ESA/CNES XFDU I/O Library – [XFDU-I/O] SAFE I/O library – [SAFE-I/O] XFDU Specifications – Issues and Proposals

3/9

SAFE Format Specifications 1.5SAFE Format Specifications 1.5[SAFE-FMT][SAFE-FMT]

4/9

SAFE - SAFE - Origin and goals (reminder)Origin and goals (reminder)

Standard Archive Format for Europe Provide ESA with a single format for all EO data

All Level 0 (Raw) and Level 1 Level 2 (as far as applicable) Ground Segment Auxiliary files)

Should become an ESA AIP Candidate for SIP and DIP (TBV) Shall be compatible with XFDU

5/9

SAFE – 111 Data types to be supportedSAFE – 111 Data types to be supported

ENVISAT (59)ASAR (23), AATSR (5), DORIS (3), GOMOS (7), MERIS (13), MIPAS (6), MWR (1), RA2 (8), SCIAMACHY (5)

ERS (25)RA (4), AMI (12), ATSR (3), MWR (1), GOME (2), PRARE (3)

Landsat (3)MSS (1), TM (1), ETM+(1), RBV (0)

JERS (3)OPS (1), SAR (2)

Terra/Aqua (1)MODIS (1)

NOAA (6)AVHRR/TOVS (6)

Orbview (5)SeaWifs (5)

NOAA (6)AVHRR/TOVS (6)

IRS P3 (1)MOS (1)

Nimbus (2)CZCS (2)

MOS (1)MESSR (1)

SPOT (3)HRV (1), HRVIR (1), HRG (1)

6/9

SAFE, a restriction of XFDUSAFE, a restriction of XFDU <!-- SAFE Root element type. The type of the SAFE root derives from XFDU root type. -->

<xsd:complexType name="safeType">

<xsd:complexContent>

<xsd:restriction base="XFDUType">

<xsd:sequence>

<xsd:element form="unqualified" name="packageHeader«  type="safe:packageHeaderType"

minOccurs="1« /> 

<xsd:element form="unqualified" name="informationPackageMap" type="safe:informationPackageMapType"

maxOccurs="1« /> 

<xsd:element form="unqualified" name="metadataSection" type="safe:metadataSectionType"

minOccurs="1« /> 

<xsd:element form="unqualified" name="dataObjectSection" type="dataObjectSectionType"

minOccurs="0« /> 

<xsd:element form="unqualified" name="behaviorSection" type="behaviorSectionType"

minOccurs="0« />

</xsd:sequence>

</xsd:restriction>

</xsd:complexContent>

</xsd:complexType>

 

<!-- SAFE Root element -->

<xsd:element name="safe" type="safe:safeType"/>

7/9

XML Schemas / Namespace policyXML Schemas / Namespace policy

http://www.esa.int/safe/1.0/core-1.0.xsd

http://www.esa.int/safe/1.0/passive-sensor-1.0.xsd

http://www.esa.int/safe/1.0/active-sensor-1.0.xsd

http://www.esa.int/safe/1.0

Specializations…

http://www.esa.int/safe/1.0/impl/xxx

8/9

contentUnitType

(from int.esa.safe)

+order:

+pdiID:

informationPackageMapType

(from int.esa.safe)

contentUnit+

1..*

contentUnitType

(from int.esa.safe)

+order:

+pdiID:

informationPackageMapType

(from int.esa.safe)

contentUnit+

1..*

SAFE Information Package Map SectionSAFE Information Package Map Section

One and only one Information Package Map Section

9/9

SAFE – Core Metadata ObjectsSAFE – Core Metadata Objects

Acquisition Period Object Platform Objects Processing Log Objects Quality Information Objects Orbit Reference Objects FrameSet Objects XML Schemas Objects

10/9

SAFE - SAFE - Acquisition Period ObjectAcquisition Period Object

acquisitionPeriodType

(from int.esa.safe)

+startTime:xsd:dateTime

+stopTime:xsd:dateTime

acquisitionPeriodType

(from int.esa.safe)

+startTime:xsd:dateTime

+stopTime:xsd:dateTime

11/9

platformType

(from int.esa.safe)

+familyName:xsd:string[0..1]

+number:xsd:positiveInteger[0..1]

instrumentType

(from int.esa.safe)

instrument+

platformType

(from int.esa.safe)

+familyName:xsd:string[0..1]

+number:xsd:positiveInteger[0..1]

instrumentType

(from int.esa.safe)

instrument+

SAFE – Platform ObjectSAFE – Platform Object

12/9

SAFE - Specialized Core ObjectsSAFE - Specialized Core Objects <!-- Platform description --> <metadataObject ID="platform" classification="DESCRIPTION" category="DMD"> <mdWrap textInfo="Platform Description" vocabluaryMdType="OTHER" otherMdType="SAFE" mimeType="text/xml"> <xmlData> <safe:platform> <!-- Platform identification --> <safe:nssdcId>2003-009A</safe:nssdcId> <safe:familyName>ENVISAT</safe:familyName> <!-- Instrument identification --> <safe:instrument> <safe:familyName>MERIS</safe:familyName> <meris:mode vcid="6" opsMode="4"/> </safe:instrument> </safe:platform> </xmlData> </mdWrap> </metadataObject>

13/9

SAFE – Orbit Reference ObjectSAFE – Orbit Reference Object

orbitReferenceType

(from int.esa.safe)

+startOrbit:xsd:positiveInteger

+stopOrbit:xsd:positiveInteger

+cycle:xsd:positiveInteger

+phase:xsd:positiveInteger

+startTrack:xsd:integer[0..1]

+stopTrack:xsd:integer[0..1]

orbitReferenceType

(from int.esa.safe)

+startOrbit:xsd:positiveInteger

+stopOrbit:xsd:positiveInteger

+cycle:xsd:positiveInteger

+phase:xsd:positiveInteger

+startTrack:xsd:integer[0..1]

+stopTrack:xsd:integer[0..1]

14/9

SAFE – Processing Log ObjectSAFE – Processing Log Object

15/9

<!-- Quality information of the measurements --> <metadataObject ID="measurementQualityInformation" classification="DESCRIPTION" category="DMD"> <mdWrap textInfo="Quality Information" vocabluaryMdType="OTHER" otherMdType="SAFE" mimeType="text/xml"> <xmlData> <safe:qualityInformation> <!-- 24 records are missing from after the 1233rd record --> <safe:missingUnits> <safe:location>mdsr[1234]</safe:location> <safe:count>24</safe:count> </safe:missingUnits> <!-- The records 456 to 468 have a bad application ID --> <safe:corruptedUnits> mdsr[456 to 468]/isp/pkHd/PkId/appId </safe:corruptedUnits> </safe:qualityInformation> </xmlData> </mdWrap>

</metadataObject>

SAFE – Quality InformationSAFE – Quality Information

qualityInformationType

(from int.esa.safe)

+corruptedUnits:xsd:string[*]

qualityInformationType

(from int.esa.safe)

+corruptedUnits:xsd:string[*]

Use of XPath/XQuery foridentifying the corrupted/

Missing areas of theData Objects

16/9

SAFE – Frame SetsSAFE – Frame Sets

frameSetType

(from int.esa.safe)

frameType

-orbit :int

-track:int

-number:int

-footPrint:gml:LinearRing

frame+

1..*

cloudVoteNotationSystem

-ceil:xs:double

-floor :xs:double

cloudVoteNotationSystem+

tile

-row:int

-column:int

-cloudVote:xs:double

##other:xs:any

tile+

*

frameSetType

(from int.esa.safe)

frameType

-orbit :int

-track:int

-number:int

-footPrint:gml:LinearRing

frame+

1..*

cloudVoteNotationSystem

-ceil:xs:double

-floor :xs:double

cloudVoteNotationSystem+

tile

-row:int

-column:int

-cloudVote:xs:double

##other:xs:any

tile+

*

17/9

SAFE – XML Schema ObjectsSAFE – XML Schema Objects

All Data Objects shall have an associated XML Schemas Post-processing (sub-setting, reformatting, …) Annotation (quality information, cloud cover) Indexing Documentation / Presentation

18/9

SAFE - Preservation of Binary ObjectsSAFE - Preservation of Binary Objects

Annotated XML Schemas Structured Data File (SDF) markups

<xsd:complexType> <xsd:sequence> <xsd:element name="record" type="xs:int" minOccurs="0" maxOccurs="unbounded"> <xsd:annotation> <xsd:appinfo> <sdf:block> <sdf:occurrence query="../header/numRec + 1"/> <sdf:length unit=‘bit’>12</sdf:length> </sdf:block> </xsd:appinfo> </xsd:annotation> </xsd:element> </xsd:sequence> <xsd:complexType/>

<xsd:complexType> <xsd:sequence> <xsd:element name="record" type="xs:int" minOccurs="0" maxOccurs="unbounded"> <xsd:annotation> <xsd:appinfo> <sdf:block> <sdf:occurrence query="../header/numRec + 1"/> <sdf:length unit=‘bit’>12</sdf:length> </sdf:block> </xsd:appinfo> </xsd:annotation> </xsd:element> </xsd:sequence> <xsd:complexType/>

19/9

SDF – Hierarchy of data blocksSDF – Hierarchy of data blocks

20/9

SDF - Location of a block within the streamSDF - Location of a block within the stream

21/9

SDF - Physical representation of a SDF - Physical representation of a nodenode

22/9

SDF - Physical representation of a SDF - Physical representation of a complex contentcomplex content

23/9

SDF - MarkupsSDF - Markups

24/9

SDF - Example of Binary interpreter SDF - Example of Binary interpreter (Derby)(Derby)

Detailed content of an ENVISAT MERIS

telemetry Source Packet(CCSDS standard)described by SDF

25/9

Active Sensors Active Sensors Metadata ObjectsMetadata Objects

Pulse Repetition Frequency Polarization Pulse Length

26/9

SAFE Behavior ObjectsSAFE Behavior Objects

No SAFE Behavior Object has been defined Some may be added for configuring Stitching? Should be investigated with CNES

27/9

XFDU I/O library – [XFDU-I/O]XFDU I/O library – [XFDU-I/O]

28/9

XFDU I/O - General architectureXFDU I/O - General architecture

29/9

Getting XFDU I/O library –Web siteGetting XFDU I/O library –Web site

http://www.gael.fr/xfdu

30/9

XFDU I/O Basic principlesXFDU I/O Basic principles

Based on interactions with the MANIFEST file Each interaction is not automatically comitted (possible

rollback): the result shall be explicitely saved 100% Java C++ Wrapper based on Java Native Interface (JNI) No support of Behavior section (to be investigated with

CNES)

31/9

XFDU I/O Implemented UsecasesXFDU I/O Implemented Usecases Creation of a new XFDU packet Opening of an existing XFDU packet Removal of a XFDU packet Moving a XFDU packet (with or without the attached objects) Management of the Content Units Management of Metadata Objects (only XML wrap) Management of Data Objects (only Flocat references) Browse Data Object Content (if associated XML Schema) Validation of XFDU packet Dumping XFDU Manifest in XML

32/9

XFDU I/O – Configuration StatusXFDU I/O – Configuration Status

Configuration identifiers: safe-1-0-beta-2: first functional release safe-1-0-beta-1: API only

Package identifier: esa.xfdu (TBV CNES)

33/9

SAFE I/O library – [SAFE-I/O]SAFE I/O library – [SAFE-I/O]

34/9

SAFE I/O - General architectureSAFE I/O - General architecture

Data Request Broker

XFDU API (ESA)

SAFE I/O Java

SAFE I/O C++ Wrapper

Java Apps C++ Apps

Data Request Broker

XFDU API (ESA)

SAFE I/O Java

SAFE I/O C++ Wrapper

Java Apps C++ Apps

35/9

Getting SAFE I/O library – SAFE Web siteGetting SAFE I/O library – SAFE Web site

http://www.gael.fr/safe

36/9

SAFE I/O Basic principles (Cont’)SAFE I/O Basic principles (Cont’)

Delegates standard XFDU access to ESA XFDU I/O Based on interactions with the MANIFEST file Each interaction is not automatically comitted (possible

rollback): the result shall be explicitely saved

37/9

SAFE I/O Implemented UsecasesSAFE I/O Implemented Usecases Creation of a new SAFE packet Opening of an existing SAFE packet Removal of a SAFE packet Moving a SAFE packet (with or without the attached objects) Management of the Content Units Management of the Core Metadata Objects Adding specialized Metadata Management of Data Objects Extraction or pointing to a subpart of a Data Object Identifying missing units and corrupted areas Browse Data Object Content Validation of SAFE packet Dumping SAFE Manifest in XML

38/9

SAFE I/O – Configuration StatusSAFE I/O – Configuration Status

Configuration identifiers: safe-1-0-release : an official release of version 1.0 safe-1-0-rc2 : second release candidate of version 1.0 safe-1-0-rc1 : a release candidate of version 1.0 safe-1-0-beta-1: unconsolidated distribution of version 1.0

Package identifier: esa.safe

39/9

XFDU SpecificationsXFDU SpecificationsIssues and ProposalsIssues and Proposals

40/9

XFDU – Environment InformationXFDU – Environment Information

What is teh exact definition of the environmentInfo/@specVersion attribute?

May the environmentInfo contain package history (e.g. SAFE processing logs)?

41/9

XFDU – Information Package MapXFDU – Information Package Map

What is the presice role of contentUnit/@order ? GAEL understanding: user defined order CNES understanding: package sequence number?

A Map section ?shall? Reference all metadata/data objects of the package?

Meaning of multiple Map sections?

42/9

XFDU – Single file packagesXFDU – Single file packages

For DIPs ESA requests single file products ZIP, Jar or other standard archive format are not

suitable for large data sets (e.g. 5 Gb for ENVISAT-ASAR format)

ESA proposes to append binary data objects to the MANIFEST file (not fully compliant with XML 1.0 or XML 1.0)

43/9

XFDU – How to design restrictions?XFDU – How to design restrictions?

The XML Schema requires global elements and attributes for restricting types in a different target namespace (E.g. SAFE)

May xfdu.xsd propose more global element for that purpose?

Should implementers redefine the xfdu.xsd? May the XFDU specification provide a

clarification/recommendation on that subject?

44/9

XFDU – Metadata/Data Object referencesXFDU – Metadata/Data Object references

metadataObject/@mdRef =? bytestream/@FLocat metadataObject/@mdWrap =? bytestream/@FContent May these differences be homogeneized?

45/9

XFDU – ChecksumsXFDU – Checksums

Should the CRC moved or add to the Flocat element (i.e. one CRC per file)?

What is the difference between dataObject/@checksum and bytestream/@checksum?

46/9

XFDU – Reference to XLinkXFDU – Reference to XLink

Requires an xlink.xsd in the same directory as the xfdu.xsd

References http://www.w3.org/TR/XLink ? http://www.w3.org/1999/XLink ?

Where to download the right XML Schema?

47/9

XFDU - TyposXFDU - Typos

vocabluaryMdType ? Comments of all items of the enumerations: e.g.EAD? mdSecType may be renamed metadataSectionType?

48/9

Thank you for your attention !Thank you for your attention !

49/9

XFDU – XML Schema OverviewXFDU – XML Schema Overview

50/9

XFDU - XFDU - Package HeaderPackage Header

51/9

XFDU - XFDU - Information Package Map SectionInformation Package Map Section

52/9

XFDU - XFDU - Metadata Object SectionMetadata Object Section

53/9

XFDU - XFDU - Data Object SectionData Object Section

54/9

XFDU - XFDU - Behaviour Object SectionBehaviour Object Section