EUFAR - EUropean Facility for Airborne Research
Workshop on Research Metadata in Context –
Sept 7/8, Nijmegen, Netherlands
Metadata Frameworks in EUFARMatt Freer, EUFAR
Overview
EUFAR Intro.
Airborne Metadata• Challenges
• Schemes
EUFAR Conv.• External
• Embedded
Summary
Overview
EUFAR Introduction
Airborne Science Metadata Overview
EUFAR Implementation
Summary
EUFAR - EUropean Facility for Airborne Research
EUFARIntegrating Activity of the EC FP7
Budget 8 M€ Duration 4 years (2008-2012) 33 Partners
7 instruments and 22 aircraft open to Trans-national Access
www.eufar.net
Overview
EUFAR Intro.
Airborne Metadata• Challenges
• Schemes
EUFAR Conv.• External
• Embedded
Summary
EUFAR Introduction
• Jülich (DE)
• ULEI (DE)
• STFC (UK)
• USZ (HU)
• UCAM-DCHEM (UK)
• UWAR (PL)
• COSINE (NL)
• IRSN (FR)
• COMAT (FR)
• VKI (BE)
• UZH (CH)
• WU (NL)
• USBE (CZ)
• TAU (IL)
• GFZ (DE)
• PML (UK)
• PTB (DE)
• ONERA (FR)
• Météo-France (FR) Coordinator
• MetOffice (UK)
• DLR (DE)
• NLR (NL)
• Enviscope (DE)
• CNRS (FR)
• NERC (UK)
• INTA (ES)
• FUB (DE)
• KIT (DE)
• AWI (DE)
• CNR (IT)
• UNIMAN (UK)
• VITO (BE)
14 aircraft or instruments operatorsand
18 experts in airborne measurements
Overview
EUFAR Intro.
Airborne Metadata• Challenges
• Schemes
EUFAR Conv.• External
• Embedded
Summary
EUFAR Introduction
EUFAR objectives:
Provide scientists with access at equal terms to the most complete range of research infrastructuresDevelop trans-national access to national infrastructuresReduce redundancy and fill the gapsPromote the use of research infrastructures, particularly among young scientists from countries where such infrastructures are lackingStrengthen expertise through exchange of knowledge, development of standards and protocols, construction of databases and joint instrumental research activities
Overview
EUFAR Intro.
Airborne Metadata• Challenges
• Schemes
EUFAR Conv.• External
• Embedded
Summary
Airborne Metadata Overview
Aircraft in EUFAR measure large range of atmospheric data:• Aircraft parameters
– Altitude/Lat/Long– Roll/Pitch/Yaw
• State parameters– Temperature– Pressure– Humidity
• Chemistry– CO2– O3– NOx
• Microphysics– Water content– Ice crystal habits – Particle concentration– Particle size distributions
• Radiation– Long wave– Short wave
• Winds• Hyperspectral imaging
– Land use– Reflectivity
Overview
EUFAR Intro.
Airborne Metadata• Challenges
• Schemes
EUFAR Conv.• External
• Embedded
Summary
Airborne Metadata Overview
Challenges:Heterogeneity:
• Instruments• Measurements• Data systems• Processing techniques• Data streams
Collaborations on projects, research and data production between varied countries and institutions (Europe, USA, etc)Many different levels of data (raw, intermediate, processed, final, etc)
All of these necessitate clearly defined, robust protocols for describing data
Overview
EUFAR Intro.
Airborne Metadata• Challenges
• Schemes
EUFAR Conv.• External
• Embedded
Summary
Airborne Metadata Overview
External metadataGives users broad overview of data
• Location• Project• Responsible party• Quality
Aids in data classification, search, retrieval, archival
Embedded metadataMakes files self-describingCan define metadata globally or per-variableReduces errors in usageHelps automate data processing tasksStores logistical information (calibration, maintenance and other historical sensor information)
Overview
EUFAR Intro.
Airborne Metadata• Challenges
• Schemes
EUFAR Conv.• External
• Embedded
Summary
EUFAR Metadata Conventions
Draft version of proposed conventions has been created and circulated to members for commentAt the International Conference on Airborne Research in the Environment (ICARE) in Toulouse, Oct 2010, version 1.0 of these standards will be finalizedEUFAR proposes:
External• INSPIRE
Embedded • NetCDF• CF Conventions• IWGADTS
Overview
EUFAR Intro.
Airborne Metadata• Challenges
• Schemes
EUFAR Conv.• External
• Embedded
Summary
External Metadata – INSPIRE Directive
Infrastructure for spatial information in Europe (INSPIRE)Directive entered into force May 2007 by the European Commission
Ensures compatibility and usability of spatial data infrastructures with the adoption of common implementing rules (e.g. Metadata)Implementation rules available at http://inspire.jrc.ec.europa.eu
INSPIRE PrinciplesData should be collected only once and kept where it can be maintained most effectivelySpatial information should be combined into seamless productData should be scalable to multiple levels – detailed enough for investigation, general enough for strategic purposesGeographic information should be readily available at all levelsAvailable data, metadata and usage conditions should be clear and easy to find
Overview
EUFAR Intro.
Airborne Metadata• Challenges
• Schemes
EUFAR Conv.• External
• Embedded
Summary
INSPIRE Metadata elements
INSPIRE Metadata regulation mainly based on ISO 19155-2Includes Information on
Data set or seriesClassification (scientific field)Geographic locationTemporal referenceQuality and validityConformityConstraints related to access and useThe responsible organisation
Descriptions of different image and instrument processing steps
List has been expanded within EUFAR by additionalquality indicators
Overview
EUFAR Intro.
Airborne Metadata• Challenges
• Schemes
EUFAR Conv.• External
• Embedded
Summary
INSPIRE Metadata rules implementation into EUFAR (1)
Identification1. Resource title: EUFAR project acronym2. Resource abstract: EUFAR abstract3. Resource type: dataset or series4. Resource locator: www.eufar.net5. Unique resource identifier: EUFAR file name specification6. Coupled resource: not applicable7. Resource language: eng
Additional:List of related available filesPlatform: aircraft all signInstrument: list of instruments incl. basic sensor characteristics (e.g. spectral coverage)
Overview
EUFAR Intro.
Airborne Metadata• Challenges
• Schemes
EUFAR Conv.• External
• Embedded
Summary
INSPIRE Metadata rules implementation into EUFAR (2)
1. Classification• Topic category: EUFAR main scientific field• Spatial data service type: not applicable
2. Keyword• Keyword value: EUFAR scientific theme• Originating controlled vocabulary: GEMET - INSPIRE themes,
version 1.0 2008-06-01
3. Geographic Location• Geographic bounding boxAdditional:
Area information: Country, Province/Region, Local name
4. Temporal Reference• Temporal extent: date of data acquisition • Date of publication: date of entry into EUFAR-DB• Date of last revision: date of last revision• Date of creation: date of data processing
Overview
EUFAR Intro.
Airborne Metadata• Challenges
• Schemes
EUFAR Conv.• External
• Embedded
Summary
INSPIRE Metadata rules implementation into EUFAR (3)
Quality and Validity1. Lineage2. Spatial resolution
Additional:Processing information (e.g. processing level)Calibration information (e.g. date of spectral calibration)Acquisition information (e.g. overall heading)Data quality layers for hyperspectral imagery
Sensor calibration and system correction Image data artefacts and processing errors GPS / IMU related errors, geometric correction Atmospheric correction and atmospheric conditions
quality control flags
hyperspectral image
attitude and position
black body information, …
Overview
EUFAR Intro.
Airborne Metadata• Challenges
• Schemes
EUFAR Conv.• External
• Embedded
Summary
INSPIRE Metadata rules implementation into EUFAR (4)
1. Conformity• Specification: INSPIRE implementing rules• Degree: degree of conformity (true or false)
2. Constraints related to access and use• Conditions applying to access and use: EUFAR consortium
agreement• Limitations on public access: EUFAR consortium agreement
3. Organisation responsible for the establishment, management, maintenance and distribution• Responsible party: [email protected]• Responsible party role: pointOfContact
4. Metadata on metadata• Metadata point of contact: e.g. DLR, [email protected], PAF• Metadata date: date when the metadata record was created
or updated• Metadata language: eng
Overview
EUFAR Intro.
Airborne Metadata• Challenges
• Schemes
EUFAR Conv.• External
• Embedded
Summary
Embedded metadata
Tangent: EUFAR recommended file format NetCDF (network Common Data Form)
Wide use in atmospheric science communityAPIs for most common programming languages (C, C++, Fortran, Java, Python, Matlab, IDL)NetCDF is:
• Binary• Portable• Scalable:
– Multi-dimensional array oriented– Subsets of large datasets easily accessible
AppendableSelf describing
Data model:DimensionsVariablesAttributes
Overview
EUFAR Intro.
Airborne Metadata• Challenges
• Schemes
EUFAR Conv.• External
• Embedded
Summary
Embedded metadata
Tangent: NetCDF (network Common Data Form)NetCDF is:
• Binary• Portable, i.e. machine independent• Scalable, I.e. multi-dimensional array oriented• Appendable• Self describing
Wide use in atmospheric science communityData model:
• Dimensions• Variables• Attributes
Overview
EUFAR Intro.
Airborne Metadata• Challenges
• Schemes
EUFAR Conv.• External
• Embedded
Summary
EUFAR NetCDF Metadata Conventions
EUFAR conventions for NetCDF extend existing, common standards:1. NetCDF Climate and Forecast (CF) Conventions
• Broad conventions, applicable across atmospheric and earth science
1. Interagency Working Group for Airborne Data and Telemetry Systems (IWGADTS) Conventions• US-based group• Tailored for aircraft
Overview
EUFAR Intro.
Airborne Metadata• Challenges
• Schemes
EUFAR Conv.• External
• Embedded
Summary
EUFAR NetCDF Metadata Conventions
Specifies file structureNames for time dimensions
• Time for 1Hz signal• Time25 for 25Hz signal• etc
Order of dimensionsDefinition of time coordinate – follow CF conventions
Defines standard file attributesConventionsTitleInstitutionSourceHistoryReferencesComment
Overview
EUFAR Intro.
Airborne Metadata• Challenges
• Schemes
EUFAR Conv.• External
• Embedded
Summary
EUFAR NetCDF Metadata Conventions
Defines standard variable attributesunitsLong nameStandard nameValid rangeSampled rateCategoryCalibration coefficientsDependenciesProcessor
Provides standard vocabularies based on CF Conventions
standard namesunits
Overview
EUFAR Intro.
Airborne Metadata• Challenges
• Schemes
EUFAR Conv.• External
• Embedded
Summary
EUFAR NetCDF Metadata Conventions
EUFAR NetCDF Metadata Conventions do not:Define or standardize variable names
Why?Unique and varied instruments and measurements between aircraft and institutionsCan be multiple measurements of the same quantity
Example: NCAR C130 Pressure measurements
Overview
EUFAR Intro.
Airborne Metadata• Challenges
• Schemes
EUFAR Conv.• External
• Embedded
Summary
NetCDF metadata rules for EUFAR
Example: NCAR C130 Pressure measurements PSFD
PSFDC
PSFRD
PSFRDC
PCAB
Solution: Data DiscoveryDefine global attribute ‘air_pressure’The value of this attribute indicates recommended fieldExample: reference:air_pressure = PSFDC
– Data discovery helps users and software choose correct measurement
All pressure: Which one to use?}
Overview
EUFAR Intro.
Airborne Metadata• Challenges
• Schemes
EUFAR Conv.• External
• Embedded
Summary
NetCDF metadata rules for EUFAR
• EUFAR standard data discovery fields– Highly recommended
• Latitude• Longitude• Altitude
– Recommended• Wind speed• Upward air velocity• Air temperature• Platform speed• Dew point temperature• Air pressure• Platform roll angle• Platform pitch angle• Platform orientation• Ground speed E/W• Ground speed N/S
Overview
EUFAR Intro.
Airborne Metadata• Challenges
• Schemes
EUFAR Conv.• External
• Embedded
Summary
Summary
• EUFAR conventions:– Proposal has been circulated– Version 1.0 being released in Oct 2010– External - INSPIRE
• General description• Archival • Search
– Embedded - NetCDF• Data usage• Processing• Accuracy