Improving Access to Biological Observations

data Dr. Hassan Moustahfid

NOAA US IOOS Program OfficeSilver Spring, MD

NMFS FMAC F2F Brief 08-16-2013

Outline

• IOOS BDP implementation • IOOC BIO TT• Animal Telemetry Data

Development• Acoustics Obs Active and Passive

2

1. Temperature2. Salinity3. Water level4. Currents5. Surface Waves6. Surface Winds7. Ocean color8. Dissolved oxygen9. pH10.pCO2

11.Heat flux12.Bottom character13.Pathogens

IOOS® Core Variables

list could be expanded based on identified needs...

14.Bathymetry15. Ice distribution16.Contaminants17.Stream flow18.Dissolved nutrients19.Optical properties20.Total suspended matter21.Colored dissolved organic matter22.Fish species23.Fish abundance24.Zooplankton species25.Phytoplankton species26.Zooplankton abundance

3

4

Interoperability of Original Data: IOOS Terminology

Biological Data Contents

Observation(e.g. Point Data)

Species Presence - Absence and Species Abundance

• Record Identity / Citation

• Spatial-Temporal-Reference

• Taxon / Identification

• Observation Details

• Sampling Details

• Details

• Record

Semantics

IOOS Biological Data Terminology

Data Mapping

Dublin Core

IOOS Terminolog

y

Original Data

“Mapping Document”

•Align originator definitions with IOOS terms

“Fallout” and Resolu-tion

Web Service

Darwin Core

IOOS-specific terms

Metadata Development

CF

Courtesy of Philip Goldstein (OBIS-USA)

Metadata Standards, Content and Access

Metadata Access

FGDCDocument of Record

Metadata Contents

FGDC is the Primary Format•ISO and ncISO Supported•Multiple Interface Methods

• Identification / Attribution

• Discovery• Understanding• Referral• History / QA• (Details of

Mapping to Terminology)

• Content Standard

• PurposeNative FGDC xml via http://

Converted ISO xml via http://

ncISO on ERDDAP

FGDC & ISO; html, xml, text

Tools transform FGDC for other formats and methods of access; tools by project team and external.

Credit to OBIS-USA, PIFSC , NPS, PMNM and PacIOOS teams

Metadata: information used to document and interpret data

IOOS BDP data conventions and services

• Metadata conventions and exchange standards- Data Model XML- http://www.ioos.gov/schema/ioosbiology/1_0/

ioos_biological_terminology20120608v1point0.xsd- Html version- http://www.ioos.gov/schema/ioosbiology/1_0/

ioos_biological_terminology20120608v1point0.xml-Data Access services

Metadata http://oos.soest.hawaii.edu/cgi-bin/get_metadata.pl?

id=PACN_FISH_TRANSECT&format=fgdcOr http://oos.soest.hawaii.edu/erddap/info/index.html?page=1&itemsPerPage=1000

5US IOOS BDP ATN TOPP POST ACOUSTIC CHALLENGES

Two parallel activities: -Implementation in IOOS Gulf of Mexico and exposing SEFSC and FWC FIM Fisheries data (Different sampling gears , Trawl and >1200 species) GCOOS ERDDAP: http://barataria.tamu.edu/erddap/info/index.html?page=1&itemsPerPage=1000-Implementation in IOOS Southeast and exposing MARMAP data – SCDNR and SEFSCSECOORA ERDDAP: http://129.252.139.124/erddap/info/index.html?page=1&itemsPerPage=1000,-NCDDC team is working closely with us to develop FGDC and ISO metadata.

6

FY12-FY13 Implementation in IOOS regions

IOOS SYSTEM ARCHITECTURE FOR

BIOLOGICAL DATA

SERVICE REGISTRYMetadata Harvester

DATA CATALOG

DATA VIEWER

US IOOS PORTAL

DATA.GOVOBIS

Others data aggregators

DATA USERS

NODC

EnrollmentFormatting to IOOS DMACStandards

9

Enhancing NMFS data standardization and integration

NMFS Data CatalogMetadata Harvester

Data Viewer Based on

Google Maps etc.

RDMS

In each NMFS SC or RO DMS

Web services

Data Encoding Or Configuring Following IOOS

DMAC standards

Data.Gov

IOOS Catalog

Animal Telemetry Observations Data Content

and Access Standards

AAT Observations System Design – Service Connections. Access to all data via ERDDAP: RDBMS > netCDF files (CDM) > ERDDAP.

Acoustic Metadata Convention for

Archiving

• Why? – Providing a central data depository so NOAA/NMFS scientists can safeguard and share their research data.

• How? – Work with data holders (e.g., NMFS Fisheries Science Centers) and NGDC

• Where? – NGDC database archiving system

Type of NMFS acoustic data

• Passive acoustics• Active fisheries acoustic• Multibeam, sidescan etc.

Working Plan for Acoustic Data Archiving:The First 3 Months

• Contact NGDC to discuss the concept of metadata standards for passive acoustic data archiving

• Work with NMFS Science Centers to finalize archiving plan that under NGDC data standards

• Pilot project to implement archiving of subset of acoustic data (if resources allow)

Which data model fit this type of data?

16

NetCDF (Network Common Data Form) may be a better format for storage of final acoustic output data.Sv (Acoustic Volume Backscatter) final data file is well suited to array-oriented acoustic data.Further, metadata contained in the NetCDF files can be automatically harvested to facilitate ready discovery of the data.

17

 

Example. Data variable for volume backscatter and ancillary data quality variable

Sv Size: 120x2154 Dimensions: DEPTH,TIME Datatype: double Attributes: name = 'Sv' long_name = 'mean_volume_backscatter_coefficient' units = 'm-1' _FillValue = 1e+04 valid_min = 0 valid_max = 1 ancillary_variables = 'Sv_quality_control'

Sv_quality_control Size: 120x2154 Dimensions: DEPTH,TIME Datatype: int8 Attributes: long_name = 'quality flag for mean_volume_backscatter_coefficient' quality_control_set = 1 quality_control_conventions = 'IOOS standard flags' _FillValue = 99 valid_min = 0 valid_max = 9 flag_values = [0 1 2 3 4 5 6 7 8 9]

flag_meanings = 'No_QC_performed, Good_data, Probably_good_data, Bad_data_that_are_potentially_correctable, Bad_data Value_changed, Not_used, Not_used, Not_used, Missing_value

18

Metadata Convention for Passive Acoustic Recordings (PAM data)

19

Metadata Convention for Active Acoustic, Fisheries Ecosounder, Sonar

A Clear View of Tomorrow

IOOS data is helping the west coast shellfish industry mitigate the effects of ocean acidification

Thank You

Hassan Moustahfid, PhD.NOAA, US IOOS

www.noaa.ioos.gov