Caro-COOPSData and Information Management

in Support of Integrated Ocean Observing System Efforts

Part I: Where we are

Scientific Advisory Board Meeting

Columbia, SC 16-18 December 2003

What is Caro-COOPS?

A wholly-integrated system for coastal observations and their application to user needs

• An extensive array of instrumented moorings• A comprehensive data and information

management system• An advanced suite of integrated models

Scientific Advisory Board Meeting

Columbia, SC 16-18 December 2003

Scientific Advisory Board Meeting

Columbia, SC 16-18 December 2003

IOOS DMAC High-priority Recommendations

Data suppliers wishing to participate in IOOS immediately should adopt initial DMAC recommendations including (IOOS DMAC, 2003):

- Creating FGDC-compliant metadata- Enabling data discovery- Making data available through the

OpeNDAP data access protocol- Making data browsable on-line through the

LAS- Ensuring long-term archival of contributed

data

A comprehensive data management system

A comprehensive data and information management system consisting of three primary components:

1. Reception, management and transfer of primary and ancillary observation data

2. Delivery of QC’d data, metadata and data products to a Web-based data and information portal

3. Provision of user-friendly tools and information products for model output

Data and Information Management Subsystem

Scientific Advisory Board Meeting

Columbia, SC 16-18 December 2003

Specifically, our system includes:

1. Offshore and shore-based near real-time data transmission via the low-orbit Iridium satellite cellular communications system, the Orbcomm low-orbit satellite communication system, and NOAA’s GOES

2. Data assimilation from observing systems and ancillary data providers (e.g. NDBC) to facilitate data parameterization for modeling and product generation

3. Archival within a relational database system for near real-time display and permanent storage in netCDF format

Data and Information Management Subsystem

Scientific Advisory Board Meeting

Columbia, SC 16-18 December 2003

Specifically, our system includes (cont.):

4. Access to data via DODS/OPenDAP5. Access to data and derived products available to

the public and research community via a web-based portal

6. Enhanced FGDC metadata development, search and query capabilities

7. Management and visualization (e.g. GIS, LAS) tools allowing users to generate products over the web

Data and Information Management Subsystem

Scientific Advisory Board Meeting

Columbia, SC 16-18 December 2003

Data and Information Management Subsystem

Current activities focus on:

- Maintain and enhance techniques for data assimilation processing and product development- Support models with improved input source data- Maintain and enhance QA/QC procedures- Enhance data documentation tools and techniques- Develop products and web access for user communities Scientific Advisory Board Meeting

Columbia, SC 16-18 December 2003

Data and Information Management Subsystem

Maintain and enhance techniques for data assimilation processing and product development- IOOS DMAC-compliant but not complacent!

- Using relational databases to address diverse dataset types and formats

- Working closely with SEA-COOS, CORMP, GoMOOS, SURA, NWLON, NDBC, NCDDC to be cognizant of and accommodate diversity of data types

Scientific Advisory Board Meeting

Columbia, SC 16-18 December 2003

Data and Information Management Subsystem

Maintain and enhance techniques … (cont.)

- Assimilating real-time observations via data telemetry, and using a variety of techniques (e.g. screen scraping, ftp) for ancillary data receipt

- Using web functionality and Internet-based GIS tools for data ingest, data browsing, product development, and data and product distribution

Scientific Advisory Board Meeting

Columbia, SC 16-18 December 2003

Data and Information Management Subsystem

Support models with improved input source data

- We recognize the importance of quality input source data for modeling and applications

- We understand the limits and implications of our source data and products

- Plans to discuss collaboration with interested groups (e.g. USCoE, SCDNR, county governments) for mutually important data development activities

Scientific Advisory Board Meeting

Columbia, SC 16-18 December 2003

Data issues of spatial and temporal concern

Data and Information Management SubsystemMaintain and enhance QA/QC procedures

Relevant QA/QC issues:

- On-instrument QA/AC

- QA/QC documentation detailing instrument calibration and maintenance procedures

- Organizational QA/QC (I.e. the QA/QC link with ancillary data providers)

- Regional QA/QC (I.e. data are checked against known or expected environmental range for geographic region)

- Feedback QA/QC (I.e. data are checked against immediate past and future readings)

Scientific Advisory Board Meeting

Columbia, SC 16-18 December 2003

Data and Information Management SubsystemMaintain and enhance QA/QC procedures (cont.)

- Obviously, the quality of our models and products are dependent upon the quality of our data

- Goal is to minimize turnaround time for release of QA/QC’d data

- Initial QA/QC efforts involve:

1.) Raw data checked for transmission errors

2.) Instrument range checks

3. Time continuity checksScientific Advisory Board Meeting

Columbia, SC 16-18 December 2003

Data and Information Management SubsystemEnhance data documentation tools and techniques

- Importance of metadata development within Caro-COOPS cannot be overstated- Metadata standards include:

1. FGDC2. EML3. SensorML

- Key operational areas supported by metadata standards include observational data, instrumentation, methods and QA/QCScientific Advisory Board Meeting

Columbia, SC 16-18 December 2003

Data and Information Management Subsystem

Enhance data documentation tools … (cont.)

- Caro-COOPS is developing a metadata editing tool to be incorporated with a data upload wizard

- Initially this tool will be a web-enabled metadata editor based on the model developed by the SURA-sponsored Cast-Net initiative.

Scientific Advisory Board Meeting

Columbia, SC 16-18 December 2003

Data and Information Management Subsystem

Develop products and web access for user communities

- Multiple user communities (e.g. researchers, managers, fisheries) requiring different products

-Website provides access not only to data but also information concerning the observational array, instrumentation, products, news and updates

- IOOS-supported data access tools (DODS, LAS) as well as GIS functionality have been implemented

Scientific Advisory Board Meeting

Columbia, SC 16-18 December 2003

Data and Information Management Subsystem

A HypotheticalCat. 3 hurricane (MCP=960 mb)moving 50 km offshore along the coast (from SW to NE)

An advanced suite of integrated models

Scientific Advisory Board Meeting

Columbia, SC 16-18 December 2003

Data and Information Management Subsystem

Products with specific audiences in mind (e.g. emergency planners)

Data and Information Management Subsystem

Scientific Advisory Board Meeting

Columbia, SC 16-18 December 2003

(Future) potential products with specific audiences in mind (e.g. recreational fishermen; courtesy NOAA website)

Data and Information Management Subsystem

Scientific Advisory Board Meeting

Columbia, SC 16-18 December 2003

Data and Information Management Subsystem

Scientific Advisory Board Meeting

Columbia, SC 16-18 December 2003

Visitors and users of the system are on the increase!

Caro-COOPSData and Information Management

in Support of Integrated Ocean Observing System Efforts

Part II: Where we are going

Scientific Advisory Board Meeting

Columbia, SC 16-18 December 2003

Caro-COOPS data and information management efforts provide for …

‘developing in situ and remote sensing techniques for near real-time provision of data on key biological and chemical variables’ (Malone 2003).

‘designing and implementing an enhanced, distributed data and information management system that links all observational and data management systems (across agencies and programs) to all data users’ (Ocean.US 2002).

‘providing an effective and sustained program for enhanced public awareness and education’ (Ocean.US 2003)

Scientific Advisory Board Meeting

Columbia, SC 16-18 December 2003

Data and Information Management Subsystem

‘enhancing existing federal networks for in situ measurements … and to improve spatial and temporal resolution … to include physical, chemical and biological variables’ (Ocean.US 2002)

‘adding to the national program of … surveys to establish … multidisciplinary observations (assess the effects of land-based sources of pollution and to improve stock assessments of exploitable fish stocks)’ (Ocean.US 2002).

Scientific Advisory Board Meeting

Columbia, SC 16-18 December 2003

Caro-COOPS data and information management efforts provide for …

Data and Information Management Subsystem

Scientific Advisory Board Meeting

Columbia, SC 16-18 December 2003

Current and proposed Caro-COOPS data and information management efforts are …

- Improving data assimilation, delivery and documentation

- Taking information and information products to the user communities

- Putting the “I” in IOOS!

Data and Information Management Subsystem

Scientific Advisory Board Meeting

Columbia, SC 16-18 December 2003

Improving data assimilation, delivery and documentation

- Program enhancements include:

establish true 24/7/365 connectivity to moorings with two-hour routine transmissions and on- demand data dumps

improvements in mechanisms for data delivery from Federal backbone data providers

enhance GIS capabilities focusing on both Internet-based GIS tools and stand-

alone GIS applications develop more rigorous QA/QC processes

and evaluate appropriate mix of automated versus human-intervention approaches

Data and Information Management Subsystem

Scientific Advisory Board Meeting

Columbia, SC 16-18 December 2003

Taking information and information products to the user communities

- Efforts will focus on developing products based on needs identified by specific user communities. Examples include:

working with state and local level emergency preparedness organizations to enhance GIS-based inundation tools

establish data kiosks at public venues including the visitors’ center at Hunting Island, and on the Sunset Pier

Data and Information Management Subsystem

Courtesy of

Scientific Advisory Board Meeting

Columbia, SC 16-18 December 2003

Putting the “I” in IOOS!

- A major thrust will be the development of Metador, which is:

an enhanced data discovery and documentation tool

focuses on the use of an XML schema (EML) and SensorML while providing for FGDC-compliant metadata

will benefit all IOOS partners, including ocean, terrestrial and estuarine-based systems

Data and Information Management Subsystem

Scientific Advisory Board Meeting

Columbia, SC 16-18 December 2003

Putting the “I” in IOOS!

Data and Information Management Subsystem

- A second major thrust will be in providing leadership via interoperability projects in designing and demonstrating how local, regional and national systems can successfully integrate

Example of an integrated data product developed within the Caro-COOPS, SEA-COOS and SURA/Inter-operability framework