MBARI Data MBARI Data Management Management

InitiativesInitiativesJohn GraybealJohn Graybeal

Information Applications Information Applications Group LeadGroup Lead

2003.07.15 2

Established in 1987

MMonterey onterey BBay ay AAquarium quarium RResearch esearch

IInstitutenstitute

David and Lucile Packard FoundationDavid and Lucile Packard Foundation

2003.07.15 3

Santa Cruz

Monterey

Monterey Canyon

MBARI

MBARI MBARI LocatioLocatio

nn

2003.07.15 4

Monterey Ocean Monterey Ocean Observing SystemObserving System

Suitable for deep ocean or coastal studiesSuitable for deep ocean or coastal studies low power, long term moorings and benthic nodeslow power, long term moorings and benthic nodes low bandwidth communication links to shorelow bandwidth communication links to shore

Configurable, re-deployable instruments and Configurable, re-deployable instruments and platforms (using ships and ROVs)platforms (using ships and ROVs)

Smart nodes on deployed platformsSmart nodes on deployed platforms some on-board data processingsome on-board data processing

facilitate autonomous event detectionfacilitate autonomous event detection perform on-board calculations/detectionsperform on-board calculations/detections handle responses from shorehandle responses from shore

2003.07.15 5

MOOS Concept of MOOS Concept of OperationsOperations

Benthic Node

Mooring

Autonomous Underwater Vehicle (AUV)

MBARI

2003.07.15 6

Data Stream Challenge Data Stream Challenge (MUSE)(MUSE)

2003.07.15 7

Data Management Data Management ChallengeChallenge

Large number of data sourcesLarge number of data sources Large variety of data sourcesLarge variety of data sources Dynamic systemsDynamic systems

Data sources may appear and disappearData sources may appear and disappear Devices & platforms reconfigured oftenDevices & platforms reconfigured often Interactions from shore and shipInteractions from shore and ship

No standard data formatNo standard data format Data can be instrument ‘native’Data can be instrument ‘native’ New sources coming on-line all the timeNew sources coming on-line all the time Streams or files, automated or manualStreams or files, automated or manual

2003.07.15 8

Example: Samples Example: Samples DatabaseDatabase

2003.07.15 9

Example: Video Example: Video and Imagesand Images

14 years, up to 300 dives/year14 years, up to 300 dives/year 14000 video tapes, 10000 hours14000 video tapes, 10000 hours 47000 frame grabs… => 900,000 47000 frame grabs… => 900,000

annotationsannotations How to manage this valuable repository?How to manage this valuable repository?

Advanced annotation system Advanced annotation system Detailed knowledge base of conceptsDetailed knowledge base of concepts Easy-to-use querying toolEasy-to-use querying tool

2003.07.15 10

Video Annotation and Video Annotation and Reference System Reference System

(VARS)(VARS)

2003.07.15 11

Notes About SSDS: TheNotes About SSDS: TheShore Side Data SystemShore Side Data System

A MOOS Development ProjectA MOOS Development Project Goals: low cost, flexible, expandable, Goals: low cost, flexible, expandable,

reliablereliable Future systems beyond MOOS (e.g., MARS)Future systems beyond MOOS (e.g., MARS) Now in 3rd year, deploying initial elementsNow in 3rd year, deploying initial elements

Key Tenets of SSDS DevelopmentKey Tenets of SSDS Development Iterative development—improve it as we goIterative development—improve it as we go Test with real data—new and archivalTest with real data—new and archival Build for change—use modular interfacesBuild for change—use modular interfaces

2003.07.15 12

Shore Side Data System:Shore Side Data System:Requirements OverviewRequirements Overview

Ingest data in Ingest data in any described formatany described format and save it and save it Capture, publish data descriptions Capture, publish data descriptions (metadata)(metadata) Provide Provide standards-based accessstandards-based access to data to data

Raw data, and other common digital formatsRaw data, and other common digital formats APIs for common visualization and analysis toolsAPIs for common visualization and analysis tools User-oriented web interfaces, quick-look plotsUser-oriented web interfaces, quick-look plots

Merge data (different sources & time intervals)Merge data (different sources & time intervals) Support data visualization & quality controlSupport data visualization & quality control Provide data access security as neededProvide data access security as needed

2003.07.15 13

Shore Side Data System:Shore Side Data System:User RequirementsUser Requirements

Raw data via device ID pages? (sort of limited)Raw data via device ID pages? (sort of limited) Standard plots like OASIS quality controlled ones?Standard plots like OASIS quality controlled ones? Access data from applications via a DODS URLs?Access data from applications via a DODS URLs?

Matlab, Ingrid, Live Access Server, Excel, IDV, FerretMatlab, Ingrid, Live Access Server, Excel, IDV, Ferret And hopefully, Ocean Data ViewAnd hopefully, Ocean Data View

Access data via returned data files (e.g., ASCII CSV Access data via returned data files (e.g., ASCII CSV w/headers) opened within desktop applications?w/headers) opened within desktop applications? Excel, ArcView, Ocean Data ViewExcel, ArcView, Ocean Data View

Delivery of data directly into an application?Delivery of data directly into an application? Ability to subset data, for example by time window?Ability to subset data, for example by time window? Ability to merge data from different data sets?Ability to merge data from different data sets?

2003.07.15 14

Data Management at Data Management at MBARI:MBARI:

SSDS EffortsSSDS Efforts Infrastructure/model developmentInfrastructure/model development OntologiesOntologies Metadata schemaMetadata schema Metadata entry/correction/annotationMetadata entry/correction/annotation User interfacesUser interfaces Data processingData processing VisualizationsVisualizations Federated access to MBARI Federated access to MBARI

data/metadatadata/metadata

2003.07.15 15

More MBARI SSDS TasksMore MBARI SSDS Tasks Legacy data migration Legacy data migration

OASIS, expd etc., Samples, Waypoints, ?OASIS, expd etc., Samples, Waypoints, ? New data sources New data sources

MTM II, AUV Sonar, CIMT, …MTM II, AUV Sonar, CIMT, … Outreach (integrating non-SSDS projects)Outreach (integrating non-SSDS projects) DocumentationDocumentation NEPTUNENEPTUNE EducationEducation Operational supportOperational support

2003.07.15 16

MOOS/SSDS MOOS/SSDS ArchitectureArchitecture

(shows data flow)            (shows data flow)            

Devices

DeployedPlatform

Shore Side Data System

User

Applications

(User Tools)

Data Tracking

Communications

Applications/ Interfaces

Archiving

101110110011

110234999

DataPresentation

Data line 1more datalast data

OceanOceanSideSide

ShoreShoreSideSide

            Portal

2003.07.15 33

SSDS Data Mgt SequenceSSDS Data Mgt Sequence

2003.07.15 34

AUV Data Sequence AUV Data Sequence DiagramDiagram

2003.07.15 35

Metadata ApproachMetadata Approach(Credit: Dan Davis)(Credit: Dan Davis)

XML suitable for MOOS metadataXML suitable for MOOS metadata Enables use of many other tools/softwareEnables use of many other tools/software But, it looks a little bit user-unfriendlyBut, it looks a little bit user-unfriendly

Use XML-driven GUI technology to Use XML-driven GUI technology to create forms to create and display create forms to create and display metadatametadata Users don’t have to directly read XMLUsers don’t have to directly read XML It’s there and easy to access if they want itIt’s there and easy to access if they want it

Bind XML metadata to each device Bind XML metadata to each device through its puckthrough its puck

2003.07.15 36

SensorPuckto host computer

serial interfaceserial interface

• During pre-deployment instrument During pre-deployment instrument configuration, and test, sensor driver configuration, and test, sensor driver and associated metadata is stored in and associated metadata is stored in compact flash memory in puckcompact flash memory in puck

Metadata stored in puck Metadata stored in puck interfaceinterface

2003.07.15 38

Metadata User Form Metadata User Form DesignDesign

User interface designer uses schema User interface designer uses schema to build a form for creation, display, to build a form for creation, display, access, of metadata instances access, of metadata instances

There may be different forms for There may be different forms for different users (e.g. scientific, different users (e.g. scientific, system, and operational) to create, system, and operational) to create, and display metadata of interestand display metadata of interest

2003.07.15 39

Metadata Form Metadata Form DesignDesign

2003.07.15 40

Instrument Instrument ConfigurationConfiguration

Metadata forms are used during device Metadata forms are used during device configuration to create metadata that configuration to create metadata that is entered into device puckis entered into device puck

Similarly metadata forms are used Similarly metadata forms are used during configuration of other system during configuration of other system elements, such as platforms, and even elements, such as platforms, and even communication links. This metadata is communication links. This metadata is maintained in system nodes.maintained in system nodes.

2003.07.15 41

Metadata Form LayoutMetadata Form Layout

2003.07.15 42

SSDS—SSDS—Metadata (Object Metadata (Object View)View)

2003.07.15 43

The data source.

SSDS tracks:• Software or hardware source• Unique identifier• Manufacturer information• References to documentation

SSDS—SSDS—Metadata (Device)Metadata (Device)

2003.07.15 44

SSDS—SSDS—Metadata Metadata (Deployment)(Deployment)

‘Deployment’ information.

SSDS tracks:• Where the data was collected.• When it was collected.• What other data was used.• Relation to other deployments

2003.07.15 45

References to the data.

SSDS tracks:• The data storage location.• How to access this data.• The deployment that produced this data.

SSDS—SSDS—Metadata Metadata (DataContainer)(DataContainer)

2003.07.15 46

Format and contents of a DataContainer.

SSDS tracks:• The contents of a data set.• The data format (to allow parsing by software).• Descriptive info like units, scale, …

SSDS—SSDS—Metadata (Records)Metadata (Records)

2003.07.15 47

Metadata and Access: Metadata and Access: Catalogs and Catalogs and RepositoriesRepositories

View From the ShoreView From the Shore Many data registries and modelsMany data registries and models

GDC, OBIS, EarthRef, NVODS, … GDC, OBIS, EarthRef, NVODS, … Many standardsMany standards

Communications protocols: SOAP, OPeNDAP, Communications protocols: SOAP, OPeNDAP, OBIS, …OBIS, …

Metadata formats (MIF, XML, NGDC, Metadata formats (MIF, XML, NGDC, NetCDF…)NetCDF…)

Metadata ontologies and effortsMetadata ontologies and efforts NGDC, MarineXML, ESRI, Metadata WranglersNGDC, MarineXML, ESRI, Metadata Wranglers

Conclusion: Watch, Learn, Try (Iterate)Conclusion: Watch, Learn, Try (Iterate)

2003.07.15 48

SSDS Data AccessSSDS Data Access Desktop Application: HOOVESDesktop Application: HOOVES

Data File Service Data File Service Quick LookQuick Look Metadata Access (and Validation)Metadata Access (and Validation) Metadata EditingMetadata Editing

Networked API: Servlet / JSP PagesNetworked API: Servlet / JSP Pages Application API (NetCDF): OPeNDAPApplication API (NetCDF): OPeNDAP Web Access (NetCDF): Live Access Web Access (NetCDF): Live Access

ServerServer Archived Files: Direct Access (?)Archived Files: Direct Access (?)

2003.07.15 49

HOOVES HelpHOOVES Help

2003.07.15 50

HOOVES Mission ViewHOOVES Mission View

2003.07.15 51

HOOVES Mission HOOVES Mission Outputs ViewOutputs View

2003.07.15 52

HOOVES Mission HOOVES Mission Resources: OverviewResources: Overview

2003.07.15 53

HOOVES Mission HOOVES Mission Resources: VehicleResources: Vehicle

2003.07.15 54

HOOVES Instrument HOOVES Instrument ViewView

2003.07.15 55

SSDS ScheduleSSDS ScheduleRequirements Development

2002 2003 2004

MTM AUV CTD

MTM II

Relational DB Schema

Infrastructure & Capabilities

Access & VisualizationExtreme Week

Tools Training

MTM Deployed

AUV Science Ops Start

SSDS Schedule

MetadataMetadata

MOOS ‘Lite’

2003.07.15 56

Prime Prime Areas for Areas for CollaborationCollaboration

Infrastructure/model developmentInfrastructure/model development OntologiesOntologies Metadata schemaMetadata schema Metadata Metadata

entry/correction/annotationentry/correction/annotation User interfacesUser interfaces Data processingData processing VisualizationsVisualizations Federated access to data/metadataFederated access to data/metadata DocumentationDocumentation

2003.07.15 57

IAG TeamIAG Team Kevin GomesKevin Gomes John GraybealJohn Graybeal Mike McCannMike McCann Brian SchliningBrian Schlining Rich SchrammRich Schramm And, a Mystery Guest (To Be And, a Mystery Guest (To Be

Determined)Determined)

Science Representative to SSDSScience Representative to SSDS John RyanJohn Ryan