Date post: | 30-Dec-2015 |
Category: |
Documents |
Upload: | gilbert-atkinson |
View: | 215 times |
Download: | 0 times |
MBARI Data MBARI Data Management Management
InitiativesInitiatives
John GraybealJohn Graybeal
Information Applications Information Applications Group LeadGroup Lead
2003.07.15 2
Established in 1987
MMonterey onterey BBay ay AAquarium quarium RResearch esearch
IInstitutenstitute
David and Lucile Packard FoundationDavid and Lucile Packard Foundation
2003.07.15 3
Santa Cruz
Monterey
Monterey Canyon
MBARI
MBARI MBARI LocatioLocatio
nn
2003.07.15 4
Monterey Ocean Monterey Ocean Observing SystemObserving System
Suitable for deep ocean or coastal studiesSuitable for deep ocean or coastal studies low power, long term moorings and benthic nodeslow power, long term moorings and benthic nodes low bandwidth communication links to shorelow bandwidth communication links to shore
Configurable, re-deployable instruments and Configurable, re-deployable instruments and platforms (using ships and ROVs)platforms (using ships and ROVs)
Smart nodes on deployed platformsSmart nodes on deployed platforms some on-board data processingsome on-board data processing
facilitate autonomous event detectionfacilitate autonomous event detection perform on-board calculations/detectionsperform on-board calculations/detections handle responses from shorehandle responses from shore
2003.07.15 5
MOOS Concept of MOOS Concept of OperationsOperations
Benthic Node
Mooring
Autonomous Underwater Vehicle (AUV)
MBARI
2003.07.15 6
Data Stream Challenge Data Stream Challenge (MUSE)(MUSE)
2003.07.15 7
Data Management Data Management ChallengeChallenge
Large number of data sourcesLarge number of data sources Large variety of data sourcesLarge variety of data sources Dynamic systemsDynamic systems
Data sources may appear and disappearData sources may appear and disappear Devices & platforms reconfigured oftenDevices & platforms reconfigured often Interactions from shore and shipInteractions from shore and ship
No standard data formatNo standard data format Data can be instrument ‘native’Data can be instrument ‘native’ New sources coming on-line all the timeNew sources coming on-line all the time Streams or files, automated or manualStreams or files, automated or manual
2003.07.15 8
Example: Samples Example: Samples DatabaseDatabase
2003.07.15 9
Example: Video Example: Video and Imagesand Images
14 years, up to 300 dives/year14 years, up to 300 dives/year 14000 video tapes, 10000 hours14000 video tapes, 10000 hours 47000 frame grabs… => 900,000 47000 frame grabs… => 900,000
annotationsannotations How to manage this valuable How to manage this valuable
repository?repository? Advanced annotation system Advanced annotation system Detailed knowledge base of conceptsDetailed knowledge base of concepts Easy-to-use querying toolEasy-to-use querying tool
2003.07.15 10
Video Annotation and Video Annotation and Reference System Reference System
(VARS)(VARS)
2003.07.15 11
Notes About SSDS: TheNotes About SSDS: TheShore Side Data SystemShore Side Data System
A MOOS Development ProjectA MOOS Development Project Goals: low cost, flexible, expandable, reliableGoals: low cost, flexible, expandable, reliable Future systems beyond MOOS (e.g., MARS)Future systems beyond MOOS (e.g., MARS) Now in 3rd year, deploying initial elementsNow in 3rd year, deploying initial elements
Key Tenets of SSDS DevelopmentKey Tenets of SSDS Development Iterative development—improve it as we goIterative development—improve it as we go Test with real data—new and archivalTest with real data—new and archival Build for change—use modular interfacesBuild for change—use modular interfaces
2003.07.15 12
Shore Side Data System:Shore Side Data System:Requirements OverviewRequirements Overview
Ingest data in Ingest data in any described formatany described format and save it and save it Capture, publish data descriptions Capture, publish data descriptions (metadata)(metadata) Provide Provide standards-based accessstandards-based access to data to data
Raw data, and other common digital formatsRaw data, and other common digital formats APIs for common visualization and analysis toolsAPIs for common visualization and analysis tools User-oriented web interfaces, quick-look plotsUser-oriented web interfaces, quick-look plots
Merge data (different sources & time intervals)Merge data (different sources & time intervals) Support data visualization & quality controlSupport data visualization & quality control Provide data access security as neededProvide data access security as needed
2003.07.15 13
Shore Side Data System:Shore Side Data System:User RequirementsUser Requirements
Raw data via device ID pages? (sort of limited)Raw data via device ID pages? (sort of limited) Standard plots like OASIS quality controlled ones?Standard plots like OASIS quality controlled ones? Access data from applications via a DODS URLs?Access data from applications via a DODS URLs?
Matlab, Ingrid, Live Access Server, Excel, IDV, FerretMatlab, Ingrid, Live Access Server, Excel, IDV, Ferret And hopefully, Ocean Data ViewAnd hopefully, Ocean Data View
Access data via returned data files (e.g., ASCII CSV Access data via returned data files (e.g., ASCII CSV w/headers) opened within desktop applications?w/headers) opened within desktop applications? Excel, ArcView, Ocean Data ViewExcel, ArcView, Ocean Data View
Delivery of data directly into an application?Delivery of data directly into an application? Ability to subset data, for example by time window?Ability to subset data, for example by time window? Ability to merge data from different data sets?Ability to merge data from different data sets?
2003.07.15 14
Data Management at Data Management at MBARI:MBARI:
SSDS EffortsSSDS Efforts Infrastructure/model developmentInfrastructure/model development OntologiesOntologies Metadata schemaMetadata schema Metadata entry/correction/annotationMetadata entry/correction/annotation User interfacesUser interfaces Data processingData processing VisualizationsVisualizations Federated access to MBARI Federated access to MBARI
data/metadatadata/metadata
2003.07.15 15
More MBARI SSDS TasksMore MBARI SSDS Tasks Legacy data migration Legacy data migration
OASIS, expd etc., Samples, Waypoints, ?OASIS, expd etc., Samples, Waypoints, ? New data sources New data sources
MTM II, AUV Sonar, CIMT, …MTM II, AUV Sonar, CIMT, … Outreach (integrating non-SSDS Outreach (integrating non-SSDS
projects)projects) DocumentationDocumentation NEPTUNENEPTUNE EducationEducation Operational supportOperational support
2003.07.15 16
MOOS/SSDS ArchitectureMOOS/SSDS Architecture(shows data flow) (shows data flow)
Devices
DeployedPlatform
Shore Side Data System
User
Applications
(User Tools)
Data Tracking
Communications
Applications/ Interfaces
Archiving
101110110011
110234999
DataPresentation
Data line 1more datalast data
OceanOceanSideSide
ShoreShoreSideSide
Portal
2003.07.15 33
SSDS Data Mgt SequenceSSDS Data Mgt Sequence
2003.07.15 34
AUV Data Sequence AUV Data Sequence DiagramDiagram
2003.07.15 35
Metadata ApproachMetadata Approach(Credit: Dan Davis)(Credit: Dan Davis)
XML suitable for MOOS metadataXML suitable for MOOS metadata Enables use of many other tools/softwareEnables use of many other tools/software But, it looks a little bit user-unfriendlyBut, it looks a little bit user-unfriendly
Use XML-driven GUI technology to Use XML-driven GUI technology to create forms to create and display create forms to create and display metadatametadata Users don’t have to directly read XMLUsers don’t have to directly read XML It’s there and easy to access if they want itIt’s there and easy to access if they want it
Bind XML metadata to each device Bind XML metadata to each device through its puckthrough its puck
2003.07.15 36
SensorPuckto host computer
serial interfaceserial interface
• During pre-deployment instrument During pre-deployment instrument configuration, and test, sensor driver configuration, and test, sensor driver and associated metadata is stored in and associated metadata is stored in compact flash memory in puckcompact flash memory in puck
Metadata stored in puck Metadata stored in puck interfaceinterface
2003.07.15 38
Metadata User Form Metadata User Form DesignDesign
User interface designer uses schema User interface designer uses schema to build a form for creation, display, to build a form for creation, display, access, of metadata instances access, of metadata instances
There may be different forms for There may be different forms for different users (e.g. scientific, different users (e.g. scientific, system, and operational) to create, system, and operational) to create, and display metadata of interestand display metadata of interest
2003.07.15 39
Metadata Form Metadata Form DesignDesign
2003.07.15 40
Instrument Instrument ConfigurationConfiguration
Metadata forms are used during device Metadata forms are used during device configuration to create metadata that configuration to create metadata that is entered into device puckis entered into device puck
Similarly metadata forms are used Similarly metadata forms are used during configuration of other system during configuration of other system elements, such as platforms, and even elements, such as platforms, and even communication links. This metadata is communication links. This metadata is maintained in system nodes.maintained in system nodes.
2003.07.15 41
Metadata Form LayoutMetadata Form Layout
2003.07.15 42
SSDS—SSDS—Metadata (Object Metadata (Object View)View)
2003.07.15 43
The data source.
SSDS tracks:• Software or hardware source• Unique identifier• Manufacturer information• References to documentation
SSDS—SSDS—Metadata (Device)Metadata (Device)
2003.07.15 44
SSDS—SSDS—Metadata Metadata (Deployment)(Deployment)
‘Deployment’ information.
SSDS tracks:• Where the data was collected.• When it was collected.• What other data was used.• Relation to other deployments
2003.07.15 45
References to the data.
SSDS tracks:• The data storage location.• How to access this data.• The deployment that produced this data.
SSDS—SSDS—Metadata Metadata (DataContainer)(DataContainer)
2003.07.15 46
Format and contents of a DataContainer.
SSDS tracks:• The contents of a data set.• The data format (to allow parsing by software).• Descriptive info like units, scale, …
SSDS—SSDS—Metadata (Records)Metadata (Records)
2003.07.15 47
Metadata and Access: Metadata and Access: Catalogs and Catalogs and RepositoriesRepositories
View From the ShoreView From the Shore Many data registries and modelsMany data registries and models
GDC, OBIS, EarthRef, NVODS, … GDC, OBIS, EarthRef, NVODS, … Many standardsMany standards
Communications protocols: SOAP, OPeNDAP, Communications protocols: SOAP, OPeNDAP, OBIS, …OBIS, …
Metadata formats (MIF, XML, NGDC, Metadata formats (MIF, XML, NGDC, NetCDF…)NetCDF…)
Metadata ontologies and effortsMetadata ontologies and efforts NGDC, MarineXML, ESRI, Metadata WranglersNGDC, MarineXML, ESRI, Metadata Wranglers
Conclusion: Watch, Learn, Try (Iterate)Conclusion: Watch, Learn, Try (Iterate)
2003.07.15 48
SSDS Data AccessSSDS Data Access
Desktop Application: HOOVESDesktop Application: HOOVES Data File Service Data File Service Quick LookQuick Look Metadata Access (and Validation)Metadata Access (and Validation) Metadata EditingMetadata Editing
Networked API: Servlet / JSP PagesNetworked API: Servlet / JSP Pages Application API (NetCDF): OPeNDAPApplication API (NetCDF): OPeNDAP Web Access (NetCDF): Live Access Web Access (NetCDF): Live Access
ServerServer Archived Files: Direct Access (?)Archived Files: Direct Access (?)
2003.07.15 49
HOOVES HelpHOOVES Help
2003.07.15 50
HOOVES Mission ViewHOOVES Mission View
2003.07.15 51
HOOVES Mission HOOVES Mission Outputs ViewOutputs View
2003.07.15 52
HOOVES Mission HOOVES Mission Resources: OverviewResources: Overview
2003.07.15 53
HOOVES Mission HOOVES Mission Resources: VehicleResources: Vehicle
2003.07.15 54
HOOVES Instrument HOOVES Instrument ViewView
2003.07.15 55
SSDS ScheduleSSDS ScheduleRequirements Development
2002 2003 2004
MTM
AUV CTD
MTM II
Relational DB Schema
Infrastructure & Capabilities
Access & VisualizationExtreme Week
Tools Training
MTM Deployed
AUV Science Ops Start
SSDS Schedule
MetadataMetadata
MOOS ‘Lite’
2003.07.15 56
Prime Prime Areas for Areas for CollaborationCollaboration
Infrastructure/model developmentInfrastructure/model development OntologiesOntologies Metadata schemaMetadata schema Metadata Metadata
entry/correction/annotationentry/correction/annotation User interfacesUser interfaces Data processingData processing VisualizationsVisualizations Federated access to data/metadataFederated access to data/metadata DocumentationDocumentation
2003.07.15 57
IAG TeamIAG Team
Kevin GomesKevin Gomes John GraybealJohn Graybeal Mike McCannMike McCann Brian SchliningBrian Schlining Rich SchrammRich Schramm And, a Mystery Guest (To Be And, a Mystery Guest (To Be
Determined)Determined)
Science Representative to SSDSScience Representative to SSDS John RyanJohn Ryan