MBARI’s Video Annotation and Reference System (VARS)

Brian Schlining and Nancy Jacobsen-Stout

MBARI

MBARI

Monterey Canyo

n

• More than 300 ROV dives per year• More than 12,000 hours of video observations

generated in 16 years• More than 1.6 million individual observations

MBARI Video Data

Video observations and related data• Used by MBARI researchers in well over 200 professional publications and

presentations, in addition to countless education and outreach projects• Span multiple sub-disciplines within biology, geology, chemistry

“A picture can be worth a thousand words”(if you can find it!)

Interesting Taxa (gorgonocephalus)Benthic Diversity

Scientific Applications

• Trained science staff identify animal taxonomy, animal behavior, geological features, evidence of human impact, and other objects recorded in the ROV video stream

• Annotations are stored in a database shared between MBARI scientists.

• Database continues to evolve as our researchers learn more

• Work closely with and rely on scientists to keep updated on new discoveries and publications in a timely fashion

Annotations

1st generation – Free Text– Inconsistent annotations– Difficult to search

2nd generation – VICKI/VIMS– Constrained annotations– Written in Smalltalk– Generated files

3rd generation – VARS– Constrained annotations– Written in Java– Writes information directly to database

Annotation History

Knowledge Base = hierarchical constraint lexicon & referenceMore than 3500 taxonomic, geologic, technical concepts. Reference media and descriptions.

Annotation = catalog video observations, inc. visual mediaUses the KB data to create and edit structured annotations for objects in the video stream.

Query = retrieve video-related data Uses the KB to structure simple to complex queries against annotations database

VARS - System Components

VARS - Knowledge BaseManages constraint lexicon and references

VARS - AnnotationGenerates video-annotations and frame-grabs

VCR Control(RS-422)

Frame capture(QuickTime for Java)

VARS - AnnotationGenerates video-annotations and frame-grabs

Customizable Interface

Knowledge Base

VARS - QuerySearches for and retrieves information and frame-grabs.

VARS - QuerySearches for and retrieves information and frame-grabs.

VARS - QuerySearches for and retrieves information and frame-grabs.

Vestimentiferan Tubeworms

Vesicomyid Clams

Example – “Cold Seeps”

“Cold Seep” Map Vintage 1999

“Cold Seep” Map Vintage 2004

Monterey bay is one of the most observed continental margins in the world

Visited 0.9% pixels in this map

• Previous “seep” models assumed seeps are associated with faults (fluid low conduits)

Example – “Cold Seeps”

Chemosynthetic Biological Communities occur preferentially on steep slopes?

Steep slopes imply areas of recent erosion.

VARS user goals include:1) Provide responsive user interfaces to maximize user efficiency and

accommodate real- and greater than real-time analyses.

2) Provide a robust system.

3) Provide clear and intuitive user interfaces.

VARS developmental goals include:1) Create maintainable (modular, understandable) software for future

developers.

2) Maximize use of existing software (e.g. old VIMS query and commercially available software)

3) Provide a solution that can be exported to other institutions.

4) The database should be simple to maintain by MBARI IS administrators.

VARS Development Goals

Vars system requirements:

1) Java 1.4 or greater

2) Java Comm (or RXTX)

3) Quicktime for Java (For frame-capture)

4) Relational Database

VARS Requirements

VARS Deployment

VARS Deployment

Shore

Point Lobos

Western Flyer

VARS Deployment

Shore

Point Lobos

Western Flyer

Database replication

Database replication

VARS - Annotation data

VARS - Annotation data

A related group of tapes.e.g. tapes from a single expedition

VARS - Annotation data

A video source, such as a tape or QuickTime movie

VARS - Annotation data

A single video frame

VARS - Annotation data

An object observed in a particular video-frame. The name is constrained by the knowledgebase

VARS - Annotation data

Descriptive information. Can also be used to link ‘concepts’. For example, ‘nanomia eating krill’

VARS - Annotation data

Position, physical data, expedition information,and camera data,

VARS - Annotation data

VARS - Knowledgebase data

VARS - Knowledgebase data

~ Hierarchical, Allows representation of relations or phylogeny~ Linked to descriptive info. (History, authors, images, video, etc. [not shown])

VARS - Knowledgebase data~ A concept may have one or more concept-names.~ One concept-name is designated as a primary name.~ User may annotate with any concept-name. However, only the primary name is stored in the database.

VARS - Knowledgebase data

VARS – Available to you

VARS is available to other science institutions.

See http://www.mbari.org/vars/

Fall 2005 – Release VARS as open-source

December 2005 – MBARI will host a developer workshop

If you are interested in using VARS contact:

Judith Connor (conn@mbari.org)

Director of Information & Technology Dissemination