Multimedia DatabasesMultimedia Databases

Prepared by Pradeep Konduri

Instructor: Dr. Yingshu LiGeorgia State University

Plan of AttackPlan of AttackIntroductionArchitectureImage Content AnalysisModeling ConstructsLogical ImplementationReal-World ApplicationsConclusion

Types of multimedia dataTypes of multimedia dataText: using a standard language

(SGML, HTML)Graphics: encoded in CGM,

postscriptImages: bitmap, JPEG, MPEGVideo: sequenced image data at

specified ratesAudio: aural recordings in a

string of bits in digitized form

Nature of Multimedia Nature of Multimedia ApplicationsApplicationsRepositories: central location for

data maintained by DBMS, organized in storage levels

Presentations: delivery of audio and video data, temporarily stored.

Collaborative: complex design, analyzing data

Management IssuesManagement IssuesModeling: complex objects, wide

range of typesDesign: still in researchStorage: representation,

compression, buffering during I/O, mapping

Queries: techniques need to be modified

Performance: physical limitations, parallel processing

Research ProblemsResearch ProblemsInformation Retrieval in Queries:

Modeling the content of documents

Multimedia/Hypermedia Data Modeling and Retrieval: Hyperlinks, Used in WWW

Text Retrieval: Use of a thesaurus

Multimedia Database Multimedia Database ApplicationsApplicationsDocumentation and keeping

RecordsKnowledge distributionEducation and TrainingMarketing, Advertisement,

Entertainment, TravelReal-time Control, Monitoring

A Generic Architecture of MMDBMS A Generic Architecture of MMDBMS

Media Object

Feature Extraction

Compression

Indexing

MMDBMS

Query feature construction

Search Engine

Feedback Query construction

Media Object

query

feedback

result

Media organization: organize the features for retrieval(i.e., indexing the features with effective structures)

Media query processing: accommodated with indexing structure, efficient search algorithm with similarity function should be designed

Multimedia Database Multimedia Database ArchitectureArchitecture

Multimedia Data Preprocessing System

Database Processing

MM DataPre-

processor

AdditionalInformation

<!ELEMENT ..>.....<!ATTLIST...>

Multimedia DBMS

Users

Que

ry I

nter

face

MM DataInstance

<article>.....</article>

Recognized components

MM DataInstance

MM Data

Meta-Data

Document Database Document Database ArchitectureArchitecture

Document Processing System

Database Processing

DTD filesD

TD

Par

ser<!ELEMENT ..>

.....<!ATTLIST...>

DTDManager

TypeGenerator

Multimedia DBMS

Users

Que

ry I

nter

faceDTD

Document content

SGML/XMLDocuments

XML or SGMLDocumentInstance

ParseTree

<article>.....</article>

<!ELEMENT ..>.....<!ATTLIST...> DTD

C++ Types

C++ Objects

SGML/XMLParser

InstanceGenerator

Image Database Architecture Image Database Architecture

Image Processing System

Database Processing

Image Analysis and Pattern Recognition

ImageAnnotation

Multimedia DBMS

Users

Que

ry I

nter

face

Image ContentDescription

Image

Image

SyntacticObjects

SemanticObjects

<article>.....</article>

Meta-Data

Image Content AnalysisImage Content AnalysisImage content analysis can be

categorized in 2 groups:◦Low-level features: vectors in a

multi-dimensional space Color Texture Shape

◦Mid- to high-level features: Try to infer semantics

◦Semantic Gap

Image Content Analysis: Image Content Analysis: ColorColorColor space:

◦ Multidimensional space◦ A dimension is a color component◦ Examples of color space: RGB, HSV◦ RGB space: A color is a linear combination of 3

primary colors (Red, Green and Blue)Color Quantization

◦ Used to reduce the color resolution of an imageThree widely used color features

◦ Global color histogram◦ Local color histogram◦ Dominant color

Color HistogramsColor Histograms Color histograms indicate color distribution

without spatial information◦ Color histogram distance metrics

0

10

20

30

40

50

Red Orange Yellow Green Blue Indigo Violet

Image Content Analysis: Image Content Analysis: TextureTextureRefers to visual patterns with properties of

homogeneity that do not result from the presence of only a single color

Examples of texture: Tree barks, clouds, water, bricks and fabrics

Texture features: Contrast, uniformity, coarseness, roughness, frequency, density and directionality

Two types of texture descriptors◦ Statistical model-based

Explores the gray level spatial dependence of texture and extracts meaningful statistics as texture representation

◦ Transform-based DCT transform, Fourier-Mellin transform, Polar Fourier

transform, Gabor and wavelet transform

Image Content Analysis: Image Content Analysis: ShapeShapeObject segmentation

◦Approaches: Global threshold-based approach Region growing, Split and merge approach, Edge detection app

◦Still a difficult problem in computer vision. Generally speaking it is difficult to achieve perfect segmentation

Salient Objects vs. Salient Salient Objects vs. Salient PointsPoints

Original images

Segmented images with region boundaries

Extracted salient points

Generic low-level description of images into salient objects and salient points

Modeling Images – Modeling Images – PrinciplesPrinciples

Support for multiple representations of an image

Support for user-defined categorization of images

Well-defined set of operations on imagesAn image can have (semantic, functional,

spatial) relationships with other images (or documents) which should be represented in the DBMS

An image is composed of salient objects (meaningful image components)

Salient Object ModelingSalient Object Modeling

Multiple representations of a salient object (grid, vector) are allowed

A salient object O is of a particular type which belongs to a user defined salient object types hierarchy

An image component may have some (semantic, functional, spatial) relationships with other salient objects

““Semantic GapSemantic Gap””

semantics-intensive multimedia systems & applications

non-semanticmultimedia data models

require

model

semantic meaning of the

data

raw data,primitive

properties (size, format,

etc)

Semantic Gap

Semantic modeling of Semantic modeling of multimedia multimedia -- Why hard?-- Why hard?

Context-dependency◦Semantics is not a static and intrinsic

property◦The semantics of an object often depends on:

the application/user who manipulate the object the role that the object plays other objects in the same “context”

Van Gogh’s

paintings

flower

Example:

Why hard? (cont.)Why hard? (cont.)Modality-independency

◦Media objects of different modalities may suggest the similar/related semantic meanings.

◦Example:Harry Potter has never been the star of a Quidditch team, scoring points while riding a broom far above the ground. He knows no spells, has never helped to hatch a dragon, and has never worn a cloak of invisibility.

Query:

Results:

image video text

MediaViewMediaView –– A A ““Semantic Semantic BridgeBridge””

An object-oriented view mechanism that bridges the semantic gap between multimedia systems and databases

Core concept – media view (MV)◦a customized context for semantic

interpretation of media objects (text docs, images, video, etc)

◦collectively constitute the conceptual infrastructure of a multimedia system & application

ArchitectureArchitecture

External Schema

mediaview 1

Internal Schema

mediaview 2

mediaview n

. . .

Object-oriented Database

Multimedia Systems

Conceptual Schema

. . .

MediaView Mechanism

Basic ConceptsBasic Concepts

An example…

Image

MultimediaObject

TextDocument

AudioClip

VideoClip

Image

BitmapImage

JPEGImage

keyframe

audiotrack

ImpressionisticArtworks

Name

Artist

Type

Style

Wavelet-Texture

Dominant-Shape

Color-Histogram

Artworks

RealisticArtworks

ImpressionisticPaintings

ImpressionisticArtworks

Post-modernArtworks

(a) Base Class (B) Media View

(d) View Schema(c) Base Schema

SongSpeechImpressionistic

Sculptures

subclasssubclass

subviewsubview

Basic ConceptsBasic Concepts

Semantics-based data reorganization via media views

text

audio

video

image

media view

View OperatorsView OperatorsA set of operators that take

media views and view instances as operands.

Focus on the operators that are indispensable in supporting queries and navigation over multimedia objects.

View OperatorsView Operators

type-level

V-overlapsyntax<boolean>:= v-overlap (<media view1, media view2 >)semantics true, if and only if ( o O)(oextent(<media view1>) and oextent(<media view2>))

Crosssyntax{<object>}:= cross (<media view1, media view2 >)semantics{<object>} := {o O | o extent(<media view1>) and oextent(<media view2>)}

Sumsyntax{<object>}:= sum (<media view1, meida-view2 >)semantics{<object>} := {o O | o extent(<media view1>) or oextent(<media view2>)}

Subtractsyntax{<object>}:= subtract (<media view1, media view2>)semantics{<object>}:= {o O | o extent(<media view1>) and oextent(<media view2>)}

View OperatorsView Operators

instance-level

Classsyntax<base class> := class(<view instance>)semantics<view instance> is a instance of <base class>

componentssyntax{<object>} := components (<view instance>) semantics {<object>} := { oO | o is a component (direct or indirect) of <view instance>}

i-overlapsyntax<boolean> := i-overlap (<view instnace1>, <view instance2>)semantics true, if and only if ( o O) (o components (<view instance1>) and o components(<view instance2>))

View AlgebraView Algebra Functions

-- derivation of new MVs from existing MVs

Heuristic Enumeration1. Blind enumeration 2. Content-based enumeration 3. Semantics-based enumeration

View AlgebraView AlgebraAlgebra Operators

◦select from src-MV where <predicate>

◦project <property-list> from src-MV◦intersect (src-MV1, src-MV2)◦union (src-MV1, src-MV2)◦difference (src-MV1, src-MV2)

Comparison (vs. class)Comparison (vs. class)

media view object classmembershi

pheterogeneous objects uniform objects

member acquisition

dynamic inclusion/exclusion of existing objects of other classes

creating new objects

mapping one object can belong to multiple media views

one object has exactly one class

relationship inter-member semantic relationship

N/A

Comparison (vs. traditional Comparison (vs. traditional object view)object view)

media view object viewmembershi

pheterogeneous objects uniform objects

relationship inter-member semantic relationship

N/A

member properties

instance-level properties (user-defined)

inherited or derived properties (for view

instances)global

propertiesMV-level properties (user-

defined)N/A

Logical ImplementationLogical ImplementationMediaView ConstructionMediaView CustomizationMediaView Evolution

MediaViews ConstructionMediaViews ConstructionWork with CBIR systems to

acquire the knowledge from queries◦Learn from previously performed

queries ◦A multi-system approach to support

multi-modality of media objects

Organize the semantics by following WordNet

Why WordNet?Why WordNet?

Different queries may greatly vary with the liberty of choosing query keywords

We need an approach to organize those knowledge into a logic structure◦A simple “context”: a concept in WordNet◦Common media views: corresponds to

simple contexts ◦We provide all common media views, based

on which users can build complex ones.

Navigating the Multimedia Navigating the Multimedia DatabaseDatabase

Navigating via semantic relationships of WordNetSemantic Relationship ExamplesSynonymy (similar) pipe, tubeAntonymy (opposite) fast, slowHyponymy (subordinate) tree, plantMeronymy (part) chimney, houseTroponomy (manner) march, walkEntailment drive, ride

Navigating the Multimedia Navigating the Multimedia DatabaseDatabase

Multimedia Database

MediaView 1

MediaView 2

MediaView 3

MediaView 4

Semantic Relationship in WordNet

User

browse

MediaViews Construction MediaViews Construction

CBIRSystem(Video)

CBIRSystem(Image)

CBIR System(Text)

Query

...

Multimedia Database

MediaView Engine

System Feedback

Users

User Feedback

Results

Issue

MediaView Customization MediaView Customization

Two level MediaView Framework

Basic MediaView

Customized MediaView

Simple Context Advanced Context

MediaView Customization MediaView Customization Dynamically construct complex-

context-based media views based on simple ones ◦An example complex context: “the

Grand Hall in City University” Several user-level operators are

devised to support more complex/advanced contexts, besides the basic operators

User-level OperatorsUser-level OperatorsINHERIT_MV(N: mv-name, NS:

set-of-mv-refs, VP: set-of-property-ref, MP: set-of-property-ref): mv-ref

UNION_MV(N: mv-name, NS: set-of-mv-refs): mv-ref

INTERSECTION_MV(N: mv-name, NS: set-of-mv-refs): mv-ref

DIFFERENCE_MV(N1: mv-ref, N2: mv-ref): mv-ref

Build a MediaView in Run-Build a MediaView in Run-timetime

Example: find out info about "Van Gogh"◦ Who is "Van Gogh"?◦ What is his work?◦ Know more about his

whole life.◦ Know more about his

country.◦ See his famous

painting "sunflower"

Legend

Multimedia Document

Media View 1

Text

Sound

Image

Video

Topic 1

Topic 2

Topic 3

Build MediaView

Build a MediaView in Run-Build a MediaView in Run-timetimeWho is “Van Gogh”?

◦ INHERIT_MV(“V. Gogh“, {<painter>},name=”Van Gogh” ,);

What is his work?◦ INTERSECTION_MV(“work”, {<painting>, vg});

Know more about his whole life.◦ INTERSECTION_MV(“life”, {<biography>, vg});

Know more about his country.◦ INTERSECTION_MV(“country”, {<country>, vg});

See his famous painting “sunflower”◦ Set sunflower = INTERSECTION_MV(“sunflower”,

{<sunflower>, <painting>});Set vg_sunflower = INTERSECTION_MV(“vg_sunflower”, {vg_work, sunflower});

Authoring Scenario Authoring Scenario Creates a new media view named after the

subject◦ All multimedia materials used in the document would

be put into this MediaView for further reference. To collect the most relevant materials for

authoring, the user performs the MediaView building process. ◦ Import suitable media objects by browsing media

views Reference the manner and style of authoring,

to find other media views with similar topics. ◦ Drag & Drop◦ “learning-from-references”

SummarySummaryTypes of multimedia data: Text, Audio, Video,

Images.Management issues: Design, Storage,

Modeling, QueriesImage Content Analysis: Color, Texture,

ShapeMediaView – a semantic multimedia database

modeling mechanism ◦ to bridge the semantic gap between

conventional database and semantics-intensive multimedia applications

A set of user-level operators to accommodate the specialization/generalization relationships among the media views

Summary (contd..)Summary (contd..)MediaView promises more effective access

to the content of media databases◦Users could get the right stuff and tailor it

to the context of their application easily. Providing the most relevant content from

pre-learnt semantic links between media and contexthigh performance database browsing and

multimedia authoring tools can enable more comprehensive applications to the user.

Users could customize specific media view according to their tasks, by using user-level operators

Further IssuesFurther IssuesThe development and transition

of MediaView to a fully-fledged multimedia database system supporting “declarative” queries

Intensive and extensive performance studies

Advanced semantic relations (eg. temporal and spatial ones) can also be incorporated in combining individual media views

Thank you!

Q & A

Email: pkonduri1@student.gsu.edu