1

Multimedia Databases

Prepared by Chengcui ZhangLab: KDDM www.cis.uab.edu/kddm

Email: zhang@cis.uab.eduwww.cis.uab.edu/zhang

2010 Spring

8

Trends in Internet, Mobile Phones, Mobile Internet

Smart phones! 40 million of these mobile phone users in

Europe are mobile multimedia users. The total Western European mobile market

is worth 120 billion ECU per year in 2010. The mobile multimedia segment of this

Western European market are worth 30 billion ECU in 2010.

9

Introduction

Multimedia system: A variety of information sources (text, voice, image, video,

audio, animation, etc.) Characteristics:

All the different media are brought together into one single unit, all controlled by a computer

Requirements: Management and delivery of extremely large bodies of data

at a very high rate Real-time constraints …

Challenges: Synchronization… Semantic heterogeneity

10

11

Problems of Relational Database Model

Conventional data modeling techniques lack the ability to manage the composition of multimedia objects in a heterogeneous multimedia database environment.

Relational database system is only good to manage textual and numerical data. Retrieving data is often based on simple comparisons of text or

numerical values. Relational data model has limited capabilities in modeling the

structural and behavioral properties of real-world objects. Relational data model has difficulty to model time-dependent

multimedia data (video or audio). BLOBs (Binary Large Objects) are incapable of interactively

accessing various portions of objects since a BLOB is treated as a single entity in its entirety.

12

Problems of Object-Oriented Model

It provides a better facility for managing the multimedia data.

Good features: Inheritance Information hiding Can include image data Composite object (an object consisting of other

objects) provides the capability to handle the structural complexity of the data

13

Problems of Object-Oriented Model (cont.)

Lack of facilities for the management of spatio-temporal relations.

Still, the O-O DBMS is not designed to support multimedia information management.

Multimedia extension is needed to handle the mismatch between multimedia data and conventional O-O database management systems.

14

Important Characteristics of Multimedia Objects (MO)

MO are complex and therefore less than completely captured in an MDBMS.

MO are audiovisual in nature and are amenable to multiple interpretations.

MO are content sensitive. Queries looking for MO are likely to use

subjective descriptions that are often fuzzy in their interpretation.

MO may be included in fuzzy classes. …

15

16

Requirements for Modeling Multimedia Data

1. Specify incomplete information

2. Extend the definition of some individual documents beyond the definitions of its type

3. Integrate data from various databases and handle them uniformly

4. Describe structural information

5. Distinguish between internal modeling and external presentation of objects

17

Requirements for Modeling Multimedia Data (cont.)

6. Share data among multiple documents

7. Create and control versions

8. Include appropriate operations

9. Handle document access control

New trend: SMELL!

18

http://staff.science.uva.nl/~gevers/master2007/PDF/lecture1_small_2007.pdf

20

Multimedia Database Applications

Education: CAI (Computer Assisted Instruction)

Internet search (e.g., Google image/video search)

Medical Imaging Surveillance Systems Biometrics databases Video-on-demand Game …

22

Application: Image search engines – Goggle!

http://www.google.com/mobile/goggles/#landmark

Application: Fingerprint Matching and retrieval

23

24

Application: real-time skin detection for human recognition

Are HP computer webcams really racist? http://blogs.consumerreports.org/electronics/

2009/12/racist-hp-webcam-video-blog-consumer-reports-response.html

25

26

Application: real-time object recognition and tracking

27

Application: Surveillance

http://www.nydailynews.com/ny_local/2010/01/08/2010-01-08_new_jersey_man_arrested_over_security_breach_at_newark_liberty_airport.html

Content-Based Image Retrieval

An picture is worth a thousand words!

28

29

Text-based Retrieval

Content-Based Image Retrieval

Content-Based Image Retrieval (CBIR) Image databases can be huge, containing

hundreds of thousands or millions of images. In most cases they are only indexed by

keywords that have to be decided upon and entered into the database system by a human categorizer.

However, image can be retrieved according to their content, where content might refer to color distributions, texture, region shapes, or object classification.

Image Database Examples

IBM: Query by Image Content (QBIC) Retrieves images based on visual content, including such

properties as color percentage, color layout, and texture. Virage, Inc.

Virage search engine can retrieve images based on color composition, texture, and structure.

Google Image search. National Library of Medicine provides a database of x-rays,

CT scans, MRI images, and color cross-sections, taken at very small intervals along the bodies of male and female cadaver.

The NASA collects huge databases of images from its satellites and makes them available for public acquisition. (for free )

State-of-the-Art in MDBMS

First wave – query by text In a second wave, commercial systems

were proposed which handle multimedia content by providing complex object types for various kinds of media.

Broadly used commercial MMDBMSs are extensible Object-Relational DBMS (ORDBMSs).

Oracle 10g, IBM DB2, and IBM Informix.34

DB2 Image Extender

DB2 Image Extender defines the distinct data type DB2IMAGE with associated user-defined functions for storing and manipulating image files (http://www-306.ibm.com/software/data/

db2/extenders/ ). The DB2 Image Extender provides

similarity search functionality based on the QBIC technology (http://wwwqbic.almaden.ibm.com/ )

35

36

Query By Example (QBE)

The image DB user should be able to: show the system a sample image, or Paint one interactively on the screen, or Just sketch the outline of an object.

The system should then be able to return similar images or images containing similar objects.

37

IBM-QBIC

The Hermitage Web site was voted the best in Russia. It uses the QBIC engine for searching archives of world-famous art. http://www.hermitagemuseum.org/fcgi-

bin/db2www/qbicSearch.mac/qbic?selLang=English

Color percentage Color layout

A sample query

SELECT CONTENTS(image), QBScoreFROMStr(`averageColor= <255,0,0>’, image) AS SCORE

FROM signs ORDER BY SCORE

38

Photobook System

Figure 1. The texture retrieval of PhotoBook system (http://web.media.mit.edu/~tpminka/photobook/).

Search Using Sketch

Sketch entry

Results of search

ImageScape System

Figure 2.5 The interface of ImgeScape visual query system (http://skynet.liacs.nl/imagescape/).

Relevance Feedback in CBIR Motivation:

Human perception of image similarity is subjective, semantic, and task-dependent.

The CBIR based on the similarities of pure visual features are not necessarily perceptually and semantically meaningful. Each type of visual feature tends to capture only one

aspect of image property and it is usually hard for a user to specify clearly how different aspects are combined …

Relevance Feedback is introduced to address these problems. It is possible to establish the link between high-level

concepts and low-level features.

Relevance Feedback RF (cont.)

RF is a supervised active learning technique used to improve the effectiveness of information systems. Main idea: use positive and negative

examples from the user to improve system performance.

Initial query results

Collect user’s feedback

Real-time learning

Refine query results

Query Image

Initial Query Results

User Relevance Feedback

Query Results After User Feedback

Training System Interface

Object-based Image Retrieval

Object-based CBIR: Motivation1. The basic unit of user interests usually is individual objects. 2. Images are segmented into homogeneous regions, and the

image features are extracted for each region. 3. Image similarity is then measured in term of region similarity.

Spatial Indexing

Object-Based Image Retrieval with Relevance Feedback

Techniques used: Image segmentation Neural network Multiple instance

learning …