A Stepwise Modeling Approach for Individual Media Semantics

Annett Mitschick, Klaus MeißnerTU Dresden, Department of Computer Science, Multimedia Technology Group

„Applications of Semantic Technologies“ (AST' 06)

06.10.2006 AST 2006 2

Outline

• Introduction• Approach:

- Semantics for Personal Media Management

- Information Instantiation and Integration

- Implemented System Architecture

• Conclusion and Future Work

06.10.2006 AST 2006 3

Introduction

• Media Management @ Home:- Growing amount of digital assets (text,

images, video, audio)• complex, high-dimensional

- User: • non-/semi-professional• individual conceptualizations • usually minimum efforts to

annotate/organize

- Tools mostly apply administrative (objective) metadata vs. user interpretation/conceptualization

• Goal: automated modeling of essential semantic descriptions from existing metadata to ease annotation work

06.10.2006 AST 2006 4

Introduction

• Solutions:- Automated feature extraction and indexing techniques to manage

complexity of huge multimedia collections• low-level features Similarity-based Search (Query-by-Example, etc.)• higher-level features Pattern Recognition/Classification (objects,

people, genres, etc.) requires prototype features (user context, user feedback)

“Semantic Gap”

- Media descriptions based on common metadata standards to enable interoperability & comparability

• MPEG-7 Multimedia Description Schema• RDF/OWL representation for semantic modeling

- Semantic Desktop• Personal Information Management based on Semantic Web Technologies

06.10.2006 AST 2006 5

Approach :: Semantics for Personal Media Management

• Knowledge representation - Allow description of background-knowledge (context information like

events, activities, actors, locations, etc.)

- Allow integration of domain models and individual extensions (e.g. integration of MPEG-7 MDS, FOAF for description of people, etc.)

- Manageable, straightforward for non-professionals

• ABC [Hunter, et al.]- “facilitate interoperability between metadata ontologies“, “conceptual

basis for automated mapping …”, “guidance to communities…”

- Base classes to attach domain-specific entities MPEG-7 MDS

- Temporal semantics and related actualities (who, what, when, where) evolution/transition of objects

06.10.2006 AST 2006 6

Approach :: Semantics for Personal Media Management

• Acquisition of as much information as possible from content + context

- File information, administrative metadata, structural features, etc.

• Information sources depend on application context- naming, availability/accessibility, semantics, etc. of features/attributes

e.g. “Date Time Original”/”Date Created”/…, “Creator”/”Author”/…, etc.

- properties might be relevant, or redundant in some contexte.g. “EXIF:ExposureTime” and “EXIF:ShutterSpeedValue”

Metadata/FeatureExtraction

EXIFIPTC MPEG-7 ...

Metadata/FeatureExtraction

ID3CDDBMPEG-7 ...

Metadata/FeatureExtraction

DCXMPBibTex ...

Metadata/FeatureExtraction

SMILSMPTEMPEG-21 ...

06.10.2006 AST 2006 7

• First step: - Apply sets of mapping rules to construct media-centric description

(MDS)

- Mapping rules expandable regarding supported metadata schemes( developer)

Approach :: Information Instantiation and Integration

MDS

Media-Analysis

EXIF XMP IPTCDC ...

R2

III

R1

III

ABC

RDF-Repository

Make=FUJIFILMModel=FinePix F601

[(urn:Exif:Make rdf:value ?j) (urn:Exif:Model rdf:value ?k) (urn:Filepath rdf:value ?l) hashCode(?l, ?t) makeTemp(?x) -> (?t mds:creationTool ?x) (?x mds:make ?j) (?x mds:model ?k) (?x rdf:type mds:Device)]

<rdf:Description rdf:about="urn:Exif:Make"> <rdf:value …>FUJIFILM</rdf:value></rdf:Description><rdf:Description rdf:about="urn:Exif:Model"> <rdf:value …>FinePix F601</rdf:value></rdf:Description>

<rdf:Description rdf:about="#ins61737146"> <mds:creationTool rdf:nodeID="A0"/> ...</rdf:Description> ...<rdf:Description rdf:nodeID="A0"> <mds:make>FUJIFILM</mds:make> <mds:model>FinePix F601</mds:model> <rdf:type rdf:resource="#Device"/></rdf:Description>

06.10.2006 AST 2006 8

Approach :: Information Instantiation and Integration

• Next steps:- Apply derivation rules (schema level)

to establish relationships to actualities and temporalities( developer, user)

- Apply classification rules (instance level)to refine description model regardingknown pattern ( user)

MDS

Media-Analysis

EXIF XMP IPTCDC ...

R2

III

R1

III

ABC

RDF-Repository

06.10.2006 AST 2006 9

Approach :: Implemented System Architecture

• Design rationale:- Automated import and indexing of media items triggered by file system

actions

- Registry of media type specific analyzing components providing metadata and feature extraction

- Processing of semantics and ontology model wrapped by generic model API independent from RDF/OWL storage and querying solution (Jena, Sesame, …)

- Extensible, flexible service architecture plug-in based setup, extension at run-time(OSGi service platform)

MediaSemantics

Image-Analyzer

Text-Analyzer

Audio-Analyzer Model

Image-Semantics

Text-Semantics

Audio-Semantics

Jena RDF Framework

Rule Manager

RDF-Repository

MediaImporter

GUI

Media Repository

06.10.2006 AST 2006 10

Conclusion and Future Work

• Our approach:- Part of the K-IMM (Knowledge through Intelligent Media Management)

project managing private media collections with the help of semantic technologies

- Stepwise modeling of resource descriptions to separate distinct problems of construction of media semantics

• Building objective media descriptions from various sources• Deriving subjective, user-oriented semantics

• Future Work:- Strategies to extend and refine basic rule sets by means of user

feedback, context capture- Distribution of modeling steps (Web Services/Agents)- Detailed, comprehensive evaluation

06.10.2006 AST 2006 11

Thank you for your attention!

Annett Mitschickannett.mitschick@inf.tu-dresden.de

Multimedia Technology Group, Department of Computer Science, TU Dresdenhttp://www-mmt.inf.tu-dresden.de