RUCoD in CUbRIK at Star2013

STAR 2013 Lausanne, 18-19 January 2013

Rich Unified Content Description (RUCoD)

Vincenzo Croce

Research & Development Laboratory

1


Objectives 2

To develop a formal description for each type of multimedia content (text, audio, image, video and 3D content)

To develop formal descriptions for real world and social information

To clearly specify the format of the Rich Unified Content Description (RUCoD)

To represent in the same format, the actual content (multimedia information) along with the additional contextual information (real world, user-related)


The Concept of Content Object

3

“A Content Object is the representation of a specific instance of either a physical object or a physical entity (an

entity that has physical existence), which might have multiple views (many images, videos, audio files, text, real-

world and user-related information).”

Similar approaches: Multimedia Document (MMD): a set of co-occurring

multimedia objects (e.g. images, audio and text) that are of different modalities but carry the same semantics. If two multimedia objects are in the same MMD, they can be regarded as context of each other.

Multimedia Bag: defines a container including text instances, image instances and audio instances that share the shame semantic concepts


The Concept of Content Object

4

A CO may consist of several media types, user-related information and real-world information.

A CO can be the result of an authoring process (e.g. using an authoring tool).

RUCoD is a formal representation of a CO consisting of descriptions of various media types somehow associated to each other.


Example Content Object 5

Great (Cheops) – Pyramid of Giza

CO: Great (Cheops)

– Pyramid of Giza

3D objectImage

Text:It is believed the pyramid was

built as a tomb for Fourth dynasty Egyptian pharaoh Khufu(or Cheops) and constructed over a 14 to 20 year period concluding around 2560 BC…

Real-world (location)


RUCoD Specification 6

Header Low-Level Descriptors Real World Descriptors User-related Descriptors

Text-based descriptors

Image descriptors

Video descriptors …

Low-Level Descriptors

Audio descriptors

3D descriptors

Date Time Position (GPS, etc)

Weather (temperature) …

Real World Descriptors

Sensors

CO ID Version CO Types Multimedia RW-info U-Info …Header

Creator

Expressions Emotions …User-related Descriptors

Valence Arousal


RUCoD Structure 7

He

ad

er

De

scrip

tion

L_Descriptor:

-“TextType”- “Object3D”- “ImageType”- “SoundType”- “VideoType”

R_Descriptor:

- “ContextType”

U_Descriptor:

- “Valence”

- “Arousal”


RUCoD Structure (Header) 8

<Header>

<ContentObjectType>Physical Object</ContentObjectType>

<ContentObjectName xml:lang="en-US">My Bulldog Barking</ContentObjectName>

<ContentObjectID>3577B5EF-523F-4946-9734-C974CEA6C646</ContentObjectID>

<ContentObjectVersion>1</ContentObjectVersion>

<ContentObjectCreationInformation>

<Creator> <Name>CERTH</Name></Creator>

</ContentObjectCreationInformation>

<ContentObjectTypes>

<MultimediaContent type="Text">

<FreeText>It is the image, video and 3D representation… </FreeText>

</MultimediaContent>

<MultimediaContent type="Object3D">

<MediaName>Bulldog</MediaName>

<FileFormat>x-world/x-vrml</FileFormat>

<MediaLocator>

<MediaUri>http://3d-test.iti.gr/dog.wrl</MediaUri>

<MediaPreview>http://3d-test.iti.gr/dog.jpg</MediaPreview>

</MediaLocator>

</MultimediaContent>

<MultimediaContent type="ImageType">

…</Header>


RUCoD Structure (L-Descriptor) 9

<L_Descriptor type="Object3D">

<MediaName>BulldogLR</MediaName>

<Shape3DDescription type="CMVD" matching="MultiViewL2">

<LowLevelDescriptor totalNumOfViews="18" totalNumOfDescriptors="212" descriptorType="xsd:float" descriptorSize="3816">

<Store type="Text">

<DescriptorLocator>

<DescriptorUri>http://3d-test.iti.gr/CMVD.descr</DescriptorUri>

</DescriptorLocator>

</Store>

</LowLevelDescriptor>

</Shape3DDescription>

</L_Descriptor>

<L_Descriptor type="SoundType">

<MediaName>BulldogSound2</MediaName>

<AudioDescription type="BarkBands" matching="BrayCurtis">

<LowLevelDescriptor totalNumOfDescriptors="216" descriptorType="xsd:float" descriptorSize="8 27">

<DescriptorValues>

2.31510340412e-12 3.04525744899e-11 2.56972665369e-10 6.54444409776e-09 2.40772557447e-09 2.14078905714e-08 …

</DescriptorValues>


</AudioDescription>

</L_Descriptor>


RUCoD Structure (R-Descriptor) 10

<R_Descriptor>

<RealWorldDescription type="ContextType">

<ContextSlice>

<Importance>1</Importance><DateTime>

<Date>1997-07-16T19:20:30.45+01:00</Date><Length>100</Length>

</DateTime>

<SubjectPosition>

<gml:CircleByCenterPoint numArc="1"> <gml:pos>45.8419444 13.4002778</gml:pos>

<gml:radius uom="M">10</gml:radius>

</gml:CircleByCenterPoint>

</SubjectPosition>

<Weather>

<Condition>OVC RA</Condition><Temperature>20</Temperature><WindSpeed>2</WindSpeed><Humidity>94</Humidity>

</Weather>

</ContextSlice>

</RealWorldDescription>

</R_Descriptor>


RUCoD Structure (U-Descriptor) 11

<U_Descriptor type="UserType">

<MediaName>aerosmith-Aerosmith-01-Make_It.mp3</MediaName>

<UserDescription matching="L2Distance" type="AvgValenceArousal">

<LowLevelDescriptor descriptorSize="1 1“

descriptorType="xsd:float" totalNumOfDescriptors="2">

<DescriptorValues>-0.4898 0.42857</DescriptorValues>


</UserDescription>

</U_Descriptor>


RUCoD Schema 12

• RUCoD Schema Final Version (1.4.1) released in November 2011:• The RUCoD.xsd Schema file

• The RUCoD_Descriptors.xsd Schema file

http://www.isearch-project.eu/isearch/RUCoD/


Block diagram of the I-SEARCH framework 13


Authoring & Content Analytics Architecture

14

Authoring

Content Analytics

Indexing triggering


RUCoD in I-SEARCH Use Cases

15

UC1: Music retrieval•Audio (music)•Text•Images•Real-world (location, time)•Emotions

UC2: Social retrieval of music •Audio (music)•Text•Video clip•Real-world (location)•Emotions

UC3: Furniture retrieval•Text•3D objects•Images•Real-world (location, time)

UC4: Search for Multimedia using smartphone•Text•3D objects•Images•Real-world (location, time)

UC5: Search for specific product•Text•3D objects•Images•Video •Audio (sounds)•Real-world (location, time)•Emotions

UC6: 3D game component retrieval •Text•3D objects•Images•Audio (sounds)•Real-world (location, time)

UC7: Game avatar retrieval •3D objects•Images•Video•Emotions

Rich Unified Content

Description


Comparison with MPEG-7 16

What we use:• MediaLocator and MediaUri are used to describe the link

to a specific media item.

• Creator is used for description of the author of a media item.

• Annotation as a part of RUCoD represents textual information of a media item or CO.

• Image/Video/Audio Descriptors are used for the low-level descriptions of the separate media items within a CO.

• Segment is used to describe a temporal video segment.



What we adapted:• ContentObjectName,

ContentObjectCreationInformation instead of name and CreationInformation to represent the name and creators of COs.

• TextDescription, Shape3DDescription, ImageDescription and VideoDescription, similar to MPEG-7 ContentDescription to distinguish between the descriptors of different modalities inside the same RUCoD.



What is new:• Cross-modal & multimodal retrieval are not entirely supported by

the standards. • New types of information describing the COs are introduced, such

as real-world descriptors and user-related descriptors. These enrich the CO description and improve the retrieval performance, by introducing new querying capabilities.

• With respect to low-level descriptor extraction for media items, novel descriptors are introduced. As an example, for 3D content description, new state-of-the-art descriptors are introduced, which achieve higher retrieval performance than those included in MPEG-7. Similarly, new descriptors are introduced for image, video and audio content.

• The low-level description of media items is also accompanied by specification of the matching scheme for each descriptor. In this case, the description scheme does not leave the responsibility for choosing the appropriate matching method to the search engine.


Comparison with JPSearch

What is similar/different:• JPSearch is designed in a way that decouples the

components of image search and provides a standard interface between these components. Its aim is to build a standard for interoperability among image search and retrieval systems

• RUCoD specification is focused on the description of COs and it addresses a broad range of media (apart from images), real-world and user-related information


Comparison with MPEG-21

What we use:• A multimodal approach to media (which can be of any

type)• Allowing the creation and attachment of rich metadata

to digital objects



What is similar:• Ability to create multimedia content objects (Digital

Items in MPEG-21)• Ability for content adaptation, although achieved

differently (e.g. through FileFormat elements in the RUCoD)

• L-Descriptors and R-Descriptors of RUCoD could be attached to MPEG-21 objects although not directly foreseen by the standard



What is different/new:• The CO broadens the concept of Digital Item making it

more general and flexible• Unification of the actual metadata and descriptors

(e.g. L-Descriptors) together with real world and user-related parts in the same format.

• RUCoD is particularly targeted at indexing, sharing, search and retrieval

• RUCoD overcomes the traditional hierarchical object model (also foreseen in MPEG-21) allowing for more flexible and user-centric 'connections' (e.g. RelatedSemanticConcepts field)


Ongoing Work

• RUCoD initially designed to serve the needs of the I-SEARCH framework

• However, it is not mature enough to be used in a wider range of applications.

• RUCoD will be extended within the EU-funded project CUbRIK (CERTH and ENG are participants).

• RUCoD will be adopted in CUbRIK.• A first attempt: SMILA Hackathlon, November

2011, Keiserslautern. • RUCoD was presented in the workshop• RUCoD was used as descriptor scheme to check indexing and

search within the SMILA engine


24

Questions?

Date post:	19-Jun-2015
Category:	Technology
Upload:	cubrik-project
View:	219 times
Download:	0 times

RUCoD in CUbRIK at Star2013

Technology