Bernhard Haslhofer

The LEMO Annotation Framework

Open Annotations Meeting

Berkeley, 10/12/2009

Bernhard Haslhofer

Bernhard Haslhofer

What is LEMO?

• A Web-based, multimedia annotation framework, consisting of

• a back-end storage

• front-ends for image, audio, and video annotation

• Demo at: http://dme.arcs.ac.at/annotation/econnect-annotation-showcase.html

2

Bernhard Haslhofer

Driving Requirements...1. objects to be annotated are multimedia

‣ need a uniform model; no content-type specific solution

2. annotations address specific content parts (fragments)

‣ need a uniform fragment identification approach

3. shift towards the Web

‣ need to expose annotation data on the Web (Linked Data)

4

Bernhard Haslhofer

Reuse of existing standards

• Semantic Web standards (RDF/S, OWL, SPARQL)

• RESTful Web Service Design

• Annotea data model & protocol

• XPointer, MPEG-21, SVG, etc. for fragment identification

• ....

5

Bernhard Haslhofer

Why not use Annotea as it is?• Annotea is for the annotation of Web pages; LEMO

should also support annotation of embedded content items

• Annotea doesn’t assign datatypes to its properties; LEMO uses OWL’s (data)typing features

• Annotea doesn’t support multimedia fragment identification; LEMO uses MPEG-21 (multimedia) fragment identification

6

Bernhard Haslhofer

A Uniform Annotation Model

• annotation profiles; cf. application profiles for metadata interoperability

• an Annotea-based core model

• with content- and annotation-type specific model extensions

• implemented as light-weight software components (add-ons)

7

Bernhard Haslhofer

A Uniform Annotation Model

8

Annotea Annotation Schema: http://www.w3.org/2000/10/annotation-ns#

Annotation

annotates author body contextcreated modified related

Annotation Core Schema: http://lemo.mminf.univie.ac.at/annotation-core#

Annotation

annotates author label fragmentcreated modified

subPropertyOf subPropertyOf subPropertyOf subPropertyOf subPropertyOf subPropertyOfsubClassOf

dateTime

string stringdateTime

Key Symbols:

RDFS/OWLClass

RDFSProperty

OWLObjectProperty

OWLDatatypeProperty

XML Schema Datatype

domain range

anyURI

Bernhard Haslhofer

A Uniform Annotation Model

9

Annotation Core Schema: http://lemo.mminf.univie.ac.at/annotation-core#

Annotation

annotates author label fragmentcreated modified

dateTime

string stringdateTime

Text Annotation Schema:http://lemo.mminf.univie.ac.at/ann-text#

Text

Annotation

title description

subPropertyOf subPropertyOf

Annotation Relationship Schema:http://lemo.mminf.univie.ac.at/ann-relationship#

Relation-

ship

isLinkedTo

subClassOfsubClassOf

anyURI

Bernhard Haslhofer

Uniform Fragment Identification

• introduce a fragment element with URI range

• follow MPEG-21 media pointer scheme

• an annotation can define multiple fragments

• URI-encoding of fragments

• minimal consensus

• content-type specific extensions

10

Bernhard Haslhofer

Uniform Fragment Identification

11

12 Bernhard Haslhofer et al.

<rdf:RDFxmlns:a="http :// lemo.mminf.univie.ac.at/

annotation -core#"...">

...

<a:annotates >http ://www.univie.ac.at/test.mpg</a:annotates >

<a:fragment >http ://www.univie.ac.at/test.mpg#mp(~time

(’npt ’,’30’,’40’))</a:fragment >

...

</rdf:RDF >

Listing 1 Time fragment of a video expressed according to

the MPEG-21 fragment identification specification

<rdf:RDFxmlns:a="http :// lemo.mminf.univie.ac.at/

annotation -core#"xmlns:x="http :// lemo.mminf.univie.ac.at/

annotation -video #"...">

...

<a:annotates >http ://www.univie.ac.at/test.mpg</a:annotates >

<a:fragment >http ://www.univie.ac.at/test.mpg#mp(~time

(’npt ’,’30’,’40’))</a:fragment ><x:time_fragment xmlns:mpeg21 ="http :// lemo.

mminf.univie.ac.at/annotation_mpeg21 #"><mpeg21:uri_fid >http ://www.univie.ac.at/

test.mpg#mp(~time(’npt ’,’30’,’40’))</mpeg21:uri_fid >

<mpeg21:time_scheme >npt </ mpeg21:time_scheme >

<mpeg21:start_time >30</ mpeg21:start_time ><mpeg21:end_time >40</ mpeg21:end_time >

</x:time_fragment >

...

</rdf:RDF >

Listing 2 Alternative representation within the add-on

model

sentation while preserving the benefit of interoperablefragment identification via URIs.

Listing 1 shows an MPEG-21 fragment identifier thatlinks to a time segment that starts after 30 seconds andends after 40 seconds of the video resource test.mpg.Listing 2 refers to the same fragment, but adds an ex-panded representation of the fragment to the extendedmodel using a different namespace declaration (http://lemo.mminf.univie.ac.at/annotation-video#). Inaddition to providing better readability, it facilitates thequery process by allowing one to use the already exist-ing SPARQL query interface. Apart from the mandatoryfragment element, it is up to the extension to determinehow to handle dual representations.

Fragment definitions are only useful if a user appli-cation can interpret their meaning. This limitation holdsfor all approaches, but URI fragments have a standard-ized and widespread fallback behavior that is by defaultuseful to retain a minimum relationship. If a fragmentidentifier can not be processed by a user application, thefragment part of the respective URI is ignored and therequested resource is returned. With the limitation oflosing the exact fragment, this behavior preserves therelationship to the resource as a whole.

By using this simple method in LEMO we aim atimproving the interoperability of fragment identificationrepresentations in diverse annotation systems. We be-lieve, as Geurts et al [20] have concluded, that the ubiqui-tous use of URIs will help to solve the problem of defininginteroperable, explicit links between resources and theirannotations.

4.4 Exposing Annotations as Web Resources

Since we follow a REST-based approach, the annotationURIs (e.g., http://www.example.org/annotations/1)are in fact dereferencable URIs, which can be looked upby humans and machines. Therefore the LEMO Anno-tation framework fulfills the first and second linked dataprinciples, as described in Section 3.3.

To fulfill the third principle, LEMO must be able toexpose annotation data in different formats than RDF.Humans typically access Web resources using a browser,which in turn requires an (X)HTML representation inorder to display the returned information. We fulfill thatrequirement by relying on content negotiation, which isa built-in HTTP feature. Figure 5 illustrates how anno-tations can be retrieved in various formats by specifyingthe appropriate mime-type in the HTTP Content-Typeheader field. LEMO forwards client requests for a specificannotation (e.g., http://example.com/annotations/1)to the appropriate physical representation, i.e., http://example.com/annotations/html/1 for HTML requestsand http://example.com/annotations/rdf/1 by send-ing an HTTP 303 See Other response back to the client.

The fourth Linked-Data principle is fulfilled by theinherent nature of annotations: as already mentioned inSection 3.1, an annotation must contain at least the linkto the digital item it annotates. This could be, for in-stance, any multimedia digital item that is exposed onthe Web by a digital library system and therefore ref-erencable via its URI or an already existing annotationexposed by the LEMO Annotation Framework.

5 Existing Annotation Tool Implementations

The LEMO Annotation Framework takes the role of amiddleware that can be integrated with various storageback-ends and serve as controller component for various

Bernhard Haslhofer

Uniform Fragment Identification

12

12 Bernhard Haslhofer et al.

<rdf:RDFxmlns:a="http :// lemo.mminf.univie.ac.at/

annotation -core#"...">

...

<a:annotates >http ://www.univie.ac.at/test.mpg</a:annotates >

<a:fragment >http ://www.univie.ac.at/test.mpg#mp(~time

(’npt ’,’30’,’40’))</a:fragment >

...

</rdf:RDF >

Listing 1 Time fragment of a video expressed according to

the MPEG-21 fragment identification specification

<rdf:RDFxmlns:a="http :// lemo.mminf.univie.ac.at/

annotation -core#"xmlns:x="http :// lemo.mminf.univie.ac.at/

annotation -video #"...">

...

<a:annotates >http ://www.univie.ac.at/test.mpg</a:annotates >

<a:fragment >http ://www.univie.ac.at/test.mpg#mp(~time

(’npt ’,’30’,’40’))</a:fragment ><x:time_fragment xmlns:mpeg21 ="http :// lemo.

mminf.univie.ac.at/annotation_mpeg21 #"><mpeg21:uri_fid >http ://www.univie.ac.at/

test.mpg#mp(~time(’npt ’,’30’,’40’))</mpeg21:uri_fid >

<mpeg21:time_scheme >npt </ mpeg21:time_scheme >

<mpeg21:start_time >30</ mpeg21:start_time ><mpeg21:end_time >40</ mpeg21:end_time >

</x:time_fragment >

...

</rdf:RDF >

Listing 2 Alternative representation within the add-on

model

sentation while preserving the benefit of interoperablefragment identification via URIs.

Listing 1 shows an MPEG-21 fragment identifier thatlinks to a time segment that starts after 30 seconds andends after 40 seconds of the video resource test.mpg.Listing 2 refers to the same fragment, but adds an ex-panded representation of the fragment to the extendedmodel using a different namespace declaration (http://lemo.mminf.univie.ac.at/annotation-video#). Inaddition to providing better readability, it facilitates thequery process by allowing one to use the already exist-ing SPARQL query interface. Apart from the mandatoryfragment element, it is up to the extension to determinehow to handle dual representations.

Fragment definitions are only useful if a user appli-cation can interpret their meaning. This limitation holdsfor all approaches, but URI fragments have a standard-ized and widespread fallback behavior that is by defaultuseful to retain a minimum relationship. If a fragmentidentifier can not be processed by a user application, thefragment part of the respective URI is ignored and therequested resource is returned. With the limitation oflosing the exact fragment, this behavior preserves therelationship to the resource as a whole.

By using this simple method in LEMO we aim atimproving the interoperability of fragment identificationrepresentations in diverse annotation systems. We be-lieve, as Geurts et al [20] have concluded, that the ubiqui-tous use of URIs will help to solve the problem of defininginteroperable, explicit links between resources and theirannotations.

4.4 Exposing Annotations as Web Resources

Since we follow a REST-based approach, the annotationURIs (e.g., http://www.example.org/annotations/1)are in fact dereferencable URIs, which can be looked upby humans and machines. Therefore the LEMO Anno-tation framework fulfills the first and second linked dataprinciples, as described in Section 3.3.

To fulfill the third principle, LEMO must be able toexpose annotation data in different formats than RDF.Humans typically access Web resources using a browser,which in turn requires an (X)HTML representation inorder to display the returned information. We fulfill thatrequirement by relying on content negotiation, which isa built-in HTTP feature. Figure 5 illustrates how anno-tations can be retrieved in various formats by specifyingthe appropriate mime-type in the HTTP Content-Typeheader field. LEMO forwards client requests for a specificannotation (e.g., http://example.com/annotations/1)to the appropriate physical representation, i.e., http://example.com/annotations/html/1 for HTML requestsand http://example.com/annotations/rdf/1 by send-ing an HTTP 303 See Other response back to the client.

The fourth Linked-Data principle is fulfilled by theinherent nature of annotations: as already mentioned inSection 3.1, an annotation must contain at least the linkto the digital item it annotates. This could be, for in-stance, any multimedia digital item that is exposed onthe Web by a digital library system and therefore ref-erencable via its URI or an already existing annotationexposed by the LEMO Annotation Framework.

5 Existing Annotation Tool Implementations

The LEMO Annotation Framework takes the role of amiddleware that can be integrated with various storageback-ends and serve as controller component for various

Bernhard Haslhofer

Annotations as Web Resources

13

+ URI

Resource

Annotation

annotates

1

0..*

Bernhard Haslhofer

Annotations as Web Resources

14

Client LEMO

GET http://example.com/annotations/1Accept: text/html

303 See OtherLocation http://example.com/annotations/html/1

GET http://example.com/annotations/html/1Accept: text/html

200 OK<HTML>

...</HTML>

Dereference an annotation URI, requesting HTML content

Client LEMO

GET http://example.com/annotations/1Accept: application/rdf+xml

303 See OtherLocation http://example.com/annotations/rdf/1

GET http://example.com/annotations/rdf/1Accept: application/rdf+xml

200 OK<RDF>

...</RDF>

Dereference an annotation URI, requesting RDF content

Bernhard Haslhofer

Current LEMO developments

• integration into Europeana portal

• enhancing multimedia support on user-interface

• integration with semi-automatic (multimedia) metadata extraction techniques

• investigating adoption of fragment identification working groups recommendations

15