1

Media fragment re-mixing and playout

Lyndon NixonMODUL University Viennalyndon.nixon@modul.ac.at

RE-USING MEDIA ON THE WEBWWW2014 Tutorial, Seoul, S Korea, April 8 2014

06.01.14 Slide 2 of 32

3. Media fragment re-mixing and playout

• Storing the data: repositories• Querying the data: retrieval• Re-mixing the media: semantics• Play out the media: clients

2

06.01.14 Slide 3 of 32

Repositories

06.01.14 Slide 4 of 32

Storing your RDF data

Like any data, RDF needs to be stored. It can be serialised to different formats like XML and JSON, but this is not the same as the underlying data model.•RDF is a GRAPH, XML and JSON are TREEs -> the same RDF data can be serialised in different XML and JSON syntax•RDF datatypes have different semantics than XML and JSON datatypes

In other words: if you want to persist and manipulate your data as RDF, you should use a RDF specific solution.

06.01.14 Slide 5 of 32

RDF Stores

A RDF Store (or RDF repository, or triple store) is a database specialized for RDF triples.•Can ingest RDF in different serialisations•Supports a RDF specific query language such as SPARQL•May support RDF insert/delete (a.k.a. SPARQL UPDATE)•Can support inferencing over RDF data

Different implementations, can be broadly categorized into:•In-memory: Triples are stored in main memory (usually fastest)•Native: Persistence is provided by using a custom DB, where the database structure is optimised to the RDF model (most commonly used)•Non-native: Persistence is provided by using a third party relational DBMS (e.g. mySQL or postgres) (may be better if there are many updates or a need for good concurrency control)

06.01.14 Slide 6 of 32

RDF Store Overview

Good overview at http://www.w3.org/2001/sw/wiki/Category:Triple_Store (47 listed stores)

Scalability: http://www.w3.org/wiki/LargeTripleStores lists quoted large scale deployments of stores (up to 1+ trillion RDF triples)

Comparison from 9/2012 at http://www.garshol.priv.no/blog/231.html

06.01.14 Slide 7 of 32

OpenRDF Sesame

Sesame is an open source RDF framework for RDF storage, query and inference. •http://www.openrdf.org •Implemented in Java•Supports in-memory, native and non-native stores.

Features•Lightweight yet powerful Java API•SPARQL query language support (including SPARQL UPDATE)•Highly scalable•Can support inference via a RDF Schema reasoner•Transactionality•Context (a.k.a. Named Graphs)

RDF ModelRDF Model

RioRioSAIL APISAIL API

SAIL Query Model

SeRQLSeRQL SPARQLSPARQL

Repository Access APIRepository Access API

HTTP ServerHTTP Serverapplicationapplication

applicationapplication

HTTP / SPARQL protocol

06.01.14 Slide 8 of 32

Insert RDF into Sesame

The central concept in Sesame is the repository. You add and query data by repository.

06.01.14 Slide 9 of 32

Using context

You can distinguish data sets in one repository by using context

This is an additional URI attached to each triple you insert. (thus also referred to as quads)

You can now query or update only on triples within one named context (in RDF, this is called a named graph)Context is useful for provenance tracking and for versioning

If a query does not specify a context, the default context is used. It can be exclusive (only triples without a context) or inclusive (all triples in all contexts or none)

06.01.14 Slide 10 of 32

Browse RDF in Sesame

06.01.14 Slide 11 of 32

Retrieval

06.01.14 Slide 12 of 32

SPARQL RDF query language

RDF data is a „labeled, directed graph“ without any fixed vocabulary (this is defined in RDF Schema). A query language for RDF needs to support this data model:•Tranversing paths in a RDF graph•Being independent of any defined vocabulary•Able to query over the data or the schema

SPARQL („sparkle“) is a W3C standard supported in most RDF tools•SELECT... WHERE.... constructions like in SQL•Path expressions•Additional keywords for more query expressiveness

06.01.14 Slide 13 of 32

Examples: triple patterns

The basic idea in SPARQL is to match graph patterns against RDF graphs.To understand graph patterns we must first define triple patterns:• A triple pattern is similar to a triple (in RDF) but with one or more variables in the place of a RDF resource (URI or literal)

Triple: dbpedia:Lou_Reed foaf:givenName „Lewis Allen Reed“ .

Triple pattern: dbpedia:Lou_Reed foaf:givenName ?name .

?name is the variable. If a RDF graph with the above triple is queried with the below triple pattern, then in the SPARQL results the variable ?name would be ‚bound‘ to the value „Lewis Allen Reed“.

• A SPARQL query result is a set of bindings for the variables appearing in the SELECT clause, based on matching RDF resources to variables in triple patterns in the WHERE clause. .

06.01.14 Slide 14 of 32

Examples: conjunctive and disjunctive query

In graph patterns, several triple patterns are listed within braces { ... } and these are interpreted conjunctively:

Ex: { ?what tech:noOfWheels „4“ . ?what tech:minSpeed „180“ . }

The variable ?what will only be bound to resources which BOTH have 4 wheels AND a minimum speed of 180.

You can also join results from distinct graph patterns using the UNION keyword. Note that result sets from graph patterns and from UNIONs are different, since UNION works disjunctively:

Ex: { ?what tech:noOfWheels „4“ . } UNION { ?what tech:minSpeed „180“ . }

06.01.14 Slide 15 of 32

Examples: FILTER, ORDER BYSPARQL has many other keywords. FILTER restricts variable bindings returned in query results to those for which the filter expression evaluates to true. A filter applies to solutions over the entire graph pattern it is contained in.

Ex: { ?what tech:noOfWheels „4“ . ?what tech:minSpeed ?speed .

FILTER ( ?speed > 170 ) }

The ORDER BY keyword determines the sequence of query results returned according to a sort on the referenced variable‘s bindings, ascending by default:

Ex: { ?what tech:minSpeed ?speed . }ORDER BY ?speed

Or ORDER BY DESC(?speed)

06.01.14 Slide 16 of 32

SPARQL in Sesame

06.01.14 Slide 17 of 32

Query for media fragments: SPARQL-MMResearch work in progress: extending SPARQL to Media Fragments by adding spatio-temporal filter and aggregration functions.

Relation Function Aggregation Function

Spatial mm:rightBeside mm:spatialIntersection

mm:spatialOverlaps mm:spatialBoundingBox

… …

Temporal mm:after mm:temporalIntersection

mm:temoralOverlaps mm:temporalIntermediate

… …

Combined mm:overlaps mm:boundingBox

mm:contains mm:intersection

Courtesy Thomas Kurz (Salzburg Research) & MICO EU projecthttp://demos.mico-project.eu/sparql-mm/sparql-mm/demo/index.html

06.01.14 Slide 18 of 32

Semantics

06.01.14 Slide 19 of 32

Semantic search?

Broadly, any search paradigm which makes use of machine understanding of the data being queried.

Search term „Presidents of the United States“

CLASSICAL SEARCH

Return all media with the string „Presidents of the United States“ in their title or description

SEMANTIC SEARCH

Return all media with an occurance of an instance of the type „Presidents of the United States“ in their metadata, e.g. George W Bush, Bill Clinton, Barack Obama ...

06.01.14 Slide 20 of 32

Search by types or categories

The set of „Presidents of the United States“

The category of „Presidents of the United States“

World Leaders Presidents

Presidents of the United States Bill Clinton Barack Obama George W Bush

Barack Obama

George W Bush Bill Clinton

06.01.14 Slide 21 of 32

Search by paths in the graph

RDF can be more expressive through defining properties with specific semantics; semantic search can be more powerful by using these semantics

However, independent of a RDF vocabulary, shared graph properties can be an indicator of resource similarity & used in domain-independent semantic search

?? < 50

England

Age

BornIn

?v ?

?

?p

?p

06.01.14 Slide 22 of 32

Search with Linked Data

Linked Data: RDF metadata published on the Web, with information about each resource discoverable via its unique URI. Machines can transverse the RDF graph by looking up each URI in turn.

DBPedia: an extraction of the knowledge in Wikipedia (esp. the info boxes) as RDF and published in its own namespace (http://dbpedia.org) Each Wikipedia article of the form http://en.wikipedia.org/wiki/X has a DBPedia resource at the URI http://dbpedia.org/resource/X Has its own ontology which defines resource properties and classes, also maps the Wikipedia categories using the dcterms:subject property.

Advantages: Linked Data URIs allow an unambiguous reference to information about a concept, that information carries the authority of the data provider, all applications have the same understanding of the conceptDisadvantages: information at the URI is not under your control, sometimes messy/dirty (the case with DBPedia) use local copies, extend with own metadata

06.01.14 Slide 23 of 32

SPARQL endpoints and the SERVICE keyword

Linked Data sources usually provide a SPARQL endpoint for their datasets• A SPARQL endpoint is a SPARQL query processing service that supports the SPARQL protocol (http://w3.org/TR/rdf-sparql-protocol): a definition of how to send a SPARQL query to a Web Service and receive a query result.

e.g. DBPedia.org has a SPARQL endpoint at http://dbpedia.org/sparql See a fuller list at http://esw.w3.org/topic/SparqlEndpoints

SPARQL 1.1 includes the SERVICE keyword to include queries to SPARQL endpoints within a SPARQL query addressing a different repository. • With this in Sesame, you can SPARQL query a local repository BUT include bindings from external datasets within the query

Ex: SERVICE <http://dbpedia.org/sparql> { ?resource dcterms:subject . ?subject .

?related dcterms:subject ?subject . }

06.01.14 Slide 24 of 32

SPARQL for semantic search

06.01.14 Slide 25 of 32

Clients

06.01.14 Slide 26 of 32

What is a Media Fragment Player?Regular media players playback full media assets, only the consumer may seek within the media.

A media fragment player supports playback of (spatial and/or temporal) media fragments in place of the full media asset, the customer doesn‘t have to do anything.

We mean a player which can correctly interpret a media fragment as per the W3C Media Fragment URI specification.

06.01.14 Slide 27 of 32

Media Fragments – Web browser support

• Mozilla Firefox, since version 9• Apple Safari, since Jan 2012• Google Chrome, since Jan 2012

Focus on support for the temporal dimension in the HTML5 video player.

06.01.14 Slide 28 of 32

Media Fragments – Javascript libraries

• mediafragment.js: https://github.com/tomayac/Media-Fragments-URI

• xywh.js: https://github.com/tomayac/xywh.js

Focus on implementing the spatial and clock dimensions within browsers.

06.01.14 Slide 29 of 32

Media Fragments – SyNote playerhttp://smfplayer.synote.org/smfplayer/

•Cross-browser/device solution for Media Fragment playback via JavaScript•Supports Media Fragment playback from online video platforms such as YouTube and DailyMotion•Wider video format support than native HTML5 (e.g. also Flash or Windows Media)

06.01.14 Slide 30 of 32

What about Media Fragment servers?The advantage of media fragment playback should be also that ONLY the content of the media fragment needs to be provided by the server to the client.

(in reality, Web servers are still serving the whole media asset, the fragment is determined on the client)

Implementing Media Fragment support on the server side requires new implementations of Web (media) servers... Ninsuma Media Server is a prototypical demonstration of how a server should function with Media Fragments.

http://ninsuna.elis.ugent.be/MediaFragmentsServer

06.01.14 Slide 31 of 32

Our Media Fragment Jukebox

http://www.mediamixer.eu/jukebox/ox/

32

Wrap up