of 32
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
1/32
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
2/32
Presentation
Annotation
Collection management
Tag/Content Based
Retrieval tools
Ingestion
Transcoding
Indexing
Client-side Server-side
What is Loki?
A modular framework for the creation of web-based interfaces able tohandle different type of media
Retrieval engine
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
3/32
Who can use Loki?
! Researchers who need an interface to demo their own
multimedia annotationsystem
! Researchers who need to create ground truth annotations
to create datasetsor validate experiments(e.g. usingAmazon Mechanical Turk)
! Lecturers and teachers who need an easily installable
tool for their teaching lab activity
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
4/32
Multimedia search UI
Image component
Video component
Doc component
Audio component
Text indexing and search
LIRE: Lucene Image Retrieval
Solr / LuceneWhats inside Loki
Storage
Multimedia tools
Media transcoding
Media analysis pipelines
media, metadata, collectionsand users management
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
5/32
! Front-end development according to HTML5 standards
! AngularJS for web application using the MVC design
pattern.
! Media components developed as jQuery plugins using
properties to personalize their functionalities and
appearance.
Client side
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
6/32
Video
Synchronized annotations / speech transcription
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
7/32
Video
Frame-accurate annotation
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
8/32
Video
Frame-accurate annotations search
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
9/32
VideoSimilar frames proposed based on CBIR
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
10/32
Audio
Synchronized annotations / subtitles
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
11/32
Audio
Timecode accurate annotation and search
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
12/32
Global image annotation
Image
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
13/32
Local image annotation
Image
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
14/32
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
15/32
Page thumbnail previewDocument
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
16/32
Local textual annotationsDocument
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
17/32
Textual search with page previewDocument
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
18/32
Loki - a cross media search interface
Media-based results collection
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
19/32
Loki - a cross media search interface
Textual queries
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
20/32
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
21/32
Loki - a cross media search interface
Tag-based facets for result filtering
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
22/32
Loki - a cross media search interface
Result clusters based on similarity to enhance diversity
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
23/32
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
24/32
Enable Solr for content based image retrievalby:
1) converting image featuresinto indexed text based documents
2) hashing featuresso to allow inverted index search for sub-linear
retrieval
Lire Solr plugin supports global
image features like MPEG-7ColorLayout, EdgeHistogram,
JCD, PHOG, and color histogram
in the opponent color space.
LIRE: A Solr plugin for CBIR
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
25/32
Loki and LIRE
CBIR results
Query by example
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
26/32
Loki and LIRE
CBIR results
results cluster based
on visual similarity
Query by example
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
27/32
Upload
Manager
Media transcoding and analysis
Transcode
for HTML5
Extract
keyframes
Index keyframes
for CBIR
Loki
Videocontent
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
28/32
Upload
Manager
Media transcoding and analysis
Text
extraction
Generate
SVG pages
Loki
Docs / PPT
Indexing
Generate
JPEG thumbnails
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
29/32
Live demo
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
30/32
8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
31/32
To deploy Loki you need:
! Apache + MySQLfor web interface and storage
! Apache Tomcat for with Solrinstallation
Installation & Licenses
LOKI Github repository
https://github.com/miccunifi/Loki
Apache License 2.0
LIRE Bitbucket repository
https://bitbucket.org/dermotte/liresolr
GNU General Public License (GPL) v2
https://bitbucket.org/dermotte/liresolrhttps://github.com/miccunifi/Loki8/9/2019 Loki: A Cross-Media Search Engine: Acm2014 OSSC Presentation
32/32
Acknowledgment
Part of the research leading to these results has received funding from the EU 7th FP managed
by REA (FP7/2007-2013) under grant agreement no. 262428
G. Becchi, M. Bertini, L. Cioni, A. Del Bimbo, A. Ferracani, D. PezzatiniMedia Integration and Communication Center University of Florence, Italy.
[name.surname]@unifi.it
Mathias LuxITEC Klagenfurt Klagenfurt University, Austria
Contacts
