CUbRIK Presentation 11/5/2013

Building social graphs from images through expert-based crowdsourcing

M. Dionisio, P. Fraternali, D. Martinenghi, C. Pasini, M. Tagliasacchi, S. Zagorac (Politecnico

Di Milano, Italy)

E. Harloff, I. Micheel, J. Novak (European Institute for Participatory Media,

Germany)

1/5/2013 CUbRIK Presentation 2

The CUbRIK project

CUbRIK is a research project financed by the European Union whose main goals are:1. Advance the architecture

of multimedia search2. Exploit the human

contribution in multimedia search

3. Use open source components provided by the community

4. Start up a search business ecosystem

1/5/2013 CUbRIK Presentation 3

The CUbRIK architecture

The CUbRIK architecture is layered in four main tiers

1. Content and user acquisition tier

2. Content processing tier3. Query processing tier4. Search tier

1/5/2013 CUbRIK Presentation 4

History Of Europe use case

HoE Dataset(3924 pictures

shot from the end of World War II to the most recent

years of EU history)

Automatic face

recognition tool+

Crowdsourced validation

of face matches

Social Graph

1/5/2013 CUbRIK Presentation 5

Content processing pipeline

In the initial proof of concept we designed a prototype for a face recognition service that combined automatic mechanisms for face detection/recognition and a general purpose crowd.

Group photos

Face detection

Bounding boxes

Face matching

Annotated portraits

Face detection

Bounding boxes

Top – 10 similarities for crowd validation

1/5/2013 CUbRIK Presentation 6

Limits of a purely automatic processing

False negatives

False positives

1/5/2013 CUbRIK Presentation 7

Limits of a purely automatic processing

Matching score = 0.185

Matching score = 0.210

The matching score between two faces of the same person is not always the highest

one

1/5/2013 CUbRIK Presentation 8

Using general purpose crowds We interfaced a general purpose crowd for the validation

of the top-10 matches.

1/5/2013 CUbRIK Presentation 9

Results of the first proof of concept

574 faces extracted from group photos Only 17% of them were identified by the

crowd Of this 17% the 66% of the matches were

correct The automatic tool identified the 80% of the

faces correctly

1/5/2013 CUbRIK Presentation 10

Results of the first proof of concept

These weak results were influenced by several factors:

1. Influence of image taking times2. Limited size of the ground truth3. Image resolution constraints4. Replicability and trustworthiness of the

results

1/5/2013 CUbRIK Presentation 11

Interfacing the expert based crowd

The deficiencies encountered using a general purpose crowd can be overcome by adopting an expert-based crowdsourcing.

combined implicit and explicit expert-based crowdsourcing

interface

1/5/2013 CUbRIK Presentation 12

Interfacing the expert based crowd

Indications suggest that the expert-based strategy can succeed:

1. Experts’ knowledge can overcome the drawbacks both of the automatic tool and of the general purpose crowd

2. They can use the already existing community means to contact colleagues and cooperate to fulfill the task.

1/5/2013 CUbRIK Presentation 13

Interfacing the expert based crowd

Thank you!