MediaScope: Selective On-Demand Media Retrieval from Mobile Devices

Yurong Jiang, Xing Xu, Peter Terlecky, Tarek Abdelzaher, Amotz Bar-Noy, Ramesh Govindan

1IPSN 2013

Availability Gap 2

Trends in Photo and Video Uploads – More & More Delayed• increasing bandwidth requirements

• high-resolution cameras• restricted usage limits for cellular data plan

• restricted cellular capacityDelayed Uploads Resulting in – Availability Gap

Motivation Problem Design Evaluatio

nConclusio

n

Availability Gap in 3

Study of Uploads• 40 users• 50 images/user

50% Images are Uploaded 10+ Days Later! – Not Recent!

Motivation Problem Design Evaluatio

nConclusio

n

0 6 12 18 24 30 36 42 48 540

0.5

1

Availability Gap (Days)

%

Bridge the Availability Gap ? 4

Mall Robbery Sporting Event

Motivation Problem Design Evaluatio

nConclusio

n

Bridge the Availability Gap 5

Distributed Image Database: Most Recent, Most Diverse

Motivation Problem Design Evaluatio

nConclusio

n

MediaScope: Timely On-Demand Media Retrieval 6

Motivation Problem Design Evaluatio

nConclusi

on

On-Demand Retrieval Bridges the Availability Gap

Timely Retrieval Can be Important for Many Apps

Images from today’s game…

Respond in 30 seconds…

MediaScope Approach 7

Motivation Problem Design Evaluatio

nConclusi

on

NBA USA

MediaScope Queries – for Different Needs8Top-K

• Find K most Similar Images to Target Image

Spanner

• Find a Collection of most Dissimilar Images

Motivation Problem Design Evaluatio

nConclusi

on

Challenges and Contributions 9

Mediascope – Timely On-Demand Image Retrieval

Selecting Relevant Query Results

• Challenge: image search computationally-expensive• Contribution: adapt & generalize image-search

technique

Return Results in a Timely Manner

• Challenge: variable and limited wireless bandwidth• Contribution: optimize information-content uploaded

Motivation Problem Design Evaluatio

nConclusi

on

Img3 = { Loc = {Philly, USA}, Time = 2013-0408T09:00:00},Visual Feature = {0, 7, 0, 3, … }}

Img2 = { Loc = {L.A., USA}, Time = 2013-0316T10:00:00},Visual Feature = {1, 2, 0, 4, … }}

Img1 = { Loc = {Beijing, China}, Time = 2012-0716T19:20:30},Visual Feature = {0, 3, 2, 5, … }}

Image Search – on Image Feature Space 10

Location & Time – Filtering Out Irrelevant Media Files

Motivation Problem Design Evaluatio

nConclusi

on

Feature Vectors for Images – Support Geometric Queries

Visual Feature = {0, 3, 2, 5, 6, … }

• statistical summary• color histogram

Similarity Metric: Sim(x1, x2)

• n coefficients feature vector• a point in n-dimensional space

• Sim(x1, x2) ~ Distance(x1, x2)

Similarity on Feature Vectors 11

Motivation Problem Design Evaluatio

nConclusi

on

Medusa

MediaScope System Overview

Motivation Problem Design Evaluatio

nConclusi

on

MSCloud

MSCloudQMSCloudDB

MSMobile

ObjectUploader

FeatureExtractor

MSMobile - Feature Extractor 13

CEDD – Color & Edge Directivity Descriptor• 144 coefficients ranging from [0..7]• 54 bytes / image

Image Resizing before Feature Extraction• short feature-extraction time• acceptable error rate

Motivation Problem Design Evaluatio

nConclusi

on

Erro

r R

ate

(%)

1632x1224

1280x960 1024x768 960x720 816x612500

1500

2500

3500

4500

Resolution

Tim

e (m

s)

1632x1224

1280x960 1024x768 960x720 816x612500

1500

2500

3500

4500

0510152025

Resolution

Tim

e (m

s)

1.5s proc. time

4% error rate

Geometric Queries in MediaScope 14

Motivation Problem Design Evaluatio

nConclusi

on

Top-K Spanner Cluster Representative

max(Sim(x, query)) min(max(Sim(x1, x2))

Clusteringmax(Sim(x, center))

O(V logV) O(V2) O(IV)

Similar x1, x2 Greater Sim(x1, x2) ValueIn Particular, Sim(x, x) = ∞

MSCloud – Timely Retrieval for Concurrent Queries15

Trading off Query Completeness for Timeliness

• maximize amount of retrieved information• credit-assignment mechanism

Q1 Q1

Q1 Q1 Q1 P1

P2

Q1

Q2 Q2 Q2

Q2 Q2 Q2 Q2

Q2

Motivation Problem Design Evaluatio

nConclusi

on

1000

MSCloud – Queries & Credit Assignment 16

Each Query is Assigned Credits• divide up credits among selected images by importance

Spanner

• credit ∝ similarity to target image

• (credit)-1 ∝ average similarity to other images

Top-K

Motivation Problem Design Evaluatio

nConclusi

on

Credit Assignment ~ Feature Space Geometry

800 100 100

MSCloud – Credit Based Scheduling 17

From P1From P2

From Q1

From Q2

Optimization Goal: Maximize Uploaded Credits

Motivation Problem Design Evaluatio

nConclusi

on

MSCloud – Credit Based Scheduling 18

Q1 Q1

Q1 Q1 Q1P1

P2Q1 1. Filesize

2. Credit

3. Deadline

MSCloud

MSMobile

Motivation Problem Design Evaluatio

nConclusi

on

P1

MSCloud – Credit Based Scheduling @ Phone 19

{(Filesize, Credit, Deadline)}• max(uploaded credits on-time)

Optimal Scheduling for Same File Size (Same Uploading Time)• arrange images by deadlines• always give up smallest credit object

Motivation Problem Design Evaluatio

nConclusi

on

MediaScope Evaluation 20

Motivation Problem Design Evaluati

onConclusi

on

Prototype

MSCloud

• MSCloudQ (4300 LOC)• PHP, Python

• MSCloudDB• MySql

MSMobile

• Java (1100 LOC)

Evaluation

Setup• MSCloud - Dell XPS 7100• MSMobile - 8 Android

PhonesMetric• Query Completeness

Methodology• Query Trace Replay

Query Completeness 21

MCF: Max Credit First

EDF: Earliest Deadline First

RR: Round Robin

OMNI: Omniscient

MSC: MediaScope Credit-Based

Motivation Problem Design Evaluati

onConclusi

on

MediaScope Overhead 22

Communication & system Overhead Average Latency (ms)

C2DM Message 150Feature Vector Download 138Query Result Response 54

Upload Scheduling 46Query Parsing 24

Component overhead Energy (µAh)Feature Extraction 331

Low Latency!

10% Battery Consumption ~ 400+ images

Motivation Problem Design Evaluati

onConclusi

on

MediaScope Summary 23

Motivation Problem Design Evaluatio

nConclusi

on

MediaScope: Timely On-Demand Media Retrieval

• accurately & efficiently extracts visual features

• supports geometric queries over feature space

• timely returns informative retrieval results for queries

Bridges Availability Gap

24Query Sample

Query Completeness 25

Motivation Problem Design Evaluati

onConclusi

on

MediaScope – Related Work 26

Motivation Problem Design Evaluatio

nConclusi

on

Content Based Image Retrieval• Faced Image Search, Virage Image Search Engine,

ImgSeek• Search on Local Database vs. Mobile Setting

Image Search on Mobile Devices• CrowdSearch• Centralized Database vs. Distributed Database from

Mobile Device

General Image Search Problem• Similarity Match (k-NN with k=1)• Geometric Search Queries on Feature Space