KEYFRAME-BASED VIDEO SUMMARIZATION DESIGNER

Carlos Ramos Caballero

Advisors: Horst Eidenberger and Xavier Giró I Nieto

Contents

Introduction

State of the art

Methodology

Results assessment

Conclusions

2

The application: Designer Master

DEMONSTRATION

3

Contents

Introduction State of the art

Methodology

Results assessment

Conclusions

4

Introduction

Motivation

Designer Master: keyframe-based video summarization interface Object Maps: system for automatic video summarization

5

Graphical User Interface (Designer Master)

Computer Vision Engine (Object Maps)

Introduction

Goals of the thesis

6

Introduction

Goals of the thesis Improving the keyframe extraction module

7

Introduction

Goals of the thesis Improving the keyframe extraction module Assessing the improvement

8

Contents

Introduction

State of the art Methodology

Results assessment

Conclusions

9

State of the art

Shot segmentation

10

Hierarchical decomposition and representation of video content [1]

[1] http://www.scholarpedia.org/article/Video_Content_Structuring

State of the art

Shot segmentation example

11

Shot boundary detection example [2].

[2] Martos, M. “Content-based Video Summarization to Object Maps”, Vienna University of Technology, Austria (2013).

State of the art

Shot segmentation techniques

Pixel-to-pixel methods

• Global pixel-to-pixel • Cumulative pixel-to-pixel

Histogram-based methods • Simple histogram • Maximum histogram • Weighted histogram

Hausdorff method

12

Contents

Introduction

State of the art

Methodology Results assessment

Conclusions

13

Methodology: Implemented solution

System architecture overview

14

Methodology : Implemented solution

Uniform sampling

𝑓𝑝𝑠𝑖: frame rate of the input video.

𝐿𝑖: total number of frames of the input video.

𝑁0: total number of frames we want to keep (𝑁0=100).

15

Methodology : Implemented solution

Gray scale domain

16

Color model transformation RGB to YIQ.

Methodology : Implemented solution

Difference computation

Where 𝐼(𝑡,𝑖,𝑗) represents the intensity value at frame t in pixel(𝑖,𝑗).

X and Y are the width and height of the video frames, respectively.

17

Methodology : Implemented solution

Normalization

Where 𝑑 ̂ is the normalized value, 256 is the number of grey levels, X and Y are the width and height of the video frames, respectively.

18

Methodology : Implemented solution

Decision making

The threshold value used in our application is 𝜏 = 0.1 (as defined in [2]).

19

[2] Martos, M. “Content-based Video Summarization to Object Maps”, Vienna University of Technology, Austria (2013).

Methodology: Environment

Environment

20

Contents

Introduction

State of the art

Methodology

Results assessment Conclusions

21

Results assessment

TEST 1: Testing the applications + ‘in situ’ survey 11 participants Test data: The intouchables trailer

22

Results assessment

Example: pair of summaries

23

Designer Master v1 Designer Master v2

Results assessment

TEST 2: web-based survey 43 participants Test data: The Intouchables trailer

24

Results assessment

EVALUATION Quality of the generated summaries Representativeness of the generated summaries Mean Opinion Score

• 1. Unacceptable • 2. Poor • 3. Good • 4. Very good • 5. Excellent

25

Results assessment

Quality generated summaries

“Please, rate summary 1”

26

“Please, rate summary 2”

Results assessment

Quality generated summaries

27

MOS MOS – scores distribution

Results assessment

Representativeness of the summaries

“Which summary let you better recognize the video content?”

28

Results assessment

Representativeness of the summaries

29

Results assessment

Ease-of-use of the application

“Do you think the application is intuitive and easy to use?”

30

Results assessment

Ease-of-use of the application

31

Results assessment

Execution time

32

Contents

Introduction

State of the art

Methodology

Results assessment

Conclusions

33

Conclusions

Accomplishment of the initial goals Improving the keyframe extraction module by integrating both

projects. Assessing the improvement.

34

Conclusions

Accomplishment of the initial goals Improving the keyframe extraction module by integrating both

projects. Assessing the improvement.

Our work has slightly improved Designer Master Users can create better video summaries and easily due the better

quality of the extracted keyframes.

35

Conclusions

Accomplishment of the initial goals Improving the keyframe extraction module by integrating both

projects. Assessing the improvement.

Our work has slightly improved Designer Master Users can create better video summaries and easily due the better

quality of the extracted keyframes.

It is hoped to develop this work into a product for the Austrian Broadcasting station ORF

36

Conclusions

Accomplishment of the initial goals Improving the keyframe extraction module by integrating both

projects. Assessing the improvement.

Our work has slightly improved Designer Master Users can create better video summaries and easily due the better

quality of the extracted keyframes.

It is hoped to develop this work into a product for the Austrian Broadcasting station ORF

37

Thank you very much for your attention!

Danke schön!

Moltes gràcies!

38