1
Smart Photo Selection: Interpret Gaze as Personal Interest
Tina Walber1, Ansgar Scherp2,3, Steffen Staab1
1 Institute WeST, University of Koblenz, Germany2 Kiel University, Germany3 Leibniz Information Center for Economics, Kiel, Germany
Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 2
Managment of Digital Photos● Its a mess!● We take a lot of photos● Manually selecting photos is cumbersome● Like to have photo selections for
– Sharing photos online
– Creating photo products like photo books
– Creating presentations
Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 3
State of the Art: Automatic Creation of Photo Selections
● Content-based approaches– Analysis of low-level features
● Context-based approaches– Analysis of context information
● What about individual aestetics, personal preferences, user interests, ….?
Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 4
Interpret Gaze as Personal Interest
● Gaze delivers information on user's interest● Useful for creating individual photo selections?
● Principal approach – Merely observe what users are doing anyway
– Do not ask to perfom additional tasks
Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 5
● Starting from $99
Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 6
Research Questions
1. Is there a need for individual photo selections?
2. Does a gaze-based selection outperform selections based on content and context analysis when comparing to those created manually?
3. Does the personal interest in a viewed photo set have an impact on the obtained selection results?
Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 7
Experiment Setup
Photo Viewing
Task: „get an overview“
Step 1
Recording of the eye tracking data
Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 8
Photo Viewing
32 pages with 9 photos each
Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 9
Experiment Setup
Photo Viewing
Task: „get an overview“
Step 1
Photo Selection
Task: „select photos for your private photo
collection“
Step 2
Recording of the eye tracking data
Creation of Ground Truth
Sm
Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 10
Manual Selection Creator:LibreOffice 3.5 LanguageLevel:2
Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 11
Collection CA Collection CB 162 photos 126 photos
Experiment Data Set C = CA CB
∩
Two Data Sets and Two User Groups
Institute A Institute B
Home collectionHome collectionForeign collection
● Taken during social events of the research institutes
Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 12
Participants
● 33 participants (12 of them female)● 21 associated to Institute A, 12 to Institute B● Aged between 25 and 62 (Ø 33.5 ± 9.57)● 20 graduate students, 4 postdocs,
9 other professions
Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 13
Overview Analysis and Evaluation
Se
Collection C
GazeBased
Selection
Calculation of Precision
P
ManualSelection
Content andContext Based
Selection
Sb+e
Sb
Ground Truth
Sm
Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 14
Baseline Measures
# Name Description
1 concentrationTime
Photo was taken with other photos in a short period of time
2 sharpness Sharpness score from related work
3 numberOfFaces Number of faces
4 faceGaussian Size and position of faces
5 personsPopularity
Popularity of the depicted persons
6 faceArea Areas in pixels covered by faces
Selection of photos based on:
Calculated for each photo
Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 15
Eye Tracking Data
● Fixations and saccades● Analysed gaze data with eye tracking measures
Creator:LibreOffice 3.5 LanguageLevel:2
● Viewing duration / page: M = 12.6 s
● Number of fixations / photo: M = 3.25
Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 16
Eye Tracking Measures# Name Description
7 fixated Was the photo was fixated?
Creator:LibreOffice 3.5 LanguageLevel:2
Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 17
Eye Tracking Measures# Name Description
7 fixated Was the photo was fixated?
8 fixationCount Counts the number of fixations
Creator:LibreOffice 3.5 LanguageLevel:2
Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 18
Eye Tracking Measures# Name Description
7 fixated Was the photo was fixated?
8 fixationCount Counts the number of fixations
9 fixationDuration Sum of duration of all fixations Creator:LibreOffice 3.5 LanguageLevel:2
Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 19
Eye Tracking Measures# Name Description
7 fixated Was the photo was fixated?
8 fixationCount Counts the number of fixations
9 fixationDuration Sum of duration of all fixations
10 firstFixationDuration Duration of the first fixation
11 lastFixationDuration Duration of the last fixation
12 avgFixationDuration Average fixation duration
13 maxVisitDuration Maximum visit length
14 meanVisitDuration Average visit length
15 visitCount Number of visits
16 saccLength Mean length of the saccades
17 pupilMax Maximum pupil diameter
18 pupilMaxChange Maximum pupil diameter change
19 pupilAvg Average pupil diameter
Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 20
Combination of Measures
● Using a model learned from logistic regression● Assigns each image a probability of being
selected● 30 random splits for training and test data
Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 21
1. Is there a need for individual photo selections?
1 21 41 61 81 1011211411611812012212412612810
5
10
15
20
25
30Photo with the highest number of selections
Photos with no selections
Photos in data set C
10
40
70
100
130
160
190
220
250
280
Sel
ectio
n F
requ
ency
● Manually created photo selections are diverse
Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 22
2. Evaluation of the Photo Selections
Pre
cisi
on P
Sb Sb+e Se
**
Random Selection
P = 0.428P = 0.365 P = 0.426
● Improvement of 17% over baseline
Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 23
3. Impact of personal interest?
Pre
cisi
on P
Results for Sb+e
Foreign Collection Home Collection
P = 0.446P = 0.404
*
Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 24
Conclusion● Photo selection behavior is individual● Gaze helps capture personal preferences● Results are better for photos with personal interest
● Might work even better for real personal photos● Potential application in photo book authoring
Thank you for your attention!