Date post: | 05-Dec-2014 |
Category: |
Documents |
Upload: | dong-bach-vo |
View: | 346 times |
Download: | 3 times |
GESTURE-AWARE REMOTE CONTROLS:GUIDELINES & INTERACTION TECHNIQUES
Gilles Bailly, Dong-Bach Vo, Eric Lecolinet and Yves GuiardTelecom ParisTech, LTCI - CNRS
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 20112
More and more services and contents:• Electronic program guide
• Video on demand
• Web based widgets
• Web applications
• Personal content
Interactive television
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
More and more services and contents:• Electronic program guide
• Video on demand
• Web based widgets
• Web applications
• Personal content
3
Interactive television
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 20114
More and more services and contents:• Electronic program guide
• Video on demand
• Web based widgets
• Web applications
• Personal content
Interactive television
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 20115
More and more services and contents:• Electronic program guide
• Video on demand
• Web based widgets
• Web applications
• Personal content
Interactive television
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 20116
More and more services and contents:• Electronic program guide
• Video on demand
• Web based widgets
• Web applications
• Personal content
Interactive television
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Inspired from computer interaction:• Deep hierarchical menus
• Long lists
7
Interacting with interactive television
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 20118
Interacting with interactive television
Inspired from computer interaction:• Deep hierarchical menus
• Long lists
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 20119
Interacting with interactive television
Inspired from computer interaction:• Deep hierarchical menus
• Long lists
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201110
Inspired from computer interaction:• Deep hierarchical menus
• Long lists
Interacting with interactive television
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Problems:• Browsing
• Commands
11
Remote control
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Problems:• Browsing
• Commands
12
Remote control
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201113
Problems:• Browsing
• Commands
Remote control
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201114
Problems:• Browsing
• Commands
Remote control
Limited expressiveness
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Improving expressiveness
Solution?• Add a new button for each new
command
15
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201116
Improving expressiveness
Pultius TV remote control [Artlebedev’07]
Solution?• Add a new button for each new
command
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201117
Can we benefit from moving the remote control
in 3D space to improve expressiveness?
Solution?• Add a new button for each new
command
Improving expressiveness
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Approach
Advantages of 3D gestures:• Allow large vocabulary of gestures
• Can be easy to remember
• Free visual focus from the remote control
• Comfortable to perform gestures at home [Rico et al.’10]
• Remote controls can remain small and easy to hold
18
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Approach
Advantages of 3D gestures:• Allow large vocabulary of gestures
• Can be easy to remember
• Free visual focus from the remote control
• Comfortable to perform gestures at home [Rico et al.’10]
• Remote controls can remain small and easy to hold
18
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201119
1. Interactive television2. Related work3. Observing spontaneous gestures 4. Exploring new gesture dimension5. Designing new interaction techniques6. Conclusion
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Related work
20
3D gestures offer 6 degrees of freedom:• 3 translations
• 3 rotations
X
Y
Z
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Related work
21
3D gestures offer 6 degrees of freedom:• 3 translations
• 3 rotations
Pitch
Yaw
Roll
Z
X
Y
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Related work
22
Motion marking menu system [Oakley’03]
Pitch X
Rotations:• 1 dimension
• 2 dimensions
• Discretization of tilt angle with visual feedback [Rahman et al.’09]
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Related work
23
GesText [Jones et al.’10]
Pitch
Roll Z
X
Rotations:• 1 dimension
• 2 dimensions
• Discretization of tilt angle with visual feedback [Rahman et al.’09]
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Related work
24
Yaw
Roll
Z
Y
12 levels
16 levels
Rotations:• 1 dimension
• 2 dimensions
• Discretization of tilt angle with visual feedback [Rahman et al.’09]
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201125
1. Interactive television2. Related work3. Observing spontaneous gestures 4. Exploring new gesture dimension5. Designing new interaction techniques6. Conclusion
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Experiment 1: spontaneous gestures
What gestures users perform spontaneously for moving targets?• Which degree(s) of freedom?
• Is it context dependant (smartphone vs. remote controls)?
26
Pitch
Yaw
Roll
Z
X
Y
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Task:• Perform the gesture that would
move the target to this position
Procedure:• Smartphone vs. Remote control
• Observation from video recordings
27
Experiment 1: spontaneous gestures
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Task:• Perform a gesture which has
produced this effect
Procedure:• Smartphone vs. Remote control
• Observation from video recordings
28
Experiment 1: spontaneous gestures
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Results:• Either rotations or translations
• 14 of 18 users perform rotations
• Pitch and Roll only in smartphone condition
• Pitch and Yaw in remote control condition
• Roll can be used for something else
29
0123456789
(S) Condition (RC) Condition
Num
ber o
f partic
ipan
ts
Pitch & Roll rotations
Pitch & Yaw rotations
X & Y translations
Experiment 1: spontaneous gestures
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Results:• Either rotations or translations
• 14 of 18 users perform rotations
• Pitch and Roll only in smartphone condition
• Pitch and Yaw in remote control condition
• Roll can be used for something else
30
0123456789
(S) Condition (RC) Condition
Num
ber o
f partic
ipan
ts
Pitch & Roll rotations
Pitch & Yaw rotations
X & Y translations
Experiment 1: spontaneous gestures
Pitch
RollZ
Xmardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Results:• Either rotations or translations
• 14 of 18 users perform rotations
• Pitch and Roll only in smartphone condition
• Pitch and Yaw in remote control condition
• Roll can be used for something else ntrol condition
31
0123456789
(S) Condition (RC) Condition
Num
ber o
f partic
ipan
ts
Pitch & Roll rotations
Pitch & Yaw rotations
X & Y translations
Experiment 1: spontaneous gestures
Pitch
Yaw
X
Y
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Results:• Either rotations or translations
• 14 of 18 users perform rotations
• Pitch and Roll only in smartphone condition
• Pitch and Yaw in remote control condition
• Roll can be used for something else
32
0123456789
(S) Condition (RC) Condition
Num
ber o
f partic
ipan
ts
Pitch & Roll rotations
Pitch & Yaw rotations
X & Y translations
Experiment 1: spontaneous gestures
Pitch
Yaw
X
Y
RollZ
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201133
1. Interactive television2. Related work3. Observing spontaneous gestures 4. Exploring roll gestures5. Designing new interaction techniques6. Conclusion
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Experiment 2: exploring roll gestures
What we know:• 16 items selection with visual feedback [Rahman’09]
• 3 items selection in eyes free selection [Oakley’07]
34
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201135
Task:Use any roll amplitude you need to reach the target
• Roll - Press - Roll - Release strategy
• No feedback provided
Procedure:• 9 participants
• Items set size (5, 7, 9, 11 items)
Experiment 2: exploring roll gestures
Question: How many levels users can control without visual feedback?
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201136
Task:Use any roll amplitude you need to reach the target
• Roll - Press - Roll - Release strategy
• No feedback provided
Procedure:• 9 participants
• Items set size (5, 7, 9, 11 items)
Experiment 2: exploring roll gestures
Question: How many levels users can control without visual feedback?
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201137
Task:Use any roll amplitude you need to reach the target
• Roll - Press - Roll - Release strategy
• No feedback provided
Procedure:• 9 participants
• Items set size (5, 7, 9, 11 items)
Experiment 2: exploring roll gestures
Step 1: Roll to prepare the gesture
Question: How many levels users can control without visual feedback?
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201137
Task:Use any roll amplitude you need to reach the target
• Roll - Press - Roll - Release strategy
• No feedback provided
Procedure:• 9 participants
• Items set size (5, 7, 9, 11 items)
Experiment 2: exploring roll gestures
Step 1: Roll to prepare the gesture
Question: How many levels users can control without visual feedback?
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201138
Task:Use any roll amplitude you need to reach the target
• Roll - Press - Roll - Release strategy
• No feedback provided
Procedure:• 9 participants
• Items set size (5, 7, 9, 11 items)
Experiment 2: exploring roll gestures
Step 2: Press the trigger
Question: How many levels users can control without visual feedback?
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201139
Question: How many levels users can control without visual feedback?
Task:Use any roll amplitude you need to reach the target
• Roll - Press - Roll - Release strategy
• No feedback provided
Procedure:• 9 participants
• Items set size (5, 7, 9, 11 items)
Experiment 2: exploring roll gestures
Step 3: Roll gesture
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201140
Task:Use any roll amplitude you need to reach the target
• Roll - Press - Roll - Release strategy
• No feedback provided
Procedure:• 9 participants
• Items set size (5, 7, 9, 11 items)
Experiment 2: exploring roll gestures
Step 4: Release the trigger
Question: How many levels users can control without visual feedback?
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201141
Task:Use any roll amplitude you need to reach the target
• Roll - Press - Roll - Release strategy
• No feedback provided
Procedure:• 9 participants
• Items set size (5, 7, 9, 11 items)
Experiment 2: exploring roll gestures
Question: How many levels users can control without visual feedback?
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201142
Task:Use any roll amplitude you need to reach the target
• Roll - Press - Roll - Release strategy
• No feedback provided
Procedure:• 9 participants
• Items set size (5, 7, 9, 11 items)
Experiment 2: exploring roll gestures
Question: How many levels users can control without visual feedback?
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Angular range:• Differs largely among
participants
• Depends on item-set size
Angular variation:• Constant variation between two
contiguous items
43
162° 326°
Minimum and maximum angular range for 7 items
Experiment 2: results
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Angular range:• Differs largely among
participants
• Depends on item-set size
Angular variation:• Constant variation between two
contiguous items
44
Experiment 2: results
0
100
200
300
400
0 1 2 3 4 6 7 8 9 10 11∆
angle (in
deg
rees
)
number of items in set
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Angular range:• Differs largely among
participants
• Depends on item-set size
Angular variation:• Constant variation between two
contiguous items
45
Experiment 2: results
-200-160-120-80-4004080120160200
-3 -2 -1 0 1 2 3
Mean angular varation for 7 items
RightLeft Center
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201146
Experiment 2: results
Recognition:• KNN algorithm
• euclidian distance on angular variation
• cross-validation technique
Results:• Very good for 5 items (96.3%)
• Insufficient for 9 and 11 items
• 7 items: • 87.7% for cross-validation
• 96% with user specific training
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201147
Experiment 2: results
0
25
50
75
100
5 items 7 items 9 items 11 itemsac
curacy
(in %)
mean accuracy rate
96.3Recognition:• KNN algorithm
• euclidian distance on angular variation
• cross-validation technique
Results:• Very good for 5 items (96.3%)
• Insufficient for 9 and 11 items
• 7 items: • 87.7% for cross-validation
• 96% with user specific training
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201148
Experiment 2: results
0
25
50
75
100
5 items 7 items 9 items 11 itemsac
curacy
(in %)
mean accuracy rate
77.1
63.1
Recognition:• KNN algorithm
• euclidian distance on angular variation
• cross-validation technique
Results:• Very good for 5 items (96.3%)
• Insufficient for 9 and 11 items
• 7 items: • 87.7% for cross-validation
• 96% with user specific training
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201149
Experiment 2: results
Recognition:• KNN algorithm
• euclidian distance on angular variation
• cross-validation technique
Results:• Very good for 5 items (96.3%)
• Insufficient for 9 and 11 items
• 7 items: • 87.7% for cross-validation
• 96% with user specific training
0
25
50
75
100
5 items 7 items 9 items 11 itemsac
curacy
(in %)
mean accuracy rate
87.7
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201150
1. Interactive television2. Related work3. Observing spontaneous gestures 4. Exploring roll gestures5. Designing new interaction techniques6. Conclusion
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Designing interaction techniques
51
Goals:• two-level circular menus
• that combine gestural modalities
• Pitch & Yaw
• Roll
• Directional pad buttons
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Designing interaction techniques
52
Design space:• 1 modality per menu
level
• D-pad used as a baseline
• Unpromising techniques dismissed at pretest
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Designing interaction techniques
53
Design space:• 1 modality per menu
level
• D-pad used as a baseline
• Unpromising techniques dismissed at pretest
2nd Level
1st LevelPitch & Yaw Roll Directional Pad
Pitch & Yaw
Roll
Directional Pad
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Designing interaction techniques
54
Design space:• 1 modality per menu
level
• D-pad used as a baseline
• Unpromising techniques dismissed at pretest
2nd Level
1st LevelPitch & Yaw Roll Directional Pad
Pitch & Yaw
Roll
Directional Pad Baseline
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Designing interaction techniques
55
Design space:• 1 modality per menu
level
• D-pad used as a baseline
• Unpromising techniques dismissed at pretest
2nd Level
1st LevelPitch & Yaw Roll Directional Pad
Pitch & Yaw
Roll
Directional Pad
Gesture trigger
penalty
Biomechanical
constraint
Poor results in
pretests
Gesture trigger
penalty
Baseline
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
2nd Level
1st LevelPitch & Yaw Roll Directional Pad
Pitch & Yaw
Roll
Directional Pad
3D multi-strokes
menu
3D gestures
parallelization
Gesture trigger
penalty
Biomechanical
constraint
Poor results in
pretests
Gesture trigger
penalty
2D/3D gestures
parallelization
2D/3D gestures
parallelizationBaseline
56
Designing interaction techniques
Design space:• 1 modality per menu
level
• D-pad used as a baseline
• Unpromising techniques dismissed at pretest
• 5 techniques to evaluate
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Procedure:• Menu with 4 * 4 items
• Combination of modalities
• User feedback
• 13 participants
Task:• Perform the gesture
corresponding to the symbolic stimulus
57
Experiment 3: interaction techniques
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201158
0
20,0
40,0
60,0
80,0
100,0
Pad/Pad Pad/P&Y Pad/Roll
Mean accuracy rate by technique
Accu
racy
in %
Best modality combinations for accuracy:• Pad / Pad (baseline)
• Pad / Pitch & Yaw (better than baseline)
• Pad / Roll
Experiment 3: results
97.4 98.4 94.2
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201159
0
425
850
1275
1700
Pad/Pad Pad/P&Y Pad/Roll
Mean time selection by technique
Tim
e in m
s
Best modality combinations for speed:• Similar results
• Pad / Roll is a bit slower
Experiment 3: results
1400 13901650
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201160
User satisfaction:• Pad / Pad
• then Pad / Pitch & Yaw
• then P&Y / P&Y
• then Pad / Roll
Experiment 3: results
01234567
Speed Accuracy Pleasant Tiring Easy to learn
Subjective preferences
Pad/Pad Pad/P&Y Pad/Roll P&Y/P&Y
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Multimedia system menu
61
Novice mode:• 2-level circular menu
• 2 modalities
• Novice mode:• 1st level : Pitch and Yaw
• 2nd level : Roll
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Multimedia system menu
62
Novice mode:• 2-levels circular menu
• 1st level selection
• 2nd level selection
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201163
Multimedia system menu
Novice mode:• 2-levels circular menu
• 1st level selection
• 2nd level selection
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201164
Expert mode:• Same gestures
• But the menu does not appear
• Implicit learning of gestures
Multimedia system menu
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 201165
1. Interactive television2. Related work3. Observing spontaneous gestures 4. Exploring roll gestures5. Designing new interaction techniques6. Conclusion
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Conclusion
66
What we have learned:• Experiment 1:
• Users rather use rotations when interacting with handheld devices
• With a remote control, they use pitch and yaw to interact with a distant screen
• Experiment 2: Roll gestures without feedback work fine:
• for 5 levels
• for 7 levels with user specific training
• Using button and 3D gestures can efficiently augment remote control expressiveness
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Conclusion
67
What we have learned:• Experiment 3:
• 3D gestures provide good performance
• especially when combined with d-pad buttons
• Conclusion:• Moving the remote control into space is indeed an effective
way to improve expressiveness
mardi 15 novembre 2011
Dong-Bach Vo ‒ ICMI 2011
Thank you for your attention!
http://perso.telecom-paristech.fr/~via/[email protected]
GESTURE-AWARE REMOTE CONTROLS:GUIDELINES & INTERACTION TECHNIQUES
mardi 15 novembre 2011