1
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG Neuroinformatik
INTRODUCTION
METHOD
RESULTS
CONCLUSIONS
Alignment of Attention
in Mediated Communication
SFB 673
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG Neuroinformatik
INTRODUCTIONINTRODUCTION
METHOD
RESULTS
CONCLUSIONS
What is mediated communication?
Does a suitable research paradigm existfor investigating mediated communication?
2
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG NeuroinformatikMotivation
INTRODUCTIONINTRODUCTION
METHOD
RESULTS
CONCLUSIONS
Customer support
Source: Demag & STMZ
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG NeuroinformatikChallenge
INTRODUCTIONINTRODUCTION
METHOD
RESULTS
CONCLUSIONS
Mismatch detection in mediated communication
3
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG NeuroinformatikParadigm
Visual search
INTRODUCTIONINTRODUCTION
METHOD
RESULTS
CONCLUSIONS
Comparative visual search • Tasks
- Visual scanning
• Goal
- Mismatch identification
• Strategy / Procedure
- Analysis of individual objects or object features- Memorisation- Shift of attention- Comparison and validation
(e.g. Pomplun et al., 2006)
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG NeuroinformatikParadigm
Visual search
INTRODUCTIONINTRODUCTION
METHOD
RESULTS
CONCLUSIONS
• Tasks
- Visual scanning
- Verbal description
• Goal
- Mismatch identification
Comparative visual search
- Verbal guidance
- Collaborative effort
• Strategy / Procedure
- Analysis of individual objects or object features- Description- Identification: Matching of description and stimulus- Comparison and validation
+ +communication
“… in the top left corner …”
requires alignment
dialogue structure
4
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG Neuroinformatik
1. Can we identify typical search patterns? Do these depend on the mismatch dimension?
2. (How) do partners align their individual search patterns? Do consistent search patterns develop?
3. Do preferable, efficient search patterns exist?
4. Does non-verbal communication affect the search, e.g. gaze contact or gestures?
5. Can we compensate for unavailable communication channels?
Research Questions
INTRODUCTIONINTRODUCTION
METHOD
RESULTS
CONCLUSIONS
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG Neuroinformatik
Participants• 48 native German speakers
• 50% male-male, 50% female-female pairs
Stimuli• 16 search image pairs
• search images with four objects at quadrant locations
• exactly one mismatch between image pairs
INTRODUCTION
METHODMETHOD
RESULTS
CONCLUSIONS
Experiment Details
Subject A Subject B
5
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG Neuroinformatik
INTRODUCTION
METHODMETHOD
RESULTS
CONCLUSIONS
Experiment Details
Procedure
➵
t
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG Neuroinformatik
INTRODUCTION
METHODMETHOD
RESULTS
CONCLUSIONS
Experiment Details
Setting• Subjects’ face-to-face situation OR screen
between subjects
Apparatus• Two Interactive Minds / LC Technologies
EyeGaze eye-tracking systems
• Remote binocular tracking, 120 Hz (interlaced)
• TCP/IP link for experiment (display) synchronisation
6
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG Neuroinformatik
INTRODUCTION
METHODMETHOD
RESULTS
CONCLUSIONS
Experiment Details
SettingVi
suel
le A
ufm
erks
amke
it un
d Bl
ickb
eweg
unge
n
AG Neuroinformatik
Independent variables• Mismatch dimension
INTRODUCTION
METHODMETHOD
RESULTS
CONCLUSIONS
Experiment Details
• Group
- Face-to-face
- Behind screen
Dependent variables• Response correctness & time
• Typical EM parameters, e.g. number of fixations, gaze duration & saccade length: distribution of attention
Scan path analysis
- Colour
- Typicality
- Completeness
- Orientation
7
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG NeuroinformatikSearch Patterns
INTRODUCTION
METHOD
RESULTSRESULTS
CONCLUSIONS
Sample gaze trajectories
View subject A View subject B
Face-to-face
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG NeuroinformatikSearch Patterns
INTRODUCTION
METHOD
RESULTSRESULTS
CONCLUSIONS
Sample gaze trajectories
View subject A View subject B
Face-to-face
8
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG NeuroinformatikSearch Patterns
INTRODUCTION
METHOD
RESULTSRESULTS
CONCLUSIONS
Sample gaze trajectories
View subject B View subject A
Behind screen
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG NeuroinformatikSearch Patterns
INTRODUCTION
METHOD
RESULTSRESULTS
CONCLUSIONS
Distribution of attention
Areas of interest and transitions
9
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG NeuroinformatikSearch Patterns
INTRODUCTION
METHOD
RESULTSRESULTS
CONCLUSIONS
Distribution of attention
I
IIIIV
II
Areas of interest and transitions
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG NeuroinformatikSearch Patterns
INTRODUCTION
METHOD
RESULTSRESULTS
CONCLUSIONS
Distribution of attention
I
IIIIV
III / II
II / I
III / II II / IIIIV / I I / IV
IV / III
III / IV
IV / III / III
III / III / IV
Areas of interest and transitions
10
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG NeuroinformatikSearch Patterns
INTRODUCTION
METHOD
RESULTSRESULTS
CONCLUSIONS
Distribution of attention
Areas of interest and transitions
I
IIIIV
II I
IIIIV
II I
IIIIV
II I
IIIIV
II
I
IIIIV
II I
IIIIV
II I
IIIIV
II I
IIIIV
II
I
IIIIV
II I
IIIIV
II I
IIIIV
II I
IIIIV
II
I
IIIIV
II I
IIIIV
II I
IIIIV
II I
IIIIV
II0.15
0.15
0.13
0.12
0.03
0.08
0.040.090.06 0.04
0.02 0.05
0.15
0.12
0.03
0.08
0.040.090.06 0.04
0.02 0.05
0.15
0.15
0.03
0.08
0.040.090.06 0.04
0.02 0.05
0.15
0.12
0.03
0.08
0.040.090.06 0.04
0.02 0.05
0.13 0.130.13
0.13 0.12 0.13
0.15
0.15
0.13
0.12
0.03
0.08
0.040.090.06 0.04
0.02 0.05
0.15
0.12
0.03
0.08
0.040.090.06 0.04
0.02 0.05
0.15
0.15
0.03
0.08
0.040.090.06 0.04
0.02 0.05
0.15
0.12
0.03
0.08
0.040.090.06 0.04
0.02 0.05
0.13 0.130.13
0.13 0.12 0.13
0.15
0.15
0.13
0.12
0.03
0.08
0.040.090.06 0.04
0.02 0.05
0.15
0.12
0.03
0.08
0.040.090.06 0.04
0.02 0.05
0.15
0.15
0.03
0.08
0.040.090.06 0.04
0.02 0.05
0.15
0.12
0.03
0.08
0.040.090.06 0.04
0.02 0.05
0.13 0.130.13
0.13 0.12 0.13
0.15
0.15
0.13
0.12
0.03
0.08
0.040.090.06 0.04
0.02 0.05
0.15
0.12
0.03
0.08
0.040.090.06 0.04
0.02 0.05
0.15
0.15
0.03
0.08
0.040.090.06 0.04
0.02 0.05
0.15
0.12
0.03
0.08
0.040.090.06 0.04
0.02 0.05
0.13 0.130.13
0.13 0.12 0.13
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG NeuroinformatikSearch Patterns
INTRODUCTION
METHOD
RESULTSRESULTS
CONCLUSIONS
Distribution of attention
I
IIIIV
II0.16
0.05
0.05 0.140.12 0.08
0.08
0.15
0.030.02
0.040.08
Areas of interest and transitions: typical frequencies
11
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG NeuroinformatikSearch Patterns
INTRODUCTION
METHOD
RESULTSRESULTS
CONCLUSIONS
Distribution of attention
I
IIIIV
II0.16
0.05
0.05 0.140.12 0.08
0.08
0.15
0.030.02
0.040.08
Areas of interest and transitions: the search “loop”
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG Neuroinformatik
T(23) = 48.66; p = 0.01INTRODUCTION
METHOD
RESULTSRESULTS
CONCLUSIONS
0
0,5
1
1,5
2
1 2 3 subseq.
search "loop"
rel.
freq
uenc
y
Distribution of attention
Search Patterns
Pattern I
Fixations: Within-quadrant vs. between-quadrants F(3; 69) = 2139.90; p < 0.001
Pattern II
0
0,5
1
1,5
2
1 2 3 subseq.
search "loop"
rel.
freq
uenc
y
(76% of all trials)
(24% of all trials)
F(3; 69) = 90.87; p = 0.03
12
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG Neuroinformatik
INTRODUCTION
METHOD
RESULTSRESULTS
CONCLUSIONS
0
0,5
1
1,5
2
1 2 3 subseq.
search "loop"
rel.
freq
uenc
y
Distribution of attention
Search Patterns
Pattern I
Saccades: Within-quadrant vs. between-quadrants
0
100
200
300
400
500
600
1 2 3 subseq.
search "loop"
sacc
ade
leng
th (p
xls)
F(3; 69) = 1987.2; p < 0.001
with
in q
uadr
ant
betw
een
quad
rant
s F(3; 69) = 6.92; p = 0.09F(3; 69) = 113.61; p = 0.02
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG Neuroinformatik
T(23) = 18.57; p = 0.03
T(23) = 46.06; p = 0.01
INTRODUCTION
METHOD
RESULTSRESULTS
CONCLUSIONS
0
0,5
1
1,5
2
1 2 3 subseq.
search "loop"
rel.
freq
uenc
y
Distribution of attention
Search Patterns
Pattern I vs. II
Saccades: Within-quadrant vs. between-quadrants
0
100
200
300
400
500
600
1 2 3 subseq.
search "loop"
sacc
ade
leng
th (p
xls)
F(3; 69) = 1987.2; p < 0.001F(3; 69) = 200.71; p = 0.02
F(3; 69) = 8.11; p = 0.09F(3; 69) = 24.88; p = 0.07
13
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG Neuroinformatik
INTRODUCTION
METHOD
RESULTSRESULTS
CONCLUSIONS
0
50
100
150
200
colour orientation typicality completeness
mismatch dimension
dete
ctio
n tim
e (s
)
F(3; 33) = 14.92; p < 0.001
Detection time
Group & Mismatch Dimension Effects
0
50
100
150
200
colour orientation typicality completeness
mismatch dimension
dete
ctio
n tim
e (s
)
F(3; 33) = 4.33; p = 0.033
Face-to-face
Behind screen
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG Neuroinformatik
INTRODUCTION
METHOD
RESULTSRESULTS
CONCLUSIONS
Gaze duration
Face-to-face
Behind screen
Group & Mismatch Dimension Effects
0
50
100
150
200
colour orientation typicality completeness
mismatch dimension
gaze
dur
atio
n (s
)
F(3; 33) = 14.92; p < 0.001
0
50
100
150
200
colour orientation typicality completeness
mismatch dimension
gaze
dur
atio
n (s
)
F(3; 33) = 4.33; p = 0.033
14
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG Neuroinformatik
INTRODUCTION
METHOD
RESULTSRESULTS
CONCLUSIONS
Gaze duration
Face-to-face
Behind screen
Group & Mismatch Dimension Effects
0
50
100
150
200
colour orientation typicality completeness
mismatch dimension
gaze
dur
atio
n (s
)
F(3; 33) = 14.92; p < 0.001
0
50
100
150
200
colour orientation typicality completeness
mismatch dimension
gaze
dur
atio
n (s
)
F(3; 33) = 4.33; p = 0.033
gaze contact
“gaze contact”
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG Neuroinformatik
INTRODUCTION
METHOD
RESULTSRESULTS
CONCLUSIONS
0
100
200
300
400
500
colour orientation typicality completeness
mismatch dimension
num
ber o
f fix
atio
ns
Number of fixations
0
100
200
300
400
500
colour orientation typicality completeness
mismatch dimension
num
ber o
f fix
atio
ns
Face-to-face
Behind screen
F(3; 33) = 30.52; p < 0.001
F(3; 33) = 10.88; p = 0.02
Group & Mismatch Dimension Effects
15
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG Neuroinformatik
INTRODUCTION
METHOD
RESULTSRESULTS
CONCLUSIONS
0
20
40
60
80
100
colour orientation typicality completeness
mismatch dimension
sacc
ade
leng
th (p
xls)
Saccade length
Group & Mismatch Dimension Effects
0
20
40
60
80
100
colour orientation typicality completeness
mismatch dimension
sacc
ade
leng
th (p
xls)
Face-to-face
Behind screen
F(3; 33) = 27.33; p = 0.005
F(3; 33) = 35.38; p = 0.003
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG Neuroinformatik
INTRODUCTION
METHOD
RESULTSRESULTS
CONCLUSIONS
0
50
100
150
I II
search pattern
dete
ctio
n tim
e (s
)
T(23) = 8.92; p < 0.001
Detection time & search patterns
Search Efficiency
0
50
100
150
colour orientation typicality completeness
mismatch dimension
dete
ctio
n tim
e (s
)
Pattern I vs. II
16
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG NeuroinformatikSummary & Additional Findings
INTRODUCTION
METHOD
RESULTSRESULTS
CONCLUSIONS
• Individual differences in search patterns exist between subject pairs:
I. Partners initially scan all objects briefly and, if mismatch was not found, subsequently discuss objects in detail serially. This is generally quicker than
II. detailed serial object processing without initial “overview” scan.
• Search patterns are consistent only within trials. Partners do not usually stick to a single search pattern during the experiment.
• Colour differences are most easily detected, finding object typicality mismatches takes longest.
• Mismatch dimension from previous trial are often checked first (“recency effect”), leading to singular, unexpectedly short RTs for “difficult” mismatch dimension.
• Search patterns are guided by verbal object descriptions. Partners rarely discuss search strategies beforehand, turn-taking and verbal guidance are automised.
• Searches are rarely dominated by a single interlocutor, only if one partner is very inactive. This “monologue” pattern takes longer to accomplish mismatch detection.
• Approx. 6% of all fixations land on partner, most frequently so when partners search reassurance for verbal utterances or when verbal references are unclear.
• No significant effect on search times was found between face-to-face and behind-screen groups. Partners hardly gesture, even in face-to-face situations.
Visu
elle
Auf
mer
ksam
keit
und
Blic
kbew
egun
gen
AG Neuroinformatik
• A multi-pass search strategy with rather than without an initial coarse overview scan generally detects mismatches in collaborative visual search quicker. The rapid initial scene overview helps to detect obvious differences and should thus not be omitted.
• A colour-coding mechanism even for other object dimensions such as orientation, typicality or completeness may increase mismatch detection speeds.
• In repeated search tasks, visual search may benefit from recency effects, in particular in case of “hard-to-spot” differences.
• A natural communication pattern between partners should not be replaced by a “check-list” search initiated/dominated by one partner – unless required due to partner inactivity.
• The role of gaze contact remains unclear. Although partners more frequently look at each other in “critical” situations, this does not seem to directly affect search performance – but could be a “personal comfort” factor.
• The lack of (attending to) gestures comes as a surprise, however, might be due to experimental conditions/setting. The face-to-face scenario should be improved for future studies and ensure less interrupted views for partners.
INTRODUCTION
METHOD
RESULTS
CONCLUSIONSCONCLUSIONS
Conclusions