SummarySummaryHow can we measure eye-tracker performance?We developed a test battery with multiple tasks and measured different metrics of eye-trackers.Simultaneous data collection with the remote EyeLink 1000 against the mobile Pupil Labs glasses allow for direct and powerful within-subject comparisons.

FreeviewingFreeviewing

−10° 5°

−7.5°

7.5°

Y-po

sitio

n [°

]

X-position [°] EL PL

11.25

20

Fixa

tions

per

imag

e

Fixation duration [s]0 1 1.50

100

400EyeLink

Pupil Labs

Num

ber

of fi

xatio

ns

Pupil dilationPupil dilation

Diff

eren

ce E

yelin

k −

Pup

il La

bsno

rmal

ized

pup

il ar

ea [a

.u]

100 150 200 250

Luminance [%max]

-10%

-5%

0%

5%

pa_n

orm

0 2 4 6

60%

40%

80%

100%

Nor

mal

ized

pup

il ar

ea [a

.u.]

luminance onset luminance offset

monitor luminance

25% 50% 75% 100%

Time [s]

EyelinkPupil Labs

N = 15. For preprocessing we used the same algorithms where possible. For saccade classification we used the Engbert mergenthaler algorithm (λ = 5).

CalibrationCalibration

1 3 7Time [s]

−10

0

10

20

X-po

sitio

n [°

]

−10

0

10

20

2 6 8

binocular fusion artefact

staircase fusion artefact

no blink detected ?

?

EyeLink

Pupil Labs

fixation

saccade

Accuracy & PrecisionAccuracy & Precision

∙ Researchers have to know the strengths and weaknesses of their devices

∙ Eye-tracking data quality cannot be reduced to a single value for all tasks

∙ Concurrent measurements can reveal consistent subject-biases

Take homeTake home

EyeLink Pupil Labs

Spa

tial A

ccur

acy

[°]

Accuracy

0.50

0.75

1

0.2

0.3

0.4

Pre

cisi

on S

tand

ard

Dev

iatio

n [°

]

Precision [SD]

EyeLink Pupil Labs

EyeLink Pupil Labs

0

0.5

1

1.5

Eye

-tra

cker

repo

rted

valid

atio

n ac

cura

cy [°

]

Samples of horizontal gaze movement during the grid task. Note the sample-wiseeye movement classification. Blink samples were removed.Samples of horizontal gaze movement during the grid task. Note the sample-wiseeye movement classification. Blink samples were removed.

large grid pursuit free viewing micro saccades blinks

pupil dilation small grid small gridhead yaw head roll

calibration / validation

Repeat 6x

Psychtoolboxpowered by

0.57°

0.82°

0.19°

0.31°

Get the full paper:A new comprehensive eye-tracking test battery concurrently evaluating the Pupil Labs glasses and the EyeLink 1000doi: 10.7717/peerj.7086

Stimulus & Analysis Code:10.5281/zenodo.2553176

FundingThis work was supported by the EuropeanUnion (H2020-FETPROACT-2014,SEP-210141273, ID: 641321 socSMCs)

Validation accuracy before each block reported by the eye-tracker‘s manufacturer software.

Spa

tial A

ccur

acy

[°]

Small Grid I Small Grid II

EyeLink

Large Grid (subset)

0.5

1

1.5

Pupil Labs

On the group level, both eye-trackers report the same normalized pupil area; on the single subject level, the eye-trackers show a reliable subject-specific bias.

Pupil Labs shows much higher variance in estimated gaze positions and finds fewer and shorter sac-cades than EyeLink and, consequentially, on average longer fixation durations.

EyeLink shows 40% better accuracy and 60% better precision. Pupil Labs exhibits strong calibration decay after ~ 4 minutes of recording.

~4m 40s ~6m 20s

Katharina Groß¹,*, Benedikt V. Ehinger¹,²,*, Inga Ibs¹, and Peter König¹,³ ¹ Institute of Cognitive Science, University of Osnabrück2 Donders Institute for Brain, Cognition and Behaviour, Radboud University3 Department of Neurophysiology and Pathophysiology, University Medical Center Hamburg-Eppendorf

* Authors contributed equally

A comprehensive eye-tracking test battery:Pupil Labs vs. EyeLink 1000

Find 700 GB of eye videos:10.6084/m9.�gshare.c.4379810.v2