1
Summary Summary How can we measure eye-tracker performance? We developed a test battery with multiple tasks and measured different metrics of eye-trackers. Simultaneous data collection with the remote EyeLink 1000 against the mobile Pupil Labs glasses allow for direct and powerful within-subject comparisons. Freeviewing Freeviewing -10° 5° -7.5° 7.5° Y-position [°] X-position [°] EL PL 11.25 20 Fixations per image Fixation duration [s] 0 1 0 100 400 EyeLink Pupil Labs Number of fixations Pupil dilation Pupil dilation Difference Eyelink - Pupil Labs normalized pupil area [a.u] 100 150 200 250 Luminance [%max] -10% -5% 0% 5% 0 2 4 6 60% 40% 80% 100% Normalized pupil area [a.u.] luminance onset luminance offset monitor luminance 25% 50% 75% 100% Time [s] Eyelink Pupil Labs N = 15. For preprocessing we used the same algorithms where possible. For saccade classification we used the Engbert mergenthaler algorithm (λ = 5). Calibration Calibration 1 3 7 Time [s] -10 0 10 20 X-position [°] -10 0 10 20 2 6 8 binocular fusion artefact staircase fusion artefact no blink detected ? ? EyeLink Pupil Labs fixation saccade Accuracy & Precision Accuracy & Precision ∙ Researchers have to know the strengths and weaknesses of their devices ∙ Eye-tracking data quality cannot be reduced to a single value for all tasks ∙ Concurrent measurements can reveal consistent subject-biases Take home Take home EyeLink Pupil Labs Spatial Accuracy [°] Accuracy 0.50 0.75 1 0.2 0.3 0.4 Precision Standard Deviation [°] Precision [SD] EyeLink Pupil Labs EyeLink Pupil Labs 0 0.5 1 1.5 Eye-tracker reported validation accuracy [°] Samples of horizontal gaze movement during the grid task. Note the sample-wise eye movement classification. Blink samples were removed. Samples of horizontal gaze movement during the grid task. Note the sample-wise eye movement classification. Blink samples were removed. large grid pursuit free viewing micro saccades blinks pupil dilation small grid small grid head yaw head roll calibration / validation Repeat 6x Psychtoolbox powered by 0.57° 0.82° 0.19° 0.31° Get the full paper: A new comprehensive eye-tracking test battery concurrently evaluating the Pupil Labs glasses and the EyeLink 1000 doi: 10.7717/peerj.7086 Stimulus & Analysis Code: 10.5281/zenodo.2553176 Funding This work was supported by the European Union (H2020-FETPROACT-2014, SEP-210141273, ID: 641321 socSMCs) Validation accuracy before each block reported by the eye-tracker‘s manufacturer software. Spatial Accuracy [°] Small Grid I Small Grid II EyeLink Large Grid (subset) 0.5 1 1.5 Pupil Labs On the group level, both eye-trackers report the same normalized pupil area; on the single subject level, the eye-trackers show a reliable subject-specific bias. Pupil Labs shows much higher variance in estimated gaze positions and finds fewer and shorter sac- cades than EyeLink and, consequentially, on average longer fixation durations. EyeLink shows 40% better accuracy and 60% better precision. Pupil Labs exhibits strong calibration decay after ~ 4 minutes of recording. ~4m 40s ~6m 20s Katharina Groß¹ , *, Benedikt V. Ehinger¹ , ² , *, Inga Ibs¹, and Peter König¹ , ³ ¹ Institute of Cognitive Science, University of Osnabrück 2 Donders Institute for Brain, Cognition and Behaviour, Radboud University 3 Department of Neurophysiology and Pathophysiology, University Medical Center Hamburg-Eppendorf * Authors contributed equally A comprehensive eye-tracking test battery: Pupil Labs vs. EyeLink 1000 Find 700 GB of eye videos: 10.6084/m9.figshare.c.4379810.v2
