23/4/8 EIE426-AICV 1

Eye Tracking Techniques and applications

eie426-EyeTracking-200903.ppt

Outline

Part I :Physiology of the EYE Part II : Mechanism of Eye Movements Part III :Eye Tracking Technology Part IV :Eye Tracking Applications

23/4/8 EIE426-AICV 2

How does our eye work

23/4/8 EIE426-AICV 3

HOW DOES OUR EYE WORK

Part I :Physiology of the EYE

23/4/8 EIE426-AICV 4

23/4/8 EIE426-AICV 5

CAN YOU BELIEVE YOUR EYE

Part II : Mechanism of Eye Movements

23/4/8 EIE426-AICV 6

HOW EYE TRACKER WORKS

Part III :Eye Tracking Technology

23/4/8 EIE426-AICV 7

Outline

1. Current State and Application of Eye Gaze Tracking

(EGT) Technology

2. Theory and the Classification of EGT Technology

3. The Framework of EGT system

Ⅰ. Current State and Application of Eye Gaze Tracking (EGT) Technology

Ⅰ. Current State and Application of Eye Gaze Tracking (EGT) Technology What is EGT eye gaze——The line from the fovea through the center of

the pupil is the line of sight (LoS). Usually, we take the optical axis as line of gaze (LoG). LoG can be approximate to LoS. In fact, LoS determines a person’s visual attention.

eye gaze Tracking——By image Processing,if the LoG or LoS can be estimated, the point of regard (POR) is

computed as the intersection of the LoG (or LoS) with the object of the scene or space.

History and Development of EGT Interests of Visual attention can be traced back to 1897. At that time, it was a kind of

diagnostic research, i.e. the recording of eye movement. Those technologies include ophthalmometer, Mechanical Method, Electro-Oculography (EOG), Optical-based Method , electromagnetic Oculography

eye movement initially applied in medical research, such as brain and physiology analysis. with the development of Electronics, Computer and Image processing technology, further research focuses on Eye Gaze Tracking Technology .

Search Coil

A picture wearing the EOG apparatus

eye tracking techniques ：

1. Direct Observation ： ophthalmometer, peep-hole method

2. Mechanic Method ： use level to record eye movement

3. Optical-based Method ： Reflection Method （ mirror or prism ）、 Pupil-

Corneal reflection 、 Purkinje Image

4. Electro-Oculography (EOG): recorded the difference in the skin potential

5. Electromagnetic Oculography ： The users gaze is estimated from

measuring the voltage induced in the search coil by an external electro-magnetic

field

Except Optical-based Method, the above methods are more or less low accuracy or

high intrusiveness . As a result, Modern Eye Gaze Tracking Techniques are mostly

based on Optical-based Method.

Ⅰ. Current State and Application of Eye Gaze Tracking (EGT) Technology

Ⅱ. Theory and the Classification of EGT Technology

Ⅱ. Theory and the Classification of EGT Technology

Image-Based EGT Technology

1. Infrared Light : out of visible light; Paired with filter to eliminate the light of other wavelength ；

2. Eye camera :tracking eye movement and recording eye image sequence

3. Image Processing: detecting the visual elements

4. Further Estimation: 。 Having pre-processed, the data of eye gaze can be get from eye-movement model

System Framework

Output

Calibration

Input Image acquisition

Image Processing

FeatureEstimation

Eye Tracking

DataEstimation

Ⅱ. Theory and the Classification of EGT Technology

Ⅱ. Theory and the Classification of EGT Technology

Function of Each Module11

Image acquisition: get bright pupil or dark pupil

22Image Processing :Filtering ,Noise reducing ,difference ,thresholding

33Feature Estimation: Estimating the center of Pupil and corneal reflection

44Eye Tracking:Estimation of eye gaze

55Data Estimation:

Algorithm validation ,eye movement data analysis

Theory of EGT Technology

2D to detect the Optical Axis (basic) 2D to detect the Line of Sight (Advanced)

¹â ÖáO A 2 D ·½ ·¨

ÊÓÏß Öá LO S 3 D ·½ ·¨

Classification of EGT Technology

view of view of humanityhumanity

in tr u s iv e £º C o n tac t len s ¡¢ P h o to - tr an s m ite r ¡¢ I R lig h t em itt in g d io d esn o n - in tr u s iv e £º u s e im ag e- b as ed M eth o d

D ia g n o s t ic£º Us e th e Ey e m o v e m e n t ch a ra cte rs to Ps y ch o lo g y a n d Ne u ro s cie n ce (n e e d to dis t in g u is h dif f e re n t ty pe o f e y e m o v m e n t )I n te ra ct iv e £º Us e Ey e M o v m e n t to de cide Po in t o f R e g a rd ,u s e fo r in te ra ct io n £¨ n e e d to k n o w th e v is iu a l a t t e n t io n £©

View of View of appearanceappearance

B o dy-M o unte d£º g o g g l e ¡¢ he l m e t¡¢ bac k pac k, e tcno te : al tho ugh labe le d with inrus ive ,but m o re and m o re bo dy-M o unte d s ys te m sare be c o m ing no n- intrus ive

R e m o te s ys te m £º R e m o te E YE G aze Tr ac ke r (R E G T)

view ofview of application application

Ⅱ. Theory and the Classification of EGT Technology From View of Humanity

Usually By goggle , helmet …

features： high accuracy

defects： intrusiveness

RemoteEye Tracker

Usually on tablefeatures： non-intrusive defects： eye and head

relative motion

Head MountedEye Tracker

Ⅱ. Theory and the Classification of EGT

Technology From View of Theory

Purkinje Image The Purkinje images are reflections created at different layers of the eye structure. Eye gaze can be calculated from these relative positions of these reflections

EOG

Limbus-Scalar IR Tracking

ANN

Contact Lens

Pupil-Corneal reflection

By placing electrodes around the eye, it is possible to measure small differences in the skin potential corresponding to eye movement.

The Limbus is the boundary between the white sclera and the dark iris of the eye. By placing IR light emitting diodes and photo-transistors, respectively, above and below the eye. the resulting voltage difference is proportional to the angular deviation of the eye.

Training images are taken when the user is looking at a specific Calibration markers. Use ANN to decide the eye gaze

Use a small coil embedded into a contact lens that is tightly fit over the sclera . The user’s gaze is estimated from measuring the voltage induced in the search coil by an external electro-magnetic field.

The IR source can generate a glint on corneal and divide pupil from iris, the difference between can represent the eye gaze movement

Purkinje Image The Purkinje images are reflections created at different layers of the eye structure. Eye gaze can be calculated from these relative positions of these reflections

EOG

ANN

Contact Lens

Pupil-Corneal reflection

By placing electrodes around the eye, it is possible to measure small differences in the skin potential corresponding to eye movement.

The Limbus is the boundary between the white sclera and the dark iris of the eye. This boundary can easily be optically detected and tracked

Training images are taken when the user is looking at a specific Calibration markers. Use ANN to decide the eye gaze

Use a small coil embedded into a contact lens that is tightly fit over the sclera . The user’s gaze is estimated from measuring the voltage induced in the search coil by an external electro-magnetic field.

The IR source can generate a glint on corneal and divide pupil from iris, the difference between can represent the eye gaze movement

The Limbus is the boundary between the white sclera and the dark iris of the eye. By placing IR light emitting diodes and photo-transistors, respectively, above and below the eye. the resulting voltage difference is proportional to the angular deviation of the eye.

将两对氯化银皮肤表面电极分别置于眼睛左右、上下两侧 , 就能引起眼球变化方向上的微弱电信号 , 经放大后得到眼球运动的位置信息

The Purkinje images are reflections created at different layers of the eye structure. Eye gaze can be calculated from these relative positions of these reflections

Limbus-Scalar IR Tracking

Ⅱ. Theory and the Classification of EGT

Technology From View of Theory

Purkinje Image The Purkinje images are reflections created at different layers of the eye structure. Eye gaze can be calculated from these relative positions of these reflections

EOG

ANN

Contact Lens

Pupil-Corneal reflection

By placing electrodes around the eye, it is possible to measure small differences in the skin potential corresponding to eye movement.

The Limbus is the boundary between the white sclera and the dark iris of the eye. This boundary can easily be optically detected and tracked

Training images are taken when the user is looking at a specific Calibration markers. Use ANN to decide the eye gaze

Use a small coil embedded into a contact lens that is tightly fit over the sclera . The user’s gaze is estimated from measuring the voltage induced in the search coil by an external electro-magnetic field.

The IR source can generate a glint on corneal and divide pupil from iris, the difference between can represent the eye gaze movement

Limbus-Scalar IR Tracking

Ⅱ. Theory and the Classification of EGT

Technology From View of Theory

Purkinje Image The Purkinje images are reflections created at different layers of the eye structure. Eye gaze can be calculated from these relative positions of these reflections

EOG

ANN

Contact Lens

Pupil-Corneal reflection

By placing electrodes around the eye, it is possible to measure small differences in the skin potential corresponding to eye movement.

The Limbus is the boundary between the white sclera and the dark iris of the eye. This boundary can easily be optically detected and tracked

Training images are taken when the user is looking at a specific Calibration markers. Use ANN to decide the eye gaze

Use a small coil embedded into a contact lens that is tightly fit over the sclera . The user’s gaze is estimated from measuring the voltage induced in the search coil by an external electro-magnetic field.

The IR source can generate a glint on corneal and divide pupil from iris, the difference between can represent the eye gaze movement

将两对氯化银皮肤表面电极分别置于眼睛左右、上下两侧 , 就能引起眼球变化方向上的微弱电信号 , 经放大后得到眼球运动的位置信息

The Limbus is the boundary between the white sclera and the dark iris of the eye. By placing IR light emitting diodes and photo-transistors, respectively, above and below the eye. the resulting voltage difference is proportional to the angular deviation of the eye.

Limbus-Scalar IR Tracking

Ⅱ. Theory and the Classification of EGT

Technology From View of Theory

Purkinje Image The Purkinje images are reflections created at different layers of the eye structure. Eye gaze can be calculated from these relative positions of these reflections

EOG

ANN

Contact Lens

Pupil-Corneal reflection

By placing electrodes around the eye, it is possible to measure small differences in the skin potential corresponding to eye movement.

The Limbus is the boundary between the white sclera and the dark iris of the eye. This boundary can easily be optically detected and tracked

Training images are taken when the user is looking at a specific Calibration markers. Use ANN to decide the eye gaze

Use a small coil embedded into a contact lens that is tightly fit over the sclera . The user’s gaze is estimated from measuring the voltage induced in the search coil by an external electro-magnetic field.

The IR source can generate a glint on corneal and divide pupil from iris, the difference between can represent the eye gaze movement

Limbus-Scalar IR Tracking

Ⅱ. Theory and the Classification of EGT

Technology From View of Theory

Purkinje Image The Purkinje images are reflections created at different layers of the eye structure. Eye gaze can be calculated from these relative positions of these reflections

EOG

ANN

Contact Lens

Pupil-Corneal reflection

By placing electrodes around the eye, it is possible to measure small differences in the skin potential corresponding to eye movement.

The Limbus is the boundary between the white sclera and the dark iris of the eye. This boundary can easily be optically detected and tracked

Training images are taken when the user is looking at a specific Calibration markers. Use ANN to decide the eye gaze

Use a small coil embedded into a contact lens that is tightly fit over the sclera . The user’s gaze is estimated from measuring the voltage induced in the search coil by an external electro-magnetic field.

The IR source can generate a glint on corneal and divide pupil from iris, the difference between can represent the eye gaze movement

将两对氯化银皮肤表面电极分别置于眼睛左右、上下两侧 , 就能引起眼球变化方向上的微弱电信号 , 经放大后得到眼球运动的位置信息

Use a small coil embedded into a contact lens that is tightly fit over the sclera . The user’s gaze is estimated from measuring the voltage induced in the search coil by an external electro-magnetic field.

Limbus-Scalar IR Tracking

Ⅱ. Theory and the Classification of EGT

Technology From View of Theory

Purkinje Image The Purkinje images are reflections created at different layers of the eye structure. Eye gaze can be calculated from these relative positions of these reflections

EOG

ANN

Contact Lens

Pupil-Corneal reflection

By placing electrodes around the eye, it is possible to measure small differences in the skin potential corresponding to eye movement.

The Limbus is the boundary between the white sclera and the dark iris of the eye. This boundary can easily be optically detected and tracked

Training images are taken when the user is looking at a specific Calibration markers. Use ANN to decide the eye gaze

Use a small coil embedded into a contact lens that is tightly fit over the sclera . The user’s gaze is estimated from measuring the voltage induced in the search coil by an external electro-magnetic field.

The IR source can generate a glint on corneal and divide pupil from iris, the difference between can represent the eye gaze movement

Limbus-Scalar IR Tracking

Ⅱ. Theory and the Classification of EGT

Technology From View of Theory

Ⅱ. Theory and the Classification of EGT

Technology EGC Technology Accuracy Features

Purkinje Image 0.0170High accuracy, but the light is hard to control and can only be used in Lab

EOG 1.50-2° low robustness, low accuracy, high intrusiveness

Limbus TrackingV 10-70

H 0.50-70

Horizontal accuracy is better than Vertical, but both are low

ANN 1.3-1.8° No need of calibration,low accuracy,

Contact Lens 0.080 high accuracy, high intrusiveness

Pupil Corneal Reflection 1° The best one till now

Ⅱ. Theory and the Classification of EGT Technology Feature of Eye Movement

从系统从系统应用角度应用角度

D iag n o s tic ( Õï ÁÆʽ) £º ÊÓÏß ×· ×ٵĽá¹û ×÷ Ϊ Ò»¸ö ¶¨ Á¿ ÌØ ÐÔ£¬ ¿¼²ì È˵ÄÊÓ¾õºÍ ÐÄÀí ѧÑо¿ £¨ ¾ø¶Ô׼ȷ ÐÔ£¬ ¶ÔÑÛÇò ¶¯ ×÷ µÄÇø ·Ö £©I n ter ac tiv e ( ½»»¥ ʽ) £º Ö»Òª ×· ×Ù³ö ÊÓÏß £¬ ×÷ Ϊ ºó ¶Ë ½»»¥ µÄÊä Èë £¨ Ïà ¶Ô׼ȷ £¬ Ò»°ã ²» Çø ·Ö ÑÛÇò µÄÔ˶¯ ÌØ ÐÔ£¬ Ö»Òª Öª µÀÊÓÏß ·½ Ïò £©

view of view of humanityhumanity

in tr u s iv e £º C o n tac t len s ¡¢ P h o to - tr an s m ite r ¡¢ I R lig h t em itt in g d io d esn o n - in tr u s iv e £º u s e im ag e- b as ed M eth o d

S m o o th P urs uitvo lec ity£º 1o /s ~30o /s

F ixa tio ndura tio n:1 0 0 ~ 2 0 0 m s

S a c c a de sangle: 1o~4o

d uratio n£º 30~ 120m svo lec ity:400o /s ~600o /s

Ta x o n o m y a n d M o d e ls o f E y e M o ve m e n ts

Early stage : Direct Observation ,Mechanical Method Initial Methods, not used any more for high intrusiveness and poor accuracy

Development ： EOG 、 Electromagnetic Oculography Although have improved a lot and widely used , these method are disappearing because of the intrusiveness

Advanced ： Optical Method ：Because of its’ high accuracy and low intrusiveness, Optical Method have made rapid progress in recent years.

1. The Purkinje images are reflections created at different layers of the eye structure. and the eye gaze can be calculated from these relative positions of these reflections

2. Photo-Oculography : it measures the eye movement during it's translation/rotation. with the IR light source, shape of the pupil ,Limbus or corneal reflection is detected

3. Limbus-Scalar IR Method: They place IR light emitting diodes and IR light sensitive photo-transistors, respectively, above and below the eye. Several such IR pairs can be mounted on goggles or helmets, a photo-transistor transforms the reflected IR light into a voltage. the resulting voltage difference is proportional to the angular deviation of the eye.

4. Pupil-Corneal Reflection: The IR source can generate a glint on corneal and divide pupil from iris, the camera can extract the pupil which represent the eye gaze, on the other hand the corneal reflection represent the head motion, as a result, the difference between them represents the real eye gaze movement. 。

Ⅱ. Theory and the Classification of EGT Technology Development

Ⅱ. Theory and the Classification of EGT Technology Feature of Remote Eye Gaze Tracker

Robustness Light Source Eyelash, Eyelids and Camera Position System Error Eye station (dry or wet) head motion

Lower accuracy than Head Mounted Eye Tracker

View of View of appearanceappearance

B o dy-M o unte d£º g o g g l e ¡¢ he l m e t¡¢ bac k pac k, e tcno te : al tho ugh labe le d with inrus ive ,but m o re and m o re bo dy-M o unte d s ys te m sare be c o m ing no n- intrus ive

R e m o te s ys te m £º R e m o te E YE G aze Tr ac ke r (R E G T)

Ⅲ A Practical Eye Gaze Tracking System

Ⅲ A Practical Eye Gaze Tracking SystemFramework of EGT System

Hardware ： a Pair of Cameras eye camera scene camera

eye camera is for acquisition of pupil corneal reflection image, scene camera is for: 1 mapping from the eye image coordinate to scene image coordinate ;2 Showing the combining result of POR and scene in same image ：

V is u al S tim u lu s

T rac k in g o fP u p il

-C o rn e alR e fle c tio n

A utoM a pping

M o rnite r

P C

O utput o f E ye V ide o S tre a m

IR S o urc e

IR S o urc e

E ye C a m e ra

S c e ne C a m e ra

O ptic a l S ys te m

O utput o f S c e ne V ide o S tre a m

Ⅲ A Practical Eye Gaze Tracking Systemappearance of EGT System

Ⅲ Practical Eye Gaze Tracking SystemTheory of a Head Mounted Eye Tracker

Eye Camera Capture

Pupil-Corneal image

Scene Camera

Mapping between Eye Coordinate to Scene

Coordinate

Estimate Pupil-Corneal

reflection

22222222222

222222222D

xianglian

Eye Tracking

Result

Coordinate Mapping

Calibration

Tracking

When IR source is placed near the optical axis of the camera bright pupil as can be seen ;while IR is placed off the optical axis, a dark pupil can be seen. By Thresholding, a robust pupil contour can be extracted .

Bright Pupil

Dark Pupil

Difference Image

Ⅲ Practical Eye Gaze Tracking SystemBright Pupil and Dark Pupil

Ⅲ Practical Eye Gaze Tracking SystemBright Pupil and Dark Pupil

The IR source can generate a glint on corneal and divide pupil from iris, the camera can extract the pupil which represent the eye gaze, on the other hand the corneal reflection represent the head motion, as a result, the difference between them represents the real eye gaze movement.

Bright Pupil

Dark Pupil

Corneal ReflectionCorneal Reflection

Ⅲ Practical Eye Gaze Tracking System

Ⅲ Practical Eye Gaze Tracking System

Ⅲ Practical Eye Gaze Tracking System

HOW FAR YOU CAN FLY Part IV :Applications

23/4/8 EIE426-AICV 41

Applications

Human Computer Interaction (HCI)—— efficiency, humanity

Intelligent Control —— EGT and Weapon Control

Human Movement Study ——Typing, physical training

Psychology —— Antinational Neuroscience,

Visual attention & Driving ——Aviation, navigation, driving,

traffic accidents inspection

Scene and Image Perception—— Web, AD, Designing, Scene

23/4/8 EIE426-AICV 42

23/4/8 EIE426-AICV 52

Most Recent Application