Investigating EEG for Saliency and Segmentation Applications in Image

Processing

Eva Mohedano

1

CONTENT

1 - Problem statement

2 – Related Work

3 – Local exploration of the image

4 – Experimental set-up

5 – Signal Processing of EEG Signals

6 – Conclusions

2

CONTENT

1 - Problem statement

2 – Related Work

3 – Local exploration of the image

4 – Experimental set-up

5 – Signal Processing of EEG Signals

6 – Conclusions

3

1- PROBLEM STATEMENT

Design a system based on a Brain Computer Interface (BCI) wich measureElectroencephalography (EEG) signals to answer the following questions:

BCI Data Processing

Visual stimulus

EEG Signals

2 - Are the EEG signals useful to images segmentation?

1 - Are the EEG signals to compute Saliency Maps?

4

1- PROBLEM STATEMENT

Design a system based on a Brain Computer Interface (BCI) wich measureElectroencephalography (EEG) signals to answer the following questions:

Data Processing

2 - Are the EEG signals useful to images segmentation?

1 - Are the EEG signals to compute Saliency Maps?

Visual stimulus

BCI

5

1- PROBLEM STATEMENT

Design a system based on a Brain Computer Interface (BCI) wich measureElectroencephalography (EEG) signals to answer the following questions:

Data Processing

2 - Are the EEG signals useful to images segmentation?

1 - Are the EEG signals to compute Saliency Maps?

Visual stimulus

BCI

6

1- PROBLEM STATEMENT

1 - Are the EEG signals to compute Saliency Maps?

Data Processing

Visual stimulus

BCI

Motivation:

- New way to compute maps of the atention of the imagebased directly in the reaction of the brain and not in thefeatures of the images (Niebur and Koch (1996) algorithm).

7

1- PROBLEM STATEMENT

2 - Are the EEG signals useful to images segmentation?

- Reduce the user interaction to the minimun expression.

- Measure the brain reaction at local scale of the image.

Motivation:

Data Processing

Visual stimulus

BCI

8

CONTENT

1 - Problem statement

2 – Related Work

3 – Local exploration of the image

4 – Experimental set-up

5 – Signal Processing of EEG Signals

6 – Conclusions

9

2- RELATED WORK

2.1 – BCI in image processing applications

The oddball paradigm

10

2- RELATED WORK

2.1 – BCI in image processing applications

The oddball paradigm

P300

11

2- RELATED WORK

2.1 – BCI in image processing applications

The oddball paradigm

P300

• Speed rate around 10Hz

• Usually experiements centered tofind target images not target regions

12

2- RELATED WORK

•8 electrodes placed mainly in theposterior points on the scalp.

• Which is consistent with thediscriminating activity typicallyproduced by a P300 ERP.[Optimising the Number of Channels inEEG-Augmented Image Search. GrahamHealy]

Event-Related Potential

13

2- RELATED WORK

•8 electrodes placed mainly in theposterior points on the scalp.

• Which is consistent with thediscriminating activity typicallyproduced by a P300 ERP.[Optimising the Number of Channels inEEG-Augmented Image Search. GrahamHealy]

Event-Related PotentialHow to present the image to generate and detect this signal?

14

CONTENT

1 - Problem statement

2 – Related Work

3 – Local exploration of the image

4 – Experimental set-up

5 – Signal Processing of EEG Signals

6 – Conclusions

15

First Design: Sliding Window

http://www.youtube.com/watch?v=bKTGKVx58Ps

3- LOCAL EXPLORATION OF THE IMAGE

16

CHALLENGE 1

• Eyes movement affect to the EEG signals – Introduce Artifacts to the signal

Opened eyes / Closed eyes. Image from the slides Dr. Ranjith Polusani

3- LOCAL EXPLORATION OF THE IMAGE

17

CHALLENGE 2

• Progressive inspection may not generate a useful reaction in the EEG waves.

- Follow Oddball Paradigm and perform RSVP od the windows

P300

Suggestions meeting Thomas Ward and Nima Bidgely Shamlo :

SNAP - Simulation and Neuroscience Application Platform

3- LOCAL EXPLORATION OF THE IMAGE

18

CHALLENGE 3

•Syncronitzation Problem

3- LOCAL EXPLORATION OF THE IMAGE

19

CHALLENGE 4

•Size of the object / window

Grabcut Dataset – Objects of different size

Suggestions meeting Thomas Ward and Nima Bidgely Shamlo :

To use images with an homogeneus background with a salient object.The number of distractors (windows with background) must be higher than the number of targets (windows with object).

3- LOCAL EXPLORATION OF THE IMAGE

20

CHALLENGE 5

•What am I seeing?

3- LOCAL EXPLORATION OF THE IMAGE

21

CHALLENGE 5

•What am I seeing?

3- LOCAL EXPLORATION OF THE IMAGE

22

CHALLENGES SOLUTIONS

1 - Eyes movement

2 - Progressive inspection

3 - Syncronitzation Problem

5 - What am I seeing?• Is it just noise?• Am I able to detect something?

Display a fixed window on the screen

SNAP to perform a random RSVP

First test with flashes to find ERPS

Real time visualitzation of the signal

4 - Size of the object / window Generate my own dataset

3- LOCAL EXPLORATION OF THE IMAGE

23

CONTENT

1 - Problem statement

2 – Related Work

3 – Local exploration of the image

4 – Experimental set-up

5 – Signal Processing of EEG Signals

6 – Conclusions

24

Second Design: Starting from the easiest case

a) Device CalibrationI. Real time visualization of Alpha wavesII. Detecting ERPS

b) Synthetic ImagesI. RSVP synthetic images fitted in the window.

c) RSVP real images

4- EXPERIMENTAL SET-UP

http://www.youtube.com/watch?v=KsgtvQkOElQ&feature=youtu.be

25

a) Device Calibration

4- EXPERIMENTAL SET-UP

Is the device wellconnected?

Is the syncronitzationmethod correct ?

Am I able to detectsomething?

?

26

4- EXPERIMENTAL SET-UP

a) Device Calibration

SIGNAL EXPECTED - Closed eyes – Alpha waves (8-12 Hz)

Closed-eye EEG alpha waves (10-20 channels Pz-Top, Fz-Bottom) extracted from http://blog.grahamhealy.com/

27

4- EXPERIMENTAL SET-UP

a) Device Calibration

SIGNAL OBTAINED

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5-40

-20

0

20

40

Time (sec)

Am

plit

ude (

uV

)

5 seconds Closed Eyes

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5-40

-20

0

20

40

Time (sec)

Am

plit

ude (

uV

)

5 seconds Opened Eyes

28

4- EXPERIMENTAL SET-UP

Finding ERPS response after a white flash

Presenting a serie of white flashes (2 seconds between the flashes)

SIGNAL EXPECTED: After the flash P100 and a negative peak between 150-200ms

a) Device Calibration

29

SIGNAL OBTAINED: 60 Flashes to get the response.

Channel P100 (ms) N1 (ms)

1 130 320

2 90 210

3 90 210

4 10 220

5 110 220

6 90 210

7 10 220

8 100 22

Mean 80 23

0 100 200 300 400 500 600 700 800 900 1000-15

-10

-5

0

5

10

Time (ms)

Am

plit

ude (

uV

)

Averaged ERP waveform per channel

a) Device Calibration

Finding ERPS response after a white flash

4- EXPERIMENTAL SET-UP

30

CONTENT

1 - Problem statement

2 – Related Work

3 – Local exploration of the image

4 – Experimental set-up

5 – Signal Processing of EEG Signals

6 – Conclusions

31

5- SIGNAL PROCESSING OF EEG SIGNALS

1 Target

99 Distractors

100 windows per Image

Data adquisition

32

5- SIGNAL PROCESSING OF EEG SIGNALS

Preprocessing: Single trial

1 2

3 4

5

7

6

8

One Image

33

5- SIGNAL PROCESSING OF EEG SIGNALS

Preprocessing: Single trial - PROBLEM

- Signal very noisy

- Single Targets and Single Distractors very similar

34

5- SIGNAL PROCESSING OF EEG SIGNALS

Preprocessing: Feature

Mean Absolute Amplitude looks different

Energy from 0 to 600ms

96 Distractors

96 Targets

Feature for the window presented

35

5- SIGNAL PROCESSING OF EEG SIGNALS

Preprocessing: Averaged trials

1 averaged target

99 averaged distractors

Energy from 0-600 ms

36

5- SIGNAL PROCESSING OF EEG SIGNALS

Preprocessing: Averaged trials

1 x 32 single repeats

99 x 32 single repeats

1 x 32 window repeats

99 x 32 single repeats

1 x 32 window repeats

99 x 32 single repeats

Average by 32

1 averaged target

99 averaged distractors

1 averaged target

99 averaged distractors

1 averaged target

99 averaged distractors

SINGLE AVERAGED

37

5- SIGNAL PROCESSING OF EEG SIGNALS

Preprocessing: Averaged trials

Problem:

Too few (target) samples for training

1 averaged target

99 averaged distractors

1 averaged target

99 averaged distractors

1 averaged target

99 averaged distractors

38

5- SIGNAL PROCESSING OF EEG SIGNALS

Bootstrapping

1 x 32 single repeats

99 x 32 single repeats

1 x 32 window repeats

99 x 32 single repeats

1 x 32 window repeats

99 x 32 single repeats

SINGLE

Bootstrapping

1 x 32 averaged target

99 x 1 averaged distractors

1 x 32 averaged target

99 x 1 averaged distractors

1 x 32 averaged target

99 x 1averaged distractors

AVERAGED

39

5- SIGNAL PROCESSING OF EEG SIGNALS

Classification

Problem:

Unbalanced dataset for binary classification

1 x 32 averaged target

99 x 1 averaged distractors

1 x 32 averaged target

99 x 1 averaged distractors

1 x 32 averaged target

99 x 1averaged distractors

AVERAGED

40

5- SIGNAL PROCESSING OF EEG SIGNALS

Classification

1 x 32 averaged targets

99 x 1 averaged distractors

1 x 32 averaged targets

99 x 1 averaged distractors

1 x 32 averaged targets

99 x 1averaged distractors

AVERAGED

Subsample

Subsample

Subsample

1 x 32 averaged targets

32 x 1 avgd distractors

1 x 32 averaged targets

32 x 1 avgd distractors

1 x 32 averaged targets

32 x 1 avgd distractors

AVERAGED

41

5- SIGNAL PROCESSING OF EEG SIGNALS

Classification

1 x 32 averaged targets

32 x 1 avgd distractors

1 x 32 averaged targets

32 x 1 avgd distractors

1 x 32 averaged targets

32 x 1 avgd distractors

AVERAGED HISTOGRAM

42

5- SIGNAL PROCESSING OF EEG SIGNALS

Classification

1 x 32 averaged targets

32 x 1 avgd distractors

1 x 32 averaged targets

32 x 1 avgd distractors

1 x 32 averaged targets

32 x 1 avgd distractors

AVERAGED

SVMTRAIN(linear kernel)

Classifier

43

5- SIGNAL PROCESSING OF EEG SIGNALS

Classification

100 x 32 single repeatsAverage by 32

100 avgd windows

44

5- SIGNAL PROCESSING OF EEG SIGNALS

Classification

100 x 32 single repeats

SVMPREDICT

ClassifierAverage by 32

100 avgd samples

45

5- SIGNAL PROCESSING OF EEG SIGNALS

8 samples feature vectors

Cross validation approach 3 train + 1 test

CONTENT

1 - Problem statement

2 – Related Work

3 – Local exploration of the image

4 – Experimental set-up

5 – Signal Processing of EEG Signals

6 – Conclusions

47

48

6- CONCLUSIONS

- Results from sythetic images provide and evidence that BCI devices could beused to located an object into an image.

- Simplicity of the system: Energy value from 8 channels to train SVM withlineal kernel.

Future work

-Study the impact of the number of repetitions.

- Extract better features.

-Analize data from real images.

-Tool to evaluate and compare the EEG mask (ROC, Jaccard index)