Post on 04-Jan-2016
description
transcript
An Emotion Recognition Journey
Lucy Kuncheva
A long time agoin a galaxy far, far
away...
around the summer of 2008,we came in contact with the School of Psychology at Bangor University.
I’d better move to Cardiff...
I’d better move to Brunel...
Areas of activation in the brain in response to emotion stimuli.
Amygdala
Areas of activation in the brain in response to emotion stimuli.
The limbic system
fMRI data were acquired from 16 right-handed healthy US college male students (aged 20–25).
The Dead Salmon Lesson
How far are we from MIND READNING?
this far...
March 2009
EPSRC proposal
New approaches for fMRI data analysis
April 2010
FP7 proposal
Lost an entire month of my life to this....
1. Kuncheva L.I., J. J. Rodriguez, C. O. Plumpton, D. E. J. Linden and S. J. Johnston, Random Subspace Ensembles for fMRI Classification, IEEE Transactions on Medical Imaging, 29 (2), 2010, 531-542.
2. Kuncheva L.I., J. J. Rodriguez, Classifier Ensembles for fMRI Data Analysis: An Experiment, Magnetic Resonance Imaging, 28 (4), 2010, 583-593.
3. Kuncheva L.I. and C. O. Plumpton, Choosing parameters for Random Subspace ensembles for fMRI classification, Proc. Multiple Classifier Systems (MCS'10), Cairo, Egypt, LNCS 5997, 2010, 54-63.
4. Plumpton C. O., L. I. Kuncheva, N. N. Oosterhof and S. J. Johnston, Naive random subspace ensemble with linear classifiers for real-time classification of fMRI data, Pattern Recognition, 45 (6), 2012, 2101-2108.
5. Plumpton C. O., L. I. Kuncheva, D. E. J. Linden and S. J. Johnston, On-line fMRI Data Classification Using Linear and Ensemble Classifiers, Proc. ICPR 2010, Istanbul, Turkey, 2010, 4312-4315.
9
41
70
7
5
Cat Plumpton
PhDreal-time fMRI data analysis
Tom Gardner 10/113rd year projectEnvironments for emotion recognition
Adam Williams 09/103rd year project
Emotion recognition from fMRI data
Jamie Blacker 09/103rd year project
Fractals and fMRI
Joe Freeman 10/113rd year projectfMRI Visualiser
Joey Owen 10/113rd year projectfMRI Voxel selection
Colin Steele 08/093rd year projectfMRI Data analysis
10 20 30 40 50
10
20
30
40
50
10 20 30 40
10
20
30
40
10 20 30 40 50
10
20
30
40
50
50 100 150 200
195
200
205
210
215
2010
End of the fMRI era...
Enter Tom Christy!
Summer 2010
Brain-computer interface through EEG
Game accessories
EEG headsets
Peripheral devices
Accessible
Inexpensive
And just like that...The idea was born...
A game controlled by emotion
Summer 2010
Summer 2010
Sa’ad
Martin
Not as easy as it looked...
Autumn 2010
AFFECTIVE COMPUTING
TAC celebrates its 5th Anniversary
The Galvactivator: A glove that senses and communicates skin conductivity
“Affective Computing is an area of computing that relates to, arises from, or influences emotions.” Rosalind Picard, 1995
HAHV
HALV
POSITIVE
Arousal
LALV
LAHV happy
excited
angry
fearful
content
calm
sad
depressed
PASSIVE
ACTIVE
Valence
NEGATIVE
EXPRESSION OF EMOTION - MODALITIES
facial expression
posture
behaviouralphysiologic
al
peripheral nervous system
central nervous system
EEG
fMRI
Galvanic skin response
blood pressure
skin to
respiration
EMG
speech
gesture
interaction with
the compute
r
pressure on mouse
drag-click speed
eye tracking
fNIRS
pulse rate
pulse variation
dialogue with tutor
facial expression
posture
EEG
fMRI
Galvanic skin response
blood pressure
skin to
respiration
EMGspeech
gesture
pressure on mouse
drag-click-zoom-type
speed
eye tracking
fNIRS
pulse ratepulse variation
dialogue with tutor
Detecting Stress During Real-World Driving Tasks Using Physiological SensorsJennifer A. Healey and Rosalind W. PicardIEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 6, NO. 2, JUNE 2005
The subject wore five physiological sensors:
• an electrocardiogram (EKG) on the chest
• an electromyogram (EMG) on the left shoulder
• a chest cavity expansion respiration sensor (Resp.) around the diaphragm,
• skin conductivity sensor on the left hand
• skin conductivity sensor on the left foot
AMBER Experiment #1
Tom
NeuroSky EEG
Pulse signal
Galvanic skin response
(+) Positive emotionWhale and ocean sounds...
(-) Negative emotion• Michael Jackson• Cheeky girls
3 4 5 6 10 12 15 20
1nn 62.8 64.9 63.4 62.0 61.1 60.9 56.5 59.9
DT 64.2 58.6 67.4 65.9 58.5 62.8 70.0 58.9
RT 60.0 63.0 61.9 62.6 57.1 66.7 66.8 57.1
NB 64.7 63.8 64.5 64.5 65.0 67.8 65.4 61.1
LOG 62.0 60.4 62.6 63.3 59.3 59.2 57.6 57.5
MLP 62.5 59.4 63.3 63.4 63.4 64.2 57.1 58.5SVM-
L 62.1 61.4 63.5 62.4 62.3 59.1 58.7 56.8SVM-
R 50.8 51.2 50.6 50.5 50.2 51.2 51.7 51.3
BAG 65.6 65.6 68.3 67.1 67.4 68.8 66.5 64.4
RAF 64.5 64.7 66.1 65.3 65.9 69.6 67.3 61.6
ADA 63.4 62.2 70.0 67.6 61.1 66.3 73.8 63.3
LB 65.3 62.9 68.8 68.1 62.0 64.0 68.3 60.7
RS 65.0 64.8 66.3 68.2 64.6 67.4 69.0 61.8
ROF 66.9 65.4 66.9 67.2 67.4 69.3 65.5 62.3
Individual classifiers
Ensembles
Serious wired-up man Tom Christy
Very important supervisorLucy Kuncheva
A.M.B.E.R.Advanced Multimodal Biometric Emotion Recognition
NeuroSky
EMOTIV
Nia
My lovely plant
Tom’s R2D2
(show off!)
EDAPulse reader
Cast:
A.M.B.E.R.Advanced Multimodal Biometric Emotion Recognition
Media stars overnight!
HARDWARE
DATA&
CollaboratorsTom Lucy
Tom
HARDWARE201320122011
2013“emotional mouse”
Version 1
EDA sensors(Galvanic skin response)
pulse reader
ReincarnationVersion 2
In the meantime:
Guillaume Thierry
This is what Google returned 2nd on “Guillaume Thierry”!
Me: Would you like to collaborate on emotion recognition from EEG data? Guillaume: Yes, of course, but listen what a fantastic idea occurred to me just now!!!”
Christoph Klein
Stephan Boehm
Let’s give it a go
March 2011
EPSRC proposal
No One is a Prophet in their Own Land
Hello Salzburg!Seminar
May 2012
On-going collaboratio
n with the University of
Salzburg
Hello Juan Rodriguez!
Hello Ramon Mollineda!
happy
elated
excited
alertPOSITIVE AROUSAL
tense
nervous
stressed
upset
UNPLEASANT
sad
depressed
bored
drowsy
NEGATIVE AROUSAL
calm
relaxed
tranquil
contended
PLEASANT
Video: Arch Enemy (My Apocalypse)
Volunteers Participants All
0 1 2 3 4 5 6 7 8 9 100
1
2
3
4
5
6
7
8
9
10
0 1 2 3 4 5 6 7 8 9 100
1
2
3
4
5
6
7
8
9
10
What emotion is being provoked?!?
0 1 2 3 4 5 6 7 8 9 100
1
2
3
4
5
6
7
8
9
10
Valence
Aro
usal
T: Intended emotion
E: EXPERIENCED emotion
P: Reported emotionProvoked?
Acted?
Spontaneous?
Self-reported?
So, WHAT are we RECOGNISING?
1. Emotions are very difficult to define and explicate.2. Experiments for provoking emotion vary considerably, and so do
the results reported in the literature ( from near chance to 95% accuracy).
3. Most emotion measuring modalities are intrusive and annoying.4. Emotions are individual for each person.5. The measured signals are difficult to analyse. There is a
bottleneck of idiosyncratic feature extraction and parameter tuning.
6. There is no unified protocol. 7. Benchmark data collections are not available. 8. There is no consensus about the type of experiment to validate
a hypothesis (provoked, controlled, acted, spontaneous emotion).
Are we doomed?
Maybe not...
We can detect CHANGE in the physiological responses and the EEG, which may be associated with some emotion. If we need to ACT upon detecting an emotion rather than NAMING it, we still may have a chance.
The emotional mouse
Other ingenious input devices
User-friendly EEG headsets
Simple, transparent and generic technologies for feature extraction.
State-of-the-
art data analys
is
LucyTom
Unified protocols, benchmark data
The emotional mouse
Other ingenious input devices
User-friendly EEG headsets
Simple, transparent and generic technologies for feature extraction.
State-of-the-
art data analys
is
LucyTom
Unified protocols, benchmark data
THE END