Date post: | 14-Dec-2015 |
Category: |
Documents |
Upload: | kate-cabbage |
View: | 213 times |
Download: | 0 times |
PLS’09 Beijing, China, September 7, 2009
Sponsored by US Army Research Office STIR: Advanced Estimation of Cognitive StatusContract No. W911NF-08-C-0121 15-SEP-2008 TO14-MAR-2009
PLS Tools in ElectroencephalographyLeonard J. Trejo
PDT InstitutePalo Alto, CA 94303, USA
The 6th International Conference on Partial Least Squares and Related Methods
Sept. 4th – 7th, 2009Beijing, China
PLS’09 Beijing, China, September 7, 2009
PDT and PDT Institute
• PDT• Neuroergonomics Models and Applications
• Human-System Integration• Human Performance Optimization
• Robust Biomedical Signal Processing• Embedded and Real-time Systems for Bio-Sensing
• PDT Institute• PhD/Masters/Undergraduate Training• University Partners (UC Santa Cruz, Tsinghua
University, UC San Diego, Univ. of West Florida)
PLS’09 Beijing, China, September 7, 2009
When I am not working…
PLS’09 Beijing, China, September 7, 2009
Outline• Problem: stress, workload, fatigue and performance• Response: Neuroergonomic models and control systems
– Create useful definitions of cognitive states– Model, estimate and control cognitive states
• Background– Multimodal sensor-state models using PLS and KPLS Algorithms– Successes and failures: fatigue / BCI / engagement / workload
• New directions– Truly multidimensional sensor-process models– PARAFAC, N-PLS
• Summary
PLS’09 Beijing, China, September 7, 2009
Estimation of Cognitive States
Aroused or Overloaded
Fatigued
RestingEngaged
Working
Other States
Behavior and Performance
Rewardsystem
Executivecontrol
WorkingmemorySensation
& perception
Autonomicsystem
Other Processes
Internal Processes
Biosignals
PLS’09 Beijing, China, September 7, 2009
Useful Definitions
• Engagement: selection of a task as the focus of attention and effort
• Workload: significant commit-ment of attention and effort to task
• Overload: task demands outstrip performance capacity
• Mental Fatigue: desire to with-draw attention and effort from a task
Work-load
MentalFatigue
Non-specificFactors
Engage-ment
GeneralCognitive
Status
PLS’09 Beijing, China, September 7, 2009
ElectroencaphalogramCerebral Cortex
• the outermost layers of brain• 2-4 mm thick (human)
PLS’09 Beijing, China, September 7, 2009
EEG Sources
PLS’09 Beijing, China, September 7, 2009
EEG Sources
PLS’09 Beijing, China, September 7, 2009
Other Elements of Sensor-State Models
Modality Effect of Workload
Heart rate Increase
Heart rate variability (and HFQRS) Decrease
Vertical and horizontal EOG (eye movements)
Increase
Blinks May decrease for intake
Pupil diameter Increase
Skin conductance, SCR, GSR Increase
EMG (frontalis, temporalis, trapezius) Increase
PLS’09 Beijing, China, September 7, 2009
Successful Application 1: Mental Fatigue Black = Alert Red = Mentally Fatigued
Fz Pz
FrontalTheta
ParietalAlpha
PLS’09 Beijing, China, September 7, 2009
Robust EEG-Based Classification of Mental Fatigue
2300 (Day 1) vs. 1900 Hrs (Day 2)
40
50
60
70
80
90
100
Signal-to-noise Ratio (dB)
Te
st
Pro
po
rtio
n
Co
rre
ct
21 Channels 1001009994795650
12 Channels 98989688655150
4 Channels 87889088765450
0-3-6-9-12-15-18
PLS’09 Beijing, China, September 7, 2009
Successful Application 2: BCI
PLS’09 Beijing, China, September 7, 2009
Successful Application 2: BCI
Central sulcus Primary motor areaSecondary motor area
Secondarymotor area
Lateral sulcus
Primary motor area
Resting State Real or imaginary motion
(Adapted from Beatty, 1995)
EEG from Motor Areas
PLS’09 Beijing, China, September 7, 2009
Successful Application 2: BCI
Control System for Target PracticeTrial-by-trial classification (left, right)
– 250 ms display updateDual adaptive controller design
– Adaptive PLS pattern recognition– Adaptive gain control for motion
PLS’09 Beijing, China, September 7, 2009
Successful Application 2: BCI
PLS’09 Beijing, China, September 7, 2009
Partly Successful Application: Mental Workload Estimation
Trejo, et al. ACI 2007Trejo, et al. ACI 2007
PLS’09 Beijing, China, September 7, 2009
Stress, Workload, Fatigue and Performance
Trejo, et al. ACI 2007
PLS’09 Beijing, China, September 7, 2009
Stabilizing Classifiers
Spectral Normalization
EEG Bandwidth Limiter
EEG Spectral Features
Classifiers
2s ECG Epoch
MV EOG/ECG Regression
Filters
R-wave Detector
PSDEMG
FeaturesRMS20,
Burst Duration, Burst
Frequency…
Innovations in EEG Algorithm Stabilization
ECG
FeaturesR-wave detectHR,
HRV, STD-IBI,
…
EOG
FeaturesRMS20, Blinks, EMs ,
…
AVAS Thresholds
Filters
4- 20s ECG Epoch
Gauges
PLS Algorithm
PLS’09 Beijing, China, September 7, 2009
Multimodal Overload Patterns
0 50 100 150 200 250 300 350 400 450
50
100
RT - blue; HRstd - red
0 50 100 150 200 250 300 350 400 4500
5
10
15
Left temporalis EMG - blue; Right temporalis EMG - red
0 50 100 150 200 250 300 350 400 4500
5
10
15
Fz/theta - blue; Pz/alpha - red
0 50 100 150 200 250 300 350 400 4500
100
200
300
vEOG - blue; hEOG - redTime (s)
Val
ue
PLS’09 Beijing, China, September 7, 2009
Workload-related EEG Sources
Passive viewing: theta alpha
Engaged 5: theta alpha
10.55 Hz
10.30 Hz5.79 Hz
5.79 Hz
Anterior Cingulate Inferior Parietal Precuneus
PLS’09 Beijing, China, September 7, 2009
Application Summary• Models of engagement, fatigue & BCI:
• 90-100% accurate• Stable within a day• Stable from day to day
• Models of mental workload:• 60-90% accurate• Moderately stable within a day• Unstable from day to day
PLS’09 Beijing, China, September 7, 2009
Recommended Directions
1. Deployable multimodal sensors (EEG, fNIR, EOG,
gaze, HRV, EMG, SCR, SpO2, BP, core body
temperature, gesture, posture facial expression, ...)
2. Multimodal experimental designs and operational tests
3. Advanced neurocognitive process models
4. Multimodal sensor-process mapping algorithms
PLS’09 Beijing, China, September 7, 2009
“Atomic” EEG Elements
Atoms
Molecule
Basic Sources“atoms”
CoherentSystems
“molecules”
Coherence BondsCovalent Bonds
PLS’09 Beijing, China, September 7, 2009
“Molecular” EEG Processes
Coherence BondsAtoms
PLS’09 Beijing, China, September 7, 2009
Familiar (bilinear) Mapping Algorithms
Factor Analysis
Principal Component Analysis (PCA)
ijjf
F
fifij ebax
1
F
f 1 af
bf
0ije
PLS’09 Beijing, China, September 7, 2009
Multimodal MappingHow to generalize bilinear models to systems with more dimensions?
1. Unfolding a bilinear modela. Represent all experimental factors in one dimensionb. Observations (trials) is second dimensionc. Contrast each dimension vs. pairs of the other two
2. Multidimensional modela. Assume orthogonal factors: PARAFACb. Allow interacting factors: Tucker 3
3. Modeling approacha. Unsupervised extraction: PARAFAC, CANDECOMP, Tucker 3b. Supervised extraction: N-PLS
PLS’09 Beijing, China, September 7, 2009
Unfolding a Bilinear Model
Unfolding
Dim 1 Dim 2 Dim 3
XX
X1 X2 X3
PLS’09 Beijing, China, September 7, 2009
Multidimensional Modeling (Tucker 3 Model, unsupervised)
ijklmnknjm
F
nil
F
m
F
lijk egcbax
321
111
• xijk is an element of (l x m x n) multidimensional array
• F1, F2, F3 are the number of components extracted on the 1st, 2nd and 3rd mode
• a, b, c are elements of the A, B, C loadings matrices for the 1st, 2nd and 3rd mode
• g are the elements of the core matrix G which defines how individual loading vectors in different modes interact
• eijk is an error element (unexplained variance)
PLS’09 Beijing, China, September 7, 2009
PARAFAC (Parallel Factor Analysis, unsupervised)
ijkkfjf
F
fifijk ecbax
1
F
f 1 af
bf
cf
PARAFAC is a special case of the Tucker 3 model where F1= F2 = F3=F and G = I For a 3-way array:
PLS’09 Beijing, China, September 7, 2009
N-way PLS(supervised)
X
frequency
workload condition
X
F
f 1 af
cfbf
F
f 1 vf
uf
time
time
max. covariance
electrodes
EEG
Labels
af – spectral atom bf – spatial atom cf – temporal atom
vf – workload atom uf – temporal atom
PLS’09 Beijing, China, September 7, 2009
Demo: Workload / PARAFAC EEG
Workl
oad co
ndtions
(e.g.
, trial
s, time)
Elec
trod
es
EEG Frequency
PLS’09 Beijing, China, September 7, 2009
Summary•Successes and Failures
• Fatigue, BCI, engagement: accurate, stable• Workload: variably accurate, unstable
•Useful models of state-related EEG sources• “Atomic” EEG sources• “Molecular” EEG systems
•Approaches to multidimensional models and algorithms• Tradtional bilinear methods (PCA, factor analysis, ICA)• Truly multidimensional methods
•Correlated factors (Tucker 3)•Uncorrleated factors (PARAFAC, CANDECOMP, N-PLS)•Supervised algorithms (N-PLS)