29
The Advanced Reality Lab Understanding human-machine confluence in the 21st century Brain-computer interface, virtual reality, media & brain
The Advanced Reality Lab
Understanding human-machine confluence in the 21st century
Brain-computer interface, virtual reality, media & brain
ISVAR, April 2016
Brain-Computer Interface (BCI)
BCI
Application
closed
loop
system
feedback
brain signal control signal
BCI: Signals
ISVAR, April 2016
EEG NIRS
fMRI MEG
Electrical activity Blood oxygen level
BCI: Signals
ISVAR, April 2016
EEG NIRS
fMRI MEG
Electrical activity Blood oxygen level
expensive
BCI: Signals
ISVAR, April 2016
EEG NIRS
fMRI MEG
Electrical activity Blood oxygen level
Low spatial resolution
BCI: Signals
ISVAR, April 2016
EEG NIRS
fMRI MEG
Electrical activity Blood oxygen level
Low temporal resolution
BCI: Invasive
ISVAR, April 2016
ECoG Implanted electrode array (Hochberg et al, Nature, 2012)
BCI: Consumer level devices
ISVAR, April 2016
Thought-controlled VR: Paradigms
– Navigation (controlling the viewpoint)
– Controlling an avatar
– Controlling the world directly
• In place control
• Hybrid control
– Beyond control
ISVAR, April 2016
BCI based on EEG and motor imagery
Pfurtscheller and Neuper, Proceedings IEEE, 2001
ISVAR, April 2016
Graz-BCI and UCL-CAVE
ISVAR, April 2016
Pfurtscheller et al., Brain
Research., 2006,
Friedman et al., HCI J 2010,
Leeb et al. Presence, 2007
BCI-VR by a tetraplegic patient
ISVAR, April 2016
Leeb et al., Computational Intelligence and Neuroscience, 2007 Pfurtscheller et al., Int J Psychophysiology, 2007
Thought-controlled VR: Paradigms
– Navigation (controlling the viewpoint)
– Controlling an avatar
– Controlling the world directly
• In place control
• Hybrid control
– Beyond control
ISVAR, April 2016
Graz-BCI and UCL-CAVE
ISVAR, April 2016
Friedman et al., GRAPP, 2007
Real-time fMRI
ISVAR, April 2016
Cohen et al., IEEE Conf Biomedical Robotics, 2012 Cohen et al., J Neural Eng, 2014
robot - Video
1Video 2Video
3 classes: Left hand, right hand, and legs
ISVAR, April 2016
time
% signal change
Whole brain machine learning
ISVAR, April 2016
Classification Results
ISVAR, April 2016
Amp - 95.8%, ctrl - 97.2 % (no sig difference)
Cue-based performance (4 classes)
Cue FC
A gallery visualization of the left vs right contrast using a p<0.05 for the cue-based and free choice tasks, for all subjects from the amputees and the control group. For 'PW' the inflated brain is a mapping of the flat brain below. The red arrows represent the dominant hand and intact hand for the control and amputees groups, respectively.
Thought-controlled VR: Paradigms
– Navigation (controlling the viewpoint)
– Controlling an avatar
– Controlling the world directly
• In place control
• Hybrid control
– Beyond control
ISVAR, April 2016
Steady-state visually evoked potential (SSVEP)-based BCI
ISVAR, April 2016
ISVAR, April 2016
“Unconscious” BCI
0
10
20
30
40
50
60
70
80
90
Experiment Sham
Mean of minimum error rate
Experiment Sham
Minimum error rates between groups at optimal
accuracy time stamp (11-12 seconds, 6 seconds post
stimuli onset), t (19) = 2.225, p < 0.05
Thought-controlled VR: Paradigms
– Navigation (controlling the viewpoint)
– Controlling an avatar
– Controlling the world directly
• In place control
• Hybrid control
– Beyond control
ISVAR, April 2016
Thought-controlled VR: Paradigms
– Navigation (controlling the viewpoint)
– Controlling an avatar
– Controlling the world directly
• In place control
• Hybrid control
– Beyond control
ISVAR, April 2016
Neurofeedback
ISVAR, April 2016
New Paradigms: Beyond Control
• Passive BCI
• Affective Computing
• Next-generation Neurofeedback
ISVAR, April 2016
Thank you
Doron Friedman
http://avl.idc.ac.il
ISVAR, April 2016
