Is virtual/augmented reality beneficial for learning? EEG mental load indices
associated with learning in 2D compared to 3D displays
Miriam Reiner
The VR and NeuroCognition lab Technion, Israel Institute of Technology
http://vrneurocog.wixsite.com/[email protected]
Amit Rozen
Alex Dan
Anat Dahan
Rotem Bennet
Nehai Farag
Darian Ryder
Nir Segal
Vered Halevi
Tal Pik
Guy Zuckerman
Boris Yazmir
Lulu Watad
Avivit Dolev
Postdocs:
Dr. Chen Ryder
20-Jun-18
The lab: Technology and methodology
Sig22 Neuroscience and Education
Eye
tracker
Robotic arms
EEGPerformance/learning
Brain correlates
Virtual Reality
(stimuli)
Virtual reality
20-Jun-18
Participants perform a task in a highly immersive virtual reality,
while connected to EEG, eyetrackers etc.
Projects• The Presence group (1999) – EU initiative to plan future calls on Presence.
• PresenCia
• PRESENCCIA
• IMMERSENCE
• EUROVERCITY (current)
• BEAMING
• Emphatic
• Beaming enhanced by machine learning for remote interaction• Smaller projects:
• Remote Presence and memory modification • BITE- Brain-research Inspired Technologies for Education• Educational neuroscience: What neuroscience tells us about learning in educational contexts
(funding from the Chief scientist, Ministry of Science)
• BBC on BEAMING : BBC's Rory Cellan-Jones. www.bbc.com/news/technology-18017745
• In a related piece Laurence Peter aof the BBC talks to Mel Slater from the University of Barcelona's and Ray Purdy from UCL about "Real-world beaming: The risk of avatar and robot crime“ www.bbc.co.uk/news/world-europe-17905533
Sig22 Neuroscience and Education20-Jun-18
A typical experiment…• immerse participants in an immersive multisensory virtual world
•ask them to perform a task (e.g. a motor task, solve a quiz/problem, demonstrate a concept, etc),
• connect participants to EEG and measure the resulting EEG signals -- frequencies, or event-related-potential;
• correlate EEG/ERP’s with behavioral patterns, stimuli/ environmental situational cues.
Sig22 Neuroscience and Education20-Jun-18
Look for correlations between:
• Purpose: attempt to model conditions for learning.
Sig22 Neuroscience and Education
Indicators of learning –
behavioral (action &verbal
Brain correlat
es
Sensory, environmental
situational cues
20-Jun-18
Will talk about:
•VR/AR, brief intro.
• Is V/AR better for learning compared to a flat screen? Why?• EEG mental load indices associated with learning in
2D compared to 3D displays
Sig22 Neuroscience and Education20-Jun-18
What is virtual reality?
•Virtual Reality (VR), can be referred to as the subjective sensory experience of
being immersed in a computer-mediated world
Sig22 Neuroscience and Education20-Jun-18
Augmented reality –virtual and physical
Sig22 Neuroscience and Education
• ….special gloves, earphones, and goggles, all of which receive their input from the computer system.
• .....senses are controlled by the designer. The system monitors the user's actions.
20-Jun-18
Design of the CAVE HTC-VIVE
https://www.youtube.com/watch?v=tAr7xjKoM5A
Setup
Sig22 Neuroscience and Education
• Virtual environment
20-Jun-18
A new type of remote learning?Face-to-face like e-learning
Only about 10% is conveyed via words… What conveys everything else? Bodily cues, physiological changes – blush, pupil fluctuations… rhythm of voice, bodily dynamics….
Four examples:
• Real-time brain measures of mental load while performing a learning task under varying conditions; (or– Is VR/AR better than flats screens?
• The brain mechanism behind Insight / incremental problem. Can we enhance insight?
• The shape of misconcept in science: what are neural responses when ‘you’ see the world behaving according to your own misconcepts?
• And how is it different from error-EEG signals?
20-Jun-18 Sig22 Neuroscience and Education
• Expedited memory consolidation and implications for a potential neural model of enhanced memory;
20-Jun-18 Sig22 Neuroscience and Education
EEG measures of
mental load
during problem solving in 2D and 3D VR. Alex Dan PhD thesis
• Two groups:
• Observed a remote instructordemonstrating two origami tasks –simple/complex- in 2D flat display Thenin VR 3D.
• The second group followed the sameprocedure, reversed order
• EEG was measured in real time.
• Participants were asked to repeat thefolding of the simple/complex task.
• EEG was not taken during actual folding
20-Jun-18 Sig22 Neuroscience and Education
Method
Viewsonic 3D ready Projector
Screen Emmiter Vision Pro
EEG recording software
WinEEG 2.96.63
Pa
rtic
ipan
t
Researcher
Mistar-EEG 201
Wired monitoring device
3D NVidia GlassesDell M6800 Laptop with
Nvidia Quadro K500M Graphic
Card
And proprietary software
HDMI to DVI Connection
Nvidia Graphics setup
Stereo Mode
120 Hz. Refresh Rate
Resolution 1024/768
Only to USB port
Procedure
20-Jun-18 Sig22 Neuroscience and Education
EEG File
• select learning interval, epochs
Artifacts reduction
• ICA and Analysis
Set Fourier analysis and parametrers
• Lenght of epoch
• Polynomial trends paremetr
es
• Haning time
window
Analyze
• Table ouput of
EEG spectra, amplitud
e and average
frequency
𝐶𝐿𝐼𝑛 =𝑇ℎ𝑒𝑡𝑎 𝐹𝑧 𝑛
𝐴𝑙𝑝ℎ𝑎 𝑃𝑧 𝑛The Cognitive load
index was calculated
based on the Theta
Fz/Alpha Pz ratio
(Holm et al., 2009). For
each participant, Alpha
Pz, Theta Fz, and the
average cognitive load
index for the
observation sessions
was assessed.
Results
20-Jun-18 Sig22 Neuroscience and Education
Scenario 2D 3DS
Relative
CLI
Red dot -Average
power over four
sessions –.
Green dotted line -
total average power
of all participants;
lue and red dotted
lines - the confidence
interval of 95%. The
y axis --absolute
power in (𝛍𝐕𝟐).
Results
Task+display
Box 2D Box 3D p-value
CLI (SDE) 0.8385 (0.024) 0.4596 (0.096) < 0.0001*
Task+display
Crane 2D Crane 3D p-value
CLI (SDE) 1.2483 (0.128) 0.500 (0.094) < 0.0002**
p-value p < 0.0292* = 0.8
A significantly larger
CLI in 2D compared
to 3D
for both tasks for all
participants
(matched pairs t-test, N
= 17, t-ratio = -3.0681,
SDE = 0.09483, p <
0.0037**).
20-Jun-18 Sig22 Neuroscience and Education
Performance –folding scores
20-Jun-18 Sig22 Neuroscience and Education
Folding Test scores of participants (N=14) after observing
and performing 2D/3D instructions of folding.
The y-axis represents Folding Test scores.
The x-axis represents the groups.
The bars represent the SDE.
The box represents the distribution of participants
Meaning what…?
• Higher mental load is correlated with higher ambiguity – 2D is more ambiguous than 3D. As if– it is more difficult to decipher the semantics behind the visuals.
• Found more Mu rhythms in the 3D condition. Associated with the mirror neuron system activations –learning through mimicking is better in 3D.
• Higher mental load is associated with a larger number of items during ‘short-term-memory’. Need to remember more in 2D. 3D supports memory by reducing the number of items to be memorized
20-Jun-18 Sig22 Neuroscience and Education
Implications for V/AR in social-emotional learning
Suggests that the face to face interaction is crucial
V/AR can be designed …..To enhance emotional interaction
by mimicking the emotional expression of the teacher/student
20-Jun-18 Sig22 Neuroscience and Education
Thanks!
Questions?
20-Jun-18 Sig22 Neuroscience and Education
This project has received funding from the European Union's Horizon 2020 Research And Innovation Programme under grant agreement No. 769872