A Novel EEG-BCI Dependent on Discrimination of Imagined Stimuli in Visual Field Quadrants
by
Filip Stojic
A thesis submitted in conformity with the requirements for the degree of Master of Applied Science
Institute of Biomaterial and Biomedical Engineering University of Toronto
© Copyright by Filip Stojic 2017
ii
A Novel EEG-BCI Dependent on Discrimination of Imagined
Stimuli in Visual Field Quadrants
Filip Stojic
Master of Applied Science
Institute of Biomaterials and Biomedical Engineering
University of Toronto
2017
Abstract
Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication.
This study aimed to determine whether visuospatial imagery could be used to signify intent in an
electroencephalography (EEG)-based BCI. Eighteen healthy participants imagined stimuli in visual field quadrants
while EEG was collected. A subset of participants used visuospatial imagery to control a character’s movement in
four directions. Classifying rest against non-specific visuospatial imagery attained accuracies of 77±11.4%. Six
participants exceeded chance when discriminating imagery in diagonally-opposing quadrants. These six could be
predicted using a regression model that combined visuospatial perception and fatigue scores. The 4-class navigation
accuracy was 48.3±18.8% (max 85%). Imagery in one hemifield corresponded to significant increases in alpha
spindles in ipsilateral visual cortical regions. While further improvement is necessary to make the paradigm
generalizable, visuospatial imagery shows promise for becoming a useful system for individuals in need of a BCI
access technology.
iii
Acknowledgments
I would like to thank Dr. Tom Chau for his unrelenting support and guidance in my pursuit of this thesis. Thank
you for teaching me so much, and for always encouraging my ambitions. If I ever learn to be half as good a mentor
as you, I will have achieved the impossible. I am grateful beyond words.
Thank you to my committee members, Drs. Adrian Nestor, Deryk Beal and Cesar Marquez-Chin. You were there to
help me strive to think critically and in new ways, and have helped to shape me as a scientist.
My gratitude is also extended to the PRISM lab, in particular to Ka Lun Tam and Pierre Duez, who were always
there in a pinch with brilliance.
Thank you to my parents, whose endless interest helped me believe I was doing something worthwhile. Thank you
twice for always being there for me and for supporting my dreams.
Finally, I would like to express my immense gratitude for the generosity of the Kimel Family for their Scholarship
in Pediatric Disability, the Faculty of Engineering for their graduate student award and the Bloorview Research
Institute.
iv
Table of Contents
Acknowledgments .................................................................................................................. iii
Table of Contents .....................................................................................................................iv
List of Tables ......................................................................................................................... vii
List of Figures ....................................................................................................................... viii
List of Abbreviations ................................................................................................................ x
Chapter 1 ................................................................................................................................... 1
Introduction .......................................................................................................................... 1
1.1 Motivation..................................................................................................................... 1
1.2 Question, hypothesis and objectives ............................................................................. 1
1.3 Thesis outline ................................................................................................................ 2
Chapter 2 ................................................................................................................................... 3
Background and Literature Review ..................................................................................... 3
2.1 Brain computer interfaces (BCIs) ................................................................................. 3
2.1.1 Data Acquisition ............................................................................................... 3
2.1.2 Signal processing and analysis ......................................................................... 4
2.1.3 Feedback ........................................................................................................... 5
2.1.4 Types of BCIs ................................................................................................... 5
2.2 Visual Processing Pathway ........................................................................................... 6
2.2.1 Regions of interest ............................................................................................ 6
2.2.2 Perception, imagery, memory and attention ..................................................... 7
2.2.3 Visual tasks ....................................................................................................... 7
Chapter 3 ................................................................................................................................... 9
Development and assessment of a novel visuospatial imagery-based EEG-BCI ................ 9
3.1 Abstract ......................................................................................................................... 9
v
3.2 Introduction................................................................................................................... 9
3.2.1 BCI paradigms ................................................................................................ 10
3.3 Methods ...................................................................................................................... 12
3.3.1 Participants ..................................................................................................... 12
3.3.2 Instrumentation ............................................................................................... 12
3.3.3 Experimental protocol .................................................................................... 13
3.3.4 The visual imagery stimulus ........................................................................... 17
3.3.5 BCI pipeline .................................................................................................... 18
3.3.6 Online ............................................................................................................. 18
3.3.7 Offline analysis and online navigation game.................................................. 19
3.4 Results ........................................................................................................................ 24
3.4.1 Chance levels .................................................................................................. 24
3.4.2 Classifying perception and imagery against resting mental state ................... 24
3.4.3 Classifying quadrants in perception and imagery ........................................... 27
3.4.4 Features characteristic of visuospatial perception and imagery ..................... 30
3.4.5 Multiclassification of visuospatial imagery .................................................... 31
3.4.6 Predicting classification accuracies ................................................................ 32
3.4.7 Characterizing EEG features indicative of diverse imagery mental states ..... 35
3.4.8 Summary of key findings................................................................................ 36
3.5 Discussion ................................................................................................................... 37
3.5.1 Visuospatial perception classification ............................................................ 37
3.5.2 Visuospatial imagery classification ................................................................ 38
3.5.3 Similarities between perception and imagery ................................................. 40
3.5.4 Multiclassification .......................................................................................... 41
3.5.5 Predictors of accuracy..................................................................................... 41
3.5.6 Differences in lateralization of alpha .............................................................. 43
vi
3.5.7 Potential sources of variability ....................................................................... 44
3.5.8 Potential solutions to the variability ............................................................... 45
3.5.9 Key messages.................................................................................................. 46
3.6 Conclusion .................................................................................................................. 47
Chapter 4 Conclusion ............................................................................................................. 48
Conclusion ......................................................................................................................... 48
4.1 Contributions .............................................................................................................. 48
4.2 Future Work ................................................................................................................ 48
References............................................................................................................................... 49
Appendices ............................................................................................................................. 58
vii
List of Tables
Table 1 BCI experimental protocol for each data collection session .......................................... 13
Table 2 Confusion matrix for all participant data from sessions 4 and 5 .................................... 32
Table 3 Spearman correlation table for imagery versus rest classification accuracy .................. 33
Table 4 Spearman correlation table for imagery versus imagery ................................................ 33
Table 5 Group differences between males and females using the Wilcoxon rank sum test ........ 35
Table 6 Most significant channel pair differences in alpha spindles using the Wilcoxon rank-
sum test ........................................................................................................................................ 36
viii
List of Figures
Figure 1 Example Attention Condition trial (UL) ....................................................................... 14
Figure 2 Example Auxiliary Condition trial (UR) ....................................................................... 14
Figure 3 Example Perception condition trial (UL) ...................................................................... 14
Figure 4 Example Imagery Condition trial (LR) ......................................................................... 15
Figure 5 Example Visual Feedback Condition trial (UL), with annotated probabilities (not
presented to participants) ............................................................................................................. 16
Figure 6 Example Navigation Game trial (UL) with exaggerated fixation crosses .................... 17
Figure 7 Schematic of Navigation Game trial classification with two tiers of classifiers ........... 24
Figure 8 Classification accuracies for perception versus rest with user-optimized classifier ..... 25
Figure 9 Offline classification accuracies of rest versus any imagery with user-optimized
classifiers ..................................................................................................................................... 25
Figure 10 Classification of online trials....................................................................................... 26
Figure 11 Frequency of feature group selection by elastic net regularization for perception (left)
and imagery (right) vs. rest classification .................................................................................... 27
Figure 12 Frequency of feature group selection by elastic net regularization for perception
compared to imagery (left) and perception compared to imagery with selection adjusted for
number of sub-features (right) ..................................................................................................... 27
Figure 13 Frequency of optimal CSD spatial filter parameters: spline flexibility (left); smoothing
factor (middle); Legendre polynomial order (right) .................................................................... 28
Figure 14 Classification accuracies of perceiving stimuli in UL vs. LR and LL vs. UR quadrants
with optimized spatial filter parameters ...................................................................................... 28
Figure 15 Frequency of optimal CSD parameters: spline flexibility (left), smoothing facto
exponent (middle), Legendre polynomial (right) ........................................................................ 29
ix
Figure 16 Imagery versus imagery classification accuracies with optimized spatial filter
parameters and classifiers ............................................................................................................ 29
Figure 17 Frequency of selected feature groups in classifying different types of perception
sorted by participant .................................................................................................................... 30
Figure 18 Grand average frequency of selected feature groups in classifying perception in
diagonally-opposing quadrants .................................................................................................... 30
Figure 19 Frequency of selected feature groups in classifying different types of imagery sorted
by participant ............................................................................................................................... 31
Figure 20 Grand average frequency of selected feature groups in classifying imagery in
diagonally-opposing quadrants .................................................................................................... 31
Figure 21 Four-class classification accuracies in offline (direct and two-tier) and online
approaches ................................................................................................................................... 32
Figure 22 Linear model combining pre-session exhaustion with perception classification
accuracies to predict imagery classification accuracies ............................................................... 34
Figure 23 Linear model of perception classification accuracies predicting imagery classification
accuracies ..................................................................................................................................... 34
Figure 24 Alpha spindle differences (left-right) in channel pairs across participants and imagery
quadrants ...................................................................................................................................... 35
Figure 25 Average difference in time-frequency power for PO3 minus PO4 channel pair in
participant 17 for visuospatial imagery in the left hemifield (left) and right hemifield (right);
color bar represents power/frequency (dB/Hz)............................................................................ 36
Figure 26 UL vs. LR imagery classification accuracies with and without noise attenuation and
optimized spatial filters................................................................................................................ 58
Figure 27 LL vs. UR imagery classification accuracies with and without noise attenuation and
optimized spatial filters................................................................................................................ 58
x
List of Abbreviations
ALS Amyotrophic Lateral Sclerosis
BCI Brain Computer Interface
CPSD Cross Power Spectral Density
CRLS Conventional Recursive Least-Squares
CSD Current Source Density
CVSA Covert Visuospatial Attention
ECoG Electrocorticography
EEG Electroencephalography
EOG Electro-oculography
FCBF Fast Correlation-Based Filter
fMRI functional Magnetic Resonance Imaging
fNIRS functional Near-Infrared Spectroscopy
FWHM Full Width at Half Maximum
ITR Information Transfer Rate
KNN K-Nearest Neighbours
LDA Linear Discriminant Analysis
LGN Lateral Geniculate Nucleus
LL Lower Left
LR Lower Right
MEG Magnetencephalography
MS Magnitude-squared
PSD Power Spectral Density
xi
RBF Radial Basis Function
RMSE Root Mean Squared Error
ROCF Rey-Osterrieth Complex Figure
SFFS Sequential Forward Floating Search
SMR Sensorimotor Rhythms
SNR Signal-to-Noise Ratio
SSVEP Steady State Visually-Evoked Response Potential
STFT Short-Time Fourier Transform
SVM Support Vector Machines
UL Upper Left
UR Upper Right
VVIQ2 Vividness of Visual Imagery Questionnaire
1
Chapter 1
Introduction
1.1 Motivation
Many individuals with conditions such as stroke, cerebral palsy, spinal cord injury or amyotrophic lateral sclerosis
(ALS) live with motor impairments that prevent them from interacting or communicating with their environment.
Individuals who present as locked-in may be fully paralyzed, but can still be conscious of events around them [1].
For these individuals, brain-computer interfaces (BCIs) may restore useful function by providing an alternative
means of communication not dependent upon voluntary verbalization or motor activations [2]. In BCI studies, user
intent, which is necessary for communication, is identified by detecting neural activity elicited from different
mental tasks. While many of these tasks have been useful in healthy populations, relying on these tasks in
populations with motor impairments poses problems. For instance, tasks that require participants with ALS to
sustain attention to a computer screen have been found to induce fatigue and interfere with BCI control [3]–[6].
Additionally, BCIs that depend on visual attention require participants to voluntarily control their eye muscles,
which may be difficult in patients with more severe motor disorders [7]. Other tasks such as mentally listing words
starting with a certain letter (“verbal fluency”) or sequentially subtracting numbers (“mental arithmetic”) are
cognitively demanding, and may not be intuitive to users [8]–[10]. Tasks such as the motor imagery task
(imagining movement of a particular body part) may be more intuitive, and have been shown to reduce fatigue in
patients with disorders such as ALS [11]. However, motor imagery can also be impaired in individuals with motor
disorders [12], [13]. Ideally, tasks used to identify communicative intent in populations with motor disorders should
be intuitive, should not induce a great degree of fatigue and should not rely on potentially impaired motor neural
pathways. Visuospatial imagery (imagining stimuli in the visual field) does not depend on motor circuits or on eye
gaze and may thus be a promising task. Like its motor analog, visuospatial imagery has an intuitive appeal; for
instance, imagining an arrow in the upper right quadrant of the visual field may indicate a desire to more forward
and to the right in a motorized wheelchair. Practical BCI use in the real world necessitates the development of a
task sensitive to the needs of the user. This study aims to determine the feasibility of one such task – mental
imagery in the four quadrants of the visual field.
1.2 Question, hypothesis and objectives
Question: Which approaches can be used to effectively identify visuospatial imagery in visual field quadrants as
commands in an EEG-BCI?
Hypothesis: Mental visual imagery in the four visual quadrants can be discriminated with above-chance accuracies
for binary and multi-class problems, with both offline and online EEG-BCI paradigms, in a healthy control
population.
2
Objectives:
1. To determine the accuracies with which different classes of mental states can be discriminated in offline
and online EEG-BCI paradigms dependent on visuospatial imagery
2. To evaluate the predictive quality and influencing nature of external factors on performance in a
visuospatial imagery-based EEG-BCI
3. To characterize EEG activity indicative of visuospatial imagery as well as its relationship to visuospatial
perception
1.3 Thesis outline
Chapter 1 introduced the motivation for this thesis and outlined the research hypothesis, questions and objectives.
Chapter 2 provides a literature review on brain-computer interfaces and their components, the various mental task
paradigms within BCIs, as well as the visual system and how it might be exploited for use in a BCI. Subsequent to
this is a chapter on the development of a BCI dependent on visuospatial imagery. Finally, Chapter 4 recapitulates
the contributions of this thesis, as well as potential directions for future studies.
3
Chapter 2
Background and Literature Review
2.1 Brain computer interfaces (BCIs)
BCIs enable users to interact and communicate with their environment through cognitive activity alone [14]. They
consist of an input or data acquisition method (where a change in brain signal is evoked and detected), signal
processing (where brain signals are processed and analyzed before being translated into commands), and an output
(where detected commands cause some external change) that may be used as feedback [14].
2.1.1 Data Acquisition
Multiple modalities exist for acquiring brain signals of interest, including but not limited to, functional magnetic
resonance imaging (fMRI), functional near infrared spectroscopy (fNIRS), magnetencephalography (MEG),
electrocorticography (ECoG) and electroencephalography (EEG). Each modality has its strengths and weaknesses,
among them portability, cost, invasiveness, ease of setup, and spatial and temporal resolution. The most popular
modality used in BCI studies is EEG, discussed below.
Electroencephalography
Many BCIs use electroencephalography (EEG) to acquire brain signals due to its non-invasiveness, low cost, and
ease of setup [15]. The EEG modality works by detecting changes in synchronized electrical activity from
populations of neurons in the cortex that travel through the meninges, skull, and scalp [2]. Typically, this signal is
very small and requires amplification prior to digitization [15]. Because EEG relies on electrophysiological brain
activity, signal detection is fast, on the order of milliseconds, in contrast to other modalities such as functional near
infrared spectroscopy (fNIRS), which relies on slower hemodynamic responses [15]. However, despite its high
temporal resolution, EEG has relatively low spatial resolution, as a result of “noisy” electrical activity from the
scalp and surrounding brain areas [16], as well as artifacts from power lines (60 Hz), and myogenic noise (e.g. eye
movements) [2]. Spatial resolution can be improved somewhat with an increased number of electrodes, but this
requires increased setup time and the result remains inferior to that of modalities such as fMRI. While ECoG has
both the high temporal resolution of EEG and spatial resolution of fMRI, its invasiveness makes it impractical for
BCI studies due to the placement of electrodes directly on the cortex [15], [17]. Use of a dry EEG system
considerably reduces setup time and removes the need of electrode gel. While improving the practicality and
generalizability of BCI systems in clinical settings, dry EEG electrodes have previously been shown to yield lower
signal-to-noise ratios (SNRs) than those of standard wet electrodes [18], [19].
4
2.1.2 Signal processing and analysis
Preprocessing
Upon acquisition of brain signal, some preprocessing may be necessary in order to extract useful features. This may
include downsampling, where the frequency at which the data were initially collected (e.g. 1000 Hz) is reduced to a
lower sampling rate (e.g. 256 Hz), in part to reduce subsequent processing times. Additionally, filters may be
applied to the data to limit the content of the data to a meaningful range. For instance, bandpass filters between 1
and 50 Hz have been used for mental imagery paradigms that depend on beta oscillations (13-30 Hz) [20]. Filters
can also be used to remove specific artefacts such as those from power lines, or muscle activity. The preprocessing
step also includes removal of ocular artifacts (e.g. eye blinks), through methods such as independent component
analysis (ICA) [21] and recursive least-squares regression [22].
Feature Extraction
In feature extraction, characteristic components of the signal are sought to identify specific brain events that
correspond to user commands [14], [15]. These features may be time-dependent, such as signal amplitudes at
specific time points (e.g. P300 positive potential at 300ms post-stimulus presentation [23]), or frequency-
dependent, such as increased amplitude in beta oscillations during motor imagery [14]. Feature engineering is the
process of developing and identifying features such as these, in order to best characterize the data [24].
Classification
Next, characteristic signal features need to be translated into user commands. This can be done by training
classification algorithms to automatically differentiate user intents [25]. For instance, in binary-class problems, one
mental task (e.g. rest) corresponding to a specific command (e.g. “No”) may be characterized by high alpha
frequency band power, whereas the other task (e.g. motor imagery) corresponding to another command (e.g. “Yes”)
may be characterized by high beta frequency band power. Classification algorithms can identify these patterns of
brain activity [26]. Different algorithms for classification exist, the most popular of which include classifiers such
as linear discriminant analysis (LDA) and support vector machines (SVM) [25]. It is also during this step that the
most useful features may be selected through feature selection methods such as fast correlation based filter (FCBF)
feature selection and sequential forward floating search (SFFS) feature selection [27], [28]. Feature selection is
useful as it maximizes the classification performance while decreasing the number of terms used by only selecting
the most relevant features, thereby creating a simple and generalizable model [15].
For offline paradigms, cross-validation methods may be used in order to determine classification accuracies in a
way that reduces classifier bias (e.g. from over-fitting the data) [15]. In k-fold cross-validation, data are split into k
groups consisting of k-1 training sets and a testing set. For example, a 5-fold cross-validation will randomly split a
sample of 100 trials into groups of 20. Each of those groups will have an equal number of samples from each class.
5
In a given fold, four of the groups (total n = 80) will serve as training data for the classifier, and one group will
serve as new testing data. Cross-validation is typically repeated multiple times using randomly selected partitions of
the data.
2.1.3 Feedback
Classified output can vary in type, but in many BCI studies, it is presented visually on a computer screen and serves
as neurofeedback for the user [14]. For instance, a BCI may discriminate mental arithmetic from motor imagery,
and if it does so successfully, a checkmark may be displayed to the user. Output can also be continuous, in an effort
to provide real-time feedback to the user. Feedback may reinforce certain mental strategies that yield signals
conducive to class discrimination in machine learning, while deterring strategies that are not [9]. Feedback may
also help to keep users engaged and maintain concentration [29]. However, feedback is only possible in online BCI
paradigms, where already-trained classifiers are fed real-time data.
2.1.4 Types of BCIs
Active and reactive
Paradigms in BCI studies that use brain activity for communication fall under two broad categories: “reactive”
BCIs rely on involuntary (or modulated) brain activity elicited upon external stimulation (e.g. steady-state visually
evoked potentials - SSVEPs); “active” BCIs rely on brain activity elicited when the user voluntarily and
consciously performs a cognitive task (e.g. mental arithmetic, verbal fluency, motor imagery) [30]. In active BCIs,
different mental tasks may elicit different brain activities which in turn can be distinguished as unique
communicative intents – for instance, mental arithmetic may be classified as intent to move a cursor leftward on a
computer screen whereas verbal fluency may be classified as intent to move a cursor rightward. Reactive BCIs may
function by instructing the user to pay attention to a particular word played from a speaker, such as “Yes” or “No”,
and when the target word is heard, characteristic brain signals can be identified (e.g. such as the P300 response)
[31]. While active BCIs tend to require some training – such as the modulation of sensorimotor rhythms (SMR)
during imagined limb movements [2] – reactive BCIs need none. This is because reactive BCIs rely on evoked
potentials; involuntary changes in EEG signal amplitude resulting from presentation of stimuli in the environment.
Conversely, whereas most reactive BCIs are limited to synchronous control, active BCIs have the potential to be
used in an asynchronous paradigm, as discussed below.
Binary versus multiclass paradigms
Many BCIs to date have focused on using binary class paradigms. For instance, listing words starting with a
particular letter (verbal fluency task) may be classified against a mental rotation task to convey two different
commands (e.g. move cursor left/right). Increasing the number of classes may increase the rate at which
information can be conveyed by providing greater combinations of commands. However, increasing the number of
discriminable classes often means relying on seemingly unrelated mental tasks, both to one another, and to the
desired output. This may increase length of the training period for a user prior to which the BCI can become
6
effective. Furthermore, classification accuracy also tends to decrease with increasing output categories, thereby
limiting information transfer rates (ITRs). Ideally, a practical and useful BCI should have a high information
transfer rate (ITR) as well as a high classification accuracy.
Synchronous versus asynchronous paradigms
Synchronous control refers to BCI paradigms where mental tasks are constrained to specifically-timed cues,
whereas asynchronous control refers to BCIs where mental tasks can be initiated at any time by the user [32]. For
instance, in a synchronous BCI, a user may be cued to begin mental arithmetic, or cued to rest, and a classifier may
be able to correctly discriminate the two mental states that are restricted to specific time intervals. Conversely, an
asynchronous BCI can continuously detect which mental state the user is in and a change in mental state at any time
point should be detected by the classifier. The latter offers the advantage of executing user-paced commands.
2.2 Visual Processing Pathway
2.2.1 Regions of interest
The typical bottom-up pathway through which vision is processed begins at the retina. Changes in the visual field
cause depolarizations in photoreceptors, which send this information through the optic nerve to the lateral
geniculate nucleus (LGN) in the thalamus (which receives approximately 90% of the output) and the superior
colliculus in the midbrain (which receives 10% of the output) [33], [34]. The LGN then relays this activity to the
primary visual or “striate” cortex (V1 – Brodmann area 17). As the visual information continues up the hierarchy
from V1 to extrastriate areas V2, V3, V4, V5 and V6 (Brodmann areas 18 and 19), more and more complex
features are analyzed. For instance, V1 neurons respond to a particular orientation of a line, and neurons in V5
respond to motion [33], [34]. Importantly however, each visual cortical region possesses a topographic map of the
visual field (deemed “retinotopy”), such that stimuli in specific regions of the visual field will activate specific
neuronal populations in the visual cortex, and stimuli in adjacent regions will activate other unique neuronal
populations [33], [34]. The organization is most simple in V1, where the upper half of the visual field corresponds
to populations of neurons below the calcarine sulcus, and the lower half activates neurons above the calcarine
sulcus [33], [34]. In a similar manner, the left cortical hemisphere of V1 will depolarize in response to stimuli in the
right visual hemifield, and vice-versa. Interestingly, a bias is typically present in the lower hemifield that results in
better visual performance than in the upper hemifield [35]. Measurement of increased signal amplitudes from the
lower visual hemifield are owed in part to the proximity of lower field cortical sources to sensors on the scalp [36].
As with the increasing complexity of visual features, the retinotopic map of the visual field is altered higher up the
visual processing hierarchy, such that adjacent locations in the visual field may not correspond to adjacent locations
in the cortex [37]. Regardless, discrete locations of the visual field have unique physical representations in the
visual cortex that are detectable, and may be used in a BCI paradigm.
7
2.2.2 Perception, imagery, memory and attention
While there was a long-standing debate over whether only bottom-up processes such as perception can activate
lower visual areas such as V1, many studies have shown that top-down processes such as mental imagery or
memory also do the same (for review see [38]). It has been suggested that visual mental imagery functions as a
weaker form of perception [39]. In fact, an fMRI study by Slotnick and colleagues, discovered that mental imagery
of stimuli in various parts of the visual field activate regions of the visual cortex that are very similar to those of
perception [40]. Importantly, attention to the same locations of the visual field also activated similar areas of the
visual cortex, however, these were much less similar in pattern to perception than those elicited by imagery. Further
evidence comes from Ganis and colleagues, who also found similar cortical activation patterns between imagery
and perception of common objects [41], and Albers and colleagues, who demonstrated that a classifier trained on
perception of variously-oriented gratings could reliably identify orientations of gratings in visual imagery and
working memory in early visual cortical regions [42]. These findings go hand-in-hand with recent theories of
episodic memory, which state that frontal regions tend to activate memories, and subsequently elaborate them by
recruiting sensory areas [43]. Specifically, when asked to relive vivid autobiographical events, the visual cortices of
participants were found to activate, following top-down recruitment from frontal areas. In sum, top-down processes
such as imagery, memory and attention can reliably recruit early visual areas that typically respond to perception.
2.2.3 Visual tasks
Cognitive assessment
There are multiple standardized methods of determining visual cognitive abilities in individuals, two of which are
discussed here. The Rey-Osterrieth Complex Figure (ROCF) is a well-validated method of measuring visuospatial
and visual memory skills in participants [44]. This test relies on reproduction of a complex figure in three different
conditions: (1) Copy – where the participant is allowed to view the figure and is asked to reproduce it on paper to
the best of their ability, (2) Immediate Recall – where the participant is asked to reproduce the figure from memory
immediately after the Copy condition, (3) Delayed Recall – where the participant is asked to reproduce the figure
from memory after a longer delay of 30 minutes to 1 hour. The Vividness of Visual Imagery Questionnaire
(VVIQ2) is a subjective measure of visual imagery [45]. In this test, participants rate various scenarios on a 5-point
scale of how vividly they can see them in eye-open and eyes-closed conditions. The VVIQ2 has been revised and
validated [46]. Correlations between the VVIQ2 and activation in the visual cortex have been reported such that
greater reported vividness corresponded to greater activation in early visual cortical areas [47]. Interestingly, a
number of individuals found to have high visual imagery have also reported low auditory imagery, and vice-versa
[48], a pattern that is further discussed in Chapter 3.
In EEG and BCI studies
Thus far, few EEG-BCI studies have taken advantage of the topographical organization of the visual processing
stream. It is due to this organization that stimuli across the central visual field can be spatially and temporally
discriminated using EEG signals [49]. In fact, it has been determined that the spatial resolution of EEG is adequate
8
to discriminate between activations in the visual cortex corresponding to stimuli less than 3° apart in the visual field
[50]. This introduces the possibility of using stimuli in multiple areas of the visual field to signify multiple forms of
communicative intent. Despite this opportunity to create multi-class EEG-BCI paradigms, most studies have only
used stimuli in the left and right hemisphere [51], [52]. To our knowledge, no studies have attempted to
discriminate stimuli in all four quadrants of the visual field, although one study has successfully discriminated
perception in left and right quadrants in the upper and lower visual field with single-trial EEG [53]. In order to
reliably localize stimuli to their respective retinotopic locations, it has been found that confining them to the visual
field quadrants results in the most consistent results with fMRI data, at least when using MEG [54].
Importantly, eliciting brain activity that is measurable and differentiable with EEG does not require external visual
stimulus. This is due to the top-down modulation discussed above. It has previously been demonstrated that covert
visual attention to the left and right hemisphere can yield a classification accuracy of approximately 70% (the
standard minimum see [55]) in an online EEG-BCI system without exogenous stimulus [56]. Another study, while
not considering the topographical organization of the visual cortex, has found that discriminating brain activity
from visual mental imagery of faces and houses, and a resting state is possible with EEG. This approach has
resulted in above chance accuracies for binary (64-73%) and 3-class (54%) analyses, and information transfer rates
(ITRs) of 6 to 10 bits/min [57]. While faces and objects are processed by different extrastriate visual areas [58], it is
unlikely that the two will cause notably different activation patterns in earlier visual regions. No study has
attempted to discriminate between imagined stimuli in different areas of the visual field, despite the finding that
fMRI patterns of cortical activity elicited by imagining stimuli in various parts of the visual field are very similar to
the pattern of activity elicited by perceiving actual stimuli in the same area [40], [59]. Although this finding was
only evident in half (i.e. 3/6 of) their participants, it may have been in part due to individual differences in cognitive
ability to visualize stimuli. By taking advantage of visual mental imagery and topography of the visual cortex, it
may be possible to develop a multi-class non-invasive EEG-BCI that can discriminate between imagined stimuli in
visual field quadrants, at least in individuals with good top-down visual cognitive skills.
9
Chapter 3
Development and assessment of a novel visuospatial imagery-based EEG-BCI
3.1 Abstract
Brain computer interfaces (BCIs) can provide those living with severe motor disorders a means of communication
not dependent on motor actions or verbalization. By measuring brain activity elicited from different mental tasks,
communicative intent can be identified in these individuals. Brain activity during visual mental imagery has been
shown to be detectable and does not rely on potentially compromised neural pathways. This study aimed to
determine whether visuospatial imagery could be used to signify intent in both an offline and online
electroencephalography (EEG)-based BCI. Over the course of 3 sessions, 18 participants imagined checkerboard
arrow stimuli in the four quadrants of the visual field while having their brain activity recorded with 16 dry
electrodes over the occipital lobe. At the end of the third session, they performed an online task where they received
visual feedback on whether a BCI was able to detect any imagery or not. A subset of participants continued to the
4th and 5th sessions, where after brief offline retraining, they controlled movement of a character in an online
navigation task by imagining arrow stimuli in different quadrants. Predictors of BCI performance and characteristic
features of visuospatial imagery were further assessed through statistical means. Offline and online classification
accuracies of resting state against non-specific visuospatial imagery reached mean accuracies of 71.2% and 71.7%,
respectively. Mean online accuracies were further improved using more sophisticated signal processing and
features to 77%. When classifying diagonally-opposing quadrants, only six participants exceeded chance. These six
participants could be predicted using a linear regression model that combined scores from perception classification
tasks, and a measure of fatigue. The mean 4-class classification accuracy for Sessions 4 and 5 was 49.2%, with a
maximum of 85%. Finally, there was a significant relative increase in number of alpha spindles in visual cortical
regions contralateral to where visuospatial imagery was occurring. This study is the first to assess a BCI dependent
on visuospatial imagery. Furthermore, this is the first study to date to demonstrate posterior alpha band imbalance
with visuospatial imagery. While non-specific imagery and resting state may be used as practical commands in
binary BCIs, further improvement is necessary to increase the detection and discrimination of visuospatial imagery
in all participants. By improving signal quality, increasing the number of sensors and introducing alpha spindle
feedback in a training paradigm, visuospatial imagery may become a useful BCI control paradigm with potential for
numerous and intuitive commands for more users in need of BCI access technology.
3.2 Introduction
Many individuals with conditions such as stroke, cerebral palsy, spinal cord injury or amyotrophic lateral
sclerosis (ALS) live with motor impairments that prevent them from interacting or communicating with their
environment. Individuals who present as locked-in may be fully paralyzed, but can still be conscious of events
around them [1]. For these individuals, brain computer interfaces (BCIs) may restore useful function by providing
an alternative means of communication not dependent upon voluntary verbalization or motor activations [60]. By
10
detecting specific brain signals associated with different mental tasks, BCIs can identify and translate
communicative intent into commands such as moving a wheelchair, selecting options on a screen or enunciating
pre-selected words from a speaker. In other words, BCIs enable users to interact and communicate with their
environment through cognitive activity alone [61]. While BCIs possess great potential as assistive technologies,
the paradigms in which they have proven successful for non-disabled users have demonstrated limited translation to
target users.
3.2.1 BCI paradigms
There are a variety of BCI paradigms that each present with their own advantages and disadvantages for
target users. These can be broadly categorized into three groups: reactive, active or passive [30]. Briefly, reactive
BCIs rely on the detection of brain signals that result as a direct response of external auditory, tactile, or visual
stimulus presentation, and which are indirectly modulated by the user. Conversely, active BCIs rely on brain
signals that occur without stimulus presentation, but rather, are consciously and internally driven by the user.
Finally, passive BCIs do not require any specific mental task or stimulus response, but rather monitor the changes
in baseline brain activity over time – such as those resulting from fatigue – in order to enrich human-computer
interaction [30], [62].
One example of a reactive BCI is the well-established P300 BCI, in which individuals focus on detecting
the presentation of a specific stimulus in the environment, while ignoring irrelevant stimuli [23]. Upon detecting the
stimulus of interest, there is a characteristic change in signal amplitude that can be detected with
electroencephalography (EEG) approximately 200 to 700ms post-stimulus presentation [63]. Reactive BCIs such as
these have high accuracy rates and require little to no training [64]. However, they also require sustained attention
to external visual, tactile or auditory stimuli – which may prove difficult to individuals with motor impairments.
For instance, individuals with motor impairments may lack adequate gaze control in a visual P300 paradigm [6].
Additionally, it has been demonstrated that paradigms that require sustained attention can be fatiguing for the target
population [3]. Finally, because the P300 BCI is reactive, it can only be used in a synchronous paradigm. In other
words, the BCI constrains the user’s task performance to a specific time interval during which commands are
admissible, rather than affording user’s the freedom to generate commands at any point in time, as in asynchronous
BCIs.
Active BCIs rely on the detection of mental tasks that can themselves be categorized into two types of
cognition: executive function and sensory imagery. The tasks that rely mainly on executive function such as mental
arithmetic or verbal fluency (listing words that belong to a category) result in easily detectable and distinguishable
cortical signals. As a main source of this activity is located in the prefrontal cortex [65], the signals can be detected
through the forehead without introducing sources of noise contributed by hair follicles. Furthermore, since these
executive function tasks are independent of external stimulation, they may be used in asynchronous paradigms, and
can therefore facilitate user-paced control. However, the translation of these tasks into commands can be unintuitive
11
– for instance, listing types of fruit (as in the semantic fluency task) has no obvious relation to desired wheelchair
movement such as left or right.
Sensory imagery-based tasks, such as motor imagery (e.g. picturing movement of a limb) may be more
intuitive, and can translate into clearly related commands. For instance, imagining movement of a user’s right hand
may indicate a desire to move their wheelchair to the right. In fact, the motor imagery paradigm has been shown to
be less fatiguing and require minimal training for individuals with ALS [66]. Unfortunately, motor imagery-related
brain activations tend to be attenuated in those with spinal cord injury as well as in degenerative conditions such as
ALS [13], [67]. Thus, while this paradigm may be beneficial to some individuals, it may also be limited in others
due to the nature of their condition, which may compromise motor feedback, as well as the transmission and
processing of somatosensory and proprioceptive information [13].
An alternative active BCI sensory imagery task is covert mental speech, in which users say or repeat
words in their head (e.g. “yes”, “no”, “left”, “right”). Like its motor analog, this task has an intuitive appeal,
although it relies mainly on activations in the auditory cortex. Interestingly, not every individual is adept at verbal
imagery. In fact, it has previously been demonstrated that there is a dichotomy in cognitive thinking styles whereby
individuals are typically either verbal or visual thinkers. [68]. Importantly, these cognitive styles have
corresponding distinct anatomical activations [69]. This has considerable implications for BCI mental imagery
tasks; individuals who are visual thinkers may not engage auditory-processing cortical regions of the brain to the
same degree as verbal thinkers when performing auditory imagery. Likewise, verbal thinkers may minimally
engage visual cortical locations for visual imagery tasks. This is in line with findings demonstrating that, while top-
down processing for different types of mental imagery (and memory – see [43]) may recruit common fronto-
posterior networks, visual imagery deactivates auditory-imagery processing regions and vice-versa [70]. As a result
of high inter-subject variability in cortical activations, user-personalized BCI tasks are necessary in order to
maximize performance [71], [72].
One avenue that has not been explored extensively in BCI paradigms is visual mental imagery,
specifically, picturing stimuli in the visual field. Previously, most visual-based BCIs have focused on visuospatial
attention (i.e. shifting attention to a location of the visual field possessing a stimulus) in overt (i.e. with gaze shift)
and covert (i.e. without gaze shift) contexts [56], [73]–[79]. A major shortcoming of these studies is that even in
covert visuospatial attention tasks, where users fixate on a central cross, target locations for attention shifts require
some visual stimuli (e.g. a circle outline indicating where to focus attention). These approaches are not useful for
individuals with limited ocular motor control who cannot fixate on a single spot for covert paradigms or have
trouble shifting their gaze for overt paradigms. Although gaze dependence can be mitigated somewhat with stimuli
presented through closed eye-lids, such as in [78], such reactive paradigms only accommodate synchronous control.
Unlike attention paradigms, visual mental imagery would not require stimuli in the visual field to elicit a response.
However, at the time of writing, only one BCI study has leveraged true visual mental imagery, requiring individuals
to imagine faces or houses [57]. While imagery of faces and houses elicits differential responses in the medial and
lateral fusiform gyri [80], this imagery does not intuitively translate into a desired command (e.g. picturing a house
to indicate “yes”). Interestingly, the imagery of shapes within the visual field elicits retinotopic patterns of cortical
12
activations similar to those observed in the perception of shapes [40], [59]. In fact, imagery-related activation was
shown to be stronger and more similar to that accompanying perception than attention to those visual field areas.
Because unique locations of the visual field have corresponding physical cortical regions dedicated to their
processing in the visual cortex (deemed “retinotopy”), imagery in distinct areas of the visual field may elicit unique
activations that could map to BCI user commands. Importantly, this approach might offer the potential for multiple
BCI commands. For instance, imagery in each quadrant of the visual field could correspond to different directions
of desired motorized wheelchair movements (e.g. upper-right imagery moves the wheelchair forward and to the
right). Practical BCI use in the real world necessitates the development of a task sensitive to the needs of the user.
As visual mental imagery does not rely on vision, gaze and neuromotor control, it may form the basis of an active
asynchronous paradigm, and may provide a new avenue of BCI control for individuals who do not perform well
with other tasks (e.g. verbal imagery). This study aims to determine the potential of discriminating imagery in the
four quadrants of the visual field for control of an online EEG-BCI.
3.3 Methods
3.3.1 Participants
This study was approved by the research ethics boards at Holland Bloorview and the University of
Toronto. Twenty typically developed adults (10 female, 1 left-handed, mean age 27 ±4.15) were recruited for this
study from Holland Bloorview and the University of Toronto. Each participant provided informed written consent
prior to participating, and was compensated following each session. One participant dropped out of the study
following the second session for unreported reasons. Additionally, another participant was unable to complete
session 2 due to technical difficulties with the EEG system resulting in noise that could not be rectified. The data of
18 participants who had completed at least three sessions (8 female) are presented in the results section. Finally, 6
participants (0 female) met the criteria to continue on to Sessions 4 and 5. All six completed Sessions 4 and 5.
3.3.2 Instrumentation
EEG data were collected using the 16-electrode actiCAP Dry Xpress system with the V-amp amplifier
(Brain Products, Germany). Participants were fitted with medium, medium-large, or large EEG caps depending on
their head circumference. The caps were then sprayed with 70% ethyl-alcohol and were placed on participants such
that Cz sat precisely between the nasion and inion, and between each ear tip. The electrodes were located above and
around the occipital lobe (I1/I2, Oz, O1/O2, POz, PO1-PO4, PO7-PO10, P1/P2), with reference at FCz and ground
at AFz. The raw signal was digitally bandpass-filtered between 0.5 and 80 Hz using a 12 dB/oct Butterworth filter,
and notch filtered at 60 Hz in Brain Vision Recorder. All subsequent data preprocessing and analysis was
conducted using MATLAB 2017a and EEGLAB toolbox v 13.6.5b [81].
Electrooculography (EOG) signals were acquired with 6 wet stick-on medical grade electrodes placed
around participant eyes (one on each temple, one above each eye, one in the center of the forehead and one on the
right cheekbone), and connected to the V-amp using auxiliary ports. EEG and EOG signals were collected
simultaneously at a sampling frequency of 1 kHz, and displayed using Brain Vision Recorder. Because impedances
13
are not reported through the actiCAP dry system, each EEG electrode was physically adjusted to ensure good scalp
contact, while monitoring real time signals. EEG signals were considered adequate if amplitudes were under 100
µV [82]. If channels remained noisy despite physical adjustment (moving probes left, right, up, down; rotating
probes clockwise and counterclockwise; parting hair; roughening scalp), a small dab (< 0.1 ml) of Spectra 360 salt-
free electrode gel was placed on the tip of the noisy electrode probe. Each participant was seated comfortably
approximately 40 cm away from a 24 x 13.5” interface monitor with a 75 Hz refresh rate.
3.3.3 Experimental protocol
The format for the BCI component of the 5 data collection sessions is depicted in Table 1. Only
participants whose offline data collected from sessions 1-3 met specific criteria (outlined in later sections) returned
for sessions 4 and 5.
Table 1 BCI experimental protocol for each data collection session
Session 1
In the first session, participants completed the Rey-Osterrieth Complex Figure (ROCF) [83], [84] copy and
immediate recall tasks, followed by the Vividness of Visual Imagery Questionnaire 2 (VVIQ2) [85]. Subsequently,
participants had 5 different conditions to complete: Baseline, Attention, Auxiliary, Perception and Imagery.
Participants had control over when to begin each block. The first condition consisted of 30 seconds of Baseline
activity during which the participants were asked to sit at rest and fixate on a green fixation cross spanning 1° of the
visual field on a 100% grey background. Next, participants completed 20 trials of the Attention condition, during
which they were cued to each of the four visual field quadrants (UR – upper right; LR – lower right; LL – lower
left; UL – upper left) in random order, and were told to covertly shift their attention to the indicated quadrant
without shifting their gaze from the fixation cross until they were cued to rest (Figure 1).
14
Figure 1 Example Attention Condition trial (UL)
Following this, participants engaged in an eye-gaze control condition (Auxiliary condition), in which they
were cued to each of the four quadrants, and were told to overtly shift their gaze to the center of the arrow stimulus
when it appeared in that quadrant. After 1 s, the stimulus disappeared and the participants were asked to move their
gaze back to the fixation cross and were cued to rest for 1 s. During this condition, participants were also cued to
blink when the word “BLINK” appeared on the screen (Figure 2).
Figure 2 Example Auxiliary Condition trial (UR)
Following 20 trials of the Auxiliary condition, participants engaged in 10 blocks of alternating Perception
and Imagery conditions. In the Perception condition, participants were cued to each of the four quadrants in a
randomized order, and were asked to fixate their gaze on the cross while covertly shifting their attention to the
arrow stimulus that appeared in the cued quadrant, after which they were cued to rest (Figure 3). This was done for
8 trials (two trials in each quadrant), after which the participant engaged in the imagery task.
Figure 3 Example Perception condition trial (UL)
As with the Attention condition, in the Imagery condition, participants were cued to each quadrant and
were told not to shift their gaze from the fixation cross in this condition. However, participants were asked to
imagine the arrow stimulus they had previously seen in the quadrant, rather than simply attend to the quadrant
(Figure 4). After each Imagery trial, participants were cued to rest. Each Imagery block ended after 20 trials.
15
Figure 4 Example Imagery Condition trial (LR)
After 10 blocks of alternating Perception and Imagery, the participant completed the ROCF delayed recall
task and filled out a user-satisfaction questionnaire in which they rated the paradigm on 5-point scales for measures
including pre- and post-session fatigue, difficulty and frustration (see Appendices B and C).
Session 2
In the second session, participants completed the same tasks as in the first session, with the exception of
the ROCF, the VVIQ2 and the Auxiliary Condition (see Table 1).
Session 3
The third session consisted of Baseline, followed by 10 alternating blocks of Perception and Imagery.
Subsequently, all the offline data collected for that participant were used to train a BCI classifier (detailed below),
during which the participant was briefed on the Feedback condition. Once the classifier was trained, each
participant engaged in 2 blocks of the Feedback condition, in which they were cued to picture the arrow stimulus in
each of the four quadrants, as in the Imagery condition. This trial was then immediately classified as Rest or
Imagery, and participants were presented with 3 s of feedback. If the BCI detected Imagery with greater than 0.55
probability, participants were presented with an opaque flickering arrow. If the BCI detected Imagery with between
0.45 and 0.55 probability, participants were presented with a translucent flickering arrow. Finally, if the BCI could
not identify imagery (i.e. <0.45 probability), participants were presented with the outline of the arrow (Figure 5).
Feedback was provided to reinforce mental strategies that yielded desirable signals, and deter strategies that did not
[9].
16
Figure 5 Example Visual Feedback Condition trial (UL), with annotated probabilities (not presented to participants)
Sessions 4 and 5
The fourth and fifth sessions consisted of Baseline data collection followed by 4 alternating blocks of
Perception and Imagery (see Table 1). Subsequently, the BCI was re-trained on all the offline data from each
participant. Then each participant engaged in two blocks of a Navigation Game, in which they controlled
movement of a porcine character. The character always remained in the center of the screen under a fixation cross
and while the participant pictured arrow stimuli in quadrants where they want the character to move (Figure 6). The
environment moved around the porcine character so as to maintain its central position. The aim of the navigation
game was to guide the character to as many coins as possible in 20 moves. Coins were located at random street
junctions and stop signs (through which the character could not pass) were located in random streets. Importantly,
two measures were enacted in order to reduce the likelihood of afterimage. First, saturation and contrast were
minimized while maintaining the same 50% black 50% white background as the other tasks. Second, the
participants were told to freely explore the scene with their gaze prior to initiating the trial, as saccades have been
demonstrated to reduce the effects of afterimage [86]. In order to track intended movement, participants were asked
to press one of four buttons on the number pad once they decided on a direction. Each button corresponded to one
of the four possible directions (9 for UR, 3 for LR, 1 for LL and 7 for UL). The participants were then asked to
return their gaze to the fixation cross and hit the spacebar to initiate a trial. Subsequent to this, they were presented
with cues designating where they can perform imagery. First, black ∟-shaped cues for the UL and LR quadrants
appeared just outside the fixation cross for 500ms. Once the cues disappeared, participants rested if their intended
direction was not cued, or performed imagery in either cued quadrant. After 5.5 s, they were presented with cues for
the remaining two quadrants (LL or UR), in which they could again perform imagery in the desired quadrant or
rest. After the trial, the screen returned to the view of the game. Once the BCI classified one of the four directions
(1-2 s delay), the character moved towards the detected quadrant (independent of which button the participant
pressed).
17
Figure 6 Example Navigation Game trial (UL) with exaggerated fixation crosses
3.3.4 The visual imagery stimulus
The arrow stimulus presented to the participants, and the same asked to be imagined by them, was
designed with a number of considerations in mind. First, the shape was selected so as to be similar to the
checkerboard annulus wedge in [40], but adjusted to be more task-relevant and intuitive as a command in the
navigation game condition. Specifically, the direction in which each arrow pointed corresponded to the quadrant in
which it appeared (e.g. the UR-pointing arrow was found in the UR quadrant) as well as to the desired navigational
direction in the game (e.g. UR arrow means move the character up and to the right). Further, it has previously been
discovered that the use of more naturalistic-shaped stimuli results in greater visual discriminability than simple
square checkerboard stimuli [87]. Presumably, arrow shapes are more meaningful than annulus shapes such as the
ones used in [40], and may serve to increase differentiable cortical activation. The arrow shape also results in a
greater discrimination of shape than the annulus wedge when compared across different quadrants. It is expected
that the increased shape discriminability may help distinguish neural activity resulting from different classes of
imagery. Second, each check was adjusted to account for cortical magnification; the decrease of cortical area that
represents the visual field with increasing degrees of eccentricity [88]. As in Slotnick and Yantis’ study, a high-
contrast checkerboard pattern was used in order to elicit a maximum visual cortical response [89]. However, while
the authors used a black and white pattern reversing frequency of 8.3 Hz in order to efficiently elicits retinotopic
activation of the primary visual cortex [89], this has been shown to fall in the range of seizure-inducing stimuli
[37]. As such, the arrow stimulus frequency was reduced to 2 Hz, well below the safety threshold of 3 Hz. In an
effort to reduce the likelihood of after-image during Rest trials, which would occur with constant pattern-reversing
retinal stimulation, the stimulus flashed on and off. In other words, the arrow stimulus would appear then disappear
at a rate of 2 Hz, which is within the range of saccade frequencies known to prevent afterimages [90]. Lastly, while
many of the aforementioned studies only focused on using stimuli in the central field of vision (i.e. 1-14°), it was
18
found that in contrast to visual attention, participants performed better on visual mental imagery when stimuli are
large [91]. As a result, the length of the arrow stimulus was expanded to 18° of the visual field.
3.3.5 BCI pipeline
To provide viable feedback to participants, a preliminary BCI pipeline was designed based on data from
Sessions 1 and 2, which was used in the online neurofeedback condition in Session 3. This is detailed below,
followed by specifications of a more advanced BCI (henceforth termed “sophisticated BCI”) that was designed
using data from all 3 Sessions for the offline analysis and navigation game. Briefly, the more sophisticated BCI
involves noise attenuation, is less dependent on low-frequency (e.g. delta frequency band) features that may have
been more contaminated by artefact, and is more dependent on relevant frequency bands (e.g. alpha; 7-13 Hz).
3.3.6 Online
Signal preprocessing
Trial epochs were extracted (6 s duration from cue onset), resampled at 256 Hz, and bandpass-filtered
using a 3rd order Butterworth filter between 1-40 Hz, with stopbands at 0.5 and 45Hz. EOG activity (such as
blinking) was suppressed from the EEG data using a 3rd order conventional recursive least squares (CRLS)
regression algorithm [22] with a forgetting factor of 0.9999 and initial filter state of 0.01. To increase SNR, the
following neighboring channels were averaged to create 6 additional virtual channels: I1/PO9, O1/PO7, PO1/PO3,
I2/PO10, O2/PO8, PO2/PO4. Lastly, epochs were trimmed to remove the first and last 500ms. This removed any
distortion introduced by filtering, as well as potential reactive responses from visual cues [77].
Feature engineering
The logarithms of power spectral densities (PSDs) were computed for each channel and trial using a fast
Fourier transform with a 90% overlapping 256ms sliding window. From the log PSDs, the total delta (1-3 Hz),
theta (low: 3-5 Hz, high: 5-7 Hz), alpha (low: 7-9Hz, mid: 9-11 Hz, high: 11-13Hz and 13-15Hz, total: 7-15 Hz)
and beta (low: 13-15Hz, mid: 15-17Hz and 20-24 Hz, high: 24-27 Hz) band powers were extracted. Next, the
relative power, peak frequency, relative peak frequency, and PSD fractal exponent (the linear slope of the PSD) of
each band were computed. As discussed in more detail below, the alpha frequency band has been demonstrated to
carry useful information in CVSA paradigms [92]. As such, the ratio of peak alpha power to the power from 2 Hz
prior, and the ratios of the total alpha power to the power in each beta sub-band were computed. Additionally, as
imbalances in frequency power were expected during imagery tasks, the magnitude-squared coherence for alpha
(mid: 9-12 Hz), delta (1-4 Hz) and beta (mid: 23-26 Hz) were computed for adjacent and opposing channel pairs.
Finally, to capture potentially relevant time-domain features, spectrograms were computed applying a short-time
19
Fourier transform (STFT) on 115ms segments (110ms overlap) multiplied with a Hamming window, for each trial
and channel. Then the mean, minimum and maximum for each 256ms interval in alpha (mid: 9-12 Hz) and delta (1-
4 Hz) frequency bands were calculated.
Classification
3.3.6.3.1 Feature selection
Features were selected through elastic net regularization [93]. The alpha elastic-net mixing parameter was
tuned between 0.5 (elastic-net) and 1 (lasso) with step sizes of 0.05 over 10-fold cross-validation runs. Increasing
alpha in this manner limited the number of selected features while minimizing error. The model possessing the
lowest lambda values and the smallest deviance across all alpha values was selected. Importantly, this model also
contained the beta coefficient for each selected feature, corresponding to its individual contribution to the model.
3.3.6.3.2 Classifier
To classify non-specific imagery versus rest in the Feedback condition, a logistic regression classifier with
the model parameters determined above, was trained using all 600 trials of rest and imagery from the Imagery
conditions of Sessions 1-3. The trials from the online Feedback condition were then preprocessed as above and
classified in real time. Posterior probability scores were then used to provide the feedback described in the
experimental protocol.
3.3.7 Offline analysis and online navigation game
While designed to classify tasks of visuospatial imagery, the BCI discussed here was also used to analyze and
classify data from the visuospatial perception tasks.
Signal preprocessing
Trial epochs were extracted, resampled at 256 Hz, and bandpass filtered using a 3rd order Butterworth filter
between 1-40 Hz, with stopbands at 0.5 and 45Hz. EOG activity such as blinking was suppressed from the EEG
data using CRLS. Epochs of transient noise that resulted from temporarily increased impedances (e.g. through loss
of electrode connection) in the EEG system were identified under the assumption that EEG signal amplitudes
recorded from the scalp are below 100µV [82]. As such, noisy (i.e. “bad”) channels within each trial were flagged
if they surpassed ±50µV. Next, the bad channels were attenuated by scaling the amplitudes to the standard
deviation of “good” channels in that trial, while maintaining their means. If all channels were labelled as exceeding
either specified threshold, the signals were attenuated to have a range of ±50µV around their mean. Attenuation of
20
this dry EEG noise was necessary as the spatial filter would amplify these sources of noise, and spread them to
adjacent channels. The number of flagged noise epochs were recorded.
As different types of imagery were expected to result in unique localized changes in activity, a Surface
Laplacian/Current Source Density (CSD) spatial filter was applied using the CSD toolbox [94]. This spatial filter
suppresses spatially broad activity shared between electrodes that is a result of volume conduction across the scalp,
and amplifies localized activity [95]. As a result, CSD was applied when classifying imagery in diagonally-
opposing quadrants. However, the spatial filter was not applied when classifying non-specific visuospatial imagery
against rest since the imagery class was heterogeneous, with differing locations of activity. Instead, it was expected
that the imagery versus rest classifier would rely on activity that was common to visuospatial imagery in all
quadrants. As optimal parameter settings had not previously been established for a visual imagery paradigm with a
low-density, low-number dry electrode array, a variety of CSD parameter combinations were tested across wider
ranges of values than previously established [96]–[99], as recommended in [100]. Specifically, the lambda
smoothing factor (lam) was searched for across values of 10-4.25 to 10-6 with 10-0.25 step sizes; the Legendre
polynomial order (Pn) was searched over integers of 3 to 12 and spline flexibility (m) was searched for over
integers of 2 to 8 for each participant and each imagery classification problem. Additionally, CSD estimates were
scaled according to a realistic head size (radius of 9 cm). The parameter combination resulting in the best
classification accuracies was used for final online classification. Epochs were trimmed to remove the first and last
500ms in order to remove filter distortion and influencing reactive signals from visual cue perception.
Feature engineering
Features that were extracted for final offline analysis and the navigation game are detailed in the following
sections.
3.3.7.2.1 Baseline alpha
It was previously demonstrated that classification accuracies in a CVSA paradigm could be predicted using a
measure of baseline alpha [77]. Thus, the band power of alpha frequency (7-13 Hz) was summed for each channel
over the 30 s of baseline activity, then averaged across sessions. The channel with the maximum alpha power was
selected for further statistical analysis.
3.3.7.2.2 Alpha spindles
Posterior alpha activity located over the visual cortex is closely linked to attention, where increased alpha
from resting state indicates an inhibition of irrelevant stimuli [101]–[103]. Importantly, in visuospatial attention
paradigms, alpha power increases over regions dedicated to processing irrelevant visual field locations, and
decreases over regions dedicated to processing the attended visual field locations [56], [77], [92], [104]–[107].
Another method to characterize alpha frequency fluctuations detects short-term bursts of activity – deemed “alpha
21
spindles” [108]. Importantly, alpha-spindle detection is less sensitive to noise [108], and alpha-spindle rate has been
shown to increase during suppression of visual processing, and decrease during activation of visual processing in an
attention paradigm [101]. As this approach may characterize visual imagery in a manner that is less affected by
noise than simple band power analysis, alpha spindle features were extracted based on the methods outlined in
Simon et al. [108]. The approach was slightly modified to fit this visuospatial imagery paradigm, and each step is
briefly outlined below.
First, the spectrogram for each trial and channel was computed by applying STFT on 128ms segments
(100ms overlap) with a Hamming window. Next, each 28ms segment was tested for the presence of an alpha
spindle according to the conditions outlined in Simon et al. (2011). Briefly, a segment was considered to possess an
alpha spindle if:
i) the maximum activity between 1-40 Hz occurred within the specified alpha range (e.g. 7-13 Hz);
ii) the full width at half maximum (FWHM) was no greater than twice the bandwidth of the Hamming
window; and
iii) the area under the peak (bounded by the FWHM) of the amplitude spectral density was at least twice as
large as the area under the 1/f frequency noise curve (bounded by the FWHM).
The 1/f frequency noise curve was computed by fitting the mean amplitude spectrum of the segment to an
exponential curve, which was then multiplied by the ratio of the total segment power over the total trial power.
Finally, the total number of alpha spindles per trial per channel was calculated. As it has been found that alpha sub-
bands provide more useful information for discrimination of visuospatial attention than that of the entire band
[107], this entire process was repeated with condition i) being met if the peak frequency fell within the following
alpha sub-bands: 7-9 Hz, 8-10 Hz, 9-11 Hz, 10-12 Hz, 11-13 Hz. Additionally, as alpha power is largely variable
over time, the absolute number of alpha spindle segments per trial was considered inadequate. Thus, the relative
number of alpha spindles per trial (i.e. alpha spindle lateralization) was computed by subtracting the number of
spindles in each left hemisphere and central channel from each right hemisphere and central channel (see [92]).
This was done in addition to summing alpha spindles of left and right hemisphere channels, which was aimed to
distinguish activity that was common to both hemispheres from activity that was common to one.
3.3.7.2.3 Continuous wavelet transform
Both time and frequency domain signal characteristics were found to be important in studies of CVSA, as
discriminant patterns of alpha power change over short time periods [107]. One approach to time-frequency
analysis is the spectrogram, although it suffers from trade-offs between frequency and time resolution. Scalograms
avoid this issue by decomposing the signal using mother wavelets into additive subcomponents, which provide a
measure related to the contribution of different frequencies (pseudo-frequencies) to the original signal. While this
approach assumes a stationary signal, it also offers greater time-frequency resolution than that of spectrograms. As
a result, each trial was transformed into a scalogram using the Morlet wavelet, which was then normalized by
dividing each value by the mean of the entire scalogram. Next, we isolated alpha pseudo-frequency sub-bands (α1,
22
7.5-9.5 Hz; α2, 8.5-10.5 Hz; α3, 9.5-11.5 Hz; α4, 10.5-12.5 Hz; α5, 11.5-13.5 Hz) in addition to the entire alpha
pseudo-frequency band (7.5 – 13.5 Hz), and found relative scalograms by adding and subtracting left channels from
right channels, as with alpha spindles. The total coefficients of each relative scalograms were summed and
extracted as features. Finally, while a previous study found that it was beneficial to classify left versus right
visuospatial attention using specific time-windows [107], it is unlikely that the evolution of the alpha frequency
during visual imagery is consistent over every trial, as well as across sessions. However, extreme scalogram values
may identify key changes in relative neural activity that can discriminate different types of imagery. Thus, upper-
and lower-thresholds (± 1 standard deviation around the mean) of each scalogram were used to divide its values
into high, low or medium subgroups. Finally, the mean of each of these groups was extracted as a feature. In
addition to the lateralized scalograms derived from subtraction of channel pairs, the above-mentioned features were
extracted from the scalograms from each channel.
3.3.7.2.4 Entropies
Another feature that has previously been shown to be useful in visual attention classification paradigms, is
entropy - a measure of disorder of a signal. In fact, these studies suggested that approximate and sample entropies
would be higher in attention conditions as compared to resting tasks [109], [110]. Thus, the approximate and
sample entropies for the alpha band (7.5-13.5 Hz) and sub-bands (α1, α2, α3, α4, α5) were computed for each
channel and trial with a tolerance of 0.2 times the standard deviation of the signals, and an embedded dimension of
2 as in [109].
3.3.7.2.5 Magnitude-squared coherence/cross-power spectral density (CPSD)
As frequency imbalances between left and right- hemisphere are expected over the occipital region in
alpha frequencies during tasks of attention, the mean magnitude-squared coherence measure was calculated for the
alpha band and each alpha sub-band (α1, α2, α3, α4, α5) for each trial and channel. Likewise, the mean magnitude
of CPSD, in addition to the mean phase lag between left and right electrodes was computed for alpha (7.5-13.5 Hz),
theta (4-7 Hz) and beta (13-30 Hz) bands.
Classification
3.3.7.3.1 Feature selection
Features were selected through the elastic net regularization method over 5-fold cross-validation.
However, in the offline and navigational game BCI, the alpha parameter was tuned between 0.75 and 1 with step
sizes of 0.05. This was done to further minimize the total number of selected features to a greater degree. In this
23
manner, non-logistic regression classifiers that did not use beta coefficients (i.e. weights of features) could be
trained, with the reasonable assumption that most, if not all selected features would contribute evenly to the model.
3.3.7.3.2 Binary classifiers
Three separate binary classification problems were considered for this thesis. First, as abovementioned, it
was of interest to discriminate any type of imagery in the visual field against a resting state. Next, it has been
demonstrated that attention in one location of the visual field will result in a decrease in relative alpha power in the
cortical region processing that location, and an increase in alpha power in the opposite cortical region. As a result,
the most distinguishable signals are expected to be UL and LR imagery, or LL and UR imagery (see Treder et al.
2011). Thus, the classification problems regarding different types of imagery aimed to discriminate UL versus LR
imagery, and LL versus UR imagery. As it was unknown which classifier type would result in the best accuracies,
the following were trained and tested for each classification problem through 10-run, 5-fold cross validation with
equal representations of samples from each session: K-nearest neighbours (KNN), linear and radial-basis function
(RBF) support-vector machines (SVMs) (from the LIBSVM toolbox [111]), and logistic regression.
3.3.7.3.3 Two-tier multiclass classifier
In order to maximize discrimination between the four classes of imagery, a number of characteristics of
the data were taken advantage of. First, as the maximally discriminant imagery quadrant pairs are UL/LR and
LL/UR, cues presented to the participant in the navigation game were only for diagonally opposing quadrants.
Next, during rest, alpha power was expected to be more evenly distributed across the visual cortex, so the relative
alpha power features were expected to fall between UL and LR imagery, or LL and UR imagery. If this data was
tested using a classifier trained on UL/LR or LL/UR imagery, the probability that it belonged to either class was
expected to be close to 0.5. Thus, each subtrial in the online Navigation Game was tested with a binary imagery
(UL vs. LR or LL vs. UR) classifier (Tier #1), with the expectation that participant would be resting in one of the
two subtrials (Figure 7). Importantly, each subtrial was also tested with a Rest vs. Imagery classifier (Tier #1),
which was expected to help classification as one subtrial was necessarily imagery and the other, rest. The
probabilities returned from these four classifiers were treated as features for a 4-class classifier (Tier #2). In order to
train this classifier, probabilities were obtained by testing the trained binary classifiers on all the available offline
data. All binary problems in this approach used logistic regression, as the relative feature contributions (i.e. beta
coefficients) could be used, whereas the multiclass problem relied on an RBF-SVM classifier (as recommended in
[112]). This two-tier classification approach was compared to a direct multi-category (i.e. 4-class) classification of
imagery in each quadrant using RBF-SVM.
24
Figure 7 Schematic of Navigation Game trial classification with two tiers of classifiers
3.4 Results
3.4.1 Chance levels
All chance levels discussed below were computed using the binomial distribution thresholds based on the number
of samples [113]. When accuracies are said to exceed chance, this refers to their means minus their standard
deviations being above chance.
3.4.2 Classifying perception and imagery against resting mental state
Perception
3.4.2.1.1 Offline
The sophisticated BCI was able to discriminate resting mental state from perception in any visual field quadrant
with above-chance (55.4%) accuracies in all participants save one (Figure 8). Additionally, 14 participants met the
70% threshold for a practical BCI. The mean classification accuracy across all participants was 75.7±8.5%. The
best classifier in terms of accuracy was participant-specific. Logistic regression was the most frequently selected
optimal classifier (7 participants).
25
Figure 8 Classification accuracies for perception versus rest with user-optimized classifier
Imagery
3.4.2.2.1 Offline
When discriminating resting mental state against visuospatial imagery in any quadrant, the BCI was able to achieve
above-chance (53.3%) accuracies in 16 of 18 participants (Figure 9). Additionally, data from 7 participants met the
70% threshold for a practical BCI. The mean overall classification accuracy across all participants was 71.2±11.3%.
The most frequently selected optimal classifier was Linear SVM (7 participants).
Figure 9 Offline classification accuracies of rest versus any imagery with user-optimized classifiers
3.4.2.2.2 Online
Non-specific imagery versus resting state classification scores from online trials are depicted in Figure 10. The
preliminary BCI (with a logistic regression classifier) used in the online trials resulted in a mean accuracy of
71.7±12.3%, with 12 participants meeting the 70% practical BCI threshold. When the trials were re-evaluated using
the sophisticated BCI approach, a mean accuracy of 75.1 ±12.0% was achieved with logistic regression, and
26
77.0±11.4% with the best-performing classifiers specific to each user. These resulted in 12 and 13 participants
meeting the 70% threshold, respectively. When comparing the preliminary BCI to the sophisticated BCI with
optimized participant-specific classifiers, accuracies improved for 14 participants, and decreased marginally for 3
participants (<5%) and moderately for one (>5%). Non-parametric analysis by the Wilcoxon signed rank test
indicated that the sophisticated BCI with user-optimized classifiers significantly increased classification accuracies
by an average of 5.3% when compared to the preliminary BCI (p =0.0107). The most frequently selected optimal
classifier was Logistic Regression (7 participants).
Figure 10 Classification of online trials
Features characteristic of perception and imagery versus rest
3.4.2.3.1 Perception
The average number of selected features when classifying non-specific perception against rest was 29±15. A
normalized frequency heat map sorted from best-to-worst classification accuracies is depicted in Figure 11. As
summarized in Figure 12, the groups from which features were most commonly selected were: channel pair sums of
scalograms in the alpha (18.9%) and beta (13.4%) frequencies, magnitude squared coherence of the alpha band
(15.7%) and phase lag of alpha coherence (11.3%). When adjusted for the number of sub-features within each
feature group, the most commonly selected groups were: magnitude squared coherence of the alpha band (2.6%),
lag of alpha coherence (2.3%) and channel pair sum of alpha spindles (1.8%) (Figure 12).
3.4.2.3.2 Imagery
The average number of features selected by elastic-net regularization across participants was 39.5±25.3. The
frequency of selected feature groups across all participants (sorted from best to worst classification accuracies) is
depicted in Figure 11. The most frequently selected groups were channel pair sums of scalograms in the alpha
(14.1%) and beta (14.6%) frequencies, magnitude squared coherence of the alpha band (13.5%) and lag of alpha
27
coherence (13.0%) (Figure 12). When adjusted for the number of sub-features within each feature group, the most
commonly selected groups were magnitude squared coherence of the alpha band (2.2%), lag of alpha coherence
(2.6%) and magnitude of cross power spectral density in the theta (2.2%) and beta (1.7%) frequencies (Figure 12).
Figure 11 Frequency of feature group selection by elastic net regularization for perception (left) and imagery (right) vs. rest
classification
Figure 12 Frequency of feature group selection by elastic net regularization for perception compared to imagery (left) and perception
compared to imagery with selection adjusted for number of sub-features (right)
3.4.3 Classifying quadrants in perception and imagery
Perception
Prior to classifying diagonally-opposing quadrants (UL vs. LR, LL vs. UR), spline flexibility (m), smoothing factor
(lam) and Legendre Polynomial order (Pn) were varied in the CSD spatial filter to determine the optimal settings
for each participant, within each classification problem. On average, the user-optimized CSD spatial filters were
found to improve UL vs. LR accuracies by 13.8±8.4% (p<0.001) and LL vs. UR accuracies by 13.0±7.2%
(p<0.001) (not depicted). The pattern of optimal CSD parameters across both classification problems was not
consistent across participants (Figure 13), however the most commonly selected parameters were as follows: spline
28
flexibilities of 2 (8 times) and 3 (8 times), smoothing factors of 104.25(11 times) and 10-4.5 (7 times), and Legendre
polynomial orders of 7 (7 times) and 4 (5 times).
Figure 13 Frequency of optimal CSD spatial filter parameters: spline flexibility (left); smoothing factor (middle); Legendre polynomial
order (right)
The classification accuracies after implementation of user-optimized spatial filter parameters can be seen in Figure
14. The average classification accuracy across participants was 77.3±9.3% for UL vs. LR and 77.2±9.2% for LL vs.
UR. In total, 9 participants exceeded chance (60%) in both comparisons, 6 exceeded chance in at least one
comparison, and 3 did not exceed chance in either. The most frequently selected optimal classifiers were Linear
SVM for UL vs. LR (7 participants) and Logistic Regression for LL vs. UR (8 participants).
Figure 14 Classification accuracies of perceiving stimuli in UL vs. LR and LL vs. UR quadrants with optimized spatial filter parameters
Imagery
As with classifying perception in diagonally-opposing quadrants, CSD parameters were optimized for classifying
visuospatial imagery in opposing quadrants (UL vs. LR and LL vs. UR). The use of spatial filters optimized to each
individual improved accuracies by 8.1±7.1% for UL vs. LR (p=0.001) and 9.2±4.4% for LL vs. UR (p<0.001)
classification problems, when compared to classifications without spatial filters. The frequency of CSD parameters
resulting in optimal classification accuracies across participants in both UL vs. LR and LL vs. UR are presented in
Figure 15. The most frequently selected spline flexibility parameters were 2(10) and 3(11). The most frequently
29
selected spline flexibility parameters were 2 (10 times) and 3 (11 times). The most frequently selected smoothing
factors were 10-6 (4 times), 10-5 (3 times) and 10-4.25 (3 times). Finally, the most frequently selected Legendre
polynomial orders were 3 (7 times), 7 (6 times) and 9 (5 times).
Figure 15 Frequency of optimal CSD parameters: spline flexibility (left), smoothing facto exponent (middle), Legendre polynomial
(right)
3.4.3.2.1 Offline
The accuracies for UL vs. LR and LL vs. UR following implementation of user-optimized spatial filters and
classifiers are presented in Figure 16. The mean classification accuracies across all participants was 65.2±7.5% and
65.0±7.2% for UL vs. LR and LL vs. UR problems, respectively. In total, 6 participants achieved above-chance
(56.7%) accuracies in both classification problems, 3 exceeded chance in only one classification problem, and 9 did
not exceed chance in either. Two participants met the 70% threshold in both classification problems, with an
additional 2 meeting the 70% threshold in one of the two problems. The most frequently selected optimal classifier
was the RBF SVM for both UL vs. LR (8 participants) and LL vs. UR (9 participants).
Figure 16 Imagery versus imagery classification accuracies with optimized spatial filter parameters and classifiers
30
3.4.4 Features characteristic of visuospatial perception and imagery
Perception
The average number of selected features across participants was 6.8±5.1 for UL vs. LR and 9.4 ±8.1 for LL vs. UR.
The most selected feature groups are presented in Figure 17. The most commonly selected groups in both
classification problems were alpha spindle difference (UL vs. LR -36.0%; LL vs. UR - 41.8%) and alpha coherence
phase lag (UL vs. LR - 33.8%; LL vs. UR 28.11%) (Figure 18). Each group contained the same number of sub-
features, and as a result, adjusting for the number of sub-features was unnecessary.
Figure 17 Frequency of selected feature groups in classifying different types of perception sorted by participant
Figure 18 Grand average frequency of selected feature groups in classifying perception in diagonally-opposing quadrants
Imagery
The average number of selected features across participants was 11.2±11.0 for UL vs. LR and 9.8±9.0 for LL vs.
UR. The patterns of most selected feature groups for each classification problem can be seen in Figure 19. The
groups selected with the greatest frequency were the channel pair difference in alpha spindles (UL vs. LR -35.0%;
LL vs. UR -40.5%) and phase lag in the alpha band (UL vs. LR -31.3%; LL vs. UR -30.8%) (Figure 20).
31
Figure 19 Frequency of selected feature groups in classifying different types of imagery sorted by participant
Figure 20 Grand average frequency of selected feature groups in classifying imagery in diagonally-opposing quadrants
3.4.5 Multiclassification of visuospatial imagery
Participants
The criteria for participants to continue to Sessions 4 and 5 were that their offline binary visuospatial classification
accuracies for both UL vs. LR and LL vs. LR be above chance. In other words, this required the mean of their
accuracies minus the standard deviation to be greater than the binary distribution threshold for chance (56.7%). As
a result, only 6 participants continued to Sessions 4 and 5.
Offline and online
The offline classification accuracies, and mean online classification accuracies over Sessions 4 and 5 are presented
in Figure 21. Direct classification of imagery in the four visual field quadrants (UL vs. LL vs. LR vs. UR) resulted
in a mean of 38.4±13.0% (max 63.5%), with 3 participants exceeding chance. This accuracy was significantly
increased to a mean of 57±14.6% (p= 0.0313), and a maximum of 85% when rest was used to further divide the
32
four classes using the two-tier classification scheme. This approach also allowed all 6 participants to exceed chance
levels. The mean online scores using the two-tier approach were 49.2±18.1%. The online scores for four of the
participants fell within one standard deviation of the two-tier offline accuracy means, and the remaining two fell
below. A confusion matrix for all the data across all participants is presented in Table 2. Finally, only one
participant exceeded the practical BCI threshold of 70% in both the two-tier offline and online paradigms.
Figure 21 Four-class classification accuracies in offline (direct and two-tier) and online approaches
Table 2 Confusion matrix for all participant data from sessions 4 and 5
3.4.6 Predicting classification accuracies
Correlates of classification accuracies
Measures that were obtained external to imagery mental tasks (VVIQ2 and ROCF scores, pre-session fatigue
ratings, baseline alpha, number of noisy epochs and perception task classification accuracies) in addition to those
obtained internally from mental task trials (number of noisy epochs, number of selected features and number of
different selected feature groups) were correlated using Spearman rank to imagery versus rest classification
accuracy (Table 3), as well as to the average of UL/LR and LL/UR imagery classification accuracy (Table 4).
When assessing correlates of non-specific imagery versus rest accuracy, only the number of selected features
resulted in a significant relationship (p = 3.05x10-6, rho = 0.87) (Table 3). Similarly, the number of selected features
was found to positively correlate to imagery versus imagery accuracies (p = 0.003, rho = 0.66) (Table 4). Of the
measures extracted external to the imagery trials, pre-session fatigue and perception versus perception accuracies,
33
using optimized spatial filter parameters for both perception and imagery paradigms, were found to correlate to
imagery versus imagery classification accuracies (Table 4). However, following adjustment for multiple
comparisons, only the perception accuracies using imagery-optimized spatial filter parameters correlated to imagery
versus imagery accuracy.
Table 3 Spearman correlation table for imagery versus rest classification accuracy
Table 4 Spearman correlation table for imagery versus imagery
Regression model
As it was anticipated that a combination of factors might better predict imagery accuracies, the factors derived
external to imagery trials that were significant prior to adjustment were used as predictors in a stepwise linear
regression model. The model was then simplified to reduce the overall number of terms used, while maintaining a
high R2-adjusted value. As there were only 18 samples, 1.8 features would be ideal to use according to the 10
samples-per feature rule-of-thumb for classification– however this is a very strict threshold in linear regression,
where 2 samples per variable have been demonstrated to be adequate [114]. In attempting to achieve the simplest
model, the less forgiving threshold was followed. This reduced the number of terms in the model were to two – an
interaction term between perception classification accuracies (x1; with imagery-optimized CSD parameters) and
pre-session fatigue (x2), and a quadratic term for pre-session fatigue. This stepwise linear regression approach
34
resulted in a model with an R2-adjusted value of 83.4% (p-value = 0.00000985, RMSE = 2.93, error DF = 13, F-
statistic = 22.4) (Figure 22). The six participants with x-values larger than 1.75 (blue line) in this model are
consistent with the six participants who achieved above-chance accuracies in both binary visuospatial imagery
classification problems. As comparison, Figure 23 presents the predictive model using only perception accuracies
(optimized to imagery CSD parameters).
Figure 22 Linear model combining pre-session exhaustion with perception classification accuracies to predict imagery classification
accuracies
Figure 23 Linear model of perception classification accuracies predicting imagery classification accuracies
35
Gender differences
Gender differences were found using a rank sum test, presented in Table 5. Males had a mean imagery
classification accuracy of 68.0±8.4% significantly larger than that of females, 61.5±3.0% (p = 0.0343). However,
this became non-significant when adjusting for multiple comparisons. Only perception accuracies were found to be
significant post-adjustment (p = 0.0176), with males performing significantly better (67.5±14.6%) than females
(51.6±4.6%).
Table 5 Group differences between males and females using the Wilcoxon rank sum test
3.4.7 Characterizing EEG features indicative of diverse imagery mental states
In order to characterize the differences between imagery in visuospatial quadrants, the most selected feature (alpha
spindle differences of the 9.5-11.5 Hz sub-band) was analyzed in the six participants who met the criteria for
Sessions 4 and 5. This feature was assessed for UL/LR and LL/UR in seven left-right channel pairs (I1-I2, O1-O2,
PO1-PO2, PO3-PO4, PO7-PO8, PO9-PO10 and P1-P2). The most significant comparisons are shown in Figure 24,
and the adjusted p-values are summarized in Table 6. The mean difference of alpha spindles in best left minus right
channel pairs was 3.5±3.4 for UL-LR, and 3.1±2.6 for LL-UR. In addition, the average left minus right spectrogram
for imagery in left and right hemifields are presented in Figure 25 for the best performing participant.
Figure 24 Alpha spindle differences (left-right) in channel pairs across participants and imagery quadrants
36
Table 6 Most significant channel pair differences in alpha spindles using the Wilcoxon rank-sum test
Figure 25 Average difference in time-frequency power for PO3 minus PO4 channel pair in participant 17 for visuospatial imagery in the
left hemifield (left) and right hemifield (right); color bar represents power/frequency (dB/Hz)
3.4.8 Summary of key findings
1. For offline binary classification of either non-specific imagery or perception against rest, a BCI based on
SVM or logistic regression can generally achieve above-chance accuracies, and in fact, exceed 70% for
most participants. Alpha coherence features seem to be particularly discriminatory. (Objectives 1 and 3).
2. Online binary classification of non-specific imagery against rest also exceeded chance for most
participants using logistic regression classifiers with alpha coherence features being important for
discrimination. Post-hoc analyses indicated that accuracies could be further improved by choosing
participant-specific classification algorithms. (Objective 1)
3. When classifying visuospatial perception in diagonally-opposing quadrants, the majority of participants
exceeded chance levels in at least one comparison. Conversely, only half of participants exceeded chance
in at least one comparison when classifying visuospatial imagery in opposing quadrants. Optimal CSD
parameters for perception involved low-to-medium rigidity of spline models, and large smoothing factors,
whereas optimal CSD parameters for imagery were typically more flexible splines, with no clear pattern in
smoothing factors. The most commonly selected features in both perception vs. perception and imagery vs.
37
imagery problems were channel pair differences in alpha spindle number, and phase lag of alpha
coherence. (Objectives 1 and 3)
4. In multiclassification problems, the two-tiered approach appeared to outperform the direct classification
approach, allowing all 6 participants to exceed chance offline. Online, four of the participants achieved
accuracies in the range of their offline scores. Additionally, one participant exceeded the threshold for an
effective BCI. (Objective 1)
5. When assessing the influence of various factors on BCI performance, the number of selected features
correlated positively with both rest vs. non-specific imagery and imagery vs. imagery. Additionally, a
linear regression model combining pre-session fatigue and perception vs. perception performance (with
imagery-optimized CSD) was found to explain 83% of imagery versus imagery performance variance.
(Objective 2)
6. Significant differences were found in alpha-spindle lateralization depending on the location of visuospatial
imagery in 6 participants. While this difference was most significant in different channel pairs, the
majority of participants exhibited increase alpha spindle-lateralization over the left hemisphere when
imagining stimuli in the left visual field, and decreased lateralization over the left hemisphere when
imagining stimuli in the right visual field. (Objective 3)
3.5 Discussion
3.5.1 Visuospatial perception classification
This study aimed to develop and assess a BCI dependent on a novel paradigm - visuospatial imagery in the four
visual field quadrants. As a method of comparison, and to estimate the upper limits of visuospatial imagery
classification, perception of stimuli in the four visual field quadrants was also assessed in three binary paradigms.
Classifying perception in any quadrant against rest, as well as classifying perception in UL vs. LR and LL vs. UR
exceeded levels of chance and met the threshold for practical BCIs, with means reaching 75.7%, 77.3% and 77.2%,
respectively. While it may seem counter-intuitive that classifying any type of perception against rest resulted in a
slightly lower accuracy than classifying different types of perception, the result may have simply been caused by
the heterogeneity of trials in the perception class (UL, LR, LL and UR). In other words, it is possible that
perception in each quadrant resulted in signals that were different enough from one another to make it difficult for
the classifier to find an optimal method of separating them from signals elicited by resting mental state. In any case,
the accuracies from classifying perception in different quadrants were somewhat lower than similar SSVEP
attentional studies. For instance, Xu et al. (2016) classified left versus right CVSA to flickering stimuli at 81%
[115]. However, a number of key differences are present: firstly, Xu and colleagues had simultaneous flickering in
both hemifields at 12 Hz, whereas this study only had flickering stimuli in the target locations. Additionally, Xu et
al, only used 2 seconds of data whereas this study relied on 5 seconds of data. By these accounts, classifying
perception of stimuli in quadrants with no competing stimuli for a longer period of time should have resulted in
accuracies comparable to those reported by Xu et al. Another study by Maye and Engel achieved accuracies of 95%
38
in a 9-class paradigm where a clock face flickering at 15Hz would stimulate various retinotopic locations
depending on the user’s fixation point[116]. The lower accuracy in the current study compared to other literature
was likely largely due to the sub-optimal flickering frequency of 2Hz, which produces much weaker SSVEPs than
higher frequencies between 3-60 Hz [117], and the optimal range of 12-18Hz [118]. Additionally, the BCI in this
study was designed to discriminate visuospatial imagery rather than perception, and so did not attempt to produce
features (typically canonical-correlation analysis coefficients) that are extracted in SSVEP literature.
3.5.2 Visuospatial imagery classification
Classifying any imagery against rest resulted in offline accuracies of 71.2% ±11.3% with a sophisticated BCI,
online accuracies of 71.7% ±12.3% with a preliminary BCI, and 77% ±11.4% when the online trials were
reassessed using a sophisticated BCI approach. All of these met the threshold for a practical BCI in a binary
paradigm [119]. When classifying different quadrants of imagery, the mean offline accuracies reached 65.2±7.5%
for UL vs. LR and 65.0±7.2% for UL vs. LR. The offline results presented here are very similar to the findings of
Bobrov and colleagues, who aimed to classify imagery of faces and houses, and a resting state [57]. Bobrov and
colleagues classified offline imagery against rest at an average of 69±3% for faces and 72±2% for houses, and
imagery of faces against houses at 63±2%. The corresponding scores in their online session were 73%±3%, 70±3%
and 64±2%. One notable difference in methodology is that Bobrov and colleagues classified trials of imagery that
were significantly longer (15 s) than the ones in this study (5 s). Additionally, in terms of target stimuli, there are
neuroanatomical differences in the loci of activation when processing faces (e.g. the fusiform face area [120]),
houses (e.g. lateral occipital complex [121]) and checkered arrow stimuli (early visual cortices [89]). This results in
differing EEG dynamics, depending on the visual stimulus [122]. The current study demonstrates that it is not
necessary to imagine for very long, or to imagine targets that may engage higher processing regions, to elicit
machine-discernible brain activation. Importantly, by relying on early visual retinotopic activation, the number of
potential classes in an imagery-based BCI paradigm is only limited by the number of differentiable visual field
locations, rather than by the more restrictive number of distinct higher-processing cortical regions and their
corresponding target stimuli (e.g. faces, bodies, objects).
In another recent study, authors achieved a staggering 88% accuracy in a 3-class visual imagery of motion
paradigm [123]. In this study, Sousa and colleagues measured EEG activity using electrodes directly above the
frontal eye fields (F3, F5, FC3, FC5, C3, C5), and had participants close their eyes and imagine a static dot, a dot in
constant motion or alternating motion. One crucial flaw here is that it was not possible for participants to fixate
their gaze (e.g. on a fixation cross) with their eyes closed, while imagining the dots. While EOG activity was
subtracted from the EEG channels, eye movement could not be controlled in the study, which may have induced
activity in the frontal eye fields not due to imagery, but rather due to eye saccades – thereby biasing the results. An
analogue to this might be measuring from the motor cortex while participants moved their limbs, then subtracting
the motion artifact. Unfortunately, the authors did not provide a comparison of EOG activity between the different
mental states. Furthermore, when assessing the statistical significance of the only alpha power feature used for
classification, they could not find significance between their two moving imagery dot tasks, but only between the
static and moving dots. Following the results of the statistical test, one might think that classification of either
39
moving dot against the static dot would result in higher accuracy than classifying the different types of moving
dots, but this was not the case. In fact, misclassification was balanced across the different classes. While the authors
chalked this discrepancy up to the ability of multivariate pattern classification to correctly discriminate classes, our
data revealed strong alpha spindle lateralization (Figure 24) in multiple individuals but their mean binary
classification accuracy only reached 72.2% (SD 8.3%). Importantly, this discrepant finding was not due sample size
differences, as Sousa et al. had 60 samples per class, whereas the current study had 75 samples per class. Given
these shortcomings, the work of Sousa et al. is not directly comparable to the present study.
In a study by Tonin and colleagues (2013), covert visuospatial attention to the left or right visual fields
was classified at 70.6±1.5% in an online paradigm [56]. The participants were cued with an arrow to pay covert
attention to one of circles (lower left or lower right), and after some time a red dot would appear in the circle. The
authors used 3 seconds of alpha band power from the EEG data to classify the locus of CVSA, however, the 3
seconds included the arrow cue presentation. While this cue was only presented for 100ms, it is difficult to discount
the possibility that the mere presentation of the arrow pointing to the left or right hemifield improved classification
accuracies. In a similar study by Treder et al (2011), participants were cued to pay covert attention to one of 6 target
circles. The authors used parieto-occipital alpha spectral power from 500-2000ms post cue to identify locus of
attention. Attention to each pair of circles was classified offline, and the mean classification accuracy of the most
discriminable pairs of loci for each participant was reported to be 74.6±2.3%. While only the best classification
accuracies were reported, the study, like that of Tonin et al, did manage to classify on a relatively short period of
time. The discrepancy in accuracy between these two studies (70-75%) and the current one (~65%) may be
attributed to the face that both previous studies had stimuli (circles) in the periphery to which participants attended.
If the outline of arrow stimuli were presented in visual field quadrants, it is possible that visuospatial imagery
would classify at accuracies closer to those reported by Tonin et al. and Treder et al.; however, it could also be
argued that this would not be true imagery. While the visuospatial imagery accuracies did not reach those of CVSA
BCIs, the appeal of the current study is that no stimuli were required in the target locations. In fact, the alpha power
lateralization features in the two aforementioned CVSA studies likely relied on phenomena whereby: (1) increased
alpha power (i.e. alpha synchronization) allowed for suppression of irrelevant (i.e. non-target) stimulus-processing
in corresponding visual cortical regions, and (2) decreased alpha power (i.e. alpha de-synchronization) allowed for
processing of stimuli in relevant (i.e. target) areas by their corresponding visual cortical regions[124]. Thus, in the
aforementioned studies, the alpha lateralization might have served to suppress processing of the non-target circles
and to prioritize or increase processing of target circles [125][106]. However, in the current study, there were no
target/non-target stimuli presented, and as such, the alpha power lateralization (here characterized by alpha spindle
differences – Figure 24) appears to have been used to suppress or enhance processing of visual field locations in
general, rather than suppress or enhance stimuli within those areas. As a result, the alpha power lateralization may
have been weaker. Another possibility is that, by introducing stimulus post-CVSA (as in the abovementioned
studies), the mental activity that is discriminated is not solely due to attention, but also due to anticipation of a
stimulus appearing – which has previously been demonstrated to induce distinguishable EEG activity [126]. A final
difference of the current study is that the research studies listed so far have all used wet electrode systems, which
have a significantly higher signal-to-noise ratio than that of dry systems [18], [19]. Since imagery and attention
40
both result in fairly weak cortical activation [40], it is likely that it would be easier to detect the two mental
activities using wet rather than dry EEG acquisition systems.
3.5.3 Similarities between perception and imagery
Whereas offline perception versus rest accuracy (75.7%) was only moderately greater than offline imagery vs. rest
accuracy (71.2%), classification of perception in diagonally opposing quadrants was considerably more accurate
(UL vs. LR - 77.3%; LL vs. UR - 77.2%) than its imagery analogues (UL vs. LR - 65.2%; LL vs. UR - 65.0%). The
reason for this likely lies in the fact that perception of stimuli in the visual field reliably activates the visual cortex
through the bottom-up visual processing pathway. Conversely, recruitment of early visual cortices through top-
down neural pathways is much weaker: visuospatial imagery has previously been demonstrated to recruit similar
cortical regions as perception of stimuli in the same visual field locations, only to a much lesser degree [40]. Thus,
it may be the case that individuals can reliably activate the visual cortex when engaging in any visuospatial
imagery, but have greater difficulty in producing strongly localized cortical activations for different visuospatial
regions.
This is consistent with the results of the most commonly-selected CSD parameters. Spline flexibility
indicates the rigidity of the spherical head model used in CSD estimates, with low numbers being flexible and
higher numbers increasingly more rigid and a typical range between 2 to 6 [98], [127]. The lambda factor describes
the degree of smoothing that occurs over the spatial filter, with larger numbers indicating greater smoothing [98].
Finally, Legendre Pn define the spatial harmonic frequencies at each electrode in the spatial filter [95]. When
optimizing spatial filter parameters for perception trials, the most frequent ideal spline flexibilities were mid-range
(m = 3 and 4), and, there was a tendency for higher lambda values to be selected (lam 10-4.5, 10-4.25). Conversely,
more flexible spline models were selected for imagery trials (m =2 or 3), and the pattern of lambda values selected
was sporadic. Taken together, this may indicate that local sources of perception activity were best discriminated
when they were spread out to a greater degree, but required a more rigid spherical model to improve classification
accuracy. On the other hand, the spline model may have required more flexibility to accentuate key sources of
imagery, but due to a weaker signal, smoothing was less common. It is important to note that, while the values
above indicate some patterns in optimal CSD parameters in this group of participants, these are not necessarily the
ideal parameters for each individual – as each person has different head sizes, cortical folding and neuroanatomy
that may influence the effectiveness of specific parameters in such a spatial filter.
While classification of perception and imagery differed in accuracies and ideal CSD parameters, the two
appear to share common features that characterize their EEG signals and underlying mental activity. This is
partially evident through the frequency of features selected through elastic-net regularization in each fold of offline
cross-validation. When classified against rest, the most commonly selected features in both rest and perception
were magnitude-squared coherence of the alpha band, and phase-lag of alpha coherence (Figure 12). Likewise,
when classifying different quadrants, features that were selected to distinguish different types of imagery and
perception were alpha spindle lateralization and phase-lag of alpha coherence. As aforementioned, visuospatial
41
attention seems to be characterized by imbalances in the hemispheric alpha frequency over the occipital lobe [124],
and this is consistent with the type of features selected in both perception and imagery tasks.
3.5.4 Multiclassification
The direct multi-category approach to classifying different imagery in different quadrants resulted in above-chance
accuracies for half of the participants who continued to Sessions 4 and 5. However, this approach was likely limited
to the fact that characteristic imbalances in EEG signals over the occipital region are most evident for left-right
hemifield activity, and less so for upper-lower hemifield activity. In fact, as demonstrated by Treder and colleagues,
the most discriminable quadrants appear to be diagonally-opposing (see [77]). Finally, since rest results in a
different class of EEG activity from imagery, it is of no surprise that the two-tier classification approach increased
classification accuracies by 18.8% over that attained with the direct approach.
When applying the two-tier classification in an online paradigm, however, the accuracy decreased
moderately from offline scores. While four of the six participants had online accuracies that fell within the standard
deviation of the offline scores, two participants had online scores that decreased below this range (Figure 21). One
potential reason for the relative decrease in accuracy from offline to online is attributable to the inability to search
for more accurate CSD parameters following collection of new offline data. Indeed, while CSD parameter
optimization did improve imagery accuracies, searching through 560 parameter combinations was computationally
intractable during retraining of the classifier in Sessions 4 and 5. Thus it was assumed that the ideal CSD
parameters identified from offline Sessions 1, 2 and 3 were optimal for Sessions 4 and 5 as well, despite the real
possibility for inter-session differences such as slight shifts of electrode locations or variations in mental state.
While the CSD spatial filter did improve accuracies in both perception and imagery paradigms, presumably due to
its ability to enhance local activity – the necessity of optimizing the filter to each participant is limiting. Alternative
spatial filters such as common spatial patterning, which instead rely on covariance measures for spatial filter
design, may thus be more suitable for retraining classifiers with same-day data; however these alternatives were
inappropriate for this paradigm due to the small number of channels [128], [129].
Clearly, multiclass discrimination is limited by the accuracies achievable by the multiple binary classifiers,
which each need to be tuned to individual differences before a practical four-class paradigm can be developed. At
the very least, however, the accuracies achieved by the highest-performing participant suggest that such a four-class
visuospatial imagery classifier is indeed feasible.
3.5.5 Predictors of accuracy
As the visuospatial imagery classification accuracies were largely variable across participants, one aim of this study
was to determine whether BCI performance could be predicted. In this way, future studies might use such metrics
to determine a priori participants with potential to perform well in specific paradigms, rather than having to blindly
collect multiple sessions of data from every participant. Furthermore, the identification of potential contributors to
classification error may highlight key factors that must be experimentally controlled in this paradigm. Several
potential correlates were investigated for both imagery versus rest accuracy and imagery versus imagery accuracy.
42
The only factor that significantly correlated to offline imagery versus rest classification accuracies was the
number of features selected with elastic-net regularization during cross-validation (Table 3). The number of
selected features was also found to correlate positively and strongly to imagery vs. imagery classification
accuracies (Table 4). This suggests that the more useful features could be identified from the EEG signals, the
better the performance of the BCI (to a certain point). In other words, it is likely that individuals who performed
better simply had more features that could provide useful information for classification, as elastic-net regularization
selects features that add information while reducing redundancy [130]. This may be attributed in part to individual
degree of cortical activation from the mental task. However, to extract useful features, reliable signal acquisition is
also necessary. And while noise was not correlated to imagery vs. rest accuracies, this may have been somewhat
mitigated by the noise attenuation necessitated prior to implementing the spatial filter (see Appendix A). Thus, the
quality of the signal may have contributed to BCI performance, in addition to individual cortical activation.
As for predictors of imagery versus imagery accuracy, perception versus perception accuracies (with
imagery-optimized spatial filter parameters) was the only measure extracted external to the imagery trials to be
significant. Typically, the greater the accuracy in discriminating one quadrant of visuospatial perception to another,
the greater the accuracy in discriminating one quadrant of visuospatial imagery to another. This finding furthers the
notion that perception and imagery share common characteristics in cortical activity detectable through EEG. More
importantly, it suggests that one might predict how well an individual would do in an imagery task given how well
their EEG activity from a perception task can be discriminated. When combined with pre-session exhaustion, a
linear regression model could predict imagery classification accuracy with an adjusted r-squared of 0.834 (Figure
22). Intriguingly, this model perfectly predicted the six participants who were selected to continue on to sessions 4
and 5 based on the separate criteria that their imagery versus imagery accuracy means exceeded chance. This is
considered a more robust model than one with only perception accuracies, in which the top six participants could
not be predicted accurately (Figure 23). Prior to correction, pre-session fatigue appeared to negatively correlate
with performance, which is unsurprising given that the task requires a fair deal of focus and attention. In line with
this, alpha spindles have previously been demonstrate to reliably indicate levels of fatigue [108]. Thus, a
participant’s level of fatigue may have interfered with mental task ability. While perception trials and measures of
fatigue can be collected prior to collecting imagery trials, three important caveats to the predictive linear regression
model remain. First, the measure of pre-session exhaustion used in this study was reported after the session as part
of a questionnaire, and so may have been biased according to their level of fatigue at the time, or their perceived
change of fatigue. Second, the perception blocks were collected before each imagery block across each day, and so
this model included intersession differences that might not be predicted as well with the same number of perception
trials collected from a single session. Finally, the perception versus perception accuracies that best explained
imagery accuracies were those resulting from the use of imagery versus imagery-optimized CSD parameters, not
those optimized for visuospatial perception classification. Thus, while performance in visuospatial imagery could
be predicted with a very high degree of accuracy, some a priori knowledge about the ideal spatial filters was
necessary. In any case, the findings still suggest that approximately 83% of the variance in visuospatial imagery can
be explained by the detection of visuospatial perception mental activity, combined with perceived levels of fatigue.
43
Whereas a previous study demonstrated posterior alpha to be predictive of BCI performance in a CVSA
paradigm [77], maximum baseline alpha across three sessions was not found to be predictive of imagery accuracies
using non-parametric tests. However, the two approaches to collect and calculate baseline alpha were slightly
different: participants had their eyes closed in study by Treder et al, whereas the participants in this study had their
eyes open. Furthermore, Treder and colleagues measured a minute of baseline prior to a session whereas in this
study, the average baseline of 30 seconds across three sessions was used as a correlate. Finally, the authors used
pooled alpha power over symmetric electrode pairs, while the current study only assessed the electrode resulting in
the maximum alpha power.
The only gender difference found was that males performed significantly better in visuospatial perception.
However, whereas the classification accuracies of 6/10 males exceeded chance when classifying imagery in either
diagonally-opposing quadrants, only 3/8 females exceeded chance in only one of the two classification problems.
Thus suggests a trend towards males having slightly more distinguishable visual cortical activation during
visuospatial imagery, which is consistent with literature suggesting a gender bias in the performance of visuospatial
attention and memory [131]. This may have contributed somewhat to the high individual variability in BCI
performance.
3.5.6 Differences in lateralization of alpha
Increased alpha power over the visual cortical regions corresponding to locations of unattended stimuli, and
decreased alpha power over regions processing attended stimuli has been well established [132][124][125][106].
Here, alpha lateralization was shown for the first time as a characteristic manifestation of cortical activity induced
by visuospatial imagery (Figure 24). Imagery in upper and lower left visual field quadrants resulted in greater left
hemisphere alpha spindle lateralization (indicating suppressed processing of the right visual hemifield, and
enhanced processing of the left visual hemifield), whereas visuospatial imagery in upper and lower right visual field
quadrants resulted in decreased left hemisphere alpha spindle lateralization. This difference was significant and
visually apparent in four of six participants in UL vs. LR imagery trials and five of six participants in LL vs. UR
imagery trials (Table 6). This demonstrates that there is no necessity for stimuli in the visual field to be suppressed
or enhanced, but rather that this modulation can be fully internally driven by top-down processes eliciting activity
in early visual cortical areas. Importantly, in the context of increased CVSA classification accuracies, it is likely
that having stimuli in the visual field (such as target and non-target circles) makes it easier for this top-down
modulation to occur [125]. In any case, it has now been demonstrated that similar patterns can be induced with
imagined stimuli in target visual field locations. One implication of this is that the use of lateralized alpha spindles
may be appropriate as a measure of visuospatial imagery in a feedback training paradigm.
While the majority of participants demonstrated the typical alpha lateralization pattern (alpha increase in
ipsilateral, alpha decrease in contralateral hemispheres), one participant showed the opposite. However, the channel
pair in which this was found to be significant was I1-I2, whereas for all other participants, the most significantly
different channel pairs were in more superior (PO1-4, P1-2) and lateral (PO7-PO8) locations. Incidentally, these
regions were previously found to be important in CVSA paradigms [77][56]. It is important to note that not every
44
channel pair that was used as features in the BCI was tested for significant alpha lateralization to keep family-wise
error low. As a result, it is likely that other left-right channel pairs capture the characteristic alpha spindle
lateralization pattern better.
3.5.7 Potential sources of variability
Individual mental activation
There is high inter-individual variability in cortical activation during visual imagery [47], [59]. In fact, Slotnick et
al. demonstrated that cortical activation patterns of imagery were similar to those of perception in half their
participants, whereas in the remaining half, imagery did not significantly activate the striate cortex [40]. This is in
line with the findings of the current study, where the BCI exceeded chance levels in visuospatial imagery for at
least one pair of diagonally-opposing quadrants in half the participants (ranging between 65 and 89%), but could
not exceed chance for the remaining half. As suggested by the degree to which visuospatial perception accuracy
predicted visuospatial imagery accuracy, individual differences in cortical activity appear to play a role in
visuospatial imagery accuracy. Further, most participants achieved above-chance classification in rest versus any
imagery, yet only a third reached above-chance accuracies in both UL vs. LR and LL vs. UR imagery tasks. This
may simply be due to the higher number of samples used to train the classifier in the rest versus non-specific
imagery problem. More likely, however, is that the discriminable signals during non-specific visuospatial imagery
represent a shift from the default mode network at rest, towards some general visual cortical activation. Within or
subsequent to this general activation may lie the specific visuocortical activation unique to different visual field
locations. This is wholly consistent with the findings from Klein et al, who suggested visual imagery is a two-stage
process where first the cortex needs to be “turned on”, only after which it could be “fine-tuned” for specific
imagined shapes [59]. Thus, it is possible that the majority of participants were able to shift activity from the
default resting state to engage their visual cortex during imagery, but there was high variability in the degree to
which individuals could elicit differentiable activations when performing imagery in different quadrants.
Number and location of electrodes
It is also possible is that some individuals were able to engage their visual cortices in a way that produced
distinguishable activation patterns for different types of visuospatial imagery, however the electrodes were not in
optimal locations to pick up this (weak) activity. Although the standard electrode placement system allows for
similar regions to be recorded across participants, there are inter-individual variations in the neuroanatomical
structures located underneath [133]. For instance, Koessler et al found that parieto-occipital electrodes (e.g. PO3,
PO4) could be located over up to six different macro-anatomical cortical regions across different participants [133].
This limited generalizability of the standard electrode configuration may have contributed to the variability in ideal
spatial filter parameters, as well as the degree to which the spatial filter improved classification. Exact electrode
placement may not be necessary for stronger mental activation associated with perception, which may allow
corresponding signals to travel farther from their sources to more distal electrode sites. In contrast, a weaker signal
45
such as one resulting from imagery may require electrodes situated in very specific locations, proximal to the
source of activity. The limited multiplicity of electrodes and corresponding configuration may have thus limited the
number of participants for whom visuospatial imagery signals could be detected. In fact, this may be the reason
why in three participants, imagery versus imagery classification was significant for one pair of opposing quadrants,
but not the other. Specifically, electrodes may have been located in optimal positions to detect imagery in some but
not all of the quadrants. This is consistent with the finding that the most significant channel pairs when determining
alpha spindle lateralization varied across participants, spanning 6 of the 7 tested pairs (Table 6).
Signal quality
As aforementioned, imagery results in weaker cortical activity than perception [40], which in turn likely
results in a weaker EEG signal, thereby decreasing SNR. The idea that weak signals limited this paradigm is
supported with the finding that spatial filters optimal for perception classification used larger smoothing factors
than those for imagery, presumably to enhance and spread the key signals of interest. This is further corroborated
by the finding that a larger number of selected features yielded greater classification accuracies in both imagery
versus rest and imagery versus imagery problems. In other words, weaker signals may have limited the degree to
which useful features could be extracted. On the other hand, rather than being caused solely by weak signal, the low
SNR may have been in part induced by high levels of noise. Dry systems have much higher impedances than
traditional wet setups; slight perturbances of the electrode-skin contact contaminate the signal with visibly high
amplitude noise inconsistent with cortical activity. While scaling down the amplitude of noisy epochs may have
diminished noise, key signals indicative of mental activity may have been attenuated as well. Additionally, noise
may have masked the activity of interest in imagery, but spared strong signals associated with perception. In any
case, signal quality is a key factor in BCI performance, and thus great care should be taken to mitigate noise prior to
collecting signals related to specific mental tasks.
3.5.8 Potential solutions to the variability
Participants performed much better online (77%) than they did offline (71%) – underlining the importance of
feedback for learning and engagement. This is supported by the finding that fatigue negatively correlated with BCI
performance. While in fMRI studies, visual imagery did not significantly activate the cortex in all participants [40],
[59], the possibility that visual cortical activation may be learned or improved through a training paradigm with
appropriate feedback has yet to be explored. In fact, a recent study demonstrated that feedback consistent with
alpha spindle activity over the occipital lobe resulted in increased alpha spindle frequency when compared to
controls receiving feedback not reflective of their alpha spindle activity [134]. This is promising, as alpha-spindle
lateralization has been demonstrated in this study as a useful feature of visuospatial imagery, in turn making it
viable for use in such a feedback paradigm. Furthermore, in this study, it was not in “turning on” the visual cortex
that few participants managed to exceed chance classification (i.e. discriminating rest versus non-specific
visuospatial imagery), but rather the “fine-tuning” of the visual stimulus (i.e. discriminating imagery in different
quadrants). Thus, as the first step of visual imagery suggested by Klein et al. seems to be present in the majority of
46
participants (with 14/18 reaching the 70% threshold using the sophisticated BCI on online data) [59], a training
paradigm may serve to teach individuals to elicit differential cortical activation depending on the location of the
visual imagery stimulus.
Future works should consider a greater number of electrodes over the occipital region, as well as more
fronto-parietal regions. In fact, significant alpha spindle lateralization in this study was consistently in more
superior regions (parietal, parieto-occipital), and farther from inferior regions (inion, occipital), so more electrodes
may be necessary in more parietal areas, and more densely around parieto-occipital areas. For weaker signals such
as imagery, a greater density of electrodes and perhaps a greater number of electrodes surrounding the area of
interest may be necessary to detect the activity of interest. In addition, differences between rest and imagery would
likely be much more distinguishable by measuring the contribution of more frontal (i.e. “top”) regions engaged in
top-down activation of the visual cortex. Finally, with a larger number and greater density of electrodes, alternative
spatial filters such as common spatial patterns may be more appropriate.
Lastly, future research in this paradigm should also consider using wet electrodes as they may increase
SNR. Wet electrodes may also capture low-gamma (30-80 Hz) activity that has been linked to visual imagery and
that is indicative of fronto-occipital activation [135][136][137].
3.5.9 Key messages
1. Perception vs. rest classification was lower in this study than in previously reported SSVEP attentional
studies, likely due to several experimental differences as the present protocol was optimized for imagery
rather than perception classification. (Objective 3)
2. Imagery vs. rest classification was generally on par with previously reported research while the present
paradigm offers some distinct practical advantages, including the accommodation of dry electrodes,
relinquishing the need for visual stimuli, circumventing the modest number of discernible classes inherent
to higher-order cognitive BCIs by exploiting visual retinotopic activity, and requiring only brief durations
of imagery. Recent studies boasting higher imagery classification accuracies appear to be plagued by
confounding neural and ocular signals unrelated to imagery. (Objective 1)
3. Visuospatial perception appears to be more easily discriminated than visuospatial imagery, likely due to
the difference in cortical activation as evidenced in part by the dissimilar ideal spatial filter parameters for
each mental task. Furthermore, while fine-tuned cortical activity indicative of visuospatial imagery
location is difficult to discriminate, most participants elicit easily discriminable general activation of their
visual cortex during imagery. This is consistent with the suggested stages of visual imagery in previous
literature. (Objective 3)
4. The degree of useful information gathered from EEG signals may have been masked by noise, suggesting
signal quality may have been limiting for this such a paradigm dependent on weaker cortical signals.
Additionally, the combination of two predictors (fatigue and visuospatial perception performance)
47
explained the majority of variance in imagery vs. imagery accuracy, indicating detection of individual
cortical activation and mental state may be key factors to exploit for maximizing BCI performance in
visuospatial imagery. (Objectives 2 and 3)
5. A feature of alpha power imbalance over the occipital lobe (alpha spindle lateralization) has been
demonstrated without target/non-target stimuli in the visual field, suggesting that posterior alpha
synchronization/de-synchronization can occur entirely through internally driven top-down mechanisms,
and that stimuli may simply serve to increase this modulation. (Objective 3)
6. While only a handful of individuals were able to exceed above-chance accuracies in the 4-class BCI
described in this study, effective levels of performance such as those exemplified by one participant may
be achieved in more individuals by improving binary classification performance. This may be
accomplished by prioritizing signal quality, increasing detection of key signals, and providing feedback
aimed towards eliciting more discriminable cortical activity. (Objectives 1 and 2)
3.6 Conclusion
This study established the feasibility of a BCI dependent on visuospatial imagery. In addition, this study
demonstrated that practical BCI accuracies can be achieved in an online rest vs. imagery paradigm. The study also
realized a proof-of-concept four-class online navigation game dependent on visuospatial imagery, demonstrating
above-chance in a handful of individuals. Unlike previous research, this study used a dry system with a small
number of electrodes, rendering the proposed system conducive to real-world implementation. Finally, to the best
of the author’s knowledge, this is the first study that demonstrated lateralized alpha in visuospatial imagery without
any stimuli in target or non-target locations of the visual field.
48
Chapter 4 Conclusion
Conclusion
4.1 Contributions
This study has contributed to the field of biomedical engineering by demonstrating the potential of a novel BCI
paradigm dependent on visuospatial imagery. The specific contributions of this study are as follows:
1. Development of an effective online binary BCI with the potential to distinguish non-specific visuospatial
imagery from resting state above the 70% threshold for a practical BCI.
2. Development of an offline binary BCI with the potential to distinguish imagery in pairs of diagonally-
opposing quadrants in a handful of individuals above chance.
3. Identification of the influence of two factors (fatigue and degree of differentiable cortical activation from
perception tasks) on the variability of individual differences in discriminating visuospatial imagery.
4. Development of an online proof-of-concept 4-class BCI system dependent on visuospatial imagery in the
four quadrants for four corresponding navigational directions.
5. Identification of lateralization in an alpha frequency band feature (alpha spindle) as a key characteristic of
visuospatial imagery in opposing quadrants; similar measures of which had previously only been
demonstrated in paradigms of attention.
4.2 Future Work
This is the first study to current knowledge that has identified visuospatial imagery as a potential control paradigm
in BCIs. As a result, there are a number of potential avenues to be explored in order to improve upon this paradigm.
The first is to use higher quality signal acquisition systems such as wet EEG, rather than those susceptible to noise
(i.e. dry EEG). Such an approach may improve the SNR for the relatively weak signal that is elicited from visual
imagery. Next, future works may wish to increase the number and density of electrodes in order to capitalize on
inter-individual differences in cortical structure and activation in order to improve the generalizability of this
paradigm. This approach can also be exploited in order to explore the contributions of more frontal processing
regions on visuospatial imagery. Furthermore, in terms of making the visuospatial imagery BCI more generalizable
across individuals, it may be of interest to determine whether user-personalized imagery stimuli engage the visual
cortex to a greater degree than the neutral arrow stimulus used in this study. Finally, future studies will now be able
to use alpha spindle lateralization as a feedback tool in order to train individuals to elicit more distinguishable
visual cortical activations, and further improve the visuospatial imagery BCI paradigm.
References
[1] E. Smith and M. Delargy, “Locked-in syndrome,” Br. Med. J., vol. 330, no. February, pp. 3–6, 2005.
[2] D. J. McFarland and J. R. Wolpaw, “Brain-Computer Interfaces for Communication and Control,” Commun.
ACM, vol. 54, no. 5, pp. 60–66, 2011.
[3] P. Cipresso, L. Carelli, F. Solca, D. Meazzi, P. Meriggi, B. Poletti, D. Lul??, A. C. Ludolph, V. Silani, and
G. Riva, “The use of P300-based BCIs in amyotrophic lateral sclerosis: From augmentative and alternative
communication to cognitive assessment,” Brain Behav., vol. 2, no. 4, pp. 479–498, 2012.
[4] A. Kubler, B. Kotchoubey, T. Hinterberger, N. Ghanayim, J. Perelmouter, M. Schauer, C. Fritsch, E. Taub,
and N. Birbaumer, “The thought translation device: a neurophysiological approach to communication in
total motor paralysis.,” Exp. brain Res., vol. 124, no. 2, pp. 223–232, Jan. 1999.
[5] N. J. Hill, T. N. Lal, M. Schröder, T. Hinterberger, B. Wilhelm, F. Nijboer, U. Mochty, G. Widman, C.
Elger, B. Schölkopf, A. Kübler, and N. Birbaumer, “Training for Fast BCI Implementation : Comparison
of,” vol. 14, no. 2, pp. 183–186, 2006.
[6] E. W. Sellers and E. Donchin, “A P300-based brain-computer interface: Initial tests by ALS patients,” Clin.
Neurophysiol., vol. 117, no. 3, pp. 538–548, 2006.
[7] A. Burns, H. Adeli, and J. a. Buford, “Brain-Computer Interface after Nervous System Injury,” Neurosci.,
vol. 20, pp. 639–651, 2014.
[8] K. Tai, S. Blaine, and T. Chau, “A Review of Emerging Access Technologies for Individuals With Severe
Motor Impairments,” Assist. Technol., vol. 20, no. May, pp. 204–219, 2008.
[9] S. Weyand, K. Takehara-Nishiuchi, and T. Chau, “Weaning Off Mental Tasks to Achieve Voluntary Self-
Regulatory Control of a Near-Infrared Spectroscopy Brain-Computer Interface,” IEEE Trans. Neural Syst.
Rehabil. Eng., vol. 23, no. 4, pp. 548–561, 2015.
[10] N. Birbaumer, “Breaking the silence : Brain – computer interfaces ( BCI ) for communication and motor
control,” Psychophysiology, vol. 43, pp. 517–532, 2006.
[11] O. Bai, L. Peter, D. Huang, D.-Y. Fei, and M. K. Floeter, “Towards a User-Friendly Brain-Computer
Interface: Initial Tests in ALS and PLS Patients,” Clin. Neurophysiol., vol. 121, no. 8, pp. 1293–1303, 2010.
[12] J. Faller, R. Scherer, E. V. C. Friedrich, U. Costa, E. Opisso, J. Medina, and G. R. Müller-Putz, “Non-motor
tasks improve adaptive brain-computer interface performance in users with severe motor impairment.,”
Front. Neurosci., vol. 8, no. October, p. 320, 2014.
[13] F. Fiori, a Sedda, E. R. Ferrè, a Toraldo, M. Querzola, F. Pasotti, D. Ovadia, C. Piroddi, R. Dell’aquila, C.
Lunetta, M. Corbo, and G. Bottini, “Exploring motor and visual imagery in Amyotrophic Lateral Sclerosis.,”
Exp. brain Res., vol. 226, no. 4, pp. 537–47, 2013.
[14] J. R. Wolpaw, N. Birbaumer, D. J. Mcfarland, G. Pfurtscheller, and T. M. Vaughan, “Brain – computer
interfaces for communication and control,” Clin. Neurophysiol., vol. 113, pp. 767–791, 2002.
[15] L. F. Nicolas-Alonso and J. Gomez-Gil, “Brain computer interfaces, a review,” Sensors, vol. 12, no. 2, pp.
1211–1279, 2012.
[16] B. Burle, L. Spieser, C. Roger, L. Casini, T. Hasbroucq, and F. Vidal, “Spatial and temporal resolutions of
50
EEG: Is it really black and white? A scalp current density view,” Int. J. Psychophysiol., vol. 97, no. 3, pp.
210–220, 2015.
[17] T. Ball, M. Kern, I. Mutschler, A. Aertsen, and A. Schulze-Bonhage, “Signal quality of simultaneously
recorded invasive and non-invasive EEG,” Neuroimage, vol. 46, no. 3, pp. 708–716, 2009.
[18] M. A. Lopez-Gordo, D. Sanchez Morillo, and F. Pelayo Valle, “Dry EEG Electrodes,” Sensors, vol. 14, no.
7, pp. 12847–12870, 2014.
[19] F. Wang, G. Li, and J. Chen, “Applying dynamic data collection to improve dry electrode system
performance for a P300-based brain – computer interface,” J. Neural Eng., vol. 13, no. 6, pp. 1–11, 2016.
[20] S. Ge, R. Wang, and D. Yu, “Classification of Four-Class Motor Imagery Employing Single-Channel
Electroencephalography,” PLoS One, vol. 9, no. 6, pp. 1–7, 2014.
[21] R. N. Vigario, “Extraction of ocular artefacts from EEG using independent component analysis,”
Electroencephalogr. Clin. Neurophysiol., vol. 103, pp. 395–404, 1997.
[22] G. Gómez-Herrero, “Automatic Artifact Removal ( AAR ) toolbox Germ ´,” 2007.
[23] L. A. Farwell and E. Donchin, “Talking off the top of your head: toward a mental prosthesis utilizing event-
related brain potentials,” Electroencephalogr. Clin. Neurophysiol., vol. 70, no. 6, pp. 510–523, 1988.
[24] C. R. Turner, A. Fuggetta, L. Lavazza, and A. L. Wolf, “A conceptual basis for feature engineering,” J. Syst.
Softw., vol. 49, pp. 3–15, 1999.
[25] F. Lotte and M. Congedo, “A review of classification algorithms for EEG-based brain – computer
interfaces,” J. Neural Eng., vol. 4, pp. 1–13, 2007.
[26] D. J. McFarland, C. W. Anderson, K. R. Müller, A. Schlögl, and D. J. Krusienski, “BCI Meeting 2005 -
Workshop on BCI signal processing: Feature extraction and translation,” IEEE Trans. Neural Syst. Rehabil.
Eng., vol. 14, no. 2, pp. 135–138, 2006.
[27] L. Yu and H. Liu, “Feature Selection for High-Dimensional Data: A Fast Correlation-Based Filter Solution,”
Int. Conf. Mach. Learn., pp. 1–8, 2003.
[28] P. Pudil, J. Novovieova, and J. Kittler, “Floating search methods in feature selection,” Pattern Recognit.
Lett., vol. 15, no. June 1993, 1994.
[29] K. P. Thomas, A. P. Vinod, and C. Guan, “Design of an online EEG based neurofeedback game for
enhancing attention and memory,” Conf. Proc. ... Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. IEEE Eng.
Med. Biol. Soc. Annu. Conf., vol. 2013, no. ii, pp. 433–436, 2013.
[30] T. O. Zander and C. Kothe, “Towards passive brain–computer interfaces: applying brain–computer interface
technology to human–machine systems in general,” J. Neural Eng., vol. 8, no. 2, p. 25005, 2011.
[31] E. Yin and T. Zeyl, “An Auditory-Tactile Visual Saccade-Independent P300 Brain – Computer Interface,”
Int. J. Neural Syst., vol. 26, no. 1, pp. 1–16, 2016.
[32] G. R. Muller-Putz, R. Scherer, and G. Pfurtscheller, “Brain-computer interfaces for control of
neuroprostheses : from synchronous to asynchronous mode of operation,” Biomed. Eng. (NY)., vol. 51, pp.
57–63, 2006.
[33] M. F. Bear, B. W. Connors, and M. A. Paradiso, Neuroscience - Exploring the Brain Fourth Edition, Fourth.
Lippincott Williams & Wilkins, 2016.
51
[34] M. J. Farah, The cognitive neuroscience of vision. Blackwell Publishing, 2000.
[35] M. W. Levine and J. J. McAnany, “The relative capabilities of the upper and lower visual hemifields,”
Vision Res., vol. 45, no. 21, pp. 2820–2830, 2005.
[36] D. J. Hagler, “Visual field asymmetries in visual evoked responses,” vol. 14, no. 2014, pp. 1–19, 2014.
[37] G. Harding, P. Harding, and A. Wilkins, “Wind turbines, flicker, and photosensitive epilepsy:
Characterizing the flashing that may precipitate seizures and optimizing guidelines to prevent them,”
Epilepsia, vol. 49, no. 6, pp. 1095–1098, 2008.
[38] K. Rauss, S. Schwartz, and G. Pourtois, “Top-down effects on early visual processing in humans: A
predictive coding framework,” Neurosci. Biobehav. Rev., vol. 35, no. 5, pp. 1237–1253, 2011.
[39] J. Pearson, T. Naselaris, E. A. Holmes, and S. M. Kosslyn, “Mental Imagery: Functional Mechanisms and
Clinical Applications,” Trends Cogn. Sci., vol. 19, no. 10, pp. 590–602, 2015.
[40] S. D. Slotnick and W. L. Thompson, “Visual Mental Imagery Induces Retinotopically Organized Activation
of Early Visual Areas,” Cereb. Cortex, vol. 15, pp. 1570–1583, 2005.
[41] G. Ganis, W. L. Thompson, and S. M. Kosslyn, “Brain areas underlying visual mental imagery and visual
perception: An fMRI study,” Cogn. Brain Res., vol. 20, no. 2, pp. 226–241, 2004.
[42] A. M. Albers, P. Kok, I. Toni, H. C. Dijkerman, and F. P. De Lange, “Report Shared Representations for
Working Memory and Mental Imagery in Early Visual Cortex,” Curr. Biol., vol. 23, no. 15, pp. 1427–1431,
2013.
[43] C. McCormick, M. St-Laurent, A. Ty, T. a. Valiante, and M. P. McAndrews, “Functional and effective
hippocampal-neocortical connectivity during construction and elaboration of autobiographical memory
retrieval,” Cereb. Cortex, vol. 25, no. 5, pp. 1297–1305, 2015.
[44] M.-S. Shin, S.-Y. Park, S.-R. Park, S.-H. Seol, and J. S. Kwon, “Clinical and empirical applications of the
Rey-Osterrieth Complex Figure Test.,” Nat. Protoc., vol. 1, no. 2, pp. 892–899, 2006.
[45] D. F. Marks and A. R. Isaac, “Topographical distribution of EEG activity accompanying visual and motor
imagery in vivid and non-vivid imagers,” Br. J. Psychol., vol. 86, pp. 271–282, 1995.
[46] A. Campos, “Internal consistency and construct validity of two versions of the Revised Vividness of Visual
Imagery Questionnaire,” Percept. Mot. Skills, vol. 113, no. 2, pp. 454–460, 2011.
[47] X. Cui, C. B. Jeter, D. Yang, P. R. Montague, and D. M. Eagleman, “Vividness of mental imagery:
Individual variability can be measured objectively,” Vision Res., vol. 47, no. 4, pp. 474–478, 2007.
[48] J. Andrade, J. May, C. Deeprose, S. J. Baugh, and G. Ganis, “Assessing vividness of mental imagery: The
plymouth sensory imagery questionnaire,” Br. J. Psychol., vol. 105, no. 4, pp. 547–563, 2014.
[49] S. D. Slotnick, S. a Klein, T. Carney, E. Sutter, and S. Dastmalchi, “Using multi-stimulus VEP source
localization to obtain a retinotopic map of human primary visual cortex,” Clin. Neurophysiol., vol. 110, pp.
1793–1800, 1999.
[50] C.-H. Im, A. Gururajan, N. Zhang, W. Chen, and B. He, “Spatial Resolution of EEG Cortical Source
Imaging Revealed by Localization of Retinotopic Organization in Human Primary Visual Cortex,” J.
Neurosci. Methods, vol. 161, no. 1, pp. 142–154, 2008.
[51] Z. Yan, X. Gao, G. Bin, B. Hong, and S. Gao, “A half-field stimulation pattern for SSVEP-based brain-
52
computer interface,” Proc. 31st Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. Eng. Futur. Biomed. EMBC
2009, no. 2006, pp. 6461–6464, 2009.
[52] Y. Punsawad, S. Member, and Y. Wongsawat, “Multi-Command SSVEP-based BCI System via Single
Flickering Frequency Half-Field Stimulation Pattern,” Proc. 33rd Annu. Int. Conf. IEEE Eng. Med. Biol.
Soc. Eng. Futur. Biomed. EMBC 2011, pp. 1101–1104, 2011.
[53] A. Tzovara, M. M. Murray, G. Plomp, M. H. Herzog, C. M. Michel, and M. De Lucia, “Decoding stimulus-
related information from single-trial EEG responses based on voltage topographies,” Pattern Recognit., vol.
45, no. 6, pp. 2109–2122, 2012.
[54] N. Cicmil, H. Bridge, A. J. Parker, M. W. Woolrich, and K. Krug, “Localization of MEG human brain
responses to retinotopic visual stimuli with contrasting source reconstruction approaches,” Front. Neurosci.,
vol. 8, no. 8 MAY, pp. 1–16, 2014.
[55] A. Kubler, V. K. Mushahwar, L. R. Hochberg, and J. P. Donoghue, “BCI Meeting 2005--workshop on
clinical issues and applications.,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 14, no. 2, pp. 131–134, Jun.
2006.
[56] L. Tonin, R. Leeb, A. Sobolewski, and J. R. Mill, “An online EEG BCI based on covert visuospatial
attention in absence of exogenous stimulation,” J. Neural Eng., vol. 10, 2013.
[57] P. Bobrov, A. Frolov, C. Cantor, I. Fedulova, M. Bakhnyan, and A. Zhavoronkov, “Brain-Computer
Interface Based on Generation of Visual Images,” PLoS One, vol. 6, no. 6, 2011.
[58] M. J. Tarr and Y. D. Cheng, “Learning to see faces and objects,” Trends Cogn. Sci., vol. 7, no. 1, pp. 23–30,
2003.
[59] I. Klein, J. Dubois, F. Kherif, G. Flandin, J. Poline, M. Denis, S. M. Kosslyn, and D. Le, “Retinotopic
organization of visual mental images as revealed by functional magnetic resonance imaging,” Cogn. Brain
Res., vol. 22, pp. 26–31, 2004.
[60] J. R. Wolpaw, N. Birbaumer, D. J. McFarland, G. Pfurtscheller, and T. M. Vaughan, “Brain-computer
interfaces for communication and control.,” Clin. Neurophysiol., vol. 113, no. 6, pp. 767–91, 2002.
[61] J. Wolpaw, N. Birbaumer, D. J. McFarland, G. Pfurtscheller, and T. M. Vaughan, “Brain–computer
interfaces for communication and control,” Clin. Neurophysiol., vol. 113, no. 6, pp. 767–791, 2002.
[62] A. Myrden and T. Chau, “Effects of user mental state on EEG-BCI performance,” Front. Hum. Neurosci.,
vol. 9, no. June, pp. 1–11, 2015.
[63] S. Sutton, M. Braren, J. Zubin, and E. R. John, “Evoked-Potential Correlates of Stimulus Uncertainty
Author ( s ): Samuel Sutton , Margery Braren , Joseph Zubin and E . R . John,” Science (80-. )., vol. 150, no.
3700, pp. 1187–1188, 1965.
[64] C. Guger, S. Daban, E. Sellers, C. Holzner, G. Krausz, R. Carabalona, F. Gramatica, and G. Edlinger, “How
many people are able to control a P300-based brain-computer interface (BCI)?,” Neurosci. Lett., vol. 462,
no. 1, pp. 94–98, 2009.
[65] A. Miyake, N. P. Friedman, M. J. Emerson, A. H. Witzki, A. Howerter, and T. D. Wager, “The Unity and
Diversity of Executive Functions and Their Contributions to Complex ‘Frontal Lobe’ Tasks: A Latent
Variable Analysis,” Cogn. Psychol., vol. 41, no. 1, pp. 49–100, 2000.
53
[66] O. Bai, P. Lin, D. Huang, D. Y. Fei, and M. K. Floeter, “Towards a user-friendly brain-computer interface:
Initial tests in ALS and PLS patients,” Clin. Neurophysiol., vol. 121, no. 8, pp. 1293–1303, 2010.
[67] G. Pfurtscheller, P. Linortner, R. Winkler, G. Korisek, and G. Müller-Putz, “Discrimination of motor
imagery-induced EEG patterns in patients with complete spinal cord injury,” Comput. Intell. Neurosci., vol.
2009, pp. 1–7, 2009.
[68] J. R. Kirby, P. J. Moore, and N. J. Schofield, “Verbal and visual learning styles,” Contemp. Educ. Psychol.,
vol. 13, no. 2, pp. 169–184, 1988.
[69] D. J. M. Kraemer, L. M. Rosenberg, and S. L. Thompson-Schill, “The Neural Correlates of Visual and
Verbal Cognitive Styles,” J. Neurosci., vol. 29, no. 12, pp. 3792–3798, 2009.
[70] M. Zvyagintsev, B. Clemens, N. Chechko, K. A. Mathiak, A. T. Sack, and K. Mathiak, “Brain networks
underlying mental imagery of auditory and visual information,” Eur. J. Neurosci., vol. 37, no. 9, pp. 1421–
1434, 2013.
[71] M.-C. Dobrea and D. M. Dobrea, “The selection of proper discriminative cognitive tasks - A necessary
prerequisite in high-quality BCI applications,” Int. Symp. Appl. Sci. Biomed. Commun. Technol., pp. 1–6,
2009.
[72] S. Weyand, K. Takehara-Nishiuchi, and T. Chau, “Exploring methodological frameworks for a mental task-
based near-infrared spectroscopy brain–computer interface,” J. Neurosci. Methods, vol. 254, pp. 36–45,
2015.
[73] P. Andersson, N. F. Ramsey, J. P. W. Pluim, and M. A. Viergever, “BCI control using 4 direction spatial
visual attention and real-time fMRI at 7T,” pp. 4221–4225, 2010.
[74] P. Andersson, N. F. Ramsey, M. Raemaekers, M. A. Viergever, and J. P. W. Pluim, “Real-time decoding of
the direction of covert visuospatial attention,” J. Neural Eng., vol. 9, no. 4, 2012.
[75] M. van Gerven and O. Jensen, “Attention modulations of posterior alpha as a control signal for two-
dimensional brain-computer interfaces,” J. Neurosci. Methods, vol. 179, no. 1, pp. 78–84, 2009.
[76] E. Astrand, C. Wardak, and S. Ben Hamed, “Selective visual attention to drive cognitive brain-machine
interfaces: from concepts to neurofeedback and rehabilitation applications,” Front. Syst. Neurosci., vol. 8,
no. August, pp. 1–16, 2014.
[77] M. S. Treder, A. Bahramisharif, N. M. Schmidt, M. A. van Gerven, and B. Blankertz, “Brain-computer
interfacing using modulations of alpha activity induced by covert shifts of attention,” J. Neuroeng. Rehabil.,
vol. 8, no. 1, p. 24, 2011.
[78] H.-J. Hwang, V. Y. Ferreria, D. Ulrich, T. Kilic, X. Chatziliadis, B. Blankertz, and M. Treder, “A Gaze
Independent Brain-Computer Interface Based on Visual Stimulation through Closed Eyelids,” Sci. Rep., vol.
5, no. June, p. 15890, 2015.
[79] M. E. Thurlings, A.-M. Brouwer, J. B. F. Van Erp, and P. Werkhoven, “Gaze-independent ERP-BCIs:
augmenting performance through location-congruent bimodal stimuli,” Front. Syst. Neurosci., vol. 8, no.
September, pp. 1–14, 2014.
[80] A. Ishai, L. G. Ungerleider, and J. V. Haxby, “Distributed neural systems for the generation of visual
images,” Neuron, vol. 28, no. 3, pp. 979–990, 2000.
54
[81] A. Delorme and S. Makeig, “EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics
including independent component analysis,” J. Neurosci. Methods, vol. 134.1, no. 1, pp. 9–21, 2004.
[82] H. Aurlien, I. O. Gjerde, J. H. Aarseth, G. Eldøen, B. Karlsen, H. Skeidsvoll, and N. E. Gilhus, “EEG
background activity described by a large computerized database,” Clin. Neurophysiol., vol. 115, no. 3, pp.
665–673, 2004.
[83] P. A. Osterrieth, “Le test de copie d’une figure complexe,” Arch. Psychol. (Geneve)., vol. 30, pp. 206–356,
1944.
[84] A. Rey, “L’examen psychologique dans les cas d’encéphalopathie traumatique (Les problems),” Arch.
Psychol. (Geneve)., vol. 28, pp. 215–285, 1941.
[85] D. F. Marks, “New directions for mental imagery research,” J. Ment. Imag., vol. 19, no. 3–4, pp. 153–167,
1995.
[86] G. Powell, P. Sumner, and A. Bompas, “The effect of eye movements and blinks on afterimage appearance
and duration,” J. Vis., vol. 15, no. 3, pp. 20–20, 2015.
[87] K. B. Ng, A. P. Bradley, and R. Cunnington, “Effect of posterized naturalistic stimuli on SSVEP-based
BCI,” Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. EMBS, pp. 3105–3108, 2013.
[88] S. D. Slotnick, S. A. Klein, T. Carney, and E. E. Sutter, “Electrophysiological estimate of human cortical
magnification,” Clin. Neurophysiol., vol. 112, no. 7, pp. 1349–1356, 2001.
[89] S. D. Slotnick and S. Yantis, “Efficient acquisition of human retinotopic maps,” Hum. Brain Mapp., vol. 18,
no. 1, pp. 22–29, 2003.
[90] D. Kennard, R. Hartmann, D. Kraft, and B. Boshes, “Perceptual Suppression,” Vision Res., vol. 10, no. 7,
pp. 575–585, 1970.
[91] W. L. Thompson, Y. Hsiao, and S. M. Kosslyn, “Dissociation between visual attention and visual mental
imagery,” J. Cogn. Psychol., vol. 23, no. 2, pp. 256–263, 2011.
[92] G. Thut, A. Nietzel, S. Brandt, and A. Pascual-Leone, “a-Band Electroencephalographic Activity over
Occipital Cortex Indexes Visuospatial Attention Bias and Predicts Visual Target Detection,” J. Neurosci.,
vol. 26, no. 37, pp. 9494–9502, 2006.
[93] J. Qian, T. Hastie, J. Friedman, R. Tibshirani, and N. Simon, “Glmnet for Matlab,” 2013. .
[94] J. Kayser, “Current source density (CSD) interpolation using spherical splines - CSD Toolbox (Version
1.1),” New York State Psychiatric Institute: Division of Cognitive Neuroscience., 2009. [Online]. Available:
http://psychophysiology.cpmc.columbia.edu/Software/CSDtoolbox.
[95] M. X. Cohen, “Surface Laplacian,” in Analyzing Neural Time Series Data: Theory and Practice, MIT Press,
2014, pp. 275–290.
[96] F. Babiloni, C. Babiloni, L. Fattorini, F. Carducci, P. Onorati, and A. Urbano, “Performances of surface
Laplacian estimators: A study of simulated and real scalp potential distributions,” Brain Topogr., vol. 8, no.
1, pp. 35–45, 1995.
[97] J. Kayser and C. E. Tenke, “Principal components analysis of Laplacian waveforms as a generic method for
identifying ERP generator patterns: I. Evaluation with auditory oddball tasks,” Clin. Neurophysiol., vol. 117,
no. 2, pp. 369–380, 2006.
55
[98] J. Kayser and C. E. Tenke, “Principal components analysis of Laplacian waveforms as a generic method for
identifying ERP generator patterns: II. Adequacy of low-density estimates,” Clin. Neurophysiol., vol. 117,
no. 2, pp. 369–380, 2006.
[99] J. Kayser and C. E. Tenke, “Issues and considerations for using the scalp surface Laplacian in EEG/ERP
research: A tutorial review,” Int. J. Psychophysiol., vol. 97, no. 3, pp. 189–209, 2015.
[100] J. Kayser and C. E. Tenke, “On the benefits of using surface Laplacian (current source density) methodology
in electrophysiology,” Int. J. Psychophysiol., vol. 97, no. 3, pp. 171–173, 2015.
[101] A. Sonnleitner, M. Simon, W. E. Kincses, A. Buchner, and M. Schrauf, “Alpha spindles as
neurophysiological correlates indicating attentional shift in a simulated driving task,” Int. J. Psychophysiol.,
vol. 83, no. 1, pp. 110–118, 2012.
[102] W. Klimesch, “Alpha-band oscillations, attention, and controlled access to stored information,” Trends
Cogn. Sci., vol. 16, no. 12, pp. 606–617, 2012.
[103] J. J. Foxe and A. C. Snyder, “The role of alpha-band brain oscillations as a sensory suppression mechanism
during selective attention,” Front. Psychol., vol. 2, no. JUL, pp. 1–13, 2011.
[104] N. Schmidt, B. Blankertz, and M. S. Treder, “??-modulation induced by covert attention shifts as a new
input modality for EEG-based BCIs,” Conf. Proc. - IEEE Int. Conf. Syst. Man Cybern., pp. 481–487, 2010.
[105] A. Ikkai, S. Dandekar, and C. E. Curtis, “Lateralization in Alpha-Band Oscillations Predicts the Locus and
Spatial Distribution of Attention,” pp. 1–17, 2016.
[106] S. P. Kelly, E. C. Lalor, R. B. Reilly, and J. J. Foxe, “Increases in Alpha Oscillatory Power Reflect an
Active Retinotopic Mechanism for Distracter Suppression During Sustained Visuospatial Attention,” J.
Neurophysiol., vol. 95, no. 6, pp. 3844–3851, 2006.
[107] L. Tonin, R. Leeb, and J. del R Millán, “Time-dependent approach for single trial classification of covert
visuospatial attention,” J. Neural Eng., vol. 9, no. 4, p. 45011, 2012.
[108] M. Simon, E. A. Schmidt, W. E. Kincses, M. Fritzsche, A. Bruns, C. Aufmuth, M. Bogdan, W. Rosenstiel,
and M. Schrauf, “Eeg alpha spindle measures as indicators of driver fatigue under real traffic conditions,”
Clin. Neurophysiol., vol. 122, no. 6, pp. 1168–1178, 2011.
[109] Y. Ke, L. Chen, L. Fu, Y. Jia, P. Li, X. Zhao, H. Qi, P. Zhou, L. Zhang, B. Wan, and D. Ming, “Visual
attention recognition based on nonlinear dynamical parameters of EEG,” Biomed. Mater. Eng., vol. 24, no.
1, pp. 349–355, 2014.
[110] W. Li, D. Ming, R. Xu, H. Ding, H. Qi, and B. Wan, “Research on visual attention classification based on
EEG entropy parameters,” IFMBE Proc., vol. 39 IFMBE, no. 3, pp. 1553–1556, 2013.
[111] C. Chang and C. Lin, “LIBSVM : A Library for Support Vector Machines,” ACM Trans. Intell. Syst.
Technol., vol. 2, no. 3, pp. 1–27, 2011.
[112] and C.-J. L. Chih-Wei Hsu, Chih-Chung Chang, “A Practical Guide to Support Vector Classification,” BJU
Int., vol. 101, no. 1, pp. 1396–400, 2008.
[113] E. Combrisson and K. Jerbi, “Exceeding chance level by chance: The caveat of theoretical chance levels in
brain signal classification and statistical assessment of decoding accuracy,” J. Neurosci. Methods, vol. 250,
pp. 126–136, 2015.
56
[114] P. C. Austin and E. W. Steyerberg, “The number of subjects per variable required in linear regression
analyses,” J. Clin. Epidemiol., vol. 68, no. 6, pp. 627–636, 2015.
[115] M. Xu, Y. Wang, M. Nakanishi, and Y. Wang, “Fast detection of covert visuospatial attention using hybrid
N2pc and SSVEP features,” J. Neural Eng., vol. 13, 2016.
[116] A. Maye, D. Zhang, and A. K. Engel, “Utilizing Retinotopic Mapping for a Multi-Target SSVEP BCI With
a Single Flicker Frequency,” IEEE Trans. Biomed. Eng., vol. 25, no. 7, pp. 1026–1036, 2017.
[117] F.-B. Vialatte, M. Maurice, J. Dauwels, and A. Cichocki, “Steady State Visual Evoked Potentials in the
Delta Range (0.5-5 Hz) BT - Advances in Neuro-Information Processing: 15th International Conference,
ICONIP 2008, Auckland, New Zealand, November 25-28, 2008, Revised Selected Papers, Part I,” M.
Köppen, N. Kasabov, and G. Coghill, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2009, pp. 400–
407.
[118] A. Duszyk, P. Milanowski, M. Łabe, P. Suffczyn, M. Michalska, and Z. Jarosław, “On the Quantification of
SSVEP Frequency Responses in Human EEG in Realistic BCI Conditions,” PLoS One, vol. 8, no. 10, 2013.
[119] A. Kübler and N. Birbaumer, “Clinical Neurophysiology Brain – computer interfaces and communication in
paralysis : Extinction of goal directed thinking in completely paralysed patients ?,” Clin. Neurophysiol., vol.
119, no. 11, pp. 2658–2666, 2008.
[120] J. V Haxby, M. I. Gobbini, M. L. Furey, A. Ishai, J. L. Schouten, and P. Pietrini, “Distributed and
Overlapping Representations of Faces and Objects in Ventral Temporal Cortex,” Science (80-. )., vol. 293,
no. September, pp. 2425–2430, 2001.
[121] K. Grill-spector, Z. Kourtzi, and N. Kanwisher, “The lateral occipital complex and its role in object
recognition,” Vision Res., vol. 41, pp. 1409–1422, 2001.
[122] G. A. Rousselet, J. S. Husk, P. J. Bennett, and A. B. Sekuler, “Single-trial EEG dynamics of object and face
visual processing,” Neuroimage, vol. 36, no. 3, pp. 843–862, 2007.
[123] T. Sousa, C. Amaral, J. Andrade, G. Pires, U. Nunes, and M. Castelo-Branco, “Pure visual imagery as a
potential approach to achieve three classes of control for implementation of BCI in non-motor disorders,” J.
Neural Eng., vol. 14, 2017.
[124] T. A. Rihs, C. M. Michel, and G. Thut, “Mechanisms of selective inhibition in visual spatial attention are
indexed by alpha -band EEG synchronization Mechanisms of selective inhibition in visual spatial attention
are indexed by a -band EEG synchronization,” Eur. J. Neurosci., no. March 2014, 2007.
[125] M. J. Gray, H. Frey, T. J. Wilson, and J. J. Foxe, “Oscillatory Recruitment of Bilateral Visual Cortex during
Spatial Attention to Competing Rhythmic Inputs,” J. Neurosci., vol. 35, no. 14, pp. 5489–5503, 2015.
[126] R. Chavarriaga, X. Perrin, R. Siegwart, and R. Mill, “Anticipation- and Error-related EEG Signals during
Realistic Human-Machine interaction : A Study on Visual and Tactile Feedback,” Eng. Med. Biol. Soc.
Annu. Int. Conf. IEEE, vol. 1, pp. 1–4, 2012.
[127] F. Perrin, J. Pernier, O. Bertrand, and J. F. Echallier, “Spherical splines for scalp potential and current
density mapping,” Electroencephalogr. Clin. Neurophysiol., vol. 72, no. 2, pp. 184–187, 1989.
[128] C. Sannelli, T. Dickhaus, S. Halder, E. H. K. Mu, and B. Blankertz, “On Optimal Channel Configurations
for SMR-based Brain – Computer Interfaces,” pp. 186–193, 2010.
57
[129] C. Sannelli, C. Vidaurre, K. R. Müller, and B. Blankertz, “Common spatial pattern patches - An optimized
filter ensemble for adaptive Brain-Computer Interfaces,” 2010 Annu. Int. Conf. IEEE Eng. Med. Biol. Soc.
EMBC’10, pp. 4351–4354, 2010.
[130] H. Zou and T. Hastie, “Regularization and variable selection via the elastic net,” J. R. Stat. Soc. Ser. B
(Statistical Methodol., vol. 67, pp. 301–320, 2005.
[131] A. A. González-garrido, F. R. Gómez-velázquez, H. Sequeira, and J. Ramos-loyo, “Gender Differences in
Visuospatial Working Memory — Does Emotion Matter ?,” Int. J. Psychol. Stud., vol. 5, no. 1, 2013.
[132] T. Thiery, T. Lajnef, K. Jerbi, M. Arguin, M. Aubin, and P. Jolicoeur, “Decoding the locus of covert
visuospatial attention from EEG signals,” PLoS One, vol. 11, no. 8, pp. 1–17, 2016.
[133] L. Koessler, L. Maillard, A. Benhadid, J. P. Vignal, J. Felblinger, H. Vespignani, and M. Braun, “Automated
cortical projection of EEG sensors: Anatomical correlation via the international 10-10 system,” Neuroimage,
vol. 46, no. 1, pp. 64–72, 2009.
[134] A. Ossadtchi, T. Shamaeva, E. Okorokova, and V. Moiseeva, “Neurofeedback learning modifies the
incidence rate of alpha spindles , but not their duration and amplitude,” Nature, vol. 7, no. October 2016, pp.
1–12, 2017.
[135] T. R. Marshall, J. O. Shea, X. O. Jensen, and T. O. Bergmann, “Frontal Eye Fields Control Attentional
Modulation of Alpha and Gamma Oscillations in Contralateral Occipitoparietal Cortex,” vol. 35, no. 4, pp.
1638–1647, 2015.
[136] X. J. Daume, T. Gruber, A. K. Engel, and U. Friese, “Phase-Amplitude Coupling and Long-Range Phase
Synchronization Reveal Frontotemporal Interactions during Visual Working Memory,” J. Neurosci., vol. 37,
no. 2, pp. 313–322, 2017.
[137] C. M. Hamamé, J. R. Vidal, T. Ossandón, K. Jerbi, S. S. Dalal, L. Minotti, O. Bertrand, P. Kahane, and J.
Lachaux, “NeuroImage Reading the minds eye : Online detection of visuo-spatial working memory and
visual imagery in the inferior temporal lobe,” Neuroimage, vol. 59, no. 1, pp. 872–879, 2012.
58
Appendices
Appendix A: Necessity of noise attenuation prior to spatial filtering
The mean accuracies of UL vs. LR and LL vs. UR with and without noise attenuation and optimized spatial filters
are presented in Figures 26 and 27. The mean accuracies for no noise and no spatial filter was 57.1±10.1% for UL
vs. LR and 56.0±8.9% for LL vs. UR. Similarly, implementation of noise attenuation resulted in an accuracy of
57.2±10.4% in the UL vs. LR classification problem, and 55.8±8.7% in the LL vs. UR classification problem. Both
accuracies were found to be non-significantly different from accuracies without noise attenuation. Conversely, use
of the CSD spatial filter with noise attenuation resulted in significant increases to 65.2±7.5% (p = 0.001) and
65.0±7.2% for UL vs. LR and LL vs. UR (p<0.001), respectively. However, use of the same optimized spatial filters
without noise attenuation significantly reduced accuracies to 58.5±10.3% for UL vs. LR (p<0.001) and 57.5±9.0%
for LL vs. UR (p<0.001).
Figure 26 UL vs. LR imagery classification accuracies with and without noise attenuation and optimized spatial filters
Figure 27 LL vs. UR imagery classification accuracies with and without noise attenuation and optimized spatial filters
59
Appendix B: Offline Session Feedback Questionnaire 1. How tired were you before you began this session?
Not tired Somewhat tired Very tired
1 2 3 4 5
2. How tired are you after completing this session?
Not tired Somewhat tired Very tired
1 2 3 4 5
3. How hard did you find it to focus during the trials?
Very easy Neither easy nor hard Very hard
1 2 3 4 5
4. Do you feel that you could not focus as much by the end of the session?
No Somewhat Yes
1 2 3 4 5
5. Was the EEG cap comfortable?
No Somewhat Yes
1 2 3 4 5
6. Did you have fun?
No Somewhat Yes
1 2 3 4 5
7. How easy was it to perform the visual mental imagery task?
Very easy Neither easy nor hard Very hard
1 2 3 4 5
8. How often were you able to picture the arrow on the screen?
Very rarely Sometimes Very Often
1 2 3 4 5
9. Aside from when you were told to, how often did you move your eyes away from the
cross?
Very often Sometimes Never
1 2 3 4 5
60
10. How satisfied are you with your performance on the task?
Not at all Somewhat Very
1 2 3 4 5
11. How often did you get frustrated with this task?
Very rarely Sometimes Very Often
1 2 3 4 5
Thank you for your participation. You may provide any additional comments below:
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
61
Appendix C: Online Session Feedback Questionnaire 1. How tired were you before you began this session?
Not tired Somewhat tired Very tired
1 2 3 4 5
2. How tired are you after completing this session?
Not tired Somewhat tired Very tired
2 2 3 4 5
3. How hard did you find it to focus during the trials?
Very easy Neither easy nor hard Very hard
1 2 3 4 5
4. Do you feel that you could not focus as much by the end of the session?
No Somewhat Yes
1 2 3 4 5
5. Was the EEG cap comfortable?
No Somewhat Yes
1 2 3 4 5
6. Did you have fun?
No Somewhat Yes
2 2 3 4 5
7. How easy was it to perform the visual mental imagery task?
Very easy Neither easy nor hard Very hard
1 2 3 4 5
8. How often were you able to picture the arrow on the screen?
Very rarely Sometimes Very Often
1 2 3 4 5
62
9. Aside from when you were told to, how often did you move your eyes away from the
cross?
Very often Sometimes Never
1 2 3 4 5
10. How satisfied are you with your performance on the task?
Not at all Somewhat Very
1 2 3 4 5
11. How often did you get frustrated with this task?
Very rarely Sometimes Very Often
1 2 3 4 5
12. Was the feedback helpful?
Not helpful Somewhat Very helpful
1 2 3 4 5
13. Do you think the feedback accurately showed the effort you put in?
Not accurate Somewhat accurately Very accurately
1 2 3 4 5
14. Did you find the feedback frustrating?
Not frustrating Somewhat frustrating Very frustrating
1 2 3 4 5
Thank you for your participation. You may provide any additional comments below:
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
![SSVEP based Bremen-BCI { boosting information transfer rates · BCI systems which are based on various sensor modalities [6]. In non-invasive BCIs, electroencephalography (EEG) is](https://static.documents.pub/doc/80x56/606641a202508851e87b5167/ssvep-based-bremen-bci-boosting-information-transfer-bci-systems-which-are-based.jpg)
