+ All Categories
Home > Documents > A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can...

A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can...

Date post: 19-Aug-2020
Category:
Upload: others
View: 0 times
Download: 0 times
Share this document with a friend
73
A Novel EEG-BCI Dependent on Discrimination of Imagined Stimuli in Visual Field Quadrants by Filip Stojic A thesis submitted in conformity with the requirements for the degree of Master of Applied Science Institute of Biomaterial and Biomedical Engineering University of Toronto © Copyright by Filip Stojic 2017
Transcript
Page 1: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

A Novel EEG-BCI Dependent on Discrimination of Imagined Stimuli in Visual Field Quadrants

by

Filip Stojic

A thesis submitted in conformity with the requirements for the degree of Master of Applied Science

Institute of Biomaterial and Biomedical Engineering University of Toronto

© Copyright by Filip Stojic 2017

Page 2: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

ii

A Novel EEG-BCI Dependent on Discrimination of Imagined

Stimuli in Visual Field Quadrants

Filip Stojic

Master of Applied Science

Institute of Biomaterials and Biomedical Engineering

University of Toronto

2017

Abstract

Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication.

This study aimed to determine whether visuospatial imagery could be used to signify intent in an

electroencephalography (EEG)-based BCI. Eighteen healthy participants imagined stimuli in visual field quadrants

while EEG was collected. A subset of participants used visuospatial imagery to control a character’s movement in

four directions. Classifying rest against non-specific visuospatial imagery attained accuracies of 77±11.4%. Six

participants exceeded chance when discriminating imagery in diagonally-opposing quadrants. These six could be

predicted using a regression model that combined visuospatial perception and fatigue scores. The 4-class navigation

accuracy was 48.3±18.8% (max 85%). Imagery in one hemifield corresponded to significant increases in alpha

spindles in ipsilateral visual cortical regions. While further improvement is necessary to make the paradigm

generalizable, visuospatial imagery shows promise for becoming a useful system for individuals in need of a BCI

access technology.

Page 3: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

iii

Acknowledgments

I would like to thank Dr. Tom Chau for his unrelenting support and guidance in my pursuit of this thesis. Thank

you for teaching me so much, and for always encouraging my ambitions. If I ever learn to be half as good a mentor

as you, I will have achieved the impossible. I am grateful beyond words.

Thank you to my committee members, Drs. Adrian Nestor, Deryk Beal and Cesar Marquez-Chin. You were there to

help me strive to think critically and in new ways, and have helped to shape me as a scientist.

My gratitude is also extended to the PRISM lab, in particular to Ka Lun Tam and Pierre Duez, who were always

there in a pinch with brilliance.

Thank you to my parents, whose endless interest helped me believe I was doing something worthwhile. Thank you

twice for always being there for me and for supporting my dreams.

Finally, I would like to express my immense gratitude for the generosity of the Kimel Family for their Scholarship

in Pediatric Disability, the Faculty of Engineering for their graduate student award and the Bloorview Research

Institute.

Page 4: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

iv

Table of Contents

Acknowledgments .................................................................................................................. iii

Table of Contents .....................................................................................................................iv

List of Tables ......................................................................................................................... vii

List of Figures ....................................................................................................................... viii

List of Abbreviations ................................................................................................................ x

Chapter 1 ................................................................................................................................... 1

Introduction .......................................................................................................................... 1

1.1 Motivation..................................................................................................................... 1

1.2 Question, hypothesis and objectives ............................................................................. 1

1.3 Thesis outline ................................................................................................................ 2

Chapter 2 ................................................................................................................................... 3

Background and Literature Review ..................................................................................... 3

2.1 Brain computer interfaces (BCIs) ................................................................................. 3

2.1.1 Data Acquisition ............................................................................................... 3

2.1.2 Signal processing and analysis ......................................................................... 4

2.1.3 Feedback ........................................................................................................... 5

2.1.4 Types of BCIs ................................................................................................... 5

2.2 Visual Processing Pathway ........................................................................................... 6

2.2.1 Regions of interest ............................................................................................ 6

2.2.2 Perception, imagery, memory and attention ..................................................... 7

2.2.3 Visual tasks ....................................................................................................... 7

Chapter 3 ................................................................................................................................... 9

Development and assessment of a novel visuospatial imagery-based EEG-BCI ................ 9

3.1 Abstract ......................................................................................................................... 9

Page 5: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

v

3.2 Introduction................................................................................................................... 9

3.2.1 BCI paradigms ................................................................................................ 10

3.3 Methods ...................................................................................................................... 12

3.3.1 Participants ..................................................................................................... 12

3.3.2 Instrumentation ............................................................................................... 12

3.3.3 Experimental protocol .................................................................................... 13

3.3.4 The visual imagery stimulus ........................................................................... 17

3.3.5 BCI pipeline .................................................................................................... 18

3.3.6 Online ............................................................................................................. 18

3.3.7 Offline analysis and online navigation game.................................................. 19

3.4 Results ........................................................................................................................ 24

3.4.1 Chance levels .................................................................................................. 24

3.4.2 Classifying perception and imagery against resting mental state ................... 24

3.4.3 Classifying quadrants in perception and imagery ........................................... 27

3.4.4 Features characteristic of visuospatial perception and imagery ..................... 30

3.4.5 Multiclassification of visuospatial imagery .................................................... 31

3.4.6 Predicting classification accuracies ................................................................ 32

3.4.7 Characterizing EEG features indicative of diverse imagery mental states ..... 35

3.4.8 Summary of key findings................................................................................ 36

3.5 Discussion ................................................................................................................... 37

3.5.1 Visuospatial perception classification ............................................................ 37

3.5.2 Visuospatial imagery classification ................................................................ 38

3.5.3 Similarities between perception and imagery ................................................. 40

3.5.4 Multiclassification .......................................................................................... 41

3.5.5 Predictors of accuracy..................................................................................... 41

3.5.6 Differences in lateralization of alpha .............................................................. 43

Page 6: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

vi

3.5.7 Potential sources of variability ....................................................................... 44

3.5.8 Potential solutions to the variability ............................................................... 45

3.5.9 Key messages.................................................................................................. 46

3.6 Conclusion .................................................................................................................. 47

Chapter 4 Conclusion ............................................................................................................. 48

Conclusion ......................................................................................................................... 48

4.1 Contributions .............................................................................................................. 48

4.2 Future Work ................................................................................................................ 48

References............................................................................................................................... 49

Appendices ............................................................................................................................. 58

Page 7: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

vii

List of Tables

Table 1 BCI experimental protocol for each data collection session .......................................... 13

Table 2 Confusion matrix for all participant data from sessions 4 and 5 .................................... 32

Table 3 Spearman correlation table for imagery versus rest classification accuracy .................. 33

Table 4 Spearman correlation table for imagery versus imagery ................................................ 33

Table 5 Group differences between males and females using the Wilcoxon rank sum test ........ 35

Table 6 Most significant channel pair differences in alpha spindles using the Wilcoxon rank-

sum test ........................................................................................................................................ 36

Page 8: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

viii

List of Figures

Figure 1 Example Attention Condition trial (UL) ....................................................................... 14

Figure 2 Example Auxiliary Condition trial (UR) ....................................................................... 14

Figure 3 Example Perception condition trial (UL) ...................................................................... 14

Figure 4 Example Imagery Condition trial (LR) ......................................................................... 15

Figure 5 Example Visual Feedback Condition trial (UL), with annotated probabilities (not

presented to participants) ............................................................................................................. 16

Figure 6 Example Navigation Game trial (UL) with exaggerated fixation crosses .................... 17

Figure 7 Schematic of Navigation Game trial classification with two tiers of classifiers ........... 24

Figure 8 Classification accuracies for perception versus rest with user-optimized classifier ..... 25

Figure 9 Offline classification accuracies of rest versus any imagery with user-optimized

classifiers ..................................................................................................................................... 25

Figure 10 Classification of online trials....................................................................................... 26

Figure 11 Frequency of feature group selection by elastic net regularization for perception (left)

and imagery (right) vs. rest classification .................................................................................... 27

Figure 12 Frequency of feature group selection by elastic net regularization for perception

compared to imagery (left) and perception compared to imagery with selection adjusted for

number of sub-features (right) ..................................................................................................... 27

Figure 13 Frequency of optimal CSD spatial filter parameters: spline flexibility (left); smoothing

factor (middle); Legendre polynomial order (right) .................................................................... 28

Figure 14 Classification accuracies of perceiving stimuli in UL vs. LR and LL vs. UR quadrants

with optimized spatial filter parameters ...................................................................................... 28

Figure 15 Frequency of optimal CSD parameters: spline flexibility (left), smoothing facto

exponent (middle), Legendre polynomial (right) ........................................................................ 29

Page 9: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

ix

Figure 16 Imagery versus imagery classification accuracies with optimized spatial filter

parameters and classifiers ............................................................................................................ 29

Figure 17 Frequency of selected feature groups in classifying different types of perception

sorted by participant .................................................................................................................... 30

Figure 18 Grand average frequency of selected feature groups in classifying perception in

diagonally-opposing quadrants .................................................................................................... 30

Figure 19 Frequency of selected feature groups in classifying different types of imagery sorted

by participant ............................................................................................................................... 31

Figure 20 Grand average frequency of selected feature groups in classifying imagery in

diagonally-opposing quadrants .................................................................................................... 31

Figure 21 Four-class classification accuracies in offline (direct and two-tier) and online

approaches ................................................................................................................................... 32

Figure 22 Linear model combining pre-session exhaustion with perception classification

accuracies to predict imagery classification accuracies ............................................................... 34

Figure 23 Linear model of perception classification accuracies predicting imagery classification

accuracies ..................................................................................................................................... 34

Figure 24 Alpha spindle differences (left-right) in channel pairs across participants and imagery

quadrants ...................................................................................................................................... 35

Figure 25 Average difference in time-frequency power for PO3 minus PO4 channel pair in

participant 17 for visuospatial imagery in the left hemifield (left) and right hemifield (right);

color bar represents power/frequency (dB/Hz)............................................................................ 36

Figure 26 UL vs. LR imagery classification accuracies with and without noise attenuation and

optimized spatial filters................................................................................................................ 58

Figure 27 LL vs. UR imagery classification accuracies with and without noise attenuation and

optimized spatial filters................................................................................................................ 58

Page 10: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

x

List of Abbreviations

ALS Amyotrophic Lateral Sclerosis

BCI Brain Computer Interface

CPSD Cross Power Spectral Density

CRLS Conventional Recursive Least-Squares

CSD Current Source Density

CVSA Covert Visuospatial Attention

ECoG Electrocorticography

EEG Electroencephalography

EOG Electro-oculography

FCBF Fast Correlation-Based Filter

fMRI functional Magnetic Resonance Imaging

fNIRS functional Near-Infrared Spectroscopy

FWHM Full Width at Half Maximum

ITR Information Transfer Rate

KNN K-Nearest Neighbours

LDA Linear Discriminant Analysis

LGN Lateral Geniculate Nucleus

LL Lower Left

LR Lower Right

MEG Magnetencephalography

MS Magnitude-squared

PSD Power Spectral Density

Page 11: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

xi

RBF Radial Basis Function

RMSE Root Mean Squared Error

ROCF Rey-Osterrieth Complex Figure

SFFS Sequential Forward Floating Search

SMR Sensorimotor Rhythms

SNR Signal-to-Noise Ratio

SSVEP Steady State Visually-Evoked Response Potential

STFT Short-Time Fourier Transform

SVM Support Vector Machines

UL Upper Left

UR Upper Right

VVIQ2 Vividness of Visual Imagery Questionnaire

Page 12: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

1

Chapter 1

Introduction

1.1 Motivation

Many individuals with conditions such as stroke, cerebral palsy, spinal cord injury or amyotrophic lateral sclerosis

(ALS) live with motor impairments that prevent them from interacting or communicating with their environment.

Individuals who present as locked-in may be fully paralyzed, but can still be conscious of events around them [1].

For these individuals, brain-computer interfaces (BCIs) may restore useful function by providing an alternative

means of communication not dependent upon voluntary verbalization or motor activations [2]. In BCI studies, user

intent, which is necessary for communication, is identified by detecting neural activity elicited from different

mental tasks. While many of these tasks have been useful in healthy populations, relying on these tasks in

populations with motor impairments poses problems. For instance, tasks that require participants with ALS to

sustain attention to a computer screen have been found to induce fatigue and interfere with BCI control [3]–[6].

Additionally, BCIs that depend on visual attention require participants to voluntarily control their eye muscles,

which may be difficult in patients with more severe motor disorders [7]. Other tasks such as mentally listing words

starting with a certain letter (“verbal fluency”) or sequentially subtracting numbers (“mental arithmetic”) are

cognitively demanding, and may not be intuitive to users [8]–[10]. Tasks such as the motor imagery task

(imagining movement of a particular body part) may be more intuitive, and have been shown to reduce fatigue in

patients with disorders such as ALS [11]. However, motor imagery can also be impaired in individuals with motor

disorders [12], [13]. Ideally, tasks used to identify communicative intent in populations with motor disorders should

be intuitive, should not induce a great degree of fatigue and should not rely on potentially impaired motor neural

pathways. Visuospatial imagery (imagining stimuli in the visual field) does not depend on motor circuits or on eye

gaze and may thus be a promising task. Like its motor analog, visuospatial imagery has an intuitive appeal; for

instance, imagining an arrow in the upper right quadrant of the visual field may indicate a desire to more forward

and to the right in a motorized wheelchair. Practical BCI use in the real world necessitates the development of a

task sensitive to the needs of the user. This study aims to determine the feasibility of one such task – mental

imagery in the four quadrants of the visual field.

1.2 Question, hypothesis and objectives

Question: Which approaches can be used to effectively identify visuospatial imagery in visual field quadrants as

commands in an EEG-BCI?

Hypothesis: Mental visual imagery in the four visual quadrants can be discriminated with above-chance accuracies

for binary and multi-class problems, with both offline and online EEG-BCI paradigms, in a healthy control

population.

Page 13: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

2

Objectives:

1. To determine the accuracies with which different classes of mental states can be discriminated in offline

and online EEG-BCI paradigms dependent on visuospatial imagery

2. To evaluate the predictive quality and influencing nature of external factors on performance in a

visuospatial imagery-based EEG-BCI

3. To characterize EEG activity indicative of visuospatial imagery as well as its relationship to visuospatial

perception

1.3 Thesis outline

Chapter 1 introduced the motivation for this thesis and outlined the research hypothesis, questions and objectives.

Chapter 2 provides a literature review on brain-computer interfaces and their components, the various mental task

paradigms within BCIs, as well as the visual system and how it might be exploited for use in a BCI. Subsequent to

this is a chapter on the development of a BCI dependent on visuospatial imagery. Finally, Chapter 4 recapitulates

the contributions of this thesis, as well as potential directions for future studies.

Page 14: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

3

Chapter 2

Background and Literature Review

2.1 Brain computer interfaces (BCIs)

BCIs enable users to interact and communicate with their environment through cognitive activity alone [14]. They

consist of an input or data acquisition method (where a change in brain signal is evoked and detected), signal

processing (where brain signals are processed and analyzed before being translated into commands), and an output

(where detected commands cause some external change) that may be used as feedback [14].

2.1.1 Data Acquisition

Multiple modalities exist for acquiring brain signals of interest, including but not limited to, functional magnetic

resonance imaging (fMRI), functional near infrared spectroscopy (fNIRS), magnetencephalography (MEG),

electrocorticography (ECoG) and electroencephalography (EEG). Each modality has its strengths and weaknesses,

among them portability, cost, invasiveness, ease of setup, and spatial and temporal resolution. The most popular

modality used in BCI studies is EEG, discussed below.

Electroencephalography

Many BCIs use electroencephalography (EEG) to acquire brain signals due to its non-invasiveness, low cost, and

ease of setup [15]. The EEG modality works by detecting changes in synchronized electrical activity from

populations of neurons in the cortex that travel through the meninges, skull, and scalp [2]. Typically, this signal is

very small and requires amplification prior to digitization [15]. Because EEG relies on electrophysiological brain

activity, signal detection is fast, on the order of milliseconds, in contrast to other modalities such as functional near

infrared spectroscopy (fNIRS), which relies on slower hemodynamic responses [15]. However, despite its high

temporal resolution, EEG has relatively low spatial resolution, as a result of “noisy” electrical activity from the

scalp and surrounding brain areas [16], as well as artifacts from power lines (60 Hz), and myogenic noise (e.g. eye

movements) [2]. Spatial resolution can be improved somewhat with an increased number of electrodes, but this

requires increased setup time and the result remains inferior to that of modalities such as fMRI. While ECoG has

both the high temporal resolution of EEG and spatial resolution of fMRI, its invasiveness makes it impractical for

BCI studies due to the placement of electrodes directly on the cortex [15], [17]. Use of a dry EEG system

considerably reduces setup time and removes the need of electrode gel. While improving the practicality and

generalizability of BCI systems in clinical settings, dry EEG electrodes have previously been shown to yield lower

signal-to-noise ratios (SNRs) than those of standard wet electrodes [18], [19].

Page 15: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

4

2.1.2 Signal processing and analysis

Preprocessing

Upon acquisition of brain signal, some preprocessing may be necessary in order to extract useful features. This may

include downsampling, where the frequency at which the data were initially collected (e.g. 1000 Hz) is reduced to a

lower sampling rate (e.g. 256 Hz), in part to reduce subsequent processing times. Additionally, filters may be

applied to the data to limit the content of the data to a meaningful range. For instance, bandpass filters between 1

and 50 Hz have been used for mental imagery paradigms that depend on beta oscillations (13-30 Hz) [20]. Filters

can also be used to remove specific artefacts such as those from power lines, or muscle activity. The preprocessing

step also includes removal of ocular artifacts (e.g. eye blinks), through methods such as independent component

analysis (ICA) [21] and recursive least-squares regression [22].

Feature Extraction

In feature extraction, characteristic components of the signal are sought to identify specific brain events that

correspond to user commands [14], [15]. These features may be time-dependent, such as signal amplitudes at

specific time points (e.g. P300 positive potential at 300ms post-stimulus presentation [23]), or frequency-

dependent, such as increased amplitude in beta oscillations during motor imagery [14]. Feature engineering is the

process of developing and identifying features such as these, in order to best characterize the data [24].

Classification

Next, characteristic signal features need to be translated into user commands. This can be done by training

classification algorithms to automatically differentiate user intents [25]. For instance, in binary-class problems, one

mental task (e.g. rest) corresponding to a specific command (e.g. “No”) may be characterized by high alpha

frequency band power, whereas the other task (e.g. motor imagery) corresponding to another command (e.g. “Yes”)

may be characterized by high beta frequency band power. Classification algorithms can identify these patterns of

brain activity [26]. Different algorithms for classification exist, the most popular of which include classifiers such

as linear discriminant analysis (LDA) and support vector machines (SVM) [25]. It is also during this step that the

most useful features may be selected through feature selection methods such as fast correlation based filter (FCBF)

feature selection and sequential forward floating search (SFFS) feature selection [27], [28]. Feature selection is

useful as it maximizes the classification performance while decreasing the number of terms used by only selecting

the most relevant features, thereby creating a simple and generalizable model [15].

For offline paradigms, cross-validation methods may be used in order to determine classification accuracies in a

way that reduces classifier bias (e.g. from over-fitting the data) [15]. In k-fold cross-validation, data are split into k

groups consisting of k-1 training sets and a testing set. For example, a 5-fold cross-validation will randomly split a

sample of 100 trials into groups of 20. Each of those groups will have an equal number of samples from each class.

Page 16: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

5

In a given fold, four of the groups (total n = 80) will serve as training data for the classifier, and one group will

serve as new testing data. Cross-validation is typically repeated multiple times using randomly selected partitions of

the data.

2.1.3 Feedback

Classified output can vary in type, but in many BCI studies, it is presented visually on a computer screen and serves

as neurofeedback for the user [14]. For instance, a BCI may discriminate mental arithmetic from motor imagery,

and if it does so successfully, a checkmark may be displayed to the user. Output can also be continuous, in an effort

to provide real-time feedback to the user. Feedback may reinforce certain mental strategies that yield signals

conducive to class discrimination in machine learning, while deterring strategies that are not [9]. Feedback may

also help to keep users engaged and maintain concentration [29]. However, feedback is only possible in online BCI

paradigms, where already-trained classifiers are fed real-time data.

2.1.4 Types of BCIs

Active and reactive

Paradigms in BCI studies that use brain activity for communication fall under two broad categories: “reactive”

BCIs rely on involuntary (or modulated) brain activity elicited upon external stimulation (e.g. steady-state visually

evoked potentials - SSVEPs); “active” BCIs rely on brain activity elicited when the user voluntarily and

consciously performs a cognitive task (e.g. mental arithmetic, verbal fluency, motor imagery) [30]. In active BCIs,

different mental tasks may elicit different brain activities which in turn can be distinguished as unique

communicative intents – for instance, mental arithmetic may be classified as intent to move a cursor leftward on a

computer screen whereas verbal fluency may be classified as intent to move a cursor rightward. Reactive BCIs may

function by instructing the user to pay attention to a particular word played from a speaker, such as “Yes” or “No”,

and when the target word is heard, characteristic brain signals can be identified (e.g. such as the P300 response)

[31]. While active BCIs tend to require some training – such as the modulation of sensorimotor rhythms (SMR)

during imagined limb movements [2] – reactive BCIs need none. This is because reactive BCIs rely on evoked

potentials; involuntary changes in EEG signal amplitude resulting from presentation of stimuli in the environment.

Conversely, whereas most reactive BCIs are limited to synchronous control, active BCIs have the potential to be

used in an asynchronous paradigm, as discussed below.

Binary versus multiclass paradigms

Many BCIs to date have focused on using binary class paradigms. For instance, listing words starting with a

particular letter (verbal fluency task) may be classified against a mental rotation task to convey two different

commands (e.g. move cursor left/right). Increasing the number of classes may increase the rate at which

information can be conveyed by providing greater combinations of commands. However, increasing the number of

discriminable classes often means relying on seemingly unrelated mental tasks, both to one another, and to the

desired output. This may increase length of the training period for a user prior to which the BCI can become

Page 17: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

6

effective. Furthermore, classification accuracy also tends to decrease with increasing output categories, thereby

limiting information transfer rates (ITRs). Ideally, a practical and useful BCI should have a high information

transfer rate (ITR) as well as a high classification accuracy.

Synchronous versus asynchronous paradigms

Synchronous control refers to BCI paradigms where mental tasks are constrained to specifically-timed cues,

whereas asynchronous control refers to BCIs where mental tasks can be initiated at any time by the user [32]. For

instance, in a synchronous BCI, a user may be cued to begin mental arithmetic, or cued to rest, and a classifier may

be able to correctly discriminate the two mental states that are restricted to specific time intervals. Conversely, an

asynchronous BCI can continuously detect which mental state the user is in and a change in mental state at any time

point should be detected by the classifier. The latter offers the advantage of executing user-paced commands.

2.2 Visual Processing Pathway

2.2.1 Regions of interest

The typical bottom-up pathway through which vision is processed begins at the retina. Changes in the visual field

cause depolarizations in photoreceptors, which send this information through the optic nerve to the lateral

geniculate nucleus (LGN) in the thalamus (which receives approximately 90% of the output) and the superior

colliculus in the midbrain (which receives 10% of the output) [33], [34]. The LGN then relays this activity to the

primary visual or “striate” cortex (V1 – Brodmann area 17). As the visual information continues up the hierarchy

from V1 to extrastriate areas V2, V3, V4, V5 and V6 (Brodmann areas 18 and 19), more and more complex

features are analyzed. For instance, V1 neurons respond to a particular orientation of a line, and neurons in V5

respond to motion [33], [34]. Importantly however, each visual cortical region possesses a topographic map of the

visual field (deemed “retinotopy”), such that stimuli in specific regions of the visual field will activate specific

neuronal populations in the visual cortex, and stimuli in adjacent regions will activate other unique neuronal

populations [33], [34]. The organization is most simple in V1, where the upper half of the visual field corresponds

to populations of neurons below the calcarine sulcus, and the lower half activates neurons above the calcarine

sulcus [33], [34]. In a similar manner, the left cortical hemisphere of V1 will depolarize in response to stimuli in the

right visual hemifield, and vice-versa. Interestingly, a bias is typically present in the lower hemifield that results in

better visual performance than in the upper hemifield [35]. Measurement of increased signal amplitudes from the

lower visual hemifield are owed in part to the proximity of lower field cortical sources to sensors on the scalp [36].

As with the increasing complexity of visual features, the retinotopic map of the visual field is altered higher up the

visual processing hierarchy, such that adjacent locations in the visual field may not correspond to adjacent locations

in the cortex [37]. Regardless, discrete locations of the visual field have unique physical representations in the

visual cortex that are detectable, and may be used in a BCI paradigm.

Page 18: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

7

2.2.2 Perception, imagery, memory and attention

While there was a long-standing debate over whether only bottom-up processes such as perception can activate

lower visual areas such as V1, many studies have shown that top-down processes such as mental imagery or

memory also do the same (for review see [38]). It has been suggested that visual mental imagery functions as a

weaker form of perception [39]. In fact, an fMRI study by Slotnick and colleagues, discovered that mental imagery

of stimuli in various parts of the visual field activate regions of the visual cortex that are very similar to those of

perception [40]. Importantly, attention to the same locations of the visual field also activated similar areas of the

visual cortex, however, these were much less similar in pattern to perception than those elicited by imagery. Further

evidence comes from Ganis and colleagues, who also found similar cortical activation patterns between imagery

and perception of common objects [41], and Albers and colleagues, who demonstrated that a classifier trained on

perception of variously-oriented gratings could reliably identify orientations of gratings in visual imagery and

working memory in early visual cortical regions [42]. These findings go hand-in-hand with recent theories of

episodic memory, which state that frontal regions tend to activate memories, and subsequently elaborate them by

recruiting sensory areas [43]. Specifically, when asked to relive vivid autobiographical events, the visual cortices of

participants were found to activate, following top-down recruitment from frontal areas. In sum, top-down processes

such as imagery, memory and attention can reliably recruit early visual areas that typically respond to perception.

2.2.3 Visual tasks

Cognitive assessment

There are multiple standardized methods of determining visual cognitive abilities in individuals, two of which are

discussed here. The Rey-Osterrieth Complex Figure (ROCF) is a well-validated method of measuring visuospatial

and visual memory skills in participants [44]. This test relies on reproduction of a complex figure in three different

conditions: (1) Copy – where the participant is allowed to view the figure and is asked to reproduce it on paper to

the best of their ability, (2) Immediate Recall – where the participant is asked to reproduce the figure from memory

immediately after the Copy condition, (3) Delayed Recall – where the participant is asked to reproduce the figure

from memory after a longer delay of 30 minutes to 1 hour. The Vividness of Visual Imagery Questionnaire

(VVIQ2) is a subjective measure of visual imagery [45]. In this test, participants rate various scenarios on a 5-point

scale of how vividly they can see them in eye-open and eyes-closed conditions. The VVIQ2 has been revised and

validated [46]. Correlations between the VVIQ2 and activation in the visual cortex have been reported such that

greater reported vividness corresponded to greater activation in early visual cortical areas [47]. Interestingly, a

number of individuals found to have high visual imagery have also reported low auditory imagery, and vice-versa

[48], a pattern that is further discussed in Chapter 3.

In EEG and BCI studies

Thus far, few EEG-BCI studies have taken advantage of the topographical organization of the visual processing

stream. It is due to this organization that stimuli across the central visual field can be spatially and temporally

discriminated using EEG signals [49]. In fact, it has been determined that the spatial resolution of EEG is adequate

Page 19: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

8

to discriminate between activations in the visual cortex corresponding to stimuli less than 3° apart in the visual field

[50]. This introduces the possibility of using stimuli in multiple areas of the visual field to signify multiple forms of

communicative intent. Despite this opportunity to create multi-class EEG-BCI paradigms, most studies have only

used stimuli in the left and right hemisphere [51], [52]. To our knowledge, no studies have attempted to

discriminate stimuli in all four quadrants of the visual field, although one study has successfully discriminated

perception in left and right quadrants in the upper and lower visual field with single-trial EEG [53]. In order to

reliably localize stimuli to their respective retinotopic locations, it has been found that confining them to the visual

field quadrants results in the most consistent results with fMRI data, at least when using MEG [54].

Importantly, eliciting brain activity that is measurable and differentiable with EEG does not require external visual

stimulus. This is due to the top-down modulation discussed above. It has previously been demonstrated that covert

visual attention to the left and right hemisphere can yield a classification accuracy of approximately 70% (the

standard minimum see [55]) in an online EEG-BCI system without exogenous stimulus [56]. Another study, while

not considering the topographical organization of the visual cortex, has found that discriminating brain activity

from visual mental imagery of faces and houses, and a resting state is possible with EEG. This approach has

resulted in above chance accuracies for binary (64-73%) and 3-class (54%) analyses, and information transfer rates

(ITRs) of 6 to 10 bits/min [57]. While faces and objects are processed by different extrastriate visual areas [58], it is

unlikely that the two will cause notably different activation patterns in earlier visual regions. No study has

attempted to discriminate between imagined stimuli in different areas of the visual field, despite the finding that

fMRI patterns of cortical activity elicited by imagining stimuli in various parts of the visual field are very similar to

the pattern of activity elicited by perceiving actual stimuli in the same area [40], [59]. Although this finding was

only evident in half (i.e. 3/6 of) their participants, it may have been in part due to individual differences in cognitive

ability to visualize stimuli. By taking advantage of visual mental imagery and topography of the visual cortex, it

may be possible to develop a multi-class non-invasive EEG-BCI that can discriminate between imagined stimuli in

visual field quadrants, at least in individuals with good top-down visual cognitive skills.

Page 20: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

9

Chapter 3

Development and assessment of a novel visuospatial imagery-based EEG-BCI

3.1 Abstract

Brain computer interfaces (BCIs) can provide those living with severe motor disorders a means of communication

not dependent on motor actions or verbalization. By measuring brain activity elicited from different mental tasks,

communicative intent can be identified in these individuals. Brain activity during visual mental imagery has been

shown to be detectable and does not rely on potentially compromised neural pathways. This study aimed to

determine whether visuospatial imagery could be used to signify intent in both an offline and online

electroencephalography (EEG)-based BCI. Over the course of 3 sessions, 18 participants imagined checkerboard

arrow stimuli in the four quadrants of the visual field while having their brain activity recorded with 16 dry

electrodes over the occipital lobe. At the end of the third session, they performed an online task where they received

visual feedback on whether a BCI was able to detect any imagery or not. A subset of participants continued to the

4th and 5th sessions, where after brief offline retraining, they controlled movement of a character in an online

navigation task by imagining arrow stimuli in different quadrants. Predictors of BCI performance and characteristic

features of visuospatial imagery were further assessed through statistical means. Offline and online classification

accuracies of resting state against non-specific visuospatial imagery reached mean accuracies of 71.2% and 71.7%,

respectively. Mean online accuracies were further improved using more sophisticated signal processing and

features to 77%. When classifying diagonally-opposing quadrants, only six participants exceeded chance. These six

participants could be predicted using a linear regression model that combined scores from perception classification

tasks, and a measure of fatigue. The mean 4-class classification accuracy for Sessions 4 and 5 was 49.2%, with a

maximum of 85%. Finally, there was a significant relative increase in number of alpha spindles in visual cortical

regions contralateral to where visuospatial imagery was occurring. This study is the first to assess a BCI dependent

on visuospatial imagery. Furthermore, this is the first study to date to demonstrate posterior alpha band imbalance

with visuospatial imagery. While non-specific imagery and resting state may be used as practical commands in

binary BCIs, further improvement is necessary to increase the detection and discrimination of visuospatial imagery

in all participants. By improving signal quality, increasing the number of sensors and introducing alpha spindle

feedback in a training paradigm, visuospatial imagery may become a useful BCI control paradigm with potential for

numerous and intuitive commands for more users in need of BCI access technology.

3.2 Introduction

Many individuals with conditions such as stroke, cerebral palsy, spinal cord injury or amyotrophic lateral

sclerosis (ALS) live with motor impairments that prevent them from interacting or communicating with their

environment. Individuals who present as locked-in may be fully paralyzed, but can still be conscious of events

around them [1]. For these individuals, brain computer interfaces (BCIs) may restore useful function by providing

an alternative means of communication not dependent upon voluntary verbalization or motor activations [60]. By

Page 21: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

10

detecting specific brain signals associated with different mental tasks, BCIs can identify and translate

communicative intent into commands such as moving a wheelchair, selecting options on a screen or enunciating

pre-selected words from a speaker. In other words, BCIs enable users to interact and communicate with their

environment through cognitive activity alone [61]. While BCIs possess great potential as assistive technologies,

the paradigms in which they have proven successful for non-disabled users have demonstrated limited translation to

target users.

3.2.1 BCI paradigms

There are a variety of BCI paradigms that each present with their own advantages and disadvantages for

target users. These can be broadly categorized into three groups: reactive, active or passive [30]. Briefly, reactive

BCIs rely on the detection of brain signals that result as a direct response of external auditory, tactile, or visual

stimulus presentation, and which are indirectly modulated by the user. Conversely, active BCIs rely on brain

signals that occur without stimulus presentation, but rather, are consciously and internally driven by the user.

Finally, passive BCIs do not require any specific mental task or stimulus response, but rather monitor the changes

in baseline brain activity over time – such as those resulting from fatigue – in order to enrich human-computer

interaction [30], [62].

One example of a reactive BCI is the well-established P300 BCI, in which individuals focus on detecting

the presentation of a specific stimulus in the environment, while ignoring irrelevant stimuli [23]. Upon detecting the

stimulus of interest, there is a characteristic change in signal amplitude that can be detected with

electroencephalography (EEG) approximately 200 to 700ms post-stimulus presentation [63]. Reactive BCIs such as

these have high accuracy rates and require little to no training [64]. However, they also require sustained attention

to external visual, tactile or auditory stimuli – which may prove difficult to individuals with motor impairments.

For instance, individuals with motor impairments may lack adequate gaze control in a visual P300 paradigm [6].

Additionally, it has been demonstrated that paradigms that require sustained attention can be fatiguing for the target

population [3]. Finally, because the P300 BCI is reactive, it can only be used in a synchronous paradigm. In other

words, the BCI constrains the user’s task performance to a specific time interval during which commands are

admissible, rather than affording user’s the freedom to generate commands at any point in time, as in asynchronous

BCIs.

Active BCIs rely on the detection of mental tasks that can themselves be categorized into two types of

cognition: executive function and sensory imagery. The tasks that rely mainly on executive function such as mental

arithmetic or verbal fluency (listing words that belong to a category) result in easily detectable and distinguishable

cortical signals. As a main source of this activity is located in the prefrontal cortex [65], the signals can be detected

through the forehead without introducing sources of noise contributed by hair follicles. Furthermore, since these

executive function tasks are independent of external stimulation, they may be used in asynchronous paradigms, and

can therefore facilitate user-paced control. However, the translation of these tasks into commands can be unintuitive

Page 22: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

11

– for instance, listing types of fruit (as in the semantic fluency task) has no obvious relation to desired wheelchair

movement such as left or right.

Sensory imagery-based tasks, such as motor imagery (e.g. picturing movement of a limb) may be more

intuitive, and can translate into clearly related commands. For instance, imagining movement of a user’s right hand

may indicate a desire to move their wheelchair to the right. In fact, the motor imagery paradigm has been shown to

be less fatiguing and require minimal training for individuals with ALS [66]. Unfortunately, motor imagery-related

brain activations tend to be attenuated in those with spinal cord injury as well as in degenerative conditions such as

ALS [13], [67]. Thus, while this paradigm may be beneficial to some individuals, it may also be limited in others

due to the nature of their condition, which may compromise motor feedback, as well as the transmission and

processing of somatosensory and proprioceptive information [13].

An alternative active BCI sensory imagery task is covert mental speech, in which users say or repeat

words in their head (e.g. “yes”, “no”, “left”, “right”). Like its motor analog, this task has an intuitive appeal,

although it relies mainly on activations in the auditory cortex. Interestingly, not every individual is adept at verbal

imagery. In fact, it has previously been demonstrated that there is a dichotomy in cognitive thinking styles whereby

individuals are typically either verbal or visual thinkers. [68]. Importantly, these cognitive styles have

corresponding distinct anatomical activations [69]. This has considerable implications for BCI mental imagery

tasks; individuals who are visual thinkers may not engage auditory-processing cortical regions of the brain to the

same degree as verbal thinkers when performing auditory imagery. Likewise, verbal thinkers may minimally

engage visual cortical locations for visual imagery tasks. This is in line with findings demonstrating that, while top-

down processing for different types of mental imagery (and memory – see [43]) may recruit common fronto-

posterior networks, visual imagery deactivates auditory-imagery processing regions and vice-versa [70]. As a result

of high inter-subject variability in cortical activations, user-personalized BCI tasks are necessary in order to

maximize performance [71], [72].

One avenue that has not been explored extensively in BCI paradigms is visual mental imagery,

specifically, picturing stimuli in the visual field. Previously, most visual-based BCIs have focused on visuospatial

attention (i.e. shifting attention to a location of the visual field possessing a stimulus) in overt (i.e. with gaze shift)

and covert (i.e. without gaze shift) contexts [56], [73]–[79]. A major shortcoming of these studies is that even in

covert visuospatial attention tasks, where users fixate on a central cross, target locations for attention shifts require

some visual stimuli (e.g. a circle outline indicating where to focus attention). These approaches are not useful for

individuals with limited ocular motor control who cannot fixate on a single spot for covert paradigms or have

trouble shifting their gaze for overt paradigms. Although gaze dependence can be mitigated somewhat with stimuli

presented through closed eye-lids, such as in [78], such reactive paradigms only accommodate synchronous control.

Unlike attention paradigms, visual mental imagery would not require stimuli in the visual field to elicit a response.

However, at the time of writing, only one BCI study has leveraged true visual mental imagery, requiring individuals

to imagine faces or houses [57]. While imagery of faces and houses elicits differential responses in the medial and

lateral fusiform gyri [80], this imagery does not intuitively translate into a desired command (e.g. picturing a house

to indicate “yes”). Interestingly, the imagery of shapes within the visual field elicits retinotopic patterns of cortical

Page 23: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

12

activations similar to those observed in the perception of shapes [40], [59]. In fact, imagery-related activation was

shown to be stronger and more similar to that accompanying perception than attention to those visual field areas.

Because unique locations of the visual field have corresponding physical cortical regions dedicated to their

processing in the visual cortex (deemed “retinotopy”), imagery in distinct areas of the visual field may elicit unique

activations that could map to BCI user commands. Importantly, this approach might offer the potential for multiple

BCI commands. For instance, imagery in each quadrant of the visual field could correspond to different directions

of desired motorized wheelchair movements (e.g. upper-right imagery moves the wheelchair forward and to the

right). Practical BCI use in the real world necessitates the development of a task sensitive to the needs of the user.

As visual mental imagery does not rely on vision, gaze and neuromotor control, it may form the basis of an active

asynchronous paradigm, and may provide a new avenue of BCI control for individuals who do not perform well

with other tasks (e.g. verbal imagery). This study aims to determine the potential of discriminating imagery in the

four quadrants of the visual field for control of an online EEG-BCI.

3.3 Methods

3.3.1 Participants

This study was approved by the research ethics boards at Holland Bloorview and the University of

Toronto. Twenty typically developed adults (10 female, 1 left-handed, mean age 27 ±4.15) were recruited for this

study from Holland Bloorview and the University of Toronto. Each participant provided informed written consent

prior to participating, and was compensated following each session. One participant dropped out of the study

following the second session for unreported reasons. Additionally, another participant was unable to complete

session 2 due to technical difficulties with the EEG system resulting in noise that could not be rectified. The data of

18 participants who had completed at least three sessions (8 female) are presented in the results section. Finally, 6

participants (0 female) met the criteria to continue on to Sessions 4 and 5. All six completed Sessions 4 and 5.

3.3.2 Instrumentation

EEG data were collected using the 16-electrode actiCAP Dry Xpress system with the V-amp amplifier

(Brain Products, Germany). Participants were fitted with medium, medium-large, or large EEG caps depending on

their head circumference. The caps were then sprayed with 70% ethyl-alcohol and were placed on participants such

that Cz sat precisely between the nasion and inion, and between each ear tip. The electrodes were located above and

around the occipital lobe (I1/I2, Oz, O1/O2, POz, PO1-PO4, PO7-PO10, P1/P2), with reference at FCz and ground

at AFz. The raw signal was digitally bandpass-filtered between 0.5 and 80 Hz using a 12 dB/oct Butterworth filter,

and notch filtered at 60 Hz in Brain Vision Recorder. All subsequent data preprocessing and analysis was

conducted using MATLAB 2017a and EEGLAB toolbox v 13.6.5b [81].

Electrooculography (EOG) signals were acquired with 6 wet stick-on medical grade electrodes placed

around participant eyes (one on each temple, one above each eye, one in the center of the forehead and one on the

right cheekbone), and connected to the V-amp using auxiliary ports. EEG and EOG signals were collected

simultaneously at a sampling frequency of 1 kHz, and displayed using Brain Vision Recorder. Because impedances

Page 24: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

13

are not reported through the actiCAP dry system, each EEG electrode was physically adjusted to ensure good scalp

contact, while monitoring real time signals. EEG signals were considered adequate if amplitudes were under 100

µV [82]. If channels remained noisy despite physical adjustment (moving probes left, right, up, down; rotating

probes clockwise and counterclockwise; parting hair; roughening scalp), a small dab (< 0.1 ml) of Spectra 360 salt-

free electrode gel was placed on the tip of the noisy electrode probe. Each participant was seated comfortably

approximately 40 cm away from a 24 x 13.5” interface monitor with a 75 Hz refresh rate.

3.3.3 Experimental protocol

The format for the BCI component of the 5 data collection sessions is depicted in Table 1. Only

participants whose offline data collected from sessions 1-3 met specific criteria (outlined in later sections) returned

for sessions 4 and 5.

Table 1 BCI experimental protocol for each data collection session

Session 1

In the first session, participants completed the Rey-Osterrieth Complex Figure (ROCF) [83], [84] copy and

immediate recall tasks, followed by the Vividness of Visual Imagery Questionnaire 2 (VVIQ2) [85]. Subsequently,

participants had 5 different conditions to complete: Baseline, Attention, Auxiliary, Perception and Imagery.

Participants had control over when to begin each block. The first condition consisted of 30 seconds of Baseline

activity during which the participants were asked to sit at rest and fixate on a green fixation cross spanning 1° of the

visual field on a 100% grey background. Next, participants completed 20 trials of the Attention condition, during

which they were cued to each of the four visual field quadrants (UR – upper right; LR – lower right; LL – lower

left; UL – upper left) in random order, and were told to covertly shift their attention to the indicated quadrant

without shifting their gaze from the fixation cross until they were cued to rest (Figure 1).

Page 25: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

14

Figure 1 Example Attention Condition trial (UL)

Following this, participants engaged in an eye-gaze control condition (Auxiliary condition), in which they

were cued to each of the four quadrants, and were told to overtly shift their gaze to the center of the arrow stimulus

when it appeared in that quadrant. After 1 s, the stimulus disappeared and the participants were asked to move their

gaze back to the fixation cross and were cued to rest for 1 s. During this condition, participants were also cued to

blink when the word “BLINK” appeared on the screen (Figure 2).

Figure 2 Example Auxiliary Condition trial (UR)

Following 20 trials of the Auxiliary condition, participants engaged in 10 blocks of alternating Perception

and Imagery conditions. In the Perception condition, participants were cued to each of the four quadrants in a

randomized order, and were asked to fixate their gaze on the cross while covertly shifting their attention to the

arrow stimulus that appeared in the cued quadrant, after which they were cued to rest (Figure 3). This was done for

8 trials (two trials in each quadrant), after which the participant engaged in the imagery task.

Figure 3 Example Perception condition trial (UL)

As with the Attention condition, in the Imagery condition, participants were cued to each quadrant and

were told not to shift their gaze from the fixation cross in this condition. However, participants were asked to

imagine the arrow stimulus they had previously seen in the quadrant, rather than simply attend to the quadrant

(Figure 4). After each Imagery trial, participants were cued to rest. Each Imagery block ended after 20 trials.

Page 26: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

15

Figure 4 Example Imagery Condition trial (LR)

After 10 blocks of alternating Perception and Imagery, the participant completed the ROCF delayed recall

task and filled out a user-satisfaction questionnaire in which they rated the paradigm on 5-point scales for measures

including pre- and post-session fatigue, difficulty and frustration (see Appendices B and C).

Session 2

In the second session, participants completed the same tasks as in the first session, with the exception of

the ROCF, the VVIQ2 and the Auxiliary Condition (see Table 1).

Session 3

The third session consisted of Baseline, followed by 10 alternating blocks of Perception and Imagery.

Subsequently, all the offline data collected for that participant were used to train a BCI classifier (detailed below),

during which the participant was briefed on the Feedback condition. Once the classifier was trained, each

participant engaged in 2 blocks of the Feedback condition, in which they were cued to picture the arrow stimulus in

each of the four quadrants, as in the Imagery condition. This trial was then immediately classified as Rest or

Imagery, and participants were presented with 3 s of feedback. If the BCI detected Imagery with greater than 0.55

probability, participants were presented with an opaque flickering arrow. If the BCI detected Imagery with between

0.45 and 0.55 probability, participants were presented with a translucent flickering arrow. Finally, if the BCI could

not identify imagery (i.e. <0.45 probability), participants were presented with the outline of the arrow (Figure 5).

Feedback was provided to reinforce mental strategies that yielded desirable signals, and deter strategies that did not

[9].

Page 27: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

16

Figure 5 Example Visual Feedback Condition trial (UL), with annotated probabilities (not presented to participants)

Sessions 4 and 5

The fourth and fifth sessions consisted of Baseline data collection followed by 4 alternating blocks of

Perception and Imagery (see Table 1). Subsequently, the BCI was re-trained on all the offline data from each

participant. Then each participant engaged in two blocks of a Navigation Game, in which they controlled

movement of a porcine character. The character always remained in the center of the screen under a fixation cross

and while the participant pictured arrow stimuli in quadrants where they want the character to move (Figure 6). The

environment moved around the porcine character so as to maintain its central position. The aim of the navigation

game was to guide the character to as many coins as possible in 20 moves. Coins were located at random street

junctions and stop signs (through which the character could not pass) were located in random streets. Importantly,

two measures were enacted in order to reduce the likelihood of afterimage. First, saturation and contrast were

minimized while maintaining the same 50% black 50% white background as the other tasks. Second, the

participants were told to freely explore the scene with their gaze prior to initiating the trial, as saccades have been

demonstrated to reduce the effects of afterimage [86]. In order to track intended movement, participants were asked

to press one of four buttons on the number pad once they decided on a direction. Each button corresponded to one

of the four possible directions (9 for UR, 3 for LR, 1 for LL and 7 for UL). The participants were then asked to

return their gaze to the fixation cross and hit the spacebar to initiate a trial. Subsequent to this, they were presented

with cues designating where they can perform imagery. First, black ∟-shaped cues for the UL and LR quadrants

appeared just outside the fixation cross for 500ms. Once the cues disappeared, participants rested if their intended

direction was not cued, or performed imagery in either cued quadrant. After 5.5 s, they were presented with cues for

the remaining two quadrants (LL or UR), in which they could again perform imagery in the desired quadrant or

rest. After the trial, the screen returned to the view of the game. Once the BCI classified one of the four directions

(1-2 s delay), the character moved towards the detected quadrant (independent of which button the participant

pressed).

Page 28: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

17

Figure 6 Example Navigation Game trial (UL) with exaggerated fixation crosses

3.3.4 The visual imagery stimulus

The arrow stimulus presented to the participants, and the same asked to be imagined by them, was

designed with a number of considerations in mind. First, the shape was selected so as to be similar to the

checkerboard annulus wedge in [40], but adjusted to be more task-relevant and intuitive as a command in the

navigation game condition. Specifically, the direction in which each arrow pointed corresponded to the quadrant in

which it appeared (e.g. the UR-pointing arrow was found in the UR quadrant) as well as to the desired navigational

direction in the game (e.g. UR arrow means move the character up and to the right). Further, it has previously been

discovered that the use of more naturalistic-shaped stimuli results in greater visual discriminability than simple

square checkerboard stimuli [87]. Presumably, arrow shapes are more meaningful than annulus shapes such as the

ones used in [40], and may serve to increase differentiable cortical activation. The arrow shape also results in a

greater discrimination of shape than the annulus wedge when compared across different quadrants. It is expected

that the increased shape discriminability may help distinguish neural activity resulting from different classes of

imagery. Second, each check was adjusted to account for cortical magnification; the decrease of cortical area that

represents the visual field with increasing degrees of eccentricity [88]. As in Slotnick and Yantis’ study, a high-

contrast checkerboard pattern was used in order to elicit a maximum visual cortical response [89]. However, while

the authors used a black and white pattern reversing frequency of 8.3 Hz in order to efficiently elicits retinotopic

activation of the primary visual cortex [89], this has been shown to fall in the range of seizure-inducing stimuli

[37]. As such, the arrow stimulus frequency was reduced to 2 Hz, well below the safety threshold of 3 Hz. In an

effort to reduce the likelihood of after-image during Rest trials, which would occur with constant pattern-reversing

retinal stimulation, the stimulus flashed on and off. In other words, the arrow stimulus would appear then disappear

at a rate of 2 Hz, which is within the range of saccade frequencies known to prevent afterimages [90]. Lastly, while

many of the aforementioned studies only focused on using stimuli in the central field of vision (i.e. 1-14°), it was

Page 29: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

18

found that in contrast to visual attention, participants performed better on visual mental imagery when stimuli are

large [91]. As a result, the length of the arrow stimulus was expanded to 18° of the visual field.

3.3.5 BCI pipeline

To provide viable feedback to participants, a preliminary BCI pipeline was designed based on data from

Sessions 1 and 2, which was used in the online neurofeedback condition in Session 3. This is detailed below,

followed by specifications of a more advanced BCI (henceforth termed “sophisticated BCI”) that was designed

using data from all 3 Sessions for the offline analysis and navigation game. Briefly, the more sophisticated BCI

involves noise attenuation, is less dependent on low-frequency (e.g. delta frequency band) features that may have

been more contaminated by artefact, and is more dependent on relevant frequency bands (e.g. alpha; 7-13 Hz).

3.3.6 Online

Signal preprocessing

Trial epochs were extracted (6 s duration from cue onset), resampled at 256 Hz, and bandpass-filtered

using a 3rd order Butterworth filter between 1-40 Hz, with stopbands at 0.5 and 45Hz. EOG activity (such as

blinking) was suppressed from the EEG data using a 3rd order conventional recursive least squares (CRLS)

regression algorithm [22] with a forgetting factor of 0.9999 and initial filter state of 0.01. To increase SNR, the

following neighboring channels were averaged to create 6 additional virtual channels: I1/PO9, O1/PO7, PO1/PO3,

I2/PO10, O2/PO8, PO2/PO4. Lastly, epochs were trimmed to remove the first and last 500ms. This removed any

distortion introduced by filtering, as well as potential reactive responses from visual cues [77].

Feature engineering

The logarithms of power spectral densities (PSDs) were computed for each channel and trial using a fast

Fourier transform with a 90% overlapping 256ms sliding window. From the log PSDs, the total delta (1-3 Hz),

theta (low: 3-5 Hz, high: 5-7 Hz), alpha (low: 7-9Hz, mid: 9-11 Hz, high: 11-13Hz and 13-15Hz, total: 7-15 Hz)

and beta (low: 13-15Hz, mid: 15-17Hz and 20-24 Hz, high: 24-27 Hz) band powers were extracted. Next, the

relative power, peak frequency, relative peak frequency, and PSD fractal exponent (the linear slope of the PSD) of

each band were computed. As discussed in more detail below, the alpha frequency band has been demonstrated to

carry useful information in CVSA paradigms [92]. As such, the ratio of peak alpha power to the power from 2 Hz

prior, and the ratios of the total alpha power to the power in each beta sub-band were computed. Additionally, as

imbalances in frequency power were expected during imagery tasks, the magnitude-squared coherence for alpha

(mid: 9-12 Hz), delta (1-4 Hz) and beta (mid: 23-26 Hz) were computed for adjacent and opposing channel pairs.

Finally, to capture potentially relevant time-domain features, spectrograms were computed applying a short-time

Page 30: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

19

Fourier transform (STFT) on 115ms segments (110ms overlap) multiplied with a Hamming window, for each trial

and channel. Then the mean, minimum and maximum for each 256ms interval in alpha (mid: 9-12 Hz) and delta (1-

4 Hz) frequency bands were calculated.

Classification

3.3.6.3.1 Feature selection

Features were selected through elastic net regularization [93]. The alpha elastic-net mixing parameter was

tuned between 0.5 (elastic-net) and 1 (lasso) with step sizes of 0.05 over 10-fold cross-validation runs. Increasing

alpha in this manner limited the number of selected features while minimizing error. The model possessing the

lowest lambda values and the smallest deviance across all alpha values was selected. Importantly, this model also

contained the beta coefficient for each selected feature, corresponding to its individual contribution to the model.

3.3.6.3.2 Classifier

To classify non-specific imagery versus rest in the Feedback condition, a logistic regression classifier with

the model parameters determined above, was trained using all 600 trials of rest and imagery from the Imagery

conditions of Sessions 1-3. The trials from the online Feedback condition were then preprocessed as above and

classified in real time. Posterior probability scores were then used to provide the feedback described in the

experimental protocol.

3.3.7 Offline analysis and online navigation game

While designed to classify tasks of visuospatial imagery, the BCI discussed here was also used to analyze and

classify data from the visuospatial perception tasks.

Signal preprocessing

Trial epochs were extracted, resampled at 256 Hz, and bandpass filtered using a 3rd order Butterworth filter

between 1-40 Hz, with stopbands at 0.5 and 45Hz. EOG activity such as blinking was suppressed from the EEG

data using CRLS. Epochs of transient noise that resulted from temporarily increased impedances (e.g. through loss

of electrode connection) in the EEG system were identified under the assumption that EEG signal amplitudes

recorded from the scalp are below 100µV [82]. As such, noisy (i.e. “bad”) channels within each trial were flagged

if they surpassed ±50µV. Next, the bad channels were attenuated by scaling the amplitudes to the standard

deviation of “good” channels in that trial, while maintaining their means. If all channels were labelled as exceeding

either specified threshold, the signals were attenuated to have a range of ±50µV around their mean. Attenuation of

Page 31: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

20

this dry EEG noise was necessary as the spatial filter would amplify these sources of noise, and spread them to

adjacent channels. The number of flagged noise epochs were recorded.

As different types of imagery were expected to result in unique localized changes in activity, a Surface

Laplacian/Current Source Density (CSD) spatial filter was applied using the CSD toolbox [94]. This spatial filter

suppresses spatially broad activity shared between electrodes that is a result of volume conduction across the scalp,

and amplifies localized activity [95]. As a result, CSD was applied when classifying imagery in diagonally-

opposing quadrants. However, the spatial filter was not applied when classifying non-specific visuospatial imagery

against rest since the imagery class was heterogeneous, with differing locations of activity. Instead, it was expected

that the imagery versus rest classifier would rely on activity that was common to visuospatial imagery in all

quadrants. As optimal parameter settings had not previously been established for a visual imagery paradigm with a

low-density, low-number dry electrode array, a variety of CSD parameter combinations were tested across wider

ranges of values than previously established [96]–[99], as recommended in [100]. Specifically, the lambda

smoothing factor (lam) was searched for across values of 10-4.25 to 10-6 with 10-0.25 step sizes; the Legendre

polynomial order (Pn) was searched over integers of 3 to 12 and spline flexibility (m) was searched for over

integers of 2 to 8 for each participant and each imagery classification problem. Additionally, CSD estimates were

scaled according to a realistic head size (radius of 9 cm). The parameter combination resulting in the best

classification accuracies was used for final online classification. Epochs were trimmed to remove the first and last

500ms in order to remove filter distortion and influencing reactive signals from visual cue perception.

Feature engineering

Features that were extracted for final offline analysis and the navigation game are detailed in the following

sections.

3.3.7.2.1 Baseline alpha

It was previously demonstrated that classification accuracies in a CVSA paradigm could be predicted using a

measure of baseline alpha [77]. Thus, the band power of alpha frequency (7-13 Hz) was summed for each channel

over the 30 s of baseline activity, then averaged across sessions. The channel with the maximum alpha power was

selected for further statistical analysis.

3.3.7.2.2 Alpha spindles

Posterior alpha activity located over the visual cortex is closely linked to attention, where increased alpha

from resting state indicates an inhibition of irrelevant stimuli [101]–[103]. Importantly, in visuospatial attention

paradigms, alpha power increases over regions dedicated to processing irrelevant visual field locations, and

decreases over regions dedicated to processing the attended visual field locations [56], [77], [92], [104]–[107].

Another method to characterize alpha frequency fluctuations detects short-term bursts of activity – deemed “alpha

Page 32: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

21

spindles” [108]. Importantly, alpha-spindle detection is less sensitive to noise [108], and alpha-spindle rate has been

shown to increase during suppression of visual processing, and decrease during activation of visual processing in an

attention paradigm [101]. As this approach may characterize visual imagery in a manner that is less affected by

noise than simple band power analysis, alpha spindle features were extracted based on the methods outlined in

Simon et al. [108]. The approach was slightly modified to fit this visuospatial imagery paradigm, and each step is

briefly outlined below.

First, the spectrogram for each trial and channel was computed by applying STFT on 128ms segments

(100ms overlap) with a Hamming window. Next, each 28ms segment was tested for the presence of an alpha

spindle according to the conditions outlined in Simon et al. (2011). Briefly, a segment was considered to possess an

alpha spindle if:

i) the maximum activity between 1-40 Hz occurred within the specified alpha range (e.g. 7-13 Hz);

ii) the full width at half maximum (FWHM) was no greater than twice the bandwidth of the Hamming

window; and

iii) the area under the peak (bounded by the FWHM) of the amplitude spectral density was at least twice as

large as the area under the 1/f frequency noise curve (bounded by the FWHM).

The 1/f frequency noise curve was computed by fitting the mean amplitude spectrum of the segment to an

exponential curve, which was then multiplied by the ratio of the total segment power over the total trial power.

Finally, the total number of alpha spindles per trial per channel was calculated. As it has been found that alpha sub-

bands provide more useful information for discrimination of visuospatial attention than that of the entire band

[107], this entire process was repeated with condition i) being met if the peak frequency fell within the following

alpha sub-bands: 7-9 Hz, 8-10 Hz, 9-11 Hz, 10-12 Hz, 11-13 Hz. Additionally, as alpha power is largely variable

over time, the absolute number of alpha spindle segments per trial was considered inadequate. Thus, the relative

number of alpha spindles per trial (i.e. alpha spindle lateralization) was computed by subtracting the number of

spindles in each left hemisphere and central channel from each right hemisphere and central channel (see [92]).

This was done in addition to summing alpha spindles of left and right hemisphere channels, which was aimed to

distinguish activity that was common to both hemispheres from activity that was common to one.

3.3.7.2.3 Continuous wavelet transform

Both time and frequency domain signal characteristics were found to be important in studies of CVSA, as

discriminant patterns of alpha power change over short time periods [107]. One approach to time-frequency

analysis is the spectrogram, although it suffers from trade-offs between frequency and time resolution. Scalograms

avoid this issue by decomposing the signal using mother wavelets into additive subcomponents, which provide a

measure related to the contribution of different frequencies (pseudo-frequencies) to the original signal. While this

approach assumes a stationary signal, it also offers greater time-frequency resolution than that of spectrograms. As

a result, each trial was transformed into a scalogram using the Morlet wavelet, which was then normalized by

dividing each value by the mean of the entire scalogram. Next, we isolated alpha pseudo-frequency sub-bands (α1,

Page 33: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

22

7.5-9.5 Hz; α2, 8.5-10.5 Hz; α3, 9.5-11.5 Hz; α4, 10.5-12.5 Hz; α5, 11.5-13.5 Hz) in addition to the entire alpha

pseudo-frequency band (7.5 – 13.5 Hz), and found relative scalograms by adding and subtracting left channels from

right channels, as with alpha spindles. The total coefficients of each relative scalograms were summed and

extracted as features. Finally, while a previous study found that it was beneficial to classify left versus right

visuospatial attention using specific time-windows [107], it is unlikely that the evolution of the alpha frequency

during visual imagery is consistent over every trial, as well as across sessions. However, extreme scalogram values

may identify key changes in relative neural activity that can discriminate different types of imagery. Thus, upper-

and lower-thresholds (± 1 standard deviation around the mean) of each scalogram were used to divide its values

into high, low or medium subgroups. Finally, the mean of each of these groups was extracted as a feature. In

addition to the lateralized scalograms derived from subtraction of channel pairs, the above-mentioned features were

extracted from the scalograms from each channel.

3.3.7.2.4 Entropies

Another feature that has previously been shown to be useful in visual attention classification paradigms, is

entropy - a measure of disorder of a signal. In fact, these studies suggested that approximate and sample entropies

would be higher in attention conditions as compared to resting tasks [109], [110]. Thus, the approximate and

sample entropies for the alpha band (7.5-13.5 Hz) and sub-bands (α1, α2, α3, α4, α5) were computed for each

channel and trial with a tolerance of 0.2 times the standard deviation of the signals, and an embedded dimension of

2 as in [109].

3.3.7.2.5 Magnitude-squared coherence/cross-power spectral density (CPSD)

As frequency imbalances between left and right- hemisphere are expected over the occipital region in

alpha frequencies during tasks of attention, the mean magnitude-squared coherence measure was calculated for the

alpha band and each alpha sub-band (α1, α2, α3, α4, α5) for each trial and channel. Likewise, the mean magnitude

of CPSD, in addition to the mean phase lag between left and right electrodes was computed for alpha (7.5-13.5 Hz),

theta (4-7 Hz) and beta (13-30 Hz) bands.

Classification

3.3.7.3.1 Feature selection

Features were selected through the elastic net regularization method over 5-fold cross-validation.

However, in the offline and navigational game BCI, the alpha parameter was tuned between 0.75 and 1 with step

sizes of 0.05. This was done to further minimize the total number of selected features to a greater degree. In this

Page 34: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

23

manner, non-logistic regression classifiers that did not use beta coefficients (i.e. weights of features) could be

trained, with the reasonable assumption that most, if not all selected features would contribute evenly to the model.

3.3.7.3.2 Binary classifiers

Three separate binary classification problems were considered for this thesis. First, as abovementioned, it

was of interest to discriminate any type of imagery in the visual field against a resting state. Next, it has been

demonstrated that attention in one location of the visual field will result in a decrease in relative alpha power in the

cortical region processing that location, and an increase in alpha power in the opposite cortical region. As a result,

the most distinguishable signals are expected to be UL and LR imagery, or LL and UR imagery (see Treder et al.

2011). Thus, the classification problems regarding different types of imagery aimed to discriminate UL versus LR

imagery, and LL versus UR imagery. As it was unknown which classifier type would result in the best accuracies,

the following were trained and tested for each classification problem through 10-run, 5-fold cross validation with

equal representations of samples from each session: K-nearest neighbours (KNN), linear and radial-basis function

(RBF) support-vector machines (SVMs) (from the LIBSVM toolbox [111]), and logistic regression.

3.3.7.3.3 Two-tier multiclass classifier

In order to maximize discrimination between the four classes of imagery, a number of characteristics of

the data were taken advantage of. First, as the maximally discriminant imagery quadrant pairs are UL/LR and

LL/UR, cues presented to the participant in the navigation game were only for diagonally opposing quadrants.

Next, during rest, alpha power was expected to be more evenly distributed across the visual cortex, so the relative

alpha power features were expected to fall between UL and LR imagery, or LL and UR imagery. If this data was

tested using a classifier trained on UL/LR or LL/UR imagery, the probability that it belonged to either class was

expected to be close to 0.5. Thus, each subtrial in the online Navigation Game was tested with a binary imagery

(UL vs. LR or LL vs. UR) classifier (Tier #1), with the expectation that participant would be resting in one of the

two subtrials (Figure 7). Importantly, each subtrial was also tested with a Rest vs. Imagery classifier (Tier #1),

which was expected to help classification as one subtrial was necessarily imagery and the other, rest. The

probabilities returned from these four classifiers were treated as features for a 4-class classifier (Tier #2). In order to

train this classifier, probabilities were obtained by testing the trained binary classifiers on all the available offline

data. All binary problems in this approach used logistic regression, as the relative feature contributions (i.e. beta

coefficients) could be used, whereas the multiclass problem relied on an RBF-SVM classifier (as recommended in

[112]). This two-tier classification approach was compared to a direct multi-category (i.e. 4-class) classification of

imagery in each quadrant using RBF-SVM.

Page 35: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

24

Figure 7 Schematic of Navigation Game trial classification with two tiers of classifiers

3.4 Results

3.4.1 Chance levels

All chance levels discussed below were computed using the binomial distribution thresholds based on the number

of samples [113]. When accuracies are said to exceed chance, this refers to their means minus their standard

deviations being above chance.

3.4.2 Classifying perception and imagery against resting mental state

Perception

3.4.2.1.1 Offline

The sophisticated BCI was able to discriminate resting mental state from perception in any visual field quadrant

with above-chance (55.4%) accuracies in all participants save one (Figure 8). Additionally, 14 participants met the

70% threshold for a practical BCI. The mean classification accuracy across all participants was 75.7±8.5%. The

best classifier in terms of accuracy was participant-specific. Logistic regression was the most frequently selected

optimal classifier (7 participants).

Page 36: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

25

Figure 8 Classification accuracies for perception versus rest with user-optimized classifier

Imagery

3.4.2.2.1 Offline

When discriminating resting mental state against visuospatial imagery in any quadrant, the BCI was able to achieve

above-chance (53.3%) accuracies in 16 of 18 participants (Figure 9). Additionally, data from 7 participants met the

70% threshold for a practical BCI. The mean overall classification accuracy across all participants was 71.2±11.3%.

The most frequently selected optimal classifier was Linear SVM (7 participants).

Figure 9 Offline classification accuracies of rest versus any imagery with user-optimized classifiers

3.4.2.2.2 Online

Non-specific imagery versus resting state classification scores from online trials are depicted in Figure 10. The

preliminary BCI (with a logistic regression classifier) used in the online trials resulted in a mean accuracy of

71.7±12.3%, with 12 participants meeting the 70% practical BCI threshold. When the trials were re-evaluated using

the sophisticated BCI approach, a mean accuracy of 75.1 ±12.0% was achieved with logistic regression, and

Page 37: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

26

77.0±11.4% with the best-performing classifiers specific to each user. These resulted in 12 and 13 participants

meeting the 70% threshold, respectively. When comparing the preliminary BCI to the sophisticated BCI with

optimized participant-specific classifiers, accuracies improved for 14 participants, and decreased marginally for 3

participants (<5%) and moderately for one (>5%). Non-parametric analysis by the Wilcoxon signed rank test

indicated that the sophisticated BCI with user-optimized classifiers significantly increased classification accuracies

by an average of 5.3% when compared to the preliminary BCI (p =0.0107). The most frequently selected optimal

classifier was Logistic Regression (7 participants).

Figure 10 Classification of online trials

Features characteristic of perception and imagery versus rest

3.4.2.3.1 Perception

The average number of selected features when classifying non-specific perception against rest was 29±15. A

normalized frequency heat map sorted from best-to-worst classification accuracies is depicted in Figure 11. As

summarized in Figure 12, the groups from which features were most commonly selected were: channel pair sums of

scalograms in the alpha (18.9%) and beta (13.4%) frequencies, magnitude squared coherence of the alpha band

(15.7%) and phase lag of alpha coherence (11.3%). When adjusted for the number of sub-features within each

feature group, the most commonly selected groups were: magnitude squared coherence of the alpha band (2.6%),

lag of alpha coherence (2.3%) and channel pair sum of alpha spindles (1.8%) (Figure 12).

3.4.2.3.2 Imagery

The average number of features selected by elastic-net regularization across participants was 39.5±25.3. The

frequency of selected feature groups across all participants (sorted from best to worst classification accuracies) is

depicted in Figure 11. The most frequently selected groups were channel pair sums of scalograms in the alpha

(14.1%) and beta (14.6%) frequencies, magnitude squared coherence of the alpha band (13.5%) and lag of alpha

Page 38: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

27

coherence (13.0%) (Figure 12). When adjusted for the number of sub-features within each feature group, the most

commonly selected groups were magnitude squared coherence of the alpha band (2.2%), lag of alpha coherence

(2.6%) and magnitude of cross power spectral density in the theta (2.2%) and beta (1.7%) frequencies (Figure 12).

Figure 11 Frequency of feature group selection by elastic net regularization for perception (left) and imagery (right) vs. rest

classification

Figure 12 Frequency of feature group selection by elastic net regularization for perception compared to imagery (left) and perception

compared to imagery with selection adjusted for number of sub-features (right)

3.4.3 Classifying quadrants in perception and imagery

Perception

Prior to classifying diagonally-opposing quadrants (UL vs. LR, LL vs. UR), spline flexibility (m), smoothing factor

(lam) and Legendre Polynomial order (Pn) were varied in the CSD spatial filter to determine the optimal settings

for each participant, within each classification problem. On average, the user-optimized CSD spatial filters were

found to improve UL vs. LR accuracies by 13.8±8.4% (p<0.001) and LL vs. UR accuracies by 13.0±7.2%

(p<0.001) (not depicted). The pattern of optimal CSD parameters across both classification problems was not

consistent across participants (Figure 13), however the most commonly selected parameters were as follows: spline

Page 39: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

28

flexibilities of 2 (8 times) and 3 (8 times), smoothing factors of 104.25(11 times) and 10-4.5 (7 times), and Legendre

polynomial orders of 7 (7 times) and 4 (5 times).

Figure 13 Frequency of optimal CSD spatial filter parameters: spline flexibility (left); smoothing factor (middle); Legendre polynomial

order (right)

The classification accuracies after implementation of user-optimized spatial filter parameters can be seen in Figure

14. The average classification accuracy across participants was 77.3±9.3% for UL vs. LR and 77.2±9.2% for LL vs.

UR. In total, 9 participants exceeded chance (60%) in both comparisons, 6 exceeded chance in at least one

comparison, and 3 did not exceed chance in either. The most frequently selected optimal classifiers were Linear

SVM for UL vs. LR (7 participants) and Logistic Regression for LL vs. UR (8 participants).

Figure 14 Classification accuracies of perceiving stimuli in UL vs. LR and LL vs. UR quadrants with optimized spatial filter parameters

Imagery

As with classifying perception in diagonally-opposing quadrants, CSD parameters were optimized for classifying

visuospatial imagery in opposing quadrants (UL vs. LR and LL vs. UR). The use of spatial filters optimized to each

individual improved accuracies by 8.1±7.1% for UL vs. LR (p=0.001) and 9.2±4.4% for LL vs. UR (p<0.001)

classification problems, when compared to classifications without spatial filters. The frequency of CSD parameters

resulting in optimal classification accuracies across participants in both UL vs. LR and LL vs. UR are presented in

Figure 15. The most frequently selected spline flexibility parameters were 2(10) and 3(11). The most frequently

Page 40: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

29

selected spline flexibility parameters were 2 (10 times) and 3 (11 times). The most frequently selected smoothing

factors were 10-6 (4 times), 10-5 (3 times) and 10-4.25 (3 times). Finally, the most frequently selected Legendre

polynomial orders were 3 (7 times), 7 (6 times) and 9 (5 times).

Figure 15 Frequency of optimal CSD parameters: spline flexibility (left), smoothing facto exponent (middle), Legendre polynomial

(right)

3.4.3.2.1 Offline

The accuracies for UL vs. LR and LL vs. UR following implementation of user-optimized spatial filters and

classifiers are presented in Figure 16. The mean classification accuracies across all participants was 65.2±7.5% and

65.0±7.2% for UL vs. LR and LL vs. UR problems, respectively. In total, 6 participants achieved above-chance

(56.7%) accuracies in both classification problems, 3 exceeded chance in only one classification problem, and 9 did

not exceed chance in either. Two participants met the 70% threshold in both classification problems, with an

additional 2 meeting the 70% threshold in one of the two problems. The most frequently selected optimal classifier

was the RBF SVM for both UL vs. LR (8 participants) and LL vs. UR (9 participants).

Figure 16 Imagery versus imagery classification accuracies with optimized spatial filter parameters and classifiers

Page 41: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

30

3.4.4 Features characteristic of visuospatial perception and imagery

Perception

The average number of selected features across participants was 6.8±5.1 for UL vs. LR and 9.4 ±8.1 for LL vs. UR.

The most selected feature groups are presented in Figure 17. The most commonly selected groups in both

classification problems were alpha spindle difference (UL vs. LR -36.0%; LL vs. UR - 41.8%) and alpha coherence

phase lag (UL vs. LR - 33.8%; LL vs. UR 28.11%) (Figure 18). Each group contained the same number of sub-

features, and as a result, adjusting for the number of sub-features was unnecessary.

Figure 17 Frequency of selected feature groups in classifying different types of perception sorted by participant

Figure 18 Grand average frequency of selected feature groups in classifying perception in diagonally-opposing quadrants

Imagery

The average number of selected features across participants was 11.2±11.0 for UL vs. LR and 9.8±9.0 for LL vs.

UR. The patterns of most selected feature groups for each classification problem can be seen in Figure 19. The

groups selected with the greatest frequency were the channel pair difference in alpha spindles (UL vs. LR -35.0%;

LL vs. UR -40.5%) and phase lag in the alpha band (UL vs. LR -31.3%; LL vs. UR -30.8%) (Figure 20).

Page 42: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

31

Figure 19 Frequency of selected feature groups in classifying different types of imagery sorted by participant

Figure 20 Grand average frequency of selected feature groups in classifying imagery in diagonally-opposing quadrants

3.4.5 Multiclassification of visuospatial imagery

Participants

The criteria for participants to continue to Sessions 4 and 5 were that their offline binary visuospatial classification

accuracies for both UL vs. LR and LL vs. LR be above chance. In other words, this required the mean of their

accuracies minus the standard deviation to be greater than the binary distribution threshold for chance (56.7%). As

a result, only 6 participants continued to Sessions 4 and 5.

Offline and online

The offline classification accuracies, and mean online classification accuracies over Sessions 4 and 5 are presented

in Figure 21. Direct classification of imagery in the four visual field quadrants (UL vs. LL vs. LR vs. UR) resulted

in a mean of 38.4±13.0% (max 63.5%), with 3 participants exceeding chance. This accuracy was significantly

increased to a mean of 57±14.6% (p= 0.0313), and a maximum of 85% when rest was used to further divide the

Page 43: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

32

four classes using the two-tier classification scheme. This approach also allowed all 6 participants to exceed chance

levels. The mean online scores using the two-tier approach were 49.2±18.1%. The online scores for four of the

participants fell within one standard deviation of the two-tier offline accuracy means, and the remaining two fell

below. A confusion matrix for all the data across all participants is presented in Table 2. Finally, only one

participant exceeded the practical BCI threshold of 70% in both the two-tier offline and online paradigms.

Figure 21 Four-class classification accuracies in offline (direct and two-tier) and online approaches

Table 2 Confusion matrix for all participant data from sessions 4 and 5

3.4.6 Predicting classification accuracies

Correlates of classification accuracies

Measures that were obtained external to imagery mental tasks (VVIQ2 and ROCF scores, pre-session fatigue

ratings, baseline alpha, number of noisy epochs and perception task classification accuracies) in addition to those

obtained internally from mental task trials (number of noisy epochs, number of selected features and number of

different selected feature groups) were correlated using Spearman rank to imagery versus rest classification

accuracy (Table 3), as well as to the average of UL/LR and LL/UR imagery classification accuracy (Table 4).

When assessing correlates of non-specific imagery versus rest accuracy, only the number of selected features

resulted in a significant relationship (p = 3.05x10-6, rho = 0.87) (Table 3). Similarly, the number of selected features

was found to positively correlate to imagery versus imagery accuracies (p = 0.003, rho = 0.66) (Table 4). Of the

measures extracted external to the imagery trials, pre-session fatigue and perception versus perception accuracies,

Page 44: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

33

using optimized spatial filter parameters for both perception and imagery paradigms, were found to correlate to

imagery versus imagery classification accuracies (Table 4). However, following adjustment for multiple

comparisons, only the perception accuracies using imagery-optimized spatial filter parameters correlated to imagery

versus imagery accuracy.

Table 3 Spearman correlation table for imagery versus rest classification accuracy

Table 4 Spearman correlation table for imagery versus imagery

Regression model

As it was anticipated that a combination of factors might better predict imagery accuracies, the factors derived

external to imagery trials that were significant prior to adjustment were used as predictors in a stepwise linear

regression model. The model was then simplified to reduce the overall number of terms used, while maintaining a

high R2-adjusted value. As there were only 18 samples, 1.8 features would be ideal to use according to the 10

samples-per feature rule-of-thumb for classification– however this is a very strict threshold in linear regression,

where 2 samples per variable have been demonstrated to be adequate [114]. In attempting to achieve the simplest

model, the less forgiving threshold was followed. This reduced the number of terms in the model were to two – an

interaction term between perception classification accuracies (x1; with imagery-optimized CSD parameters) and

pre-session fatigue (x2), and a quadratic term for pre-session fatigue. This stepwise linear regression approach

Page 45: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

34

resulted in a model with an R2-adjusted value of 83.4% (p-value = 0.00000985, RMSE = 2.93, error DF = 13, F-

statistic = 22.4) (Figure 22). The six participants with x-values larger than 1.75 (blue line) in this model are

consistent with the six participants who achieved above-chance accuracies in both binary visuospatial imagery

classification problems. As comparison, Figure 23 presents the predictive model using only perception accuracies

(optimized to imagery CSD parameters).

Figure 22 Linear model combining pre-session exhaustion with perception classification accuracies to predict imagery classification

accuracies

Figure 23 Linear model of perception classification accuracies predicting imagery classification accuracies

Page 46: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

35

Gender differences

Gender differences were found using a rank sum test, presented in Table 5. Males had a mean imagery

classification accuracy of 68.0±8.4% significantly larger than that of females, 61.5±3.0% (p = 0.0343). However,

this became non-significant when adjusting for multiple comparisons. Only perception accuracies were found to be

significant post-adjustment (p = 0.0176), with males performing significantly better (67.5±14.6%) than females

(51.6±4.6%).

Table 5 Group differences between males and females using the Wilcoxon rank sum test

3.4.7 Characterizing EEG features indicative of diverse imagery mental states

In order to characterize the differences between imagery in visuospatial quadrants, the most selected feature (alpha

spindle differences of the 9.5-11.5 Hz sub-band) was analyzed in the six participants who met the criteria for

Sessions 4 and 5. This feature was assessed for UL/LR and LL/UR in seven left-right channel pairs (I1-I2, O1-O2,

PO1-PO2, PO3-PO4, PO7-PO8, PO9-PO10 and P1-P2). The most significant comparisons are shown in Figure 24,

and the adjusted p-values are summarized in Table 6. The mean difference of alpha spindles in best left minus right

channel pairs was 3.5±3.4 for UL-LR, and 3.1±2.6 for LL-UR. In addition, the average left minus right spectrogram

for imagery in left and right hemifields are presented in Figure 25 for the best performing participant.

Figure 24 Alpha spindle differences (left-right) in channel pairs across participants and imagery quadrants

Page 47: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

36

Table 6 Most significant channel pair differences in alpha spindles using the Wilcoxon rank-sum test

Figure 25 Average difference in time-frequency power for PO3 minus PO4 channel pair in participant 17 for visuospatial imagery in the

left hemifield (left) and right hemifield (right); color bar represents power/frequency (dB/Hz)

3.4.8 Summary of key findings

1. For offline binary classification of either non-specific imagery or perception against rest, a BCI based on

SVM or logistic regression can generally achieve above-chance accuracies, and in fact, exceed 70% for

most participants. Alpha coherence features seem to be particularly discriminatory. (Objectives 1 and 3).

2. Online binary classification of non-specific imagery against rest also exceeded chance for most

participants using logistic regression classifiers with alpha coherence features being important for

discrimination. Post-hoc analyses indicated that accuracies could be further improved by choosing

participant-specific classification algorithms. (Objective 1)

3. When classifying visuospatial perception in diagonally-opposing quadrants, the majority of participants

exceeded chance levels in at least one comparison. Conversely, only half of participants exceeded chance

in at least one comparison when classifying visuospatial imagery in opposing quadrants. Optimal CSD

parameters for perception involved low-to-medium rigidity of spline models, and large smoothing factors,

whereas optimal CSD parameters for imagery were typically more flexible splines, with no clear pattern in

smoothing factors. The most commonly selected features in both perception vs. perception and imagery vs.

Page 48: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

37

imagery problems were channel pair differences in alpha spindle number, and phase lag of alpha

coherence. (Objectives 1 and 3)

4. In multiclassification problems, the two-tiered approach appeared to outperform the direct classification

approach, allowing all 6 participants to exceed chance offline. Online, four of the participants achieved

accuracies in the range of their offline scores. Additionally, one participant exceeded the threshold for an

effective BCI. (Objective 1)

5. When assessing the influence of various factors on BCI performance, the number of selected features

correlated positively with both rest vs. non-specific imagery and imagery vs. imagery. Additionally, a

linear regression model combining pre-session fatigue and perception vs. perception performance (with

imagery-optimized CSD) was found to explain 83% of imagery versus imagery performance variance.

(Objective 2)

6. Significant differences were found in alpha-spindle lateralization depending on the location of visuospatial

imagery in 6 participants. While this difference was most significant in different channel pairs, the

majority of participants exhibited increase alpha spindle-lateralization over the left hemisphere when

imagining stimuli in the left visual field, and decreased lateralization over the left hemisphere when

imagining stimuli in the right visual field. (Objective 3)

3.5 Discussion

3.5.1 Visuospatial perception classification

This study aimed to develop and assess a BCI dependent on a novel paradigm - visuospatial imagery in the four

visual field quadrants. As a method of comparison, and to estimate the upper limits of visuospatial imagery

classification, perception of stimuli in the four visual field quadrants was also assessed in three binary paradigms.

Classifying perception in any quadrant against rest, as well as classifying perception in UL vs. LR and LL vs. UR

exceeded levels of chance and met the threshold for practical BCIs, with means reaching 75.7%, 77.3% and 77.2%,

respectively. While it may seem counter-intuitive that classifying any type of perception against rest resulted in a

slightly lower accuracy than classifying different types of perception, the result may have simply been caused by

the heterogeneity of trials in the perception class (UL, LR, LL and UR). In other words, it is possible that

perception in each quadrant resulted in signals that were different enough from one another to make it difficult for

the classifier to find an optimal method of separating them from signals elicited by resting mental state. In any case,

the accuracies from classifying perception in different quadrants were somewhat lower than similar SSVEP

attentional studies. For instance, Xu et al. (2016) classified left versus right CVSA to flickering stimuli at 81%

[115]. However, a number of key differences are present: firstly, Xu and colleagues had simultaneous flickering in

both hemifields at 12 Hz, whereas this study only had flickering stimuli in the target locations. Additionally, Xu et

al, only used 2 seconds of data whereas this study relied on 5 seconds of data. By these accounts, classifying

perception of stimuli in quadrants with no competing stimuli for a longer period of time should have resulted in

accuracies comparable to those reported by Xu et al. Another study by Maye and Engel achieved accuracies of 95%

Page 49: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

38

in a 9-class paradigm where a clock face flickering at 15Hz would stimulate various retinotopic locations

depending on the user’s fixation point[116]. The lower accuracy in the current study compared to other literature

was likely largely due to the sub-optimal flickering frequency of 2Hz, which produces much weaker SSVEPs than

higher frequencies between 3-60 Hz [117], and the optimal range of 12-18Hz [118]. Additionally, the BCI in this

study was designed to discriminate visuospatial imagery rather than perception, and so did not attempt to produce

features (typically canonical-correlation analysis coefficients) that are extracted in SSVEP literature.

3.5.2 Visuospatial imagery classification

Classifying any imagery against rest resulted in offline accuracies of 71.2% ±11.3% with a sophisticated BCI,

online accuracies of 71.7% ±12.3% with a preliminary BCI, and 77% ±11.4% when the online trials were

reassessed using a sophisticated BCI approach. All of these met the threshold for a practical BCI in a binary

paradigm [119]. When classifying different quadrants of imagery, the mean offline accuracies reached 65.2±7.5%

for UL vs. LR and 65.0±7.2% for UL vs. LR. The offline results presented here are very similar to the findings of

Bobrov and colleagues, who aimed to classify imagery of faces and houses, and a resting state [57]. Bobrov and

colleagues classified offline imagery against rest at an average of 69±3% for faces and 72±2% for houses, and

imagery of faces against houses at 63±2%. The corresponding scores in their online session were 73%±3%, 70±3%

and 64±2%. One notable difference in methodology is that Bobrov and colleagues classified trials of imagery that

were significantly longer (15 s) than the ones in this study (5 s). Additionally, in terms of target stimuli, there are

neuroanatomical differences in the loci of activation when processing faces (e.g. the fusiform face area [120]),

houses (e.g. lateral occipital complex [121]) and checkered arrow stimuli (early visual cortices [89]). This results in

differing EEG dynamics, depending on the visual stimulus [122]. The current study demonstrates that it is not

necessary to imagine for very long, or to imagine targets that may engage higher processing regions, to elicit

machine-discernible brain activation. Importantly, by relying on early visual retinotopic activation, the number of

potential classes in an imagery-based BCI paradigm is only limited by the number of differentiable visual field

locations, rather than by the more restrictive number of distinct higher-processing cortical regions and their

corresponding target stimuli (e.g. faces, bodies, objects).

In another recent study, authors achieved a staggering 88% accuracy in a 3-class visual imagery of motion

paradigm [123]. In this study, Sousa and colleagues measured EEG activity using electrodes directly above the

frontal eye fields (F3, F5, FC3, FC5, C3, C5), and had participants close their eyes and imagine a static dot, a dot in

constant motion or alternating motion. One crucial flaw here is that it was not possible for participants to fixate

their gaze (e.g. on a fixation cross) with their eyes closed, while imagining the dots. While EOG activity was

subtracted from the EEG channels, eye movement could not be controlled in the study, which may have induced

activity in the frontal eye fields not due to imagery, but rather due to eye saccades – thereby biasing the results. An

analogue to this might be measuring from the motor cortex while participants moved their limbs, then subtracting

the motion artifact. Unfortunately, the authors did not provide a comparison of EOG activity between the different

mental states. Furthermore, when assessing the statistical significance of the only alpha power feature used for

classification, they could not find significance between their two moving imagery dot tasks, but only between the

static and moving dots. Following the results of the statistical test, one might think that classification of either

Page 50: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

39

moving dot against the static dot would result in higher accuracy than classifying the different types of moving

dots, but this was not the case. In fact, misclassification was balanced across the different classes. While the authors

chalked this discrepancy up to the ability of multivariate pattern classification to correctly discriminate classes, our

data revealed strong alpha spindle lateralization (Figure 24) in multiple individuals but their mean binary

classification accuracy only reached 72.2% (SD 8.3%). Importantly, this discrepant finding was not due sample size

differences, as Sousa et al. had 60 samples per class, whereas the current study had 75 samples per class. Given

these shortcomings, the work of Sousa et al. is not directly comparable to the present study.

In a study by Tonin and colleagues (2013), covert visuospatial attention to the left or right visual fields

was classified at 70.6±1.5% in an online paradigm [56]. The participants were cued with an arrow to pay covert

attention to one of circles (lower left or lower right), and after some time a red dot would appear in the circle. The

authors used 3 seconds of alpha band power from the EEG data to classify the locus of CVSA, however, the 3

seconds included the arrow cue presentation. While this cue was only presented for 100ms, it is difficult to discount

the possibility that the mere presentation of the arrow pointing to the left or right hemifield improved classification

accuracies. In a similar study by Treder et al (2011), participants were cued to pay covert attention to one of 6 target

circles. The authors used parieto-occipital alpha spectral power from 500-2000ms post cue to identify locus of

attention. Attention to each pair of circles was classified offline, and the mean classification accuracy of the most

discriminable pairs of loci for each participant was reported to be 74.6±2.3%. While only the best classification

accuracies were reported, the study, like that of Tonin et al, did manage to classify on a relatively short period of

time. The discrepancy in accuracy between these two studies (70-75%) and the current one (~65%) may be

attributed to the face that both previous studies had stimuli (circles) in the periphery to which participants attended.

If the outline of arrow stimuli were presented in visual field quadrants, it is possible that visuospatial imagery

would classify at accuracies closer to those reported by Tonin et al. and Treder et al.; however, it could also be

argued that this would not be true imagery. While the visuospatial imagery accuracies did not reach those of CVSA

BCIs, the appeal of the current study is that no stimuli were required in the target locations. In fact, the alpha power

lateralization features in the two aforementioned CVSA studies likely relied on phenomena whereby: (1) increased

alpha power (i.e. alpha synchronization) allowed for suppression of irrelevant (i.e. non-target) stimulus-processing

in corresponding visual cortical regions, and (2) decreased alpha power (i.e. alpha de-synchronization) allowed for

processing of stimuli in relevant (i.e. target) areas by their corresponding visual cortical regions[124]. Thus, in the

aforementioned studies, the alpha lateralization might have served to suppress processing of the non-target circles

and to prioritize or increase processing of target circles [125][106]. However, in the current study, there were no

target/non-target stimuli presented, and as such, the alpha power lateralization (here characterized by alpha spindle

differences – Figure 24) appears to have been used to suppress or enhance processing of visual field locations in

general, rather than suppress or enhance stimuli within those areas. As a result, the alpha power lateralization may

have been weaker. Another possibility is that, by introducing stimulus post-CVSA (as in the abovementioned

studies), the mental activity that is discriminated is not solely due to attention, but also due to anticipation of a

stimulus appearing – which has previously been demonstrated to induce distinguishable EEG activity [126]. A final

difference of the current study is that the research studies listed so far have all used wet electrode systems, which

have a significantly higher signal-to-noise ratio than that of dry systems [18], [19]. Since imagery and attention

Page 51: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

40

both result in fairly weak cortical activation [40], it is likely that it would be easier to detect the two mental

activities using wet rather than dry EEG acquisition systems.

3.5.3 Similarities between perception and imagery

Whereas offline perception versus rest accuracy (75.7%) was only moderately greater than offline imagery vs. rest

accuracy (71.2%), classification of perception in diagonally opposing quadrants was considerably more accurate

(UL vs. LR - 77.3%; LL vs. UR - 77.2%) than its imagery analogues (UL vs. LR - 65.2%; LL vs. UR - 65.0%). The

reason for this likely lies in the fact that perception of stimuli in the visual field reliably activates the visual cortex

through the bottom-up visual processing pathway. Conversely, recruitment of early visual cortices through top-

down neural pathways is much weaker: visuospatial imagery has previously been demonstrated to recruit similar

cortical regions as perception of stimuli in the same visual field locations, only to a much lesser degree [40]. Thus,

it may be the case that individuals can reliably activate the visual cortex when engaging in any visuospatial

imagery, but have greater difficulty in producing strongly localized cortical activations for different visuospatial

regions.

This is consistent with the results of the most commonly-selected CSD parameters. Spline flexibility

indicates the rigidity of the spherical head model used in CSD estimates, with low numbers being flexible and

higher numbers increasingly more rigid and a typical range between 2 to 6 [98], [127]. The lambda factor describes

the degree of smoothing that occurs over the spatial filter, with larger numbers indicating greater smoothing [98].

Finally, Legendre Pn define the spatial harmonic frequencies at each electrode in the spatial filter [95]. When

optimizing spatial filter parameters for perception trials, the most frequent ideal spline flexibilities were mid-range

(m = 3 and 4), and, there was a tendency for higher lambda values to be selected (lam 10-4.5, 10-4.25). Conversely,

more flexible spline models were selected for imagery trials (m =2 or 3), and the pattern of lambda values selected

was sporadic. Taken together, this may indicate that local sources of perception activity were best discriminated

when they were spread out to a greater degree, but required a more rigid spherical model to improve classification

accuracy. On the other hand, the spline model may have required more flexibility to accentuate key sources of

imagery, but due to a weaker signal, smoothing was less common. It is important to note that, while the values

above indicate some patterns in optimal CSD parameters in this group of participants, these are not necessarily the

ideal parameters for each individual – as each person has different head sizes, cortical folding and neuroanatomy

that may influence the effectiveness of specific parameters in such a spatial filter.

While classification of perception and imagery differed in accuracies and ideal CSD parameters, the two

appear to share common features that characterize their EEG signals and underlying mental activity. This is

partially evident through the frequency of features selected through elastic-net regularization in each fold of offline

cross-validation. When classified against rest, the most commonly selected features in both rest and perception

were magnitude-squared coherence of the alpha band, and phase-lag of alpha coherence (Figure 12). Likewise,

when classifying different quadrants, features that were selected to distinguish different types of imagery and

perception were alpha spindle lateralization and phase-lag of alpha coherence. As aforementioned, visuospatial

Page 52: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

41

attention seems to be characterized by imbalances in the hemispheric alpha frequency over the occipital lobe [124],

and this is consistent with the type of features selected in both perception and imagery tasks.

3.5.4 Multiclassification

The direct multi-category approach to classifying different imagery in different quadrants resulted in above-chance

accuracies for half of the participants who continued to Sessions 4 and 5. However, this approach was likely limited

to the fact that characteristic imbalances in EEG signals over the occipital region are most evident for left-right

hemifield activity, and less so for upper-lower hemifield activity. In fact, as demonstrated by Treder and colleagues,

the most discriminable quadrants appear to be diagonally-opposing (see [77]). Finally, since rest results in a

different class of EEG activity from imagery, it is of no surprise that the two-tier classification approach increased

classification accuracies by 18.8% over that attained with the direct approach.

When applying the two-tier classification in an online paradigm, however, the accuracy decreased

moderately from offline scores. While four of the six participants had online accuracies that fell within the standard

deviation of the offline scores, two participants had online scores that decreased below this range (Figure 21). One

potential reason for the relative decrease in accuracy from offline to online is attributable to the inability to search

for more accurate CSD parameters following collection of new offline data. Indeed, while CSD parameter

optimization did improve imagery accuracies, searching through 560 parameter combinations was computationally

intractable during retraining of the classifier in Sessions 4 and 5. Thus it was assumed that the ideal CSD

parameters identified from offline Sessions 1, 2 and 3 were optimal for Sessions 4 and 5 as well, despite the real

possibility for inter-session differences such as slight shifts of electrode locations or variations in mental state.

While the CSD spatial filter did improve accuracies in both perception and imagery paradigms, presumably due to

its ability to enhance local activity – the necessity of optimizing the filter to each participant is limiting. Alternative

spatial filters such as common spatial patterning, which instead rely on covariance measures for spatial filter

design, may thus be more suitable for retraining classifiers with same-day data; however these alternatives were

inappropriate for this paradigm due to the small number of channels [128], [129].

Clearly, multiclass discrimination is limited by the accuracies achievable by the multiple binary classifiers,

which each need to be tuned to individual differences before a practical four-class paradigm can be developed. At

the very least, however, the accuracies achieved by the highest-performing participant suggest that such a four-class

visuospatial imagery classifier is indeed feasible.

3.5.5 Predictors of accuracy

As the visuospatial imagery classification accuracies were largely variable across participants, one aim of this study

was to determine whether BCI performance could be predicted. In this way, future studies might use such metrics

to determine a priori participants with potential to perform well in specific paradigms, rather than having to blindly

collect multiple sessions of data from every participant. Furthermore, the identification of potential contributors to

classification error may highlight key factors that must be experimentally controlled in this paradigm. Several

potential correlates were investigated for both imagery versus rest accuracy and imagery versus imagery accuracy.

Page 53: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

42

The only factor that significantly correlated to offline imagery versus rest classification accuracies was the

number of features selected with elastic-net regularization during cross-validation (Table 3). The number of

selected features was also found to correlate positively and strongly to imagery vs. imagery classification

accuracies (Table 4). This suggests that the more useful features could be identified from the EEG signals, the

better the performance of the BCI (to a certain point). In other words, it is likely that individuals who performed

better simply had more features that could provide useful information for classification, as elastic-net regularization

selects features that add information while reducing redundancy [130]. This may be attributed in part to individual

degree of cortical activation from the mental task. However, to extract useful features, reliable signal acquisition is

also necessary. And while noise was not correlated to imagery vs. rest accuracies, this may have been somewhat

mitigated by the noise attenuation necessitated prior to implementing the spatial filter (see Appendix A). Thus, the

quality of the signal may have contributed to BCI performance, in addition to individual cortical activation.

As for predictors of imagery versus imagery accuracy, perception versus perception accuracies (with

imagery-optimized spatial filter parameters) was the only measure extracted external to the imagery trials to be

significant. Typically, the greater the accuracy in discriminating one quadrant of visuospatial perception to another,

the greater the accuracy in discriminating one quadrant of visuospatial imagery to another. This finding furthers the

notion that perception and imagery share common characteristics in cortical activity detectable through EEG. More

importantly, it suggests that one might predict how well an individual would do in an imagery task given how well

their EEG activity from a perception task can be discriminated. When combined with pre-session exhaustion, a

linear regression model could predict imagery classification accuracy with an adjusted r-squared of 0.834 (Figure

22). Intriguingly, this model perfectly predicted the six participants who were selected to continue on to sessions 4

and 5 based on the separate criteria that their imagery versus imagery accuracy means exceeded chance. This is

considered a more robust model than one with only perception accuracies, in which the top six participants could

not be predicted accurately (Figure 23). Prior to correction, pre-session fatigue appeared to negatively correlate

with performance, which is unsurprising given that the task requires a fair deal of focus and attention. In line with

this, alpha spindles have previously been demonstrate to reliably indicate levels of fatigue [108]. Thus, a

participant’s level of fatigue may have interfered with mental task ability. While perception trials and measures of

fatigue can be collected prior to collecting imagery trials, three important caveats to the predictive linear regression

model remain. First, the measure of pre-session exhaustion used in this study was reported after the session as part

of a questionnaire, and so may have been biased according to their level of fatigue at the time, or their perceived

change of fatigue. Second, the perception blocks were collected before each imagery block across each day, and so

this model included intersession differences that might not be predicted as well with the same number of perception

trials collected from a single session. Finally, the perception versus perception accuracies that best explained

imagery accuracies were those resulting from the use of imagery versus imagery-optimized CSD parameters, not

those optimized for visuospatial perception classification. Thus, while performance in visuospatial imagery could

be predicted with a very high degree of accuracy, some a priori knowledge about the ideal spatial filters was

necessary. In any case, the findings still suggest that approximately 83% of the variance in visuospatial imagery can

be explained by the detection of visuospatial perception mental activity, combined with perceived levels of fatigue.

Page 54: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

43

Whereas a previous study demonstrated posterior alpha to be predictive of BCI performance in a CVSA

paradigm [77], maximum baseline alpha across three sessions was not found to be predictive of imagery accuracies

using non-parametric tests. However, the two approaches to collect and calculate baseline alpha were slightly

different: participants had their eyes closed in study by Treder et al, whereas the participants in this study had their

eyes open. Furthermore, Treder and colleagues measured a minute of baseline prior to a session whereas in this

study, the average baseline of 30 seconds across three sessions was used as a correlate. Finally, the authors used

pooled alpha power over symmetric electrode pairs, while the current study only assessed the electrode resulting in

the maximum alpha power.

The only gender difference found was that males performed significantly better in visuospatial perception.

However, whereas the classification accuracies of 6/10 males exceeded chance when classifying imagery in either

diagonally-opposing quadrants, only 3/8 females exceeded chance in only one of the two classification problems.

Thus suggests a trend towards males having slightly more distinguishable visual cortical activation during

visuospatial imagery, which is consistent with literature suggesting a gender bias in the performance of visuospatial

attention and memory [131]. This may have contributed somewhat to the high individual variability in BCI

performance.

3.5.6 Differences in lateralization of alpha

Increased alpha power over the visual cortical regions corresponding to locations of unattended stimuli, and

decreased alpha power over regions processing attended stimuli has been well established [132][124][125][106].

Here, alpha lateralization was shown for the first time as a characteristic manifestation of cortical activity induced

by visuospatial imagery (Figure 24). Imagery in upper and lower left visual field quadrants resulted in greater left

hemisphere alpha spindle lateralization (indicating suppressed processing of the right visual hemifield, and

enhanced processing of the left visual hemifield), whereas visuospatial imagery in upper and lower right visual field

quadrants resulted in decreased left hemisphere alpha spindle lateralization. This difference was significant and

visually apparent in four of six participants in UL vs. LR imagery trials and five of six participants in LL vs. UR

imagery trials (Table 6). This demonstrates that there is no necessity for stimuli in the visual field to be suppressed

or enhanced, but rather that this modulation can be fully internally driven by top-down processes eliciting activity

in early visual cortical areas. Importantly, in the context of increased CVSA classification accuracies, it is likely

that having stimuli in the visual field (such as target and non-target circles) makes it easier for this top-down

modulation to occur [125]. In any case, it has now been demonstrated that similar patterns can be induced with

imagined stimuli in target visual field locations. One implication of this is that the use of lateralized alpha spindles

may be appropriate as a measure of visuospatial imagery in a feedback training paradigm.

While the majority of participants demonstrated the typical alpha lateralization pattern (alpha increase in

ipsilateral, alpha decrease in contralateral hemispheres), one participant showed the opposite. However, the channel

pair in which this was found to be significant was I1-I2, whereas for all other participants, the most significantly

different channel pairs were in more superior (PO1-4, P1-2) and lateral (PO7-PO8) locations. Incidentally, these

regions were previously found to be important in CVSA paradigms [77][56]. It is important to note that not every

Page 55: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

44

channel pair that was used as features in the BCI was tested for significant alpha lateralization to keep family-wise

error low. As a result, it is likely that other left-right channel pairs capture the characteristic alpha spindle

lateralization pattern better.

3.5.7 Potential sources of variability

Individual mental activation

There is high inter-individual variability in cortical activation during visual imagery [47], [59]. In fact, Slotnick et

al. demonstrated that cortical activation patterns of imagery were similar to those of perception in half their

participants, whereas in the remaining half, imagery did not significantly activate the striate cortex [40]. This is in

line with the findings of the current study, where the BCI exceeded chance levels in visuospatial imagery for at

least one pair of diagonally-opposing quadrants in half the participants (ranging between 65 and 89%), but could

not exceed chance for the remaining half. As suggested by the degree to which visuospatial perception accuracy

predicted visuospatial imagery accuracy, individual differences in cortical activity appear to play a role in

visuospatial imagery accuracy. Further, most participants achieved above-chance classification in rest versus any

imagery, yet only a third reached above-chance accuracies in both UL vs. LR and LL vs. UR imagery tasks. This

may simply be due to the higher number of samples used to train the classifier in the rest versus non-specific

imagery problem. More likely, however, is that the discriminable signals during non-specific visuospatial imagery

represent a shift from the default mode network at rest, towards some general visual cortical activation. Within or

subsequent to this general activation may lie the specific visuocortical activation unique to different visual field

locations. This is wholly consistent with the findings from Klein et al, who suggested visual imagery is a two-stage

process where first the cortex needs to be “turned on”, only after which it could be “fine-tuned” for specific

imagined shapes [59]. Thus, it is possible that the majority of participants were able to shift activity from the

default resting state to engage their visual cortex during imagery, but there was high variability in the degree to

which individuals could elicit differentiable activations when performing imagery in different quadrants.

Number and location of electrodes

It is also possible is that some individuals were able to engage their visual cortices in a way that produced

distinguishable activation patterns for different types of visuospatial imagery, however the electrodes were not in

optimal locations to pick up this (weak) activity. Although the standard electrode placement system allows for

similar regions to be recorded across participants, there are inter-individual variations in the neuroanatomical

structures located underneath [133]. For instance, Koessler et al found that parieto-occipital electrodes (e.g. PO3,

PO4) could be located over up to six different macro-anatomical cortical regions across different participants [133].

This limited generalizability of the standard electrode configuration may have contributed to the variability in ideal

spatial filter parameters, as well as the degree to which the spatial filter improved classification. Exact electrode

placement may not be necessary for stronger mental activation associated with perception, which may allow

corresponding signals to travel farther from their sources to more distal electrode sites. In contrast, a weaker signal

Page 56: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

45

such as one resulting from imagery may require electrodes situated in very specific locations, proximal to the

source of activity. The limited multiplicity of electrodes and corresponding configuration may have thus limited the

number of participants for whom visuospatial imagery signals could be detected. In fact, this may be the reason

why in three participants, imagery versus imagery classification was significant for one pair of opposing quadrants,

but not the other. Specifically, electrodes may have been located in optimal positions to detect imagery in some but

not all of the quadrants. This is consistent with the finding that the most significant channel pairs when determining

alpha spindle lateralization varied across participants, spanning 6 of the 7 tested pairs (Table 6).

Signal quality

As aforementioned, imagery results in weaker cortical activity than perception [40], which in turn likely

results in a weaker EEG signal, thereby decreasing SNR. The idea that weak signals limited this paradigm is

supported with the finding that spatial filters optimal for perception classification used larger smoothing factors

than those for imagery, presumably to enhance and spread the key signals of interest. This is further corroborated

by the finding that a larger number of selected features yielded greater classification accuracies in both imagery

versus rest and imagery versus imagery problems. In other words, weaker signals may have limited the degree to

which useful features could be extracted. On the other hand, rather than being caused solely by weak signal, the low

SNR may have been in part induced by high levels of noise. Dry systems have much higher impedances than

traditional wet setups; slight perturbances of the electrode-skin contact contaminate the signal with visibly high

amplitude noise inconsistent with cortical activity. While scaling down the amplitude of noisy epochs may have

diminished noise, key signals indicative of mental activity may have been attenuated as well. Additionally, noise

may have masked the activity of interest in imagery, but spared strong signals associated with perception. In any

case, signal quality is a key factor in BCI performance, and thus great care should be taken to mitigate noise prior to

collecting signals related to specific mental tasks.

3.5.8 Potential solutions to the variability

Participants performed much better online (77%) than they did offline (71%) – underlining the importance of

feedback for learning and engagement. This is supported by the finding that fatigue negatively correlated with BCI

performance. While in fMRI studies, visual imagery did not significantly activate the cortex in all participants [40],

[59], the possibility that visual cortical activation may be learned or improved through a training paradigm with

appropriate feedback has yet to be explored. In fact, a recent study demonstrated that feedback consistent with

alpha spindle activity over the occipital lobe resulted in increased alpha spindle frequency when compared to

controls receiving feedback not reflective of their alpha spindle activity [134]. This is promising, as alpha-spindle

lateralization has been demonstrated in this study as a useful feature of visuospatial imagery, in turn making it

viable for use in such a feedback paradigm. Furthermore, in this study, it was not in “turning on” the visual cortex

that few participants managed to exceed chance classification (i.e. discriminating rest versus non-specific

visuospatial imagery), but rather the “fine-tuning” of the visual stimulus (i.e. discriminating imagery in different

quadrants). Thus, as the first step of visual imagery suggested by Klein et al. seems to be present in the majority of

Page 57: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

46

participants (with 14/18 reaching the 70% threshold using the sophisticated BCI on online data) [59], a training

paradigm may serve to teach individuals to elicit differential cortical activation depending on the location of the

visual imagery stimulus.

Future works should consider a greater number of electrodes over the occipital region, as well as more

fronto-parietal regions. In fact, significant alpha spindle lateralization in this study was consistently in more

superior regions (parietal, parieto-occipital), and farther from inferior regions (inion, occipital), so more electrodes

may be necessary in more parietal areas, and more densely around parieto-occipital areas. For weaker signals such

as imagery, a greater density of electrodes and perhaps a greater number of electrodes surrounding the area of

interest may be necessary to detect the activity of interest. In addition, differences between rest and imagery would

likely be much more distinguishable by measuring the contribution of more frontal (i.e. “top”) regions engaged in

top-down activation of the visual cortex. Finally, with a larger number and greater density of electrodes, alternative

spatial filters such as common spatial patterns may be more appropriate.

Lastly, future research in this paradigm should also consider using wet electrodes as they may increase

SNR. Wet electrodes may also capture low-gamma (30-80 Hz) activity that has been linked to visual imagery and

that is indicative of fronto-occipital activation [135][136][137].

3.5.9 Key messages

1. Perception vs. rest classification was lower in this study than in previously reported SSVEP attentional

studies, likely due to several experimental differences as the present protocol was optimized for imagery

rather than perception classification. (Objective 3)

2. Imagery vs. rest classification was generally on par with previously reported research while the present

paradigm offers some distinct practical advantages, including the accommodation of dry electrodes,

relinquishing the need for visual stimuli, circumventing the modest number of discernible classes inherent

to higher-order cognitive BCIs by exploiting visual retinotopic activity, and requiring only brief durations

of imagery. Recent studies boasting higher imagery classification accuracies appear to be plagued by

confounding neural and ocular signals unrelated to imagery. (Objective 1)

3. Visuospatial perception appears to be more easily discriminated than visuospatial imagery, likely due to

the difference in cortical activation as evidenced in part by the dissimilar ideal spatial filter parameters for

each mental task. Furthermore, while fine-tuned cortical activity indicative of visuospatial imagery

location is difficult to discriminate, most participants elicit easily discriminable general activation of their

visual cortex during imagery. This is consistent with the suggested stages of visual imagery in previous

literature. (Objective 3)

4. The degree of useful information gathered from EEG signals may have been masked by noise, suggesting

signal quality may have been limiting for this such a paradigm dependent on weaker cortical signals.

Additionally, the combination of two predictors (fatigue and visuospatial perception performance)

Page 58: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

47

explained the majority of variance in imagery vs. imagery accuracy, indicating detection of individual

cortical activation and mental state may be key factors to exploit for maximizing BCI performance in

visuospatial imagery. (Objectives 2 and 3)

5. A feature of alpha power imbalance over the occipital lobe (alpha spindle lateralization) has been

demonstrated without target/non-target stimuli in the visual field, suggesting that posterior alpha

synchronization/de-synchronization can occur entirely through internally driven top-down mechanisms,

and that stimuli may simply serve to increase this modulation. (Objective 3)

6. While only a handful of individuals were able to exceed above-chance accuracies in the 4-class BCI

described in this study, effective levels of performance such as those exemplified by one participant may

be achieved in more individuals by improving binary classification performance. This may be

accomplished by prioritizing signal quality, increasing detection of key signals, and providing feedback

aimed towards eliciting more discriminable cortical activity. (Objectives 1 and 2)

3.6 Conclusion

This study established the feasibility of a BCI dependent on visuospatial imagery. In addition, this study

demonstrated that practical BCI accuracies can be achieved in an online rest vs. imagery paradigm. The study also

realized a proof-of-concept four-class online navigation game dependent on visuospatial imagery, demonstrating

above-chance in a handful of individuals. Unlike previous research, this study used a dry system with a small

number of electrodes, rendering the proposed system conducive to real-world implementation. Finally, to the best

of the author’s knowledge, this is the first study that demonstrated lateralized alpha in visuospatial imagery without

any stimuli in target or non-target locations of the visual field.

Page 59: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

48

Chapter 4 Conclusion

Conclusion

4.1 Contributions

This study has contributed to the field of biomedical engineering by demonstrating the potential of a novel BCI

paradigm dependent on visuospatial imagery. The specific contributions of this study are as follows:

1. Development of an effective online binary BCI with the potential to distinguish non-specific visuospatial

imagery from resting state above the 70% threshold for a practical BCI.

2. Development of an offline binary BCI with the potential to distinguish imagery in pairs of diagonally-

opposing quadrants in a handful of individuals above chance.

3. Identification of the influence of two factors (fatigue and degree of differentiable cortical activation from

perception tasks) on the variability of individual differences in discriminating visuospatial imagery.

4. Development of an online proof-of-concept 4-class BCI system dependent on visuospatial imagery in the

four quadrants for four corresponding navigational directions.

5. Identification of lateralization in an alpha frequency band feature (alpha spindle) as a key characteristic of

visuospatial imagery in opposing quadrants; similar measures of which had previously only been

demonstrated in paradigms of attention.

4.2 Future Work

This is the first study to current knowledge that has identified visuospatial imagery as a potential control paradigm

in BCIs. As a result, there are a number of potential avenues to be explored in order to improve upon this paradigm.

The first is to use higher quality signal acquisition systems such as wet EEG, rather than those susceptible to noise

(i.e. dry EEG). Such an approach may improve the SNR for the relatively weak signal that is elicited from visual

imagery. Next, future works may wish to increase the number and density of electrodes in order to capitalize on

inter-individual differences in cortical structure and activation in order to improve the generalizability of this

paradigm. This approach can also be exploited in order to explore the contributions of more frontal processing

regions on visuospatial imagery. Furthermore, in terms of making the visuospatial imagery BCI more generalizable

across individuals, it may be of interest to determine whether user-personalized imagery stimuli engage the visual

cortex to a greater degree than the neutral arrow stimulus used in this study. Finally, future studies will now be able

to use alpha spindle lateralization as a feedback tool in order to train individuals to elicit more distinguishable

visual cortical activations, and further improve the visuospatial imagery BCI paradigm.

Page 60: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

References

[1] E. Smith and M. Delargy, “Locked-in syndrome,” Br. Med. J., vol. 330, no. February, pp. 3–6, 2005.

[2] D. J. McFarland and J. R. Wolpaw, “Brain-Computer Interfaces for Communication and Control,” Commun.

ACM, vol. 54, no. 5, pp. 60–66, 2011.

[3] P. Cipresso, L. Carelli, F. Solca, D. Meazzi, P. Meriggi, B. Poletti, D. Lul??, A. C. Ludolph, V. Silani, and

G. Riva, “The use of P300-based BCIs in amyotrophic lateral sclerosis: From augmentative and alternative

communication to cognitive assessment,” Brain Behav., vol. 2, no. 4, pp. 479–498, 2012.

[4] A. Kubler, B. Kotchoubey, T. Hinterberger, N. Ghanayim, J. Perelmouter, M. Schauer, C. Fritsch, E. Taub,

and N. Birbaumer, “The thought translation device: a neurophysiological approach to communication in

total motor paralysis.,” Exp. brain Res., vol. 124, no. 2, pp. 223–232, Jan. 1999.

[5] N. J. Hill, T. N. Lal, M. Schröder, T. Hinterberger, B. Wilhelm, F. Nijboer, U. Mochty, G. Widman, C.

Elger, B. Schölkopf, A. Kübler, and N. Birbaumer, “Training for Fast BCI Implementation : Comparison

of,” vol. 14, no. 2, pp. 183–186, 2006.

[6] E. W. Sellers and E. Donchin, “A P300-based brain-computer interface: Initial tests by ALS patients,” Clin.

Neurophysiol., vol. 117, no. 3, pp. 538–548, 2006.

[7] A. Burns, H. Adeli, and J. a. Buford, “Brain-Computer Interface after Nervous System Injury,” Neurosci.,

vol. 20, pp. 639–651, 2014.

[8] K. Tai, S. Blaine, and T. Chau, “A Review of Emerging Access Technologies for Individuals With Severe

Motor Impairments,” Assist. Technol., vol. 20, no. May, pp. 204–219, 2008.

[9] S. Weyand, K. Takehara-Nishiuchi, and T. Chau, “Weaning Off Mental Tasks to Achieve Voluntary Self-

Regulatory Control of a Near-Infrared Spectroscopy Brain-Computer Interface,” IEEE Trans. Neural Syst.

Rehabil. Eng., vol. 23, no. 4, pp. 548–561, 2015.

[10] N. Birbaumer, “Breaking the silence : Brain – computer interfaces ( BCI ) for communication and motor

control,” Psychophysiology, vol. 43, pp. 517–532, 2006.

[11] O. Bai, L. Peter, D. Huang, D.-Y. Fei, and M. K. Floeter, “Towards a User-Friendly Brain-Computer

Interface: Initial Tests in ALS and PLS Patients,” Clin. Neurophysiol., vol. 121, no. 8, pp. 1293–1303, 2010.

[12] J. Faller, R. Scherer, E. V. C. Friedrich, U. Costa, E. Opisso, J. Medina, and G. R. Müller-Putz, “Non-motor

tasks improve adaptive brain-computer interface performance in users with severe motor impairment.,”

Front. Neurosci., vol. 8, no. October, p. 320, 2014.

[13] F. Fiori, a Sedda, E. R. Ferrè, a Toraldo, M. Querzola, F. Pasotti, D. Ovadia, C. Piroddi, R. Dell’aquila, C.

Lunetta, M. Corbo, and G. Bottini, “Exploring motor and visual imagery in Amyotrophic Lateral Sclerosis.,”

Exp. brain Res., vol. 226, no. 4, pp. 537–47, 2013.

[14] J. R. Wolpaw, N. Birbaumer, D. J. Mcfarland, G. Pfurtscheller, and T. M. Vaughan, “Brain – computer

interfaces for communication and control,” Clin. Neurophysiol., vol. 113, pp. 767–791, 2002.

[15] L. F. Nicolas-Alonso and J. Gomez-Gil, “Brain computer interfaces, a review,” Sensors, vol. 12, no. 2, pp.

1211–1279, 2012.

[16] B. Burle, L. Spieser, C. Roger, L. Casini, T. Hasbroucq, and F. Vidal, “Spatial and temporal resolutions of

Page 61: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

50

EEG: Is it really black and white? A scalp current density view,” Int. J. Psychophysiol., vol. 97, no. 3, pp.

210–220, 2015.

[17] T. Ball, M. Kern, I. Mutschler, A. Aertsen, and A. Schulze-Bonhage, “Signal quality of simultaneously

recorded invasive and non-invasive EEG,” Neuroimage, vol. 46, no. 3, pp. 708–716, 2009.

[18] M. A. Lopez-Gordo, D. Sanchez Morillo, and F. Pelayo Valle, “Dry EEG Electrodes,” Sensors, vol. 14, no.

7, pp. 12847–12870, 2014.

[19] F. Wang, G. Li, and J. Chen, “Applying dynamic data collection to improve dry electrode system

performance for a P300-based brain – computer interface,” J. Neural Eng., vol. 13, no. 6, pp. 1–11, 2016.

[20] S. Ge, R. Wang, and D. Yu, “Classification of Four-Class Motor Imagery Employing Single-Channel

Electroencephalography,” PLoS One, vol. 9, no. 6, pp. 1–7, 2014.

[21] R. N. Vigario, “Extraction of ocular artefacts from EEG using independent component analysis,”

Electroencephalogr. Clin. Neurophysiol., vol. 103, pp. 395–404, 1997.

[22] G. Gómez-Herrero, “Automatic Artifact Removal ( AAR ) toolbox Germ ´,” 2007.

[23] L. A. Farwell and E. Donchin, “Talking off the top of your head: toward a mental prosthesis utilizing event-

related brain potentials,” Electroencephalogr. Clin. Neurophysiol., vol. 70, no. 6, pp. 510–523, 1988.

[24] C. R. Turner, A. Fuggetta, L. Lavazza, and A. L. Wolf, “A conceptual basis for feature engineering,” J. Syst.

Softw., vol. 49, pp. 3–15, 1999.

[25] F. Lotte and M. Congedo, “A review of classification algorithms for EEG-based brain – computer

interfaces,” J. Neural Eng., vol. 4, pp. 1–13, 2007.

[26] D. J. McFarland, C. W. Anderson, K. R. Müller, A. Schlögl, and D. J. Krusienski, “BCI Meeting 2005 -

Workshop on BCI signal processing: Feature extraction and translation,” IEEE Trans. Neural Syst. Rehabil.

Eng., vol. 14, no. 2, pp. 135–138, 2006.

[27] L. Yu and H. Liu, “Feature Selection for High-Dimensional Data: A Fast Correlation-Based Filter Solution,”

Int. Conf. Mach. Learn., pp. 1–8, 2003.

[28] P. Pudil, J. Novovieova, and J. Kittler, “Floating search methods in feature selection,” Pattern Recognit.

Lett., vol. 15, no. June 1993, 1994.

[29] K. P. Thomas, A. P. Vinod, and C. Guan, “Design of an online EEG based neurofeedback game for

enhancing attention and memory,” Conf. Proc. ... Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. IEEE Eng.

Med. Biol. Soc. Annu. Conf., vol. 2013, no. ii, pp. 433–436, 2013.

[30] T. O. Zander and C. Kothe, “Towards passive brain–computer interfaces: applying brain–computer interface

technology to human–machine systems in general,” J. Neural Eng., vol. 8, no. 2, p. 25005, 2011.

[31] E. Yin and T. Zeyl, “An Auditory-Tactile Visual Saccade-Independent P300 Brain – Computer Interface,”

Int. J. Neural Syst., vol. 26, no. 1, pp. 1–16, 2016.

[32] G. R. Muller-Putz, R. Scherer, and G. Pfurtscheller, “Brain-computer interfaces for control of

neuroprostheses : from synchronous to asynchronous mode of operation,” Biomed. Eng. (NY)., vol. 51, pp.

57–63, 2006.

[33] M. F. Bear, B. W. Connors, and M. A. Paradiso, Neuroscience - Exploring the Brain Fourth Edition, Fourth.

Lippincott Williams & Wilkins, 2016.

Page 62: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

51

[34] M. J. Farah, The cognitive neuroscience of vision. Blackwell Publishing, 2000.

[35] M. W. Levine and J. J. McAnany, “The relative capabilities of the upper and lower visual hemifields,”

Vision Res., vol. 45, no. 21, pp. 2820–2830, 2005.

[36] D. J. Hagler, “Visual field asymmetries in visual evoked responses,” vol. 14, no. 2014, pp. 1–19, 2014.

[37] G. Harding, P. Harding, and A. Wilkins, “Wind turbines, flicker, and photosensitive epilepsy:

Characterizing the flashing that may precipitate seizures and optimizing guidelines to prevent them,”

Epilepsia, vol. 49, no. 6, pp. 1095–1098, 2008.

[38] K. Rauss, S. Schwartz, and G. Pourtois, “Top-down effects on early visual processing in humans: A

predictive coding framework,” Neurosci. Biobehav. Rev., vol. 35, no. 5, pp. 1237–1253, 2011.

[39] J. Pearson, T. Naselaris, E. A. Holmes, and S. M. Kosslyn, “Mental Imagery: Functional Mechanisms and

Clinical Applications,” Trends Cogn. Sci., vol. 19, no. 10, pp. 590–602, 2015.

[40] S. D. Slotnick and W. L. Thompson, “Visual Mental Imagery Induces Retinotopically Organized Activation

of Early Visual Areas,” Cereb. Cortex, vol. 15, pp. 1570–1583, 2005.

[41] G. Ganis, W. L. Thompson, and S. M. Kosslyn, “Brain areas underlying visual mental imagery and visual

perception: An fMRI study,” Cogn. Brain Res., vol. 20, no. 2, pp. 226–241, 2004.

[42] A. M. Albers, P. Kok, I. Toni, H. C. Dijkerman, and F. P. De Lange, “Report Shared Representations for

Working Memory and Mental Imagery in Early Visual Cortex,” Curr. Biol., vol. 23, no. 15, pp. 1427–1431,

2013.

[43] C. McCormick, M. St-Laurent, A. Ty, T. a. Valiante, and M. P. McAndrews, “Functional and effective

hippocampal-neocortical connectivity during construction and elaboration of autobiographical memory

retrieval,” Cereb. Cortex, vol. 25, no. 5, pp. 1297–1305, 2015.

[44] M.-S. Shin, S.-Y. Park, S.-R. Park, S.-H. Seol, and J. S. Kwon, “Clinical and empirical applications of the

Rey-Osterrieth Complex Figure Test.,” Nat. Protoc., vol. 1, no. 2, pp. 892–899, 2006.

[45] D. F. Marks and A. R. Isaac, “Topographical distribution of EEG activity accompanying visual and motor

imagery in vivid and non-vivid imagers,” Br. J. Psychol., vol. 86, pp. 271–282, 1995.

[46] A. Campos, “Internal consistency and construct validity of two versions of the Revised Vividness of Visual

Imagery Questionnaire,” Percept. Mot. Skills, vol. 113, no. 2, pp. 454–460, 2011.

[47] X. Cui, C. B. Jeter, D. Yang, P. R. Montague, and D. M. Eagleman, “Vividness of mental imagery:

Individual variability can be measured objectively,” Vision Res., vol. 47, no. 4, pp. 474–478, 2007.

[48] J. Andrade, J. May, C. Deeprose, S. J. Baugh, and G. Ganis, “Assessing vividness of mental imagery: The

plymouth sensory imagery questionnaire,” Br. J. Psychol., vol. 105, no. 4, pp. 547–563, 2014.

[49] S. D. Slotnick, S. a Klein, T. Carney, E. Sutter, and S. Dastmalchi, “Using multi-stimulus VEP source

localization to obtain a retinotopic map of human primary visual cortex,” Clin. Neurophysiol., vol. 110, pp.

1793–1800, 1999.

[50] C.-H. Im, A. Gururajan, N. Zhang, W. Chen, and B. He, “Spatial Resolution of EEG Cortical Source

Imaging Revealed by Localization of Retinotopic Organization in Human Primary Visual Cortex,” J.

Neurosci. Methods, vol. 161, no. 1, pp. 142–154, 2008.

[51] Z. Yan, X. Gao, G. Bin, B. Hong, and S. Gao, “A half-field stimulation pattern for SSVEP-based brain-

Page 63: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

52

computer interface,” Proc. 31st Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. Eng. Futur. Biomed. EMBC

2009, no. 2006, pp. 6461–6464, 2009.

[52] Y. Punsawad, S. Member, and Y. Wongsawat, “Multi-Command SSVEP-based BCI System via Single

Flickering Frequency Half-Field Stimulation Pattern,” Proc. 33rd Annu. Int. Conf. IEEE Eng. Med. Biol.

Soc. Eng. Futur. Biomed. EMBC 2011, pp. 1101–1104, 2011.

[53] A. Tzovara, M. M. Murray, G. Plomp, M. H. Herzog, C. M. Michel, and M. De Lucia, “Decoding stimulus-

related information from single-trial EEG responses based on voltage topographies,” Pattern Recognit., vol.

45, no. 6, pp. 2109–2122, 2012.

[54] N. Cicmil, H. Bridge, A. J. Parker, M. W. Woolrich, and K. Krug, “Localization of MEG human brain

responses to retinotopic visual stimuli with contrasting source reconstruction approaches,” Front. Neurosci.,

vol. 8, no. 8 MAY, pp. 1–16, 2014.

[55] A. Kubler, V. K. Mushahwar, L. R. Hochberg, and J. P. Donoghue, “BCI Meeting 2005--workshop on

clinical issues and applications.,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 14, no. 2, pp. 131–134, Jun.

2006.

[56] L. Tonin, R. Leeb, A. Sobolewski, and J. R. Mill, “An online EEG BCI based on covert visuospatial

attention in absence of exogenous stimulation,” J. Neural Eng., vol. 10, 2013.

[57] P. Bobrov, A. Frolov, C. Cantor, I. Fedulova, M. Bakhnyan, and A. Zhavoronkov, “Brain-Computer

Interface Based on Generation of Visual Images,” PLoS One, vol. 6, no. 6, 2011.

[58] M. J. Tarr and Y. D. Cheng, “Learning to see faces and objects,” Trends Cogn. Sci., vol. 7, no. 1, pp. 23–30,

2003.

[59] I. Klein, J. Dubois, F. Kherif, G. Flandin, J. Poline, M. Denis, S. M. Kosslyn, and D. Le, “Retinotopic

organization of visual mental images as revealed by functional magnetic resonance imaging,” Cogn. Brain

Res., vol. 22, pp. 26–31, 2004.

[60] J. R. Wolpaw, N. Birbaumer, D. J. McFarland, G. Pfurtscheller, and T. M. Vaughan, “Brain-computer

interfaces for communication and control.,” Clin. Neurophysiol., vol. 113, no. 6, pp. 767–91, 2002.

[61] J. Wolpaw, N. Birbaumer, D. J. McFarland, G. Pfurtscheller, and T. M. Vaughan, “Brain–computer

interfaces for communication and control,” Clin. Neurophysiol., vol. 113, no. 6, pp. 767–791, 2002.

[62] A. Myrden and T. Chau, “Effects of user mental state on EEG-BCI performance,” Front. Hum. Neurosci.,

vol. 9, no. June, pp. 1–11, 2015.

[63] S. Sutton, M. Braren, J. Zubin, and E. R. John, “Evoked-Potential Correlates of Stimulus Uncertainty

Author ( s ): Samuel Sutton , Margery Braren , Joseph Zubin and E . R . John,” Science (80-. )., vol. 150, no.

3700, pp. 1187–1188, 1965.

[64] C. Guger, S. Daban, E. Sellers, C. Holzner, G. Krausz, R. Carabalona, F. Gramatica, and G. Edlinger, “How

many people are able to control a P300-based brain-computer interface (BCI)?,” Neurosci. Lett., vol. 462,

no. 1, pp. 94–98, 2009.

[65] A. Miyake, N. P. Friedman, M. J. Emerson, A. H. Witzki, A. Howerter, and T. D. Wager, “The Unity and

Diversity of Executive Functions and Their Contributions to Complex ‘Frontal Lobe’ Tasks: A Latent

Variable Analysis,” Cogn. Psychol., vol. 41, no. 1, pp. 49–100, 2000.

Page 64: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

53

[66] O. Bai, P. Lin, D. Huang, D. Y. Fei, and M. K. Floeter, “Towards a user-friendly brain-computer interface:

Initial tests in ALS and PLS patients,” Clin. Neurophysiol., vol. 121, no. 8, pp. 1293–1303, 2010.

[67] G. Pfurtscheller, P. Linortner, R. Winkler, G. Korisek, and G. Müller-Putz, “Discrimination of motor

imagery-induced EEG patterns in patients with complete spinal cord injury,” Comput. Intell. Neurosci., vol.

2009, pp. 1–7, 2009.

[68] J. R. Kirby, P. J. Moore, and N. J. Schofield, “Verbal and visual learning styles,” Contemp. Educ. Psychol.,

vol. 13, no. 2, pp. 169–184, 1988.

[69] D. J. M. Kraemer, L. M. Rosenberg, and S. L. Thompson-Schill, “The Neural Correlates of Visual and

Verbal Cognitive Styles,” J. Neurosci., vol. 29, no. 12, pp. 3792–3798, 2009.

[70] M. Zvyagintsev, B. Clemens, N. Chechko, K. A. Mathiak, A. T. Sack, and K. Mathiak, “Brain networks

underlying mental imagery of auditory and visual information,” Eur. J. Neurosci., vol. 37, no. 9, pp. 1421–

1434, 2013.

[71] M.-C. Dobrea and D. M. Dobrea, “The selection of proper discriminative cognitive tasks - A necessary

prerequisite in high-quality BCI applications,” Int. Symp. Appl. Sci. Biomed. Commun. Technol., pp. 1–6,

2009.

[72] S. Weyand, K. Takehara-Nishiuchi, and T. Chau, “Exploring methodological frameworks for a mental task-

based near-infrared spectroscopy brain–computer interface,” J. Neurosci. Methods, vol. 254, pp. 36–45,

2015.

[73] P. Andersson, N. F. Ramsey, J. P. W. Pluim, and M. A. Viergever, “BCI control using 4 direction spatial

visual attention and real-time fMRI at 7T,” pp. 4221–4225, 2010.

[74] P. Andersson, N. F. Ramsey, M. Raemaekers, M. A. Viergever, and J. P. W. Pluim, “Real-time decoding of

the direction of covert visuospatial attention,” J. Neural Eng., vol. 9, no. 4, 2012.

[75] M. van Gerven and O. Jensen, “Attention modulations of posterior alpha as a control signal for two-

dimensional brain-computer interfaces,” J. Neurosci. Methods, vol. 179, no. 1, pp. 78–84, 2009.

[76] E. Astrand, C. Wardak, and S. Ben Hamed, “Selective visual attention to drive cognitive brain-machine

interfaces: from concepts to neurofeedback and rehabilitation applications,” Front. Syst. Neurosci., vol. 8,

no. August, pp. 1–16, 2014.

[77] M. S. Treder, A. Bahramisharif, N. M. Schmidt, M. A. van Gerven, and B. Blankertz, “Brain-computer

interfacing using modulations of alpha activity induced by covert shifts of attention,” J. Neuroeng. Rehabil.,

vol. 8, no. 1, p. 24, 2011.

[78] H.-J. Hwang, V. Y. Ferreria, D. Ulrich, T. Kilic, X. Chatziliadis, B. Blankertz, and M. Treder, “A Gaze

Independent Brain-Computer Interface Based on Visual Stimulation through Closed Eyelids,” Sci. Rep., vol.

5, no. June, p. 15890, 2015.

[79] M. E. Thurlings, A.-M. Brouwer, J. B. F. Van Erp, and P. Werkhoven, “Gaze-independent ERP-BCIs:

augmenting performance through location-congruent bimodal stimuli,” Front. Syst. Neurosci., vol. 8, no.

September, pp. 1–14, 2014.

[80] A. Ishai, L. G. Ungerleider, and J. V. Haxby, “Distributed neural systems for the generation of visual

images,” Neuron, vol. 28, no. 3, pp. 979–990, 2000.

Page 65: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

54

[81] A. Delorme and S. Makeig, “EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics

including independent component analysis,” J. Neurosci. Methods, vol. 134.1, no. 1, pp. 9–21, 2004.

[82] H. Aurlien, I. O. Gjerde, J. H. Aarseth, G. Eldøen, B. Karlsen, H. Skeidsvoll, and N. E. Gilhus, “EEG

background activity described by a large computerized database,” Clin. Neurophysiol., vol. 115, no. 3, pp.

665–673, 2004.

[83] P. A. Osterrieth, “Le test de copie d’une figure complexe,” Arch. Psychol. (Geneve)., vol. 30, pp. 206–356,

1944.

[84] A. Rey, “L’examen psychologique dans les cas d’encéphalopathie traumatique (Les problems),” Arch.

Psychol. (Geneve)., vol. 28, pp. 215–285, 1941.

[85] D. F. Marks, “New directions for mental imagery research,” J. Ment. Imag., vol. 19, no. 3–4, pp. 153–167,

1995.

[86] G. Powell, P. Sumner, and A. Bompas, “The effect of eye movements and blinks on afterimage appearance

and duration,” J. Vis., vol. 15, no. 3, pp. 20–20, 2015.

[87] K. B. Ng, A. P. Bradley, and R. Cunnington, “Effect of posterized naturalistic stimuli on SSVEP-based

BCI,” Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. EMBS, pp. 3105–3108, 2013.

[88] S. D. Slotnick, S. A. Klein, T. Carney, and E. E. Sutter, “Electrophysiological estimate of human cortical

magnification,” Clin. Neurophysiol., vol. 112, no. 7, pp. 1349–1356, 2001.

[89] S. D. Slotnick and S. Yantis, “Efficient acquisition of human retinotopic maps,” Hum. Brain Mapp., vol. 18,

no. 1, pp. 22–29, 2003.

[90] D. Kennard, R. Hartmann, D. Kraft, and B. Boshes, “Perceptual Suppression,” Vision Res., vol. 10, no. 7,

pp. 575–585, 1970.

[91] W. L. Thompson, Y. Hsiao, and S. M. Kosslyn, “Dissociation between visual attention and visual mental

imagery,” J. Cogn. Psychol., vol. 23, no. 2, pp. 256–263, 2011.

[92] G. Thut, A. Nietzel, S. Brandt, and A. Pascual-Leone, “a-Band Electroencephalographic Activity over

Occipital Cortex Indexes Visuospatial Attention Bias and Predicts Visual Target Detection,” J. Neurosci.,

vol. 26, no. 37, pp. 9494–9502, 2006.

[93] J. Qian, T. Hastie, J. Friedman, R. Tibshirani, and N. Simon, “Glmnet for Matlab,” 2013. .

[94] J. Kayser, “Current source density (CSD) interpolation using spherical splines - CSD Toolbox (Version

1.1),” New York State Psychiatric Institute: Division of Cognitive Neuroscience., 2009. [Online]. Available:

http://psychophysiology.cpmc.columbia.edu/Software/CSDtoolbox.

[95] M. X. Cohen, “Surface Laplacian,” in Analyzing Neural Time Series Data: Theory and Practice, MIT Press,

2014, pp. 275–290.

[96] F. Babiloni, C. Babiloni, L. Fattorini, F. Carducci, P. Onorati, and A. Urbano, “Performances of surface

Laplacian estimators: A study of simulated and real scalp potential distributions,” Brain Topogr., vol. 8, no.

1, pp. 35–45, 1995.

[97] J. Kayser and C. E. Tenke, “Principal components analysis of Laplacian waveforms as a generic method for

identifying ERP generator patterns: I. Evaluation with auditory oddball tasks,” Clin. Neurophysiol., vol. 117,

no. 2, pp. 369–380, 2006.

Page 66: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

55

[98] J. Kayser and C. E. Tenke, “Principal components analysis of Laplacian waveforms as a generic method for

identifying ERP generator patterns: II. Adequacy of low-density estimates,” Clin. Neurophysiol., vol. 117,

no. 2, pp. 369–380, 2006.

[99] J. Kayser and C. E. Tenke, “Issues and considerations for using the scalp surface Laplacian in EEG/ERP

research: A tutorial review,” Int. J. Psychophysiol., vol. 97, no. 3, pp. 189–209, 2015.

[100] J. Kayser and C. E. Tenke, “On the benefits of using surface Laplacian (current source density) methodology

in electrophysiology,” Int. J. Psychophysiol., vol. 97, no. 3, pp. 171–173, 2015.

[101] A. Sonnleitner, M. Simon, W. E. Kincses, A. Buchner, and M. Schrauf, “Alpha spindles as

neurophysiological correlates indicating attentional shift in a simulated driving task,” Int. J. Psychophysiol.,

vol. 83, no. 1, pp. 110–118, 2012.

[102] W. Klimesch, “Alpha-band oscillations, attention, and controlled access to stored information,” Trends

Cogn. Sci., vol. 16, no. 12, pp. 606–617, 2012.

[103] J. J. Foxe and A. C. Snyder, “The role of alpha-band brain oscillations as a sensory suppression mechanism

during selective attention,” Front. Psychol., vol. 2, no. JUL, pp. 1–13, 2011.

[104] N. Schmidt, B. Blankertz, and M. S. Treder, “??-modulation induced by covert attention shifts as a new

input modality for EEG-based BCIs,” Conf. Proc. - IEEE Int. Conf. Syst. Man Cybern., pp. 481–487, 2010.

[105] A. Ikkai, S. Dandekar, and C. E. Curtis, “Lateralization in Alpha-Band Oscillations Predicts the Locus and

Spatial Distribution of Attention,” pp. 1–17, 2016.

[106] S. P. Kelly, E. C. Lalor, R. B. Reilly, and J. J. Foxe, “Increases in Alpha Oscillatory Power Reflect an

Active Retinotopic Mechanism for Distracter Suppression During Sustained Visuospatial Attention,” J.

Neurophysiol., vol. 95, no. 6, pp. 3844–3851, 2006.

[107] L. Tonin, R. Leeb, and J. del R Millán, “Time-dependent approach for single trial classification of covert

visuospatial attention,” J. Neural Eng., vol. 9, no. 4, p. 45011, 2012.

[108] M. Simon, E. A. Schmidt, W. E. Kincses, M. Fritzsche, A. Bruns, C. Aufmuth, M. Bogdan, W. Rosenstiel,

and M. Schrauf, “Eeg alpha spindle measures as indicators of driver fatigue under real traffic conditions,”

Clin. Neurophysiol., vol. 122, no. 6, pp. 1168–1178, 2011.

[109] Y. Ke, L. Chen, L. Fu, Y. Jia, P. Li, X. Zhao, H. Qi, P. Zhou, L. Zhang, B. Wan, and D. Ming, “Visual

attention recognition based on nonlinear dynamical parameters of EEG,” Biomed. Mater. Eng., vol. 24, no.

1, pp. 349–355, 2014.

[110] W. Li, D. Ming, R. Xu, H. Ding, H. Qi, and B. Wan, “Research on visual attention classification based on

EEG entropy parameters,” IFMBE Proc., vol. 39 IFMBE, no. 3, pp. 1553–1556, 2013.

[111] C. Chang and C. Lin, “LIBSVM : A Library for Support Vector Machines,” ACM Trans. Intell. Syst.

Technol., vol. 2, no. 3, pp. 1–27, 2011.

[112] and C.-J. L. Chih-Wei Hsu, Chih-Chung Chang, “A Practical Guide to Support Vector Classification,” BJU

Int., vol. 101, no. 1, pp. 1396–400, 2008.

[113] E. Combrisson and K. Jerbi, “Exceeding chance level by chance: The caveat of theoretical chance levels in

brain signal classification and statistical assessment of decoding accuracy,” J. Neurosci. Methods, vol. 250,

pp. 126–136, 2015.

Page 67: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

56

[114] P. C. Austin and E. W. Steyerberg, “The number of subjects per variable required in linear regression

analyses,” J. Clin. Epidemiol., vol. 68, no. 6, pp. 627–636, 2015.

[115] M. Xu, Y. Wang, M. Nakanishi, and Y. Wang, “Fast detection of covert visuospatial attention using hybrid

N2pc and SSVEP features,” J. Neural Eng., vol. 13, 2016.

[116] A. Maye, D. Zhang, and A. K. Engel, “Utilizing Retinotopic Mapping for a Multi-Target SSVEP BCI With

a Single Flicker Frequency,” IEEE Trans. Biomed. Eng., vol. 25, no. 7, pp. 1026–1036, 2017.

[117] F.-B. Vialatte, M. Maurice, J. Dauwels, and A. Cichocki, “Steady State Visual Evoked Potentials in the

Delta Range (0.5-5 Hz) BT - Advances in Neuro-Information Processing: 15th International Conference,

ICONIP 2008, Auckland, New Zealand, November 25-28, 2008, Revised Selected Papers, Part I,” M.

Köppen, N. Kasabov, and G. Coghill, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2009, pp. 400–

407.

[118] A. Duszyk, P. Milanowski, M. Łabe, P. Suffczyn, M. Michalska, and Z. Jarosław, “On the Quantification of

SSVEP Frequency Responses in Human EEG in Realistic BCI Conditions,” PLoS One, vol. 8, no. 10, 2013.

[119] A. Kübler and N. Birbaumer, “Clinical Neurophysiology Brain – computer interfaces and communication in

paralysis : Extinction of goal directed thinking in completely paralysed patients ?,” Clin. Neurophysiol., vol.

119, no. 11, pp. 2658–2666, 2008.

[120] J. V Haxby, M. I. Gobbini, M. L. Furey, A. Ishai, J. L. Schouten, and P. Pietrini, “Distributed and

Overlapping Representations of Faces and Objects in Ventral Temporal Cortex,” Science (80-. )., vol. 293,

no. September, pp. 2425–2430, 2001.

[121] K. Grill-spector, Z. Kourtzi, and N. Kanwisher, “The lateral occipital complex and its role in object

recognition,” Vision Res., vol. 41, pp. 1409–1422, 2001.

[122] G. A. Rousselet, J. S. Husk, P. J. Bennett, and A. B. Sekuler, “Single-trial EEG dynamics of object and face

visual processing,” Neuroimage, vol. 36, no. 3, pp. 843–862, 2007.

[123] T. Sousa, C. Amaral, J. Andrade, G. Pires, U. Nunes, and M. Castelo-Branco, “Pure visual imagery as a

potential approach to achieve three classes of control for implementation of BCI in non-motor disorders,” J.

Neural Eng., vol. 14, 2017.

[124] T. A. Rihs, C. M. Michel, and G. Thut, “Mechanisms of selective inhibition in visual spatial attention are

indexed by alpha -band EEG synchronization Mechanisms of selective inhibition in visual spatial attention

are indexed by a -band EEG synchronization,” Eur. J. Neurosci., no. March 2014, 2007.

[125] M. J. Gray, H. Frey, T. J. Wilson, and J. J. Foxe, “Oscillatory Recruitment of Bilateral Visual Cortex during

Spatial Attention to Competing Rhythmic Inputs,” J. Neurosci., vol. 35, no. 14, pp. 5489–5503, 2015.

[126] R. Chavarriaga, X. Perrin, R. Siegwart, and R. Mill, “Anticipation- and Error-related EEG Signals during

Realistic Human-Machine interaction : A Study on Visual and Tactile Feedback,” Eng. Med. Biol. Soc.

Annu. Int. Conf. IEEE, vol. 1, pp. 1–4, 2012.

[127] F. Perrin, J. Pernier, O. Bertrand, and J. F. Echallier, “Spherical splines for scalp potential and current

density mapping,” Electroencephalogr. Clin. Neurophysiol., vol. 72, no. 2, pp. 184–187, 1989.

[128] C. Sannelli, T. Dickhaus, S. Halder, E. H. K. Mu, and B. Blankertz, “On Optimal Channel Configurations

for SMR-based Brain – Computer Interfaces,” pp. 186–193, 2010.

Page 68: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

57

[129] C. Sannelli, C. Vidaurre, K. R. Müller, and B. Blankertz, “Common spatial pattern patches - An optimized

filter ensemble for adaptive Brain-Computer Interfaces,” 2010 Annu. Int. Conf. IEEE Eng. Med. Biol. Soc.

EMBC’10, pp. 4351–4354, 2010.

[130] H. Zou and T. Hastie, “Regularization and variable selection via the elastic net,” J. R. Stat. Soc. Ser. B

(Statistical Methodol., vol. 67, pp. 301–320, 2005.

[131] A. A. González-garrido, F. R. Gómez-velázquez, H. Sequeira, and J. Ramos-loyo, “Gender Differences in

Visuospatial Working Memory — Does Emotion Matter ?,” Int. J. Psychol. Stud., vol. 5, no. 1, 2013.

[132] T. Thiery, T. Lajnef, K. Jerbi, M. Arguin, M. Aubin, and P. Jolicoeur, “Decoding the locus of covert

visuospatial attention from EEG signals,” PLoS One, vol. 11, no. 8, pp. 1–17, 2016.

[133] L. Koessler, L. Maillard, A. Benhadid, J. P. Vignal, J. Felblinger, H. Vespignani, and M. Braun, “Automated

cortical projection of EEG sensors: Anatomical correlation via the international 10-10 system,” Neuroimage,

vol. 46, no. 1, pp. 64–72, 2009.

[134] A. Ossadtchi, T. Shamaeva, E. Okorokova, and V. Moiseeva, “Neurofeedback learning modifies the

incidence rate of alpha spindles , but not their duration and amplitude,” Nature, vol. 7, no. October 2016, pp.

1–12, 2017.

[135] T. R. Marshall, J. O. Shea, X. O. Jensen, and T. O. Bergmann, “Frontal Eye Fields Control Attentional

Modulation of Alpha and Gamma Oscillations in Contralateral Occipitoparietal Cortex,” vol. 35, no. 4, pp.

1638–1647, 2015.

[136] X. J. Daume, T. Gruber, A. K. Engel, and U. Friese, “Phase-Amplitude Coupling and Long-Range Phase

Synchronization Reveal Frontotemporal Interactions during Visual Working Memory,” J. Neurosci., vol. 37,

no. 2, pp. 313–322, 2017.

[137] C. M. Hamamé, J. R. Vidal, T. Ossandón, K. Jerbi, S. S. Dalal, L. Minotti, O. Bertrand, P. Kahane, and J.

Lachaux, “NeuroImage Reading the minds eye : Online detection of visuo-spatial working memory and

visual imagery in the inferior temporal lobe,” Neuroimage, vol. 59, no. 1, pp. 872–879, 2012.

Page 69: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

58

Appendices

Appendix A: Necessity of noise attenuation prior to spatial filtering

The mean accuracies of UL vs. LR and LL vs. UR with and without noise attenuation and optimized spatial filters

are presented in Figures 26 and 27. The mean accuracies for no noise and no spatial filter was 57.1±10.1% for UL

vs. LR and 56.0±8.9% for LL vs. UR. Similarly, implementation of noise attenuation resulted in an accuracy of

57.2±10.4% in the UL vs. LR classification problem, and 55.8±8.7% in the LL vs. UR classification problem. Both

accuracies were found to be non-significantly different from accuracies without noise attenuation. Conversely, use

of the CSD spatial filter with noise attenuation resulted in significant increases to 65.2±7.5% (p = 0.001) and

65.0±7.2% for UL vs. LR and LL vs. UR (p<0.001), respectively. However, use of the same optimized spatial filters

without noise attenuation significantly reduced accuracies to 58.5±10.3% for UL vs. LR (p<0.001) and 57.5±9.0%

for LL vs. UR (p<0.001).

Figure 26 UL vs. LR imagery classification accuracies with and without noise attenuation and optimized spatial filters

Figure 27 LL vs. UR imagery classification accuracies with and without noise attenuation and optimized spatial filters

Page 70: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

59

Appendix B: Offline Session Feedback Questionnaire 1. How tired were you before you began this session?

Not tired Somewhat tired Very tired

1 2 3 4 5

2. How tired are you after completing this session?

Not tired Somewhat tired Very tired

1 2 3 4 5

3. How hard did you find it to focus during the trials?

Very easy Neither easy nor hard Very hard

1 2 3 4 5

4. Do you feel that you could not focus as much by the end of the session?

No Somewhat Yes

1 2 3 4 5

5. Was the EEG cap comfortable?

No Somewhat Yes

1 2 3 4 5

6. Did you have fun?

No Somewhat Yes

1 2 3 4 5

7. How easy was it to perform the visual mental imagery task?

Very easy Neither easy nor hard Very hard

1 2 3 4 5

8. How often were you able to picture the arrow on the screen?

Very rarely Sometimes Very Often

1 2 3 4 5

9. Aside from when you were told to, how often did you move your eyes away from the

cross?

Very often Sometimes Never

1 2 3 4 5

Page 71: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

60

10. How satisfied are you with your performance on the task?

Not at all Somewhat Very

1 2 3 4 5

11. How often did you get frustrated with this task?

Very rarely Sometimes Very Often

1 2 3 4 5

Thank you for your participation. You may provide any additional comments below:

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

Page 72: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

61

Appendix C: Online Session Feedback Questionnaire 1. How tired were you before you began this session?

Not tired Somewhat tired Very tired

1 2 3 4 5

2. How tired are you after completing this session?

Not tired Somewhat tired Very tired

2 2 3 4 5

3. How hard did you find it to focus during the trials?

Very easy Neither easy nor hard Very hard

1 2 3 4 5

4. Do you feel that you could not focus as much by the end of the session?

No Somewhat Yes

1 2 3 4 5

5. Was the EEG cap comfortable?

No Somewhat Yes

1 2 3 4 5

6. Did you have fun?

No Somewhat Yes

2 2 3 4 5

7. How easy was it to perform the visual mental imagery task?

Very easy Neither easy nor hard Very hard

1 2 3 4 5

8. How often were you able to picture the arrow on the screen?

Very rarely Sometimes Very Often

1 2 3 4 5

Page 73: A Novel EEG-BCI Dependent on Discrimination of Imagined ... · Brain computer interfaces (BCIs) can provide individuals with severe motor disorders a means of communication. This

62

9. Aside from when you were told to, how often did you move your eyes away from the

cross?

Very often Sometimes Never

1 2 3 4 5

10. How satisfied are you with your performance on the task?

Not at all Somewhat Very

1 2 3 4 5

11. How often did you get frustrated with this task?

Very rarely Sometimes Very Often

1 2 3 4 5

12. Was the feedback helpful?

Not helpful Somewhat Very helpful

1 2 3 4 5

13. Do you think the feedback accurately showed the effort you put in?

Not accurate Somewhat accurately Very accurately

1 2 3 4 5

14. Did you find the feedback frustrating?

Not frustrating Somewhat frustrating Very frustrating

1 2 3 4 5

Thank you for your participation. You may provide any additional comments below:

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________


Recommended