Class separation of target and non-target responses
using unsupervised learning P300 speller-based BCIs
Completed in partial fulfillment of graduation requirements for an Honours Bachelor of Science –
Computer Science degree.
Connor Flood
Algoma University
Supervised by Dr. George Townsend
April 09, 2015
2 | P a g e
Abstract
Implementing various techniques to improve the classification of target and non-target responses
for P300 spellers shows great potential for increasing the accuracy of spelled letters and words.
There is a balanced limitation in the parameters associated with using a P300 speller, such as the
number of flashes per character, and the total number of characters on the board. Some
classification techniques, such as a brute-force style guessing approach for every character per
letter spelled, did not show to make substantial improvements to the existing classifier. However,
after modifying the P300 speller parameters to increase the number of flashes, and decrease the
number of letters on the board, meaningful results were found where letters were often spelled
correctly without any problems. The success of the classification of the target and non-target
letters were attributed both to the classifying techniques, as well as to the modification of the
parameters of the P300 speller.
Acknowledgements
I would like to sincerely thank my thesis supervisor, Dr. George Townsend, for introducing me
to the research area of BCIs, and for all of his patience and dedication in helping me through all
steps of this thesis. I would also like to thank Valerie Platsko for all of her help, and sharing the
lab resources.
Finally, a huge thank you to my parents (Glenn and Kristen Flood), and my brother Alex, for
their continued support both during the time spent on this thesis, as well throughout my
undergraduate degree and planning of further education and career paths.
3 | P a g e
Table of Contents
Abstract ................................................................................................................................................. 2
Acknowledgements ................................................................................................................................ 2
List of Figures ........................................................................................................................................ 5
List of Tables ......................................................................................................................................... 5
Part 1 – Introduction ............................................................................................................................. 6
Background Information ................................................................................................................... 6
Brain Computer Interfaces............................................................................................................ 6
EEG and 10-20 System .................................................................................................................. 7
Event Related and Steady State Visually Evoked Potentials ........................................................ 7
P300 Speller ................................................................................................................................... 8
Row-Column Paradigm ................................................................................................................. 8
Checkerboard Paradigm ............................................................................................................. 10
Amyotrophic Lateral Sclerosis (ALS) ......................................................................................... 10
Feature Extraction ....................................................................................................................... 11
Class Separation .......................................................................................................................... 12
Unsupervised Learning................................................................................................................ 12
Objective .......................................................................................................................................... 13
Rationale .......................................................................................................................................... 13
Research Question ........................................................................................................................... 14
Scope ................................................................................................................................................ 14
Limitations ....................................................................................................................................... 14
Timetable ......................................................................................................................................... 15
Methods ............................................................................................................................................... 16
Materials .......................................................................................................................................... 16
Procedure ......................................................................................................................................... 16
Procedure - Generating Features ................................................................................................ 17
Feature #1: Symmetry .................................................................................................................... 21
Feature #2: Convexity .................................................................................................................... 22
Feature #3: Median-Max Index Difference ..................................................................................... 23
Feature #4: Direction Changes ....................................................................................................... 24
Combining Features: ...................................................................................................................... 25
4 | P a g e
Procedure - Brute-Force Guessing .............................................................................................. 25
Procedure - Modifying P300 Parameters .................................................................................... 30
Results .................................................................................................................................................. 33
Results - Generating Features ......................................................................................................... 33
Results – Brute-Force Guessing ...................................................................................................... 34
Results – Modifying P300 Parameters ............................................................................................ 35
Discussion ............................................................................................................................................ 41
Discussion – Generating Features ................................................................................................... 41
Discussion - Brute-Force Guessing .................................................................................................. 42
Discussion – Modifying P300 Parameters ....................................................................................... 44
Conclusion ........................................................................................................................................... 47
Future Work ........................................................................................................................................ 48
References ............................................................................................................................................ 50
Appendix A - Graphs .......................................................................................................................... 52
Appendix B – Code .............................................................................................................................. 57
5 | P a g e
List of Figures
Figure 1: Score of Targets and Non-Targets vs Time – Page 19
Figure 2: Score of Targets and Non-Targets vs Time (zoomed in) – Page 20
Figure 3: Classifier Score vs Character Value – Page 36
Figure 4: Letters Correct vs Number Sequences – Page 38
Figure 5: Score Rank of ‘U’ vs Number Sequences – Page 39
Figures 6-13: Character Rank vs Data Sequence Number in “COMPUTER” – Appendix A
List of Tables
Table 1: Feature Score Results – Page 33
Table 2: “WADSWORTH” Target Letter Ranking – Page 35
Table 3: “COMPUTER” Target Letter Ranking – Page 36
Table 4: Letters Correct Per Sequence – Page 37
Table 5: Words Guessed Per Sequence – Page 40
6 | P a g e
Part 1 – Introduction
Background Information
1.1 Brain Computer Interfaces
1.2 EEG and 10-20 System
1.3 Event Related and Steady State Visually Evoked Potentials
1.4 P300 Speller
1.5 Row-Column Paradigm
1.6 Checkerboard Paradigm
1.7 Amyotrophic Lateral Sclerosis (ALS)
1.8 Feature Extraction
1.9 Class Separation
1.10 Unsupervised Learning
Brain Computer Interfaces
Dennis McFarland and Jonathan Wolpaw discuss in their publication “Brain-Computer
Interfaces for Communication and Control” that brain activity produces electrical signals which
are detectable on the scalp [1]. A Brain Computer Interface (BCI) is able to translate these
signals into outputs which allow users to communicate without participation of peripheral nerves
or muscles [1]. McFarland and Wolpaw further explain that electrodes on the surface of the scalp
are still fairly distant from brain tissue and are separated by coverings of the brain, skull,
subcutaneous tissue, and scalp [1]. This results in the signal being considerably degraded [1].
Only synchronized activity of large numbers of neural elements can be detected, thus limiting the
7 | P a g e
resolution with what the brain activity can be monitored in this non-invasive approach [1]. In
addition, scalp electrodes also detect activity from sources other than the brain, such as
environmental and biological noise [1].
EEG and 10-20 System
Obtaining and measuring electroencephalogram (EEG) signals is necessary in many techniques
for recording brain signals, and is important due to its non-invasive property and easy
implementation [2]. Evoked potentials are one paradigm of EEG, which contains transient
waveforms in ongoing activity [1]. These waveforms are phase-locked to an event [1]. An
international standard for arranging the electrodes placed on the scalp is referred to as the 10-20
system [3]. The 10-20 system is conventionally based on the identification of anatomical
landmarks, with consecutive placement of electrodes at fixed distances [4].
Event Related and Steady State Visually Evoked Potentials
Zhenghua Wu states that when a visual stimulus with a steady frequency appears in the visual
field of a person, that an electrical signal can be observed via the scalp, with the same frequency
as that of the stimulus [5]. This signal is referred to as steady-state visually evoked potential
(SSVEP) [5]. While there are differences in the powers of different frequency SSVEPs, all of
them are concentrated on the occipital area [5]. Wu also makes the connection that even though
SSVEP has been used in many areas since its discovery, one important area concerns BCI, in
which many flickers with different frequencies are used to stimulate the subject’s eyes [5]. By
recognizing the frequency components in the subject’s SSVEP, the flicker that the subject is
8 | P a g e
staring at is determined [5]. As a result, the task corresponding to that flicker can be completed
automatically [5].
An event-related potential (ERP) is a signal observed on the scalp of the subject that occurs when
an event such as a visual stimulus happens in a short period [5]. The ERP for each trial is
hypothesized to be synchronous with the event and similar between trials [5].
P300 Speller
The P300-based speller was originally introduced by Farwell and Donchin in 1988 [6]. Subjects
using the P300 speller select characters from a matrix presented on a computer screen by
analyzing and classifying EEG signals [6]. The flashing of a character being focused on elicits an
event-related potential (ERP) that distinguishes between target and non-target characters [6]. A
“target” refers to the character being focused on and attempting to spell, and “non-targets” are
the remaining characters. Classification usually includes features both from the occipital lobe
(visually-evoked potentials), as well as from the P300 response [6]. The P300 response is a
positive deflection in the EEG over the parietal cortex that appears around 300 ms after a
stimulus [6]. Groups of characters are successively and repeatedly flashed, but only the group
that contains the target character will elicit a response [7]. A central limitation of this application
lies in the need of a high signal-to-noise ratio in order to make an accurate decision [7].
Row-Column Paradigm
In the original implementation of a P300 BCI, characters items are grouped for flashing as rows
and columns [8]. Thus, this orientation is referred to the row-column paradigm (RCP) [8]. The
9 | P a g e
computer is able to identify the target character as the intersection of the row and column that
elicited the largest P300 [8]. The RCP however is subject to errors that slow communication,
cause frustration, and diminish attentional resources [8]. Townsend, et al grouped these errors to
have two primary causes.
The first of these causes are that errors typically occur with the greatest frequency in locations
adjacent to the target character, and nearly always in the same row or column [8]. Since a P300 is
produced for every item in a row or column, this error occurs every time a target item flashes [8].
Errors also arise when flashes of non-target rows or columns that are adjacent to the target,
attract the subject’s attention, and as a result produce P300 responses [8]. This is often referred to
as “adjacency-distraction errors.” [8]
The second of these causes are that sets of items must be intensified in random order. However,
this allows target items to sometimes flash consecutively [8]. This occurs when a row flash is
followed by a column flash (or vice-versa), and the target item is at the intersection of that
particular row and column [8]. One error that is caused by a double-flash is that the second flash
may go unnoticed by the subject, and thus no P300 response is produced [8]. The second error is
that even if the second flash is perceived, the P300 responses to the two flashes will temporarily
overlap [8].
The above mentioned double-flash issues associated with CBP are collectively referred to as the
“double-flash problem” [8]. A solution to this problem is shown in the next section, entitled the
“checkerboard paradigm.”
10 | P a g e
Checkerboard Paradigm
In recognizing the issues associated with the double-flash problem, an alternative stimulation
paradigm that is faster, more accurate, and more reliable than the RCP was designed by
Townsend, et, al.. This new paradigm is called the “checkerboard paradigm.” (CBP) [8]
The CBP uses an 8x9 matrix which is virtually superimposed on a checkerboard, that the
subjects never actually see [8]. Items in white cells of the 8x9 matrix are segregated into a white
6x6 matrix, and items in the black cells are segregated into a black 6x6 matrix [8]. The
respective items of each of these 6x6 matrices are randomly populated before each sequence of
flashes [8].
This results in the subject seeing random groups of 6 items flashing, opposed to entire rows or
columns in traditional RCP [8]. More importantly, CBP ensures that an item cannot flash again
for a minimum of 6 intervening flashes, and a maximum of 18 intervening flashes [8]. Thus, this
eliminates the double-flash problem. By maximizing the time between successive flashes of the
target item, the CBP should increase the amplitude of the P300 responses, and resultantly also
improve BCI speed and accuracy [8].
Amyotrophic Lateral Sclerosis (ALS)
Amyotrophic lateral sclerosis (ALS) is a progressive disease of the upper and lower motor
neurons that often leads to complete paralysis within 2-5 years [9]. As of 2013, about 30,000
people in the United States are living with ALS [10]. As the disease progresses, many assistive
communication devices that were once a necessity, may become ineffective [9]. BCI-technology
11 | P a g e
allows people with severe motor disabilities to use brain signals, rather than muscles, to
communicate (via a P300 speller) and control their environments [9].
Feature Extraction
Wolpaw and Wolpaw define two important concepts that are important to understand when
discussing feature extraction [11]. First, signal-to-noise ratio (SNR) is the simply the ratio to
signal power to the noise power [11]. A high SNR indicates minimal corruption of the signal of
interest by background noise [11]. An important concept is understanding the difference between
noise and artifacts. Noise is due solely to background neurological activity [11]. Artifacts
however are due to biological or external sources unrelated to neurological activity, such as eye
blinks or artificial respirator activity [11]. Both noise and artifacts can contaminate the signal
[11].
A fundamental signal feature is simply a direct measurement of a signal [11]. Alone,
fundamental signal features often provide limited relevant information about complex brain
signals [11]. But when used in linear/non-linear combinations, ratios, statistical measures, or
other transformations of multiple signal features, can provide meaningful data [11]. Most
features in BCI applications are based on spatial, temporal, and spectral analyses of brain signals
[11]. In order to determine the subject’s wishes when using a BCI to be as accurate as possible, a
number of features must be extracted simultaneously [11].
12 | P a g e
Class Separation
Classification involves separating data into two or more classes, where each data member must
belong to exactly one class. Many papers in this area explore classification techniques with
domains up to tens of thousands of features [12]. Selecting the most relevant features is usually
suboptimal for building a predictor, especially if the features are redundant [12]. However, a
subset of useful features may exclude many redundant, but also relevant, features [12]. Many
feature selection algorithms include feature ranking as a principal or auxiliary selection
mechanism because of its simplicity, scalability, and empirical success [12]. The ranking of these
features directly relates to a feature’s respective weights [12]. Noise reduction and consequently
better class separation may be obtained by adding features that are presumably redundant [12]. In
terms of unsupervised feature selection, feature ranking criteria that have been found useful
include entropy, smoothness, density, and reliability of the features.
Unsupervised Learning
Before details of unsupervised learning are discussed, a brief explanation of what supervision
learning should be defined. Supervised learning is learning in the presence of ground truth, such
that training data is labelled with respect to their class membership [13]. Thus unsupervised
learning is learning from unlabelled data [13]. In the context of classification, the goal of
unsupervised learning is to divide a set of unlabelled data into classes [13]. Unsupervised
classification can directly fit a model of a multi-modal probability density to the training data
[13]. In principle, it is not difficult to extend any probability model to a mixture model [13].
Unsupervised learning can only focus on representing the training data as well as possible [13].
13 | P a g e
Objective
This thesis will attempt to implement various class separation techniques in order to more
effectively classify target and non-target responses derived from a P300 speller-based BCI. By
being able to generate meaningful features and apply appropriate weights to those features, the
classification of targets and non-targets will be more successful, and improve the usability and
success of the P300 speller.
The analysis component will focus on the role and overall effect that added class separation
techniques play in the classification stage of the P300 speller, and consequently the measure to
which these techniques will perform better or worse than existing classifying techniques. The
analysis and results of this will be tested on unseen data, in order to mimic the functionality of
unsupervised learning.
Rationale
Proper classification of targets and non-targets is essential and is the basis for a P300 speller-
based BCI’s success in allowing its subjects to effectively communicate. Not only does this have
the potential to increase the reliability of the classification, but could also result in subjects
requiring fewer flash sequences in order for the BCI to confidently decide the character they are
attempting to spell. If the subject using the BCI suffers from late stages of ALS, the effectiveness
of the P300 speller will greatly improve their ability to communicate, and thus their quality of
life. By focusing on proper classification for unsupervised learning, results generated will be
more applicable to the BCI spelling the subject’s target letters in real time.
14 | P a g e
Research Question
This thesis will attempt to answer the following research question:
“To what extent do newly implemented class separation techniques improve the functionality of
classifying targets and non-targets in unsupervised learning, as applied to the P300 speller-based
BCIs?”
Scope
The scope will focus strictly on unsupervised learning, and implementing techniques for class
separation. Although new data was generated during the time frame of this thesis, there will be
no discussion of data acquisition methods as this was not a focus for this topic. With the
exception of the above mentioned, all data was previously generated before research and
implementation stages began, and is independent of experiments later discussed.
Limitations
A major limitation encountered was the amount of test data that the implemented classification
techniques were tested with, due to time constraints. There was much “manual labour” involved
when using the P300 GUI application with the modified code, which was very time consuming.
Future work on this topic could allow a greater amount of time to be spent towards automating
this process, and thus allowing more time to conduct further trials.
15 | P a g e
Timetable
16 | P a g e
Methods
Materials
Analysis:
Laptop/Computer (running Windows XP or later) for performing analysis
MATLAB r2014a
Sample EEG data from previous experiments
P300 GUI Application
Data Acquisition (not performed by me):
8-channel electrode cap
Conductive gel
Toothpick
Computer (running Windows XP) and monitor
BCI 2000 Software
Procedure
The overall procedure for this thesis can be broken down into three sections, with each section
examining a different purpose and producing various forms of results. This allowed for a breadth
of understanding in various sub-areas of data classification, and a greater number of conclusions
to be made.
17 | P a g e
The first section will detail manually generating features in an attempt to better classify target
and non-target responses, based on the classifier scores already obtained.
The second section will focus on a brute-force method of analysis in which each possible
character was “guessed” to be the target letter at each flash.
The third section will modify two P300 parameters: number of sequences, and number of
characters to choose from, to determine both its overall effect, as well as the effect on
increasing/decreasing number of sequences towards the success of proper classification.
Procedure - Generating Features
The process for manually generating features can be difficult, and often requires some trial and
error. The goal was to create and implement features which would be able to score targets and
non-targets in an unsupervised learning fashion. Features, in this case, would be mathematical
ways to score trials based on corresponding data. However, the data these features are scoring
wouldn’t be directly dependent on the raw EEG values. Instead, these features would be
computing scores that an existing classifier had already assigned after generating its own weights
and features. While this may seem like an unnecessary step, the accuracy of the previous
classification is not very high, as there are many target and non-target responses which were
incorrectly labelled. Therefore, it is the goal that these newly created features will be able to
improve the previous classification based on its own results.
When a trial is classified to be either a target or non-target, a numeric score is returned to
determine which class it belongs to. A positive score indicates that the trial is classified as a
18 | P a g e
target, and a negative score indicates that the trial is classified as a non-target. The range of a
score is from negative infinity to positive infinity. More realistically, the scores being examined
in the sample data given ranged between -6000 and +5000.
It would not make sense to generate a feature to improve classification based on a single score
being returned for a given trial. A single score is 1-dimensional and does not provide any
additional data or patterns which can be extracted from to generate a feature. Continuing what
will be discussed below, the classification will be based on consecutive target scores, and
consecutive non-target scores, and attempting to find consistent and distinct patterns for targets
and non-targets.
The input data used for this experiment will contain 3192 flashes, consisting of 266 targets and
2926 non-targets. The dynamic that is added to increment the dimension size of this data is
obtained from “shifting” EEG. The concept of shifting EEG refers to shifting the indices of EEG
values so that the position of those values relative to the time after a flash occurs changes. After
this shift occurs, then the data can be completely reclassified. For the data that was provided for
this experiment, there were 16 shifts of indices incrementing by 1 backwards (i.e. to the left), and
16 shifts forwards (i.e. to the right). With 32 shifts, plus the origin of the starting position, that
gives a total of 33 classification scores per trial. The research and investigation of the
effectiveness of these shifts goes outside the scope of this thesis. The implementation to generate
the data with shifted EEG classification values was previously done, and was simply provided to
perform this experiment.
19 | P a g e
Given that the input data contains 3192 flashes, and each one of those flashes also contains 33
classification scores, the structure being dealt with is a 2-dimensional 3192x34 array of type
‘double’. Below is a figure of this structure being plotted, with some additional parameters to be
discussed:
In the above figure, the blue data points are the actual targets, and the red data points are the
actual non-targets. While this seems contradictory to know what the actual targets/non-targets
are for unsupervised learning classification, this knowledge is never used to help or support the
classification. Instead, it was used as more of a visual inspection to try and detect patterns in the
data visually to make some general inferences. The above figure also contains large dots. These
dots represent the middle data point (16), out of the range of 33 scores given for each flash letter.
In reality, it was this data point which was used to perform the classification, depending on
whether it was positive or negative, as previously discussed.
20 | P a g e
What the above figure doesn’t illustrate very well, is that the data is actually graphed in
parabola-like shapes for each trial. By zooming in to a more visible scope, this is apparent in the
figure below, using a subset of the same data as above:
Here the patterns for the 33 data-points for each trial are more clearly illustrated. The challenge
now is to find consistent patterns in these “parabolas” that are both unique and contrasting
between targets and non-targets. In basic terms, the objective is to find mathematical ways to
determine which data points are red, and which data points are blue, in the above two figures.
Determining what patterns to find and implement are crucial towards the success of generating
useful features for successful classification. After visual inspection of the graphed data, the
following conclusions were made on target and non-target response score patterns:
21 | P a g e
Targets:
Often shaped like an upside-down ‘U’
An upside-down ‘U’ contains the following mathematical properties:
o Perfectly symmetrical
o Completely convex
o Highest peak occurs at middle index
o Direction changes twice (i.e. slope at point of direction change = 0)
Non-Targets:
No consistent patterns
Mostly noise with no visible resulting artifacts detected
Based on mathematical properties listed above, this inspired the features which would be used at
an attempt for improved classification. Since the non-targets had no consistent visible pattern,
then it did not make sense to create features for non-targets. Instead, features based on the
patterns of the target data was used for generating features.
Below describes the theory and implementation behind the four features which were developed,
based on the visual properties of the target score patterns.
Feature #1: Symmetry
Given 33 data points, a feature is needed to return a score on how symmetric those data points
were. The range of the score this feature would return would be between 0 and 1, where 0 was
completely asymmetric and 1 was completely symmetric. Theoretically, a perfect upside-down
22 | P a g e
‘U’ would return a value of 1, and a perfectly linear line segment with a slope of 45° would
return a value of 0.
A mathematical representation of how this algorithm is expressed is:
1 −∑ [𝑥(𝑛) − 𝑥(33 − 𝑛)16
𝑛=1 ]
16, 𝑤ℎ𝑒𝑟𝑒 𝑥 𝑖𝑠 𝑡ℎ𝑒 𝑛𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑒𝑑 𝑠𝑒𝑡 𝑜𝑓 33 𝑑𝑎𝑡𝑎 𝑝𝑜𝑖𝑛𝑡𝑠
This allows the data to be immediately normalized between 0 and 1. The differences are then
taken between each data point’s respective data point, and the sum for all 16 combinations are
found, and divided by 16 to determine the average. Since a symmetric pair would expect a
difference of 0 (normalized), the average difference is then subtracted from 1, so that as the
symmetry of the data increases, then the value being returned also increases between the range of
0 and 1.
Feature #2: Convexity
Given 33 data points, a feature is needed to return how convex those data points were. The range
of the score this feature would return would be between 0 and 1, where 0 was completely
concave and 1 was completely convex. Theoretically, a perfect upside-down ‘U’ would return a
value of 1, a perfectly shaped “U” would return a value of 0, and a horizontal line would return a
value of 0.5.
A mathematical representation of how this algorithm is expressed is:
23 | P a g e
32 − [∑ [(𝑥(𝑛) > 𝑥(𝑛 + 1)) +
𝑥(𝑛) == 𝑥(𝑛 + 1)2
] + ∑ [(𝑥(𝑛) < 𝑥(𝑛 + 1)) +𝑥(𝑛) == 𝑥(𝑛 + 1)
2]32
𝑛=1615𝑛=1
32] ,
𝑤ℎ𝑒𝑟𝑒 𝑒𝑎𝑐ℎ 𝑏𝑜𝑜𝑙𝑒𝑎𝑛 𝑠𝑡𝑎𝑡𝑒𝑚𝑒𝑛𝑡 𝑟𝑒𝑡𝑢𝑟𝑛𝑠 𝑎 𝑣𝑎𝑙𝑢𝑒 𝑜𝑓 1 𝑤ℎ𝑒𝑛 𝑡𝑟𝑢𝑒, 𝑎𝑛𝑑 0 𝑤ℎ𝑒𝑛 𝑓𝑎𝑙𝑠𝑒
The logic for this algorithm is that for the first half of the data points, every time a proceeding
point increases in value, then that increases the concavity of the series of points. For the second
half of the data points, every time a proceeding point decreases in value, then that also increases
the concavity of the data. As well, any time a proceeding point is equal to the previous point,
then that makes the data series more neutral. In the above described event of a value increasing
the concavity of the series, a value of 1 is summed to a running total. A value of 0.5 is also added
to this total when the neutral case occurs. This total is then subtracted from the total maximum
pairs of points (32), and then divided by that same value of 32, in order to return a value between
0 and 1 for convexity.
Feature #3: Median-Max Index Difference
Given 33 data points, a feature is needed which detects how close the maximum point of the
series is to the middle of the plot. Therefore, this feature would be finding the difference between
the index of the median of the un-ordered numbers (17), and the index of the maximum point.
This would then need to be returned in the range of 0 and 1, where 0 represents the furthest
possible distance between the maximum index and the centre, and 1 represents the maximum
index being equal to the centre index. An example of 0 being returned would be data shaped like
a “U”, where the maximum points are either the first or last index of the series. An example of 1
being returned would be data shaped like an upside-down “U”, there the maximum value occurs
directly in the centre.
24 | P a g e
A mathematical representation of how this algorithm is expressed is:
17 − |17 − 𝑖𝑛𝑑𝑒𝑥(max(𝑥))|
17
This allows a normalized value between 0 and 1 to be returned, based on the conditions listed
above.
Feature #4: Direction Changes
Given 33 data points, a feature is needed to return a score on how many times the direction (or
sign of the slope), changes. The range of the score this feature would return would be between 0
and 32. A score of 0 would occur for any plot, where the slope between any two points remains
either positive or negative for all combination of points. A score of 32 would occur for a
completely jagged line, where the sign of the slope switches after every data point. It would not
make sense to normalize the score of this feature between 0 and 1, because this wouldn’t
accurately represent relative differences of scores where the number of changes are low. For
example, since for the perfect upside-down “U” shape that we are predicting target scores to look
like, there would be 1 direction change in this case. However, to normalize this score, we have to
assume that it’s possible that a score of 32 can be returned. In reality, the scores are more likely
to be around the 2-5 range, and so a difference of 1/32 between each direction change does not
do the score justice. This is why just the absolute value will be returned.
25 | P a g e
Given the complexity required for iteration through the points, it is difficult to write a proper
mathematical expression to represent the algorithm without writing pseudo code. The algorithm
for this feature can be found in the appendices, entitled “num_changes.m”.
Combining Features:
With the above four features that were created, plus including the given score from the original
classifier, we now have a five-feature set to use for classification. By choosing the combination
of these five features, new classifications can be made, which will be compared to the original
classification results. The results of this new classification with the generated features are shown
in the first part of the results section.
Procedure - Brute-Force Guessing
The second section of the methods performed for this thesis is entitled “brute-force guessing”.
The reason for this title is that the logic for how the modified classifier will determine which
character is the target-letter, will be through an approach similar to a brute force algorithm.
This is accomplished by implementing a utility which alters various marking channels for 72
iterations, so each iteration will leave only the data pertaining to that character for that iteration
only. In other words, for each letter being spelled, the utility will try and give a score for that
letter by guessing which letter it is. For example, if the utility is on the first iteration and
guessing that the target letter is an ‘A’, then data corresponding to the remainder of the letters in
26 | P a g e
the word are deleted so that the classification is focused on simply the letter being spelled in the
word, and the letter which the utility is guessing the target letter is.
One of the first tasks that must be done is to generate a structure which contains all of the flash
locations for a given character in a word. One of the sample data files being used was for the
spelling of the word ‘WADSWORTH’. For this data file, there were 72 possible characters
which were flashing in the matrix (8x9) to choose from. Each character flashed twice per
sequence, and five sequences occurred for the spelling of each letter in the word. Since the word
being spelled is 9 letters long, then there are (2 flashes/sequence * 5 sequences/letter * 9
letters/word =) 90 flashes for each character throughout the entire spelling of the word. With 90
flashes occurring for each character in a word, and 72 characters total, then the structure that
contains all the flash locations would be a 2-dimensional array of size 72x90.
A function was created to handle most of the logic for generating the above mentioned structure.
The utility being created is called ‘FlashDriver.m’, and the new function created is called
‘flash.m’. In FlashDriver, a structure needs to be created that simply contains the indices for
every flash occurrence, regardless of the letter. This is accomplished by parsing through the
structure sts.StimulusType and finding where an increase occurs, as this represents the
occurrence of a flash. The ‘flash’ function is called from FlashDriver, with three arguments
being passed: ‘letter’, ‘file’, and ‘AllFlashInx’. The argument ‘letter’ represents a character
(1:72), ‘file’ represents the file location of the data to be loaded form, and ‘AllFlashInx’
represents the structure mentioned above which contains the locations of all flashes. With this
information passed in, the ‘flash’ function can return the location of all (90) flashes that occur for
27 | P a g e
that given letter in a word of a given length (9). FlashDriver uses a loop to parse through all 72
characters in order to populate the 72x90 structure. One challenge of the ‘flash’ function was that
the data being referenced was obtained from 6 different channels. The data from each channel
was found in the variable ‘sts.Targ0x’, where x ranges for the integers between 1 and 6. The
variable ‘sts.Targ01’ contains all flashes occurring for the ‘lower’ characters, such that all
flashes for the letter ‘A’ are found only in sts.Targ01 and not in any of the other channels. The
character number 72 however would be found only in sts.Targ06, and not any of the previous
channels. Characters in the middle can be found in multiple channels. This is due to the fact that
6 letters are flashing at a time, and each channel focuses on only one of those letters during a
flash. So not only does the ‘flash’ function need to find the common elements between
AllFlashInx and sts.Targ0x==letter, but then also take the union of all these results from the
various sts.Targ0x structures so that it contains indices from all of these channels without
repetition. Once implemented correctly, the cardinality of this set will be the proper length, in
this case, 90. This set will then be what is returned to FlashDriver to populate the 72x90 structure
for all 72 characters. This 72x90 structure will be referred to as ‘flash_inx_letter’.
The remainder of the work being done in FlashDriver is the ‘cropping’ of various structures so
only the letter currently being spelled is focused. For example, in the event the ‘D’ in
‘WADSWORTH’ is being spelled, then the appropriate values in the ‘sts’ variables will need to
be cropped everywhere except for where it references the ‘D’ being spelled. The following
variables are what needed to be cropped for each iteration of letters in a word:
sts.PhaseInSequence
sts.StimulusBegin
28 | P a g e
sts.Targ0x (x = range of 1:6)
sts.StimulusType
This is accomplished by defining beginning (‘beg’), and ending (‘fin’), markers for which data to
keep for a given letter. In this case, the value of letter does not range from 1:72, but instead
ranges from 1 to the length of the word being spelled. In MATLAB, these index markers are
calculated using the following syntax:
beg = 1+find(sts.PhaseInSequence(1:end-1)~=1 &
sts.PhaseInSequence(2:end)==1); fin = find(sts.PhaseInSequence(1:end-1)==3 & sts.PhaseInSequence(2:end)~=3);
This allows the ‘beg’ markers to be indices where the value in sts.PhaseInSequence is not equal
to 1, and the proceeding index is equal to 1, as this indicates the beginning of a sequence. The
‘fin’ markers are the indices in sts.PhaseInSequence which are equal to 3, and the proceeding
index is not equal to 3, as this indicates the end of a sequence. This will generate a ‘beg’ 1-D
array of the length of the word and a ‘fin’ 1-D array also of the same length.
As for actually cropping the structures, consider the following 2 lines of MATLAB code:
sts.PhaseInSequence(1:beg(letter)-1)=0; sts.PhaseInSequence(fin(letter)+1:end)=0;
Using the ‘beg’ and ‘fin’ values that were defined, they will act as the markers for what will be
cropped, or zeroed out. Since ‘beg’ and ‘fin’ are the length of the word being spelt
(‘WADSWORTH’), the index of the ‘beg’ and ‘fin’ arrays correspond to the letter being spelled.
For example, beg(4) would refer to the beginning index of the 4th letter in a word. The first line
of the above code excerpt zeroes out all of the data in sts.PhaseInSequence that occurs before the
29 | P a g e
‘beg’ index. The following line then zeroes out all of the data in sts.PhaseInSequence that occurs
after the ‘fin’ index. What remains after those two lines are executed is the data that pertains only
what ‘letter’ represents.
The cropping shown above is repeated in the exact same manner for sts.StimulusBegin, and
sts.Targ0x (x ranging from 0 to 6). The remaining structure that must be modified is
sts.StimulusType. This is done in a slightly different manner. For sts.StimulusType, we begin by
zeroing out the entire structure right away:
sts.StimulusType(:) = 0;
Now to repopulate some of the data in sts.StimulusType, consider the following two lines of
MATLAB code:
sts.StimulusType(flash_inx_letter(guess,(1+(letter-1)*NumTargFlashes):
letter*NumTargFlashes))= 1;
sts.StimulusType(flash_inx_letter(guess,(1+(letter-1)*NumTargFlashes):
letter*NumTargFlashes)+1) = 1;
The repopulation of this structure is using the 72x90 ‘flash_inx_letter’ structure generated
before. The variable ‘guess’ that is shown above, refers to a value between 1:72 that is being
guessed in a brute-force manner, in pretending that the character which ‘guess’ represents is the
target letter. I.e. if guess==7, then we are pretending that the target letter is ‘G’, and making
classifications accordingly. ‘NumTargFlashes’ is equal to the total number of target flashes that
occur for a given word.
Most of the other implementation for this section of the Method deals with the storage and
organization of data as it interacts with the P300 GUI. In order to keep track of the ‘guess’ (i.e.
30 | P a g e
the character being guessed for being the target letter at each iteration), this variable is
incremented and stored in a reloadable structure every iteration. During each iteration, the score
that this ‘guess’ letter received is also appended to a structure that is added on to every iteration.
An iteration is defined as manually selecting “Generate Feature Weights” in the P300 GUI. This
process is repeated for all 72 characters of a letter, and that process is then repeated for all the
letters in a word.
In order to generate results for this section of the Method, the scores for all 72 characters for all 9
letters of the word ‘WADSWORTH’ will be examined. This will be accomplished by
normalizing the data for each letter, and then recording how the target character for each letter in
the spelled word ranked among the 71 non-targets.
Procedure - Modifying P300 Parameters
The final method of this thesis was inspired by the results of the previous method. While the
results of the previous method will be discussed in the next section, it brought up some
interesting ideas that could be further explored. Mainly, this dealt with modifying two of the
more crucial parameters of the P300 speller in order to see how that effects results. The first one
of these parameters is the number of flash sequences per letter being spelled. In the previous
method experiment, each target letter being spelled had 5 sequences per letter, with 2 flashes per
sequence, giving a total of 10 flashes per target letter. This parameter of 5 sequences per target
letter can be changed to any desired value. The second of the parameters to be modified is the
total number of characters from the array to choose from. In the previous method experiment, an
8x9 matrix was used for data acquisition, giving a total of 72 characters to choose from. By
31 | P a g e
increasing the number of flash sequences, and decreasing the P300 matrix size, it is hypothesized
that the classification results will dramatically improve More specifically, the number of flash
sequences per target letter will increase from 5 to 15, and the matrix size will decrease from 8x9
to 5x5. With the new matrix size, the letter ‘Z’ will be removed. While this would not be desired
in a real application, this new matrix size is simply used for research and better understanding
and a better understanding of classification techniques.
After the results for spelling the word ‘COMPUTER’ will be found in a similar way to the
previous method will address, one final inquiry will be further examined. This will examine how
successful the classifier is at spelling the target word as a whole after each individual sequence.
The results of that experiment will be beneficial in discovering exactly what the minimal number
of flash sequences are required for the successful spelling of each letter.
Modifying these two parameters for data acquisition can easily be done in the BCI2000 software.
However, the utility which will be performing the classification will need to be slightly modified
to handle these new parameters. Mainly, any loops that iterate 72 times for total characters will
need to be changed to 25, and any loops that iterate 5 times for sequences will need to be
changed to 15. The ‘flash.m’ function is what will need the most modifications. All of the
experiments conducted so far have been using the checkerboard-paradigm (CBP). Thus a 9x8
matrix containing 72 total characters, is virtually split into two 6x6 black and white boards.
However, a 5x5 matrix is more difficult to split into two boards since 25 is not evenly divisible
by two. The solution is one of the two virtual boards containing empty spaces, so that there is
still a total of 25 characters. However, this makes the sts.Targ0x variables slightly more
32 | P a g e
complicated. With the new 5x5 board, some flash groups will contain 3 flashes, and other flash
groups will contain 4 flashes. As a result, the sts.Targ0x variables that we care about are between
the range of x=1:4. But while sts.Targ04 will be fully populated, some of those values are
repeated from a previous flash group, and make it appear that there were always 4 flashes per
group, even though that’s not true. The useful variable to solve this problem is found within the
contents of sts.Targ00. The value found at each index of this structure indicates the number of
letters that flashed at a given index. Thus the values of this structure will either be 3 or 4 at the
index of a flash group. Therefore, sts.Targ04 can be modified so that the values of this structure
can be set to 0 whenever the value of sts.Targ00 is equal to 3 at the same indices. This can be
accomplished in a single line of MATLAB code which is shown below:
sts.Targ04(find(sts.Targ00~=4))=0;
One final modification that must be made is for ‘sts.StimulusBegin’. This is necessary for when
we want to examine the iterative process of removing the latest flash sequence, until there is only
one flash sequence remaining. These modifications will also occur for the sts.Targ0x variables.
New structures will be saved after a sequence is removed after each iteration. These structures
will then be normalized and ranked in classification scores in the same way as the previous
methods. The results of the classification will then be compared among each sequence.
33 | P a g e
Results
The below 3 sections contain results for each of the 3 methods explained in the previous section.
Results - Generating Features
In order to obtain the results from the various features generated, a script titled
‘classifier_driver.m’ was implemented. This script outputs separate averages for targets and non-
targets for each one of the scores given for all of the features. It also outputs the scores given
from the original classifier. Based on the output of this script, the below table displays these
results:
Feature Name Target Score Non-Target Score
Original Classifier 894.1154 -65.2690
Symmetry 0.6296 0.5630
Convexity 0.6215 0.7348
Median-Max Distance 0.6798 0.3714
Direction Changes 2.2444 2.9843
The original classifier features shows a substantial difference between the target and non-target
score, as suspected. The features that have the most interest are the bottom four, the ones which
were implemented.
34 | P a g e
The symmetry feature, median-max distance feature, and direction changes feature all gave
better target scores, than the non-target scores. Even though the target average for direction
changes was lower than the non-target average, this was expected and hoped for since it was
predicted the target average would be 1, given its shape was a perfect upside-down “U”.
The only feature which produced negative results was the convexity feature. The non-target
average for the convexity feature was 0.7248 while the target average for the same feature was
0.6215. This is likely due to the unpredictability of non-target brain activity. Both background
and biological noise can cause scores to vary, and thus form patterns that cannot be easily
predicted. In this case, it seems that these random scores seemed to match a convex pattern. If
this is found to be consistent, then the knowledge of this feature can help improve classifiers in
the future.
Results – Brute-Force Guessing
Using the guessing method similar to a brute force approach, the success of this classification
was determined using obtained data for spelling the word “WADSWORTH”. For this data being
used, there were 5 sequences (10 flashes) per letter, with a total of 72 characters to choose from.
35 | P a g e
Below is a table outlining how each target letter was ranked out of the 72 possible characters for
each letter in the target word.
Target Letter Rank (out of 72)
W 1
A 8
D 26
S 7
W 32
O 15
R 34
T 49
H 24
As promising as this classification method looked after the first letter, the overall results were
fairly disappointing. Besides the ‘W’, none of the other letters were even a close contender to
being the chosen target-letter with this approach. On the other hand, a thought that should also be
considered is the importance of ranks when it’s not 1st. In terms of the end application, the
chosen letter is either right or wrong, so how important is it how close or far off it is from being
1st? This idea will be further explored in the discussion section.
Results – Modifying P300 Parameters
The results of this final method focused on spelling the newly generated data for the word
“COMPUTER”. This is different from the previous method, since the number of sequences
increased from 5 to 15, and the number of characters to choose from on the P300 matrix
decreased from 72 to 25.
36 | P a g e
Similar to the previous experiment results, below is a table outlining how each target letter was
ranked out of the 25 possible characters for each letter in the target word.
Target Letter Rank (out of 25)
C 1
O 1
M 1
P 1
U 2
T 1
E 1
R 1
It is clear from these results that by incrementing the flash sequences and decrementing the size
of the target array, this classification method worked very well. Every one of these target letters
was ranked 1st, with the exception of the letter ‘U’. While the ‘U’ was ranked 2nd, the following
graph displays how close ‘U’ (letter 21) was from being given a score higher than the winner
(letter 6):
37 | P a g e
It was not a surprise that the results obtained from this experiment were a drastic improvement
from the previous experiment, given the substantial change in its parameters. However, an
interesting component of this experiment that was to be investigated was how the results change
when sequences are removed. The iterative removal of the last sequence simulates how the
subject would have spelled the word if they were given fewer sequences (i.e. fewer flashes), per
target letter.
The below table and graph show how many letters correct (out of a possible 8), the classifier
guessed correctly after each sequence. This begins after the first sequence, and goes until
sequence number 15.
Sequence # # Letters Correct (out of 8)
1 0
2 1
3 0
4 3
5 2
6 4
7 2
8 3
9 3
10 5
11 5
12 7
13 7
14 6
15 7
38 | P a g e
Further diagrams in the appendices will show the ranks of all the target letters in “COMPUTER”
after each sequence. However, one diagram that should be analyzed now is the diagram depicting
the rank of the target letter ‘U’, since that was the letter which was incorrectly classified after the
15th sequence.
39 | P a g e
Below is the graph of the target letter ‘U’, and the rank (out of 25) it was given after each
sequence:
As displayed in the above graph, the ‘U’ was classified as the target letter (rank 1) after the 6th
sequence. The ‘U’ remained the winner at rank 1 for the following 8 sequences. It was then at the
15th sequence, that the ‘U’ become ranked #2. This is unfortunate since it would be expected that
the “confidence” or reliability of the classifier would continue to increase as the number of
sequences increase. However, for some reason this wasn’t the case for this particular target letter.
Possible reasons for this will be examined in the discussion section.
40 | P a g e
One final display of results for this experiment will show the actual word that was spelled after
each sequence. This will display realistic results in that this is what would actually be output by
the P300 speller for each sequence. This also allows focus to be on which letters were
consistently guessed correctly throughout the sequences. The display of these results are shown
below:
Sequence # Word Guessed (for ‘COMPUTER’)
1 UHIWTQXS
2 KLCGKOSR
3 KGKWIVTI
4 CBMWQTHC
5 CDMUNDDP
6 OTMEUTCR
7 JPEXUTHV
8 CUEXUTHA
9 LUMEUMWR
10 CMMUUTOR
11 CTMUUTWR
12 COMTUTER
13 CUMPUTER
14 CEMPUTTR
15 COMPFTER
Based on a visual inspection of the above table, it is clear that the word being spelled does not
begin to correctly form until after about the 10th sequence. Letters such as the ‘C’, “M’, ‘T’, and
‘R’ become consistently correct guesses after this point. However the ‘U’ is the only letter to
have been consistently correctly guessed for so many consecutive sequences, and then be
replaced by an incorrect guess. The number of appropriate sequences will be further examined in
the discussion section.
41 | P a g e
Discussion
This discussion section will be broken down into the three areas that have been outlined in the
Methods and Results sections. While these areas will be discussed separately, a joined
conclusion will be made in the Conclusion section to represent the research and experiments of
the entire paper.
Discussion – Generating Features
As shown in the Results section for ‘Generating Features’, 3 out of the 4 features generated and
implemented showed positive results. While these results were positive, they were not ground-
breaking in the sense that this would be improving the existing classifier. These were 4 features
that were manually implemented based on a visual inspection of the data. In reality, the existing
classifier chooses from thousands of possible features, and then applies weights to the features
that it wants in order generate strong results to perform an accurate classification. It would be
very difficult to manually implement generated features and expect those features to both
accurately and consistently classify this large amount of data. In reality, more focus should be
drawn to further operations and calculations that can be performed on the output of the existing
classifier, and using those results to try and improve the final classification. More than anything,
this initial experiment was helpful in better understanding the interaction between features and
the classifiers, and provide a solid direction in what future experiments can be conducted in an
attempt to improve the classification of target and non-target responses.
42 | P a g e
Discussion - Brute-Force Guessing
The results obtained from the initial brute-force guessing method were quite disappointing. With
only 1 out of 8 letters being spelled correctly, this method is nowhere near plausible for success
as a needed application. Since the first character was spelled correctly with great results, a
significant amount of time was spent attempting to detect a bug that was preventing the
remainder of the letters being spelled correctly. Specifically, there was curiosity on whether there
was a miscalculation in how the indices for cropping out data were derived, but unfortunately no
such bug or error was found.
With this brute force approach for guessing the correct target letter, each ‘guess’ had appropriate
features and generated weights created to make this ‘guess’ the classified winner. It was hoped
that even though most of the 72 characters would be shown as winners depending on their
appropriate weights and features, that the real target letter would be a much clearer winner than
the other ‘fake winners’. For example, consider the letter ‘W’ in ‘WADSWORTH’. In the brute
force approach, all 72 characters were made to look like they were the actual target letter.
However, what made the ‘W’ stand out as the clear winner was that even once all the scores
were normalized, the target score for the ‘W’ was so good that it stood out from all of the other
winners. Theoretically, despite most of the 72 characters being initially classified as the target
letter, the real target letter would still stand out from all of the other guessed letters, similar to
how the ‘W’ was classified as the target letter. If this worked for the ‘W’, then why did it fail for
the remainder of the letters? The reason is not because the real target letter was scored poorly for
each letters in the target word. The real reason is that the features and weights generated to make
43 | P a g e
the other 72 characters look like the target letter was done so successfully, that it was difficult to
differentiate between all of these plausible winners because they were all scored so well. The
hope was that even if all the guessed letters were scored highly, the real target letter would stand
out, but this was not the case since all of the guessed letters were scored just as well as the real
target letter.
Despite the above method not being successful in making a clear classification of the target letter
and non-targets, it was successful in eliminating several choices. Out of a rank of 72 choices, the
average rank for target letters in the word ‘WADSWORTH’ was around 25. While this is clearly
nowhere close to consistently guessing the letter correctly, it does consistently eliminate about
two thirds of the possibilities. This is slightly comforting, but at the same time, does it really
matter how close the guess was to being correct? Regardless of whether the guessed target letter
was ranked #2 or #72, in the end the guess was incorrect. If the subject misspells a letter, they are
likely not to care what the incorrect guess was. From their view point, the guessed letter was
either correct or incorrect. With that being said, even though the guessed letters in
‘WADSWORTH’ were ranked in the top third of possible characters, from the subject’s
standpoint it only guessed 1 out of 9 letters correctly. The only usefulness of knowing how close
in rank the real target letter was from being the chosen classified letter is being able to measure
the improvements or detriments in the classifier. If the classifier continues to be modified, even
if the end result in terms of the number of correctly guessed letters does not change, small
improvements can be detected by knowing that the target letter is getting closer to being the
classified letter.
44 | P a g e
In its current implementation, this brute force method does not work under standard conditions.
Future research could try and find unique features or techniques to improve this method. As a
proof of concept to show that this method has the potential to theoretically succeed as a
classification technique, the experiment was repeated with new data and parameters This will be
analyzed in the next section of the discussion to determine the overall effect modifying two
important parameters of the P300 had on the success for classifying targets and non-targets using
the brute-force approach.
Discussion – Modifying P300 Parameters
Due to the unsatisfactory results of the brute force method in the previous conditions, the
experiment as repeated with new test data and a modification in two P300 parameters: number of
flash sequences, and size of speller matrix. The results of this modification made a substantial
difference, as 8 out of the 9 letters were guessed correctly. The changes in parameters were quite
substantial, so it was expected to receive positive results. As mentioned in the previous section,
this was done as more of a proof of concept, but it does create a few discussion areas. One of
these is how realistic these new parameters have towards the real-life application of the P300
speller.
The first parameter that was modified which needs to be discussed is the size of the target matrix.
In the previous experiment, an 8x9 matrix containing 72 characters was used. The reduction of
45 | P a g e
the matrix to size 5x5 containing 25 characters was crucial in improving results. Of course, this
is due to there being less competition that the real target letter is facing for all of the non-target
letters. An initial thought of this experiment questioned why 72 characters was necessary to
begin with for real-life application. If a subject suffers from late stages of ALS and is using the
P300-speller as a method of communication, it is not likely necessary for the matrix board to
contain characters such as ‘Ctrl’ or “Pg. Down’. For this case, it would make more sense to
contain just the letters of the alphabet, since they are relying on this as the only form of
communication, and there would be no point in overcomplicating this process. However, in the
case where the subject using the P300-speller as a way modifying a document in a word
processor, then all of the extra options on the matrix may be necessary. In either case, if the
strength and accuracy of the classifier is high enough, then the target letters would be
consistently guessed in both situations anyway.
The second modified parameter to consider is increasing the number of target flashes from 10 to
30. It is intuitive that increasing the number of flashes per target letter will also increase the
reliability of the classification of targets and non-targets. The drawback however is the amount
of time that is required to now spell each letter. It would not make sense to force a subject to sit
through 30 flashes per target letter if another classification method can do it in far fewer flashes.
While each subject will require a different amount of target flashes to correctly spell a letter, it
makes more sense to over-estimate the amount of flashes required than it is to under-estimate
this amount. By under-estimating, the success of the classification could vary and cause the P300
to spell out gibberish. By over-estimating the required number of flashes, it may be worth the
extra amount of time when using the P300 speller if this will ensure successful results.
46 | P a g e
Therefore, this balance must be considered when deciding the number of target flashes required
by each individual subject.
The latter half of this experiment focused on a similarity to the discussion above. Using the same
data and results generated from this experiment (i.e. spelling the word “COMPUTER”), the exact
effect of the number of flash sequences required for successful classification was explored.
When the word “WADSWORTH” was being spelled, 5 sequences of flashes (2
flashes/sequence) was used, resulting in 1/9 letters spelled correctly. When the word
“COMPUTER” was spelled, 15 sequences of flashes (2 flashes/sequence) was used, resulting in
7/8 letters spelled correctly. However, the success of the spelling of this second experiment can
be broken down so that it can become apparent how well the word would have been spelled had
there been less flash sequences. For example, if only 5 flash sequences were used for spelling
“COMPUTER”, then only 2 out of the 8 letters in the word would have been spelled correctly at
this point. This improvement is very minimal to the first experiment, and the only reason for
eventual success was that 10 more flash sequences followed this.
It has already been discussed that increasing the number of flash sequences should consequently
increase the reliability of the classification. However, this was not the case for the letter ‘U’ in
“COMPUTER” since it become classified as a non-target at flash sequence #15, despite being
classified as the target letter for the previous 8 flash sequences. This is a bit of an anomaly, since
it disagrees with the theory of classification improving, given more data. One possible
explanation is that due to the subject having to concentrate for 30 flashes per letter when spelling
47 | P a g e
the target word, distraction could become more difficult to hold for a longer period of time. A
solution to this could be to have the number of flash sequences be dynamic, and calculated at
run-time of the speller. In the case of the letter ‘U’, if the classifier realizes that after 3 flash-
sequences the target letter has not changed, then it could consider the possibility of locking into
this choice and moving onto the next letter without having to complete the rest of the flash
sequences. Since the brute force method was shown to not be very successful to begin with, the
data obtained from this experiment would not be ideal to use to test this new hypothesis, but
could be worth exploring in future work.
Conclusion
Several conclusions can be drawn based on the three main experiments that were conducted. The
first conclusion is that successful classification of targets and non-targets using generated
features is a very complicated task. While manually creating features based on a visual
inspection gave positive results when contrasting the difference of scores for targets and non-
targets, this would not be near enough to base the success an entire classifier on. The existing
classifier is able to choose from a pool of thousands of different features, and then apply various
weights to the hundreds of features that it selects, in order to generate good results. Trying to
manually reproduce this task with implemented features is not only near-impossible, but goes
outside the scope of this thesis. The purpose of that experiment was to become comfortable with
how the classifier works, and to understand the steps involved in generating intelligent features.
The second conclusion that can be made is that the brute force method approach of guessing the
target letter was unsuccessful under standard conditions. Using an 8x9 matrix and 5 flash
48 | P a g e
sequences/letter, successfully classifying only 1/9 letters is nowhere near ideal. The concept of
this approach was interesting enough to conduct this experiment, but in the end was simply not
conclusive in showing any improvements in classification. This led directly into the third
experiment which discussed the importance of various parameters of the P300 speller, and the
impacts that modifying those parameters may have. While for this final experiment it was
beneficial to modify these parameters in order to obtain ideal results, in reality this would not
make sense as it is unjustifiable to hinder aspects of the speller in order to obtain positive results
from a classification method that was already proven to be unsuccessful. An overall conclusion
from the results of these experiments suggest that there is still much room for improvements to
be made in the classification of targets and non-targets for the P300-speller, both in accuracy as
well as number flash sequences required for successful classification.
Future Work
Future work on this topic could continue to explore new methods or techniques for improving
class separation. Specifically, the technique of data clustering could be worth exploring in order
see how well target and non-target responses cluster, as a means of separation. Future
consideration could also focus on how well the 2nd and 3rd ranked letters are scored relative to the
winners. More specifically, experiments could be conducted to detect if there is a larger gap in
the scores of the winner and runner-up of guessed characters for target responses, than there are
for non-target responses. One final improvement that could be made for future work could be on
improving automation of the P300 GUI. Due to many modifications made to the functionality of
the classifier, there was a lot of manual work done in order to calculate results when using the
49 | P a g e
GUI. A substantial amount of time could be saved for future result generation and analysis if the
P300 GUI application was slightly modified to directly fit the needs of the work that is being
done.
50 | P a g e
References
[1] D. McFarland, J. Wolpaw, “Brain-Computer Interfaces for Communication and Control,”
in Communications of the ACM. vol. 53, no.5, pp 60-66, 2015.
[2] A. Momennezhad, M. Shamsi, H. Ebrahimnezhad, H. Saberkari, “Classification of EEG-
P300 Signals Extracted from Brain Activities in BCI Systems using ʋ-SVM and BLDA
Algorithms,” in Applied Medical Informatics. vol. 32 no. 2, pp 23-35, 2014.
[3] C. King, P. Wang, L. Chui, A. Do, Z. Nenadic, “Operation of a Brain-Computer Interface
Walking Simulator by Users with Spinal Cord Injury,” in arXiv. 2015.
[4] U. Herwig, P. Satrapi, C. Schonfeldt-Lecuona, “Using the International 10-20 EEG
system for positioning of transcranial magnetic stimulation,” in Brain Topography. vol.
16, no. 2, pp 95-99, 2015.
[5] Z. Wu, “Studying modulation on simultaneously activated SSVEP neural networks by a
cognitive task,” in Journal of Biological Physics, no. 1, pp 55, 2014.
[6] J. Lu, W. Speier, X. Hu, N. Pouratian, “The effects of stimulus timing features on P300
speller performance,” in Clinical Neurophysiology. vol. 124, pp 306-214, 2013.
[7] J. Mattout, M. Perrin, O. Bertrand, E. Maby, “Update article: Improving BCI
performance through co-adaptation: Applications to the P300-speller,” in Annals of
Physical and Rehabilitation Medicine. 2014.
51 | P a g e
[8] G. Townsend, B. LaPallo, C. Boulay, D. Krusienski, G. Frye, C. Hauser, N. Schwartz, T.
Vaughan, J. Wolpaw, E. Sellers, “A novel P300-based brain-computer interface stimulus
presentation paradigm: Moving beyond rows and columns,” in Clinical Neurophysiology.
vol. 121, pp 1109-1120, 2010.
[9] L. McCane, S. Heckman, D. McFarland, G. Townsend, J. Mak, E. Sellers, D. Zeitlin, L.
Tenteromano, J. Wolpaw, T. Vaughan, “P300-based brain-computer interface (BCI)
event-related potentials (ERPs): People with amyotrophic lateral sclerosis (ALS) vs. age-
matched controls,” in Clinical Neurophysiology: Official Journal of the International
Federation of Clinical Neurophysiology. 2015.
[10] ALS Association, “Quick Facts about ALS & The ALS Assocation,” at
www.alsa.org/news/media/quick-facts.html. 2012.
[11] J. Wolpaw, E. Wolpaw, “Brain-Computer Interfaces: Principles and Practice,” in Oxford
University Press. 2012.
[12] I. Guyon, A. Elisseeff, “An Introduction to Variable and Feature Selection,” in Journal of
Machine Learning Research. vol. 3, no. 7/8, pp 1157-1182, 2003.
[13] M. Webber, “Unsupervised Learning of Models for Object Recognition,” at
http://www.vision.caltech.edu/publications/MarkusWeber-thesis.pdf. 2000.
52 | P a g e
Appendix A - Graphs
Character Rank vs. Data Sequence number for ‘C’ in “COMPUTER”
53 | P a g e
Character Rank vs. Data Sequence number for ‘O’ in “COMPUTER”
Character Rank vs. Data Sequence number for ‘M’ in “COMPUTER”
54 | P a g e
Character Rank vs. Data Sequence number for ‘P’ in “COMPUTER”
Character Rank vs. Data Sequence number for ‘U’ in “COMPUTER”
55 | P a g e
Character Rank vs. Data Sequence number for ‘T’ in “COMPUTER”
Character Rank vs. Data Sequence number for ‘E’ in “COMPUTER”
56 | P a g e
Character Rank vs. Data Sequence number for ‘R’ in “COMPUTER”
57 | P a g e
Appendix B – Code
symmetry.m
% this function returns a value between 0 and 1, % where 0 represents complete asymmetry of the points % and 1 represents complete symmetry. function [sym_value] = symmetry(targs) x = targs(1,:); x=x-min(x); x=x/max(x); sym_value = sum(abs(fliplr(x(1:16))-x(18:end))); sym_value = sym_value/16; sym_value = 1-sym_value; end
convexity.m
% this function returns value between 0 and 1 for convexity % of target array. 0 represents concave, and 1 represents % convex. the function assumes that target is convex by % giving a score of 33/33. Every time that a preceding point % to the left of the median decreases, so does the score. % Every time that the succeeding point to the right of the % median increases, then the score decreases. % The value from 0-1 is calculated by dividing the score by % the number of targets (33).
function [convex_value] = convexity(targs) sample = targs(1,:); num_samples = 33; convex_value = num_samples;
for i = (num_samples-1)/2 : 2 if sample(i-1) > sample(i) convex_value = convex_value - 1; end if sample(i-1) == sample(i) convex_value = convex_value - 0.5; end end
for j = (num_samples-1)/2 : num_samples - 1 if sample(j+1) < sample(j) convex_value = convex_value - 1; end if sample(j+1) == sample(j) convex_value = convex_value - 0.5; end end convex_value = (convex_value-1) / (num_samples-1); end
58 | P a g e
med_max_distance_function.m
% this function returns a value between 0 and 1, % where 0 represents the maxiumum distance between % the median index and maximum value indices of a sample, and 1 % represents no distance. this is calculated by finding % the index of the maxiumum value of a sample, and % calculating the difference between that and the median % index of a sample. this is subtrated from the median index, % and divided by the median index to return the result % between 0 and 1. function [med_max_distance_value] = med_max_distance(targs) sample = targs(1,:); num_samples = 33; median = 17;
max_targ_value = max(sample); max_targ_index = find(sample == max_targ_value);
distance = abs(median - max_targ_index); med_max_distance_value = (median-distance)/median; end
num_changes.m
% this returns the number of changes in direction the graph of % a sample takes.
function [num_changes_value] = num_changes(targs) sample = targs(1,:); num_samples = 33; num_changes_value = 0;
if sample(2)>sample(1) dir = 1; else dir = 0; end
for i = 2 : (num_samples-1) if dir == 1 if sample(i+1)<sample(i) num_changes_value = num_changes_value + 1; dir = 0; end else if sample(i+1) > sample(i) num_changes_value = num_changes_value + 1; dir = 1; end end end end
59 | P a g e
classifier_driver.m
% clear; close all; figure(1); hold on; cd('C:\Users\Connor Flood\Documents\School Work\4th Year\BCI Thesis\jitter'); load;
%mscore contains 266 items of targets, and 2926 items of non-targets. Each %item contains 33 values, or raw features, where the median (17), was the %value of classifier output used to decide if an item was target or not.
mscore(:,34) = []; targs = mscore(trigs,:); %targs contains all target items nontargs = mscore(nons,:); %nontargs contains all non-target items
% ---------------------------- % EXISTING FUNCTION BELOW % ---------------------------- for i=1:length(targs) control_targ(i) = targs(i); end
for i=1:length(nontargs) control_nontarg(i) = nontargs(i); end
control_targ = control_targ'; control_nontarg = control_nontarg'; targ_control_avg = mean(control_targ); targ_control_avg nontarg_control_avg = mean(control_nontarg); nontarg_control_avg
% plot(targs(:,17),control_targ,'b.'); % plot(nontargs(:,17),control_nontarg,'r.');
% ---------------------------- % SYMMETRY FUNCTION BELOW % ---------------------------- for i=1:length(targs) sym_targ(i) = symmetry(targs(i,:)); end
for i=1:length(nontargs) sym_nontarg(i) = symmetry(nontargs(i,:)); end
sym_targ = sym_targ'; sym_nontarg = sym_nontarg'; targ_sym_avg = mean(sym_targ); targ_sym_avg nontarg_sym_avg = mean(sym_nontarg);
60 | P a g e
nontarg_sym_avg
% plot(targs(:,17),symtarg,'b.'); % plot(nontargs(:,17),symnontarg,'r.');
% ---------------------------- % CONVEXITY FUNCTION BELOW % -----------------------------
for i=1:length(targs) convex_targ(i) = convexity(targs(i,:)); end
for i=1:length(nontargs) convex_nontarg(i) = convexity(nontargs(i,:)); end
convex_targ = convex_targ'; convex_nontarg = convex_nontarg'; targ_convex_avg = mean(convex_targ); targ_convex_avg nontarg_convex_avg = mean(convex_nontarg); nontarg_convex_avg
% plot(targs(:,17),convex_targ,'b.'); % plot(nontargs(:,17),convex_nontarg,'r.');
% ---------------------------- % MED_MAX_DISTANCE FUNCTION BELOW % ---------------------------- for i=1:length(targs) med_max_distance_targ(i) = med_max_distance(targs(i,:)); end
for i=1:length(nontargs) med_max_distance_nontarg(i) = med_max_distance(nontargs(i,:)); end
med_max_distance_targ = med_max_distance_targ'; med_max_distance_nontarg = med_max_distance_nontarg'; targ_med_max_distance_avg = mean(med_max_distance_targ); targ_med_max_distance_avg nontarg_med_max_distance_avg = mean(med_max_distance_nontarg); nontarg_med_max_distance_avg
% plot(targs(:,17),med_max_distance_targ,'b.'); % plot(nontargs(:,17),med_max_distance_nontarg,'r.');
% ---------------------------- % NUM_CHANGES FUNCTION BELOW % ----------------------------
61 | P a g e
for i=1:length(targs) num_changes_targ(i) = num_changes(targs(i,:)); end
for i=1:length(nontargs) num_changes_nontarg(i) = num_changes(nontargs(i,:)); end
num_changes_targ = num_changes_targ'; num_changes_nontarg = num_changes_nontarg'; targ_num_changes_avg = mean(num_changes_targ); targ_num_changes_avg nontarg_num_changes_avg = mean(num_changes_nontarg); nontarg_num_changes_avg
% plot(targs(:,17),num_changes_targ,'b.'); % plot(nontargs(:,17),num_changes_nontarg,'r.');
% ----------------------------------- % 2-DIMENSIONAL FUNCTION COMPARISON % ----------------------------------- % plot(symtarg, convex_targ, 'b.'); % plot(symnontarg, convex_nontarg, 'r.');
% ----------------------------------- % MDBC Trainer % ----------------------------------- % targ_class=[control_targ, sym_targ, convex_targ, ... % med_max_distance_targ, num_changes_targ]; % nontarg_class=[control_nontarg, sym_nontarg, convex_nontarg, ... % med_max_distance_nontarg, num_changes_nontarg];
targ_class=[control_targ, sym_targ, convex_targ, ... med_max_distance_targ, num_changes_targ]; nontarg_class=[control_nontarg, sym_nontarg, convex_nontarg, ... med_max_distance_nontarg, num_changes_nontarg];
[m,ico]=mdbc_train(targ_class , nontarg_class);
num_correct_targ = 0; sum_correct_targ = 0; sum_incorrect_targ = 0; for i=1:266 keep_targ=(1:266); keep_targ=setdiff(keep_targ,i); % now "keep" represents all targets
except for target #i [m,ico]=mdbc_train(targ_class(keep_targ,:),nontarg_class); %train with
all targets except #i res_targ(:,i)=mdbc_classify(m,ico,targ_class(i,:)); % test on target #i
and report the two probabilities into res(:,i) if (res_targ(1,i) >= res_targ(2,i)) num_correct_targ = num_correct_targ + 1; end sum_correct_targ = sum_correct_targ + res_targ(1,i); sum_incorrect_targ = sum_incorrect_targ + res_targ(2,i);
62 | P a g e
end
num_wrong_targ = 266 - num_correct_targ; percent_correct_targ = num_correct_targ/266 * 100; percent_correct_targ
average_correct_targ = sum_correct_targ/266; average_incorrect_targ = sum_incorrect_targ/266; average_correct_targ average_incorrect_targ
sort_correct_targ = sort(res_targ(1,:)); sort_incorrect_targ = sort(res_targ(2,:)); median_correct_targ = sort_correct_targ(266/2); median_incorrect_targ = sort_incorrect_targ(266/2);
median_correct_targ median_incorrect_targ
num_correct_nontarg = 0; sum_correct_nontarg = 0; sum_incorrect_nontarg = 0; for i=1:2926 keep_nontarg=(1:2926); keep_nontarg=setdiff(keep_nontarg,i); % now "keep" represents all targets
except for target #i [m,ico]=mdbc_train(nontarg_class(keep_nontarg,:),targ_class); %train with
all targets except #i res_nontarg(:,i)=mdbc_classify(m,ico,nontarg_class(i,:)); % test on
target #i and report the two probabilities into res(:,i) if (res_nontarg(1,i) >= res_nontarg(2,i)) num_correct_nontarg = num_correct_nontarg + 1; end sum_correct_nontarg = sum_correct_nontarg + res_nontarg(1,i); sum_incorrect_nontarg = sum_incorrect_nontarg + res_nontarg(2,i); end
num_wrong_nontarg = 2926 - num_correct_nontarg; percent_correct_nontarg = num_correct_nontarg/2926 * 100; percent_correct_nontarg
average_correct_nontarg = sum_correct_nontarg/2926; average_incorrect_nontarg = sum_incorrect_nontarg/2926; average_correct_nontarg average_incorrect_nontarg
sort_correct_nontarg = sort(res_nontarg(1,:)); sort_incorrect_nontarg = sort(res_nontarg(2,:)); median_correct_nontarg = sort_correct_nontarg(2926/2); median_incorrect_nontarg = sort_incorrect_nontarg(2926/2);
median_correct_nontarg median_incorrect_nontarg
63 | P a g e
FlashDriver.m
%utility for locating flash indices and cropping appropriate areas of %various structures to focus on specific letters load(fname); sig = eeg; num_letters = size(prm.TargetDefinitions,1);
trials=length(find(sts.PhaseInSequence(1:end-1)==1 &
sts.PhaseInSequence(2:end)==2)); NumTargFlashes=length(find(sts.StimulusType(1:end-
1)<sts.StimulusType(2:end)))/trials;
AllFlashInx=find(sts.StimulusBegin(1:end-1)<sts.StimulusBegin(2:end))+1; for i=1:num_letters flash_inx_letter(i,:) = flash(i, fname, AllFlashInx); end
word_length = length(prm.TextToSpell{1}); increment = length(AllFlashInx)/word_length; letter = 1; start_inx = (letter-1)*increment +1; end_inx = letter*increment;
beg = 1+find(sts.PhaseInSequence(1:end-1)~=1 &
sts.PhaseInSequence(2:end)==1); fin = find(sts.PhaseInSequence(1:end-1)==3 & sts.PhaseInSequence(2:end)~=3);
sts.PhaseInSequence(1:beg(letter)-1)=0; sts.PhaseInSequence(fin(letter)+1:end)=0;
sts.StimulusBegin(1:beg(letter)-1)=0; sts.StimulusBegin(fin(letter)+1:end)=0;
sts.Targ00(1:beg(letter)-1)=0; sts.Targ00(fin(letter)+1:end)=0;
sts.Targ01(1:beg(letter)-1)=0; sts.Targ01(fin(letter)+1:end)=0;
sts.Targ02(1:beg(letter)-1)=0; sts.Targ02(fin(letter)+1:end)=0;
sts.Targ03(1:beg(letter)-1)=0; sts.Targ03(fin(letter)+1:end)=0;
sts.Targ04(1:beg(letter)-1)=0; sts.Targ04(fin(letter)+1:end)=0;
sts.Targ05(1:beg(letter)-1)=0; sts.Targ05(fin(letter)+1:end)=0;
sts.Targ06(1:beg(letter)-1)=0; sts.Targ06(fin(letter)+1:end)=0;
64 | P a g e
if exist('guess.mat', 'file') load guess else guess=0; save guess guess end
guess = guess +1; save guess guess
sts.StimulusType(:) = 0; sts.StimulusType(flash_inx_letter(guess,(1+(letter-
1)*NumTargFlashes):letter*NumTargFlashes))= 1; sts.StimulusType(flash_inx_letter(guess,(1+(letter-
1)*NumTargFlashes):letter*NumTargFlashes)+1) = 1;
flash.m
%Function which returns a vector of indices for all of the occurences %of 'flashes' for a given letter function [LetterFlashInx] = flash(letter, file, AllFlashInx) load(file); %/AllLetterFlashInx1 = find((sts.Targ01==letter)&AllFlashInx);
% sts.Targ04 = int16( sts.Targ04 & sts.Targ00>=4);
sts.Targ04(find(sts.Targ00~=4))=0;
for i=1:length(sts.Targ04) if sts.Targ00(i)<4 sts.Targ04(i) = 0; end end
AllLetterFlashInx1 = find(sts.Targ01==letter); AllLetterFlashInx2 = find(sts.Targ02==letter); AllLetterFlashInx3 = find(sts.Targ03==letter); AllLetterFlashInx4 = find(sts.Targ04==letter);
%indices for all values of a letter occuring on a flash FlashInx1 = intersect(AllFlashInx, AllLetterFlashInx1); FlashInx2 = intersect(AllFlashInx, AllLetterFlashInx2); FlashInx3 = intersect(AllFlashInx, AllLetterFlashInx3); FlashInx4 = intersect(AllFlashInx, AllLetterFlashInx4);
%union of all 6 targets for a letter LetterFlashInx = union(FlashInx1, FlashInx2); LetterFlashInx = union(LetterFlashInx, FlashInx3); LetterFlashInx = union(LetterFlashInx, FlashInx4);
65 | P a g e
% BELOW FOR 9X8 GRID % % %indices for all values of a letter % %/AllLetterFlashInx1 = find((sts.Targ01==letter)&AllFlashInx); % AllLetterFlashInx1 = find(sts.Targ01==letter); % AllLetterFlashInx2 = find(sts.Targ02==letter); % AllLetterFlashInx3 = find(sts.Targ03==letter); % AllLetterFlashInx4 = find(sts.Targ04==letter); % AllLetterFlashInx5 = find(sts.Targ05==letter); % AllLetterFlashInx6 = find(sts.Targ06==letter); % % %indices for all values of a letter occuring on a flash % FlashInx1 = intersect(AllFlashInx, AllLetterFlashInx1); % FlashInx2 = intersect(AllFlashInx, AllLetterFlashInx2); % FlashInx3 = intersect(AllFlashInx, AllLetterFlashInx3); % FlashInx4 = intersect(AllFlashInx, AllLetterFlashInx4); % FlashInx5 = intersect(AllFlashInx, AllLetterFlashInx5); % FlashInx6 = intersect(AllFlashInx, AllLetterFlashInx6); % % %union of all 6 targets for a letter % LetterFlashInx = union(FlashInx1, FlashInx2); % LetterFlashInx = union(LetterFlashInx, FlashInx3); % LetterFlashInx = union(LetterFlashInx, FlashInx4); % LetterFlashInx = union(LetterFlashInx, FlashInx5); % LetterFlashInx = union(LetterFlashInx, FlashInx6); % % % hold on; % % plot(sts.Targ01) % % plot(AllFlashInx,0,'r.') % % plot(LetterFlashInx,0,'black.')
end
P3Classify.m (added modifications)
function
[predicted,result,score,resultthresh,tar]=P3Classify(Responses,Code,Type,MUD,
NumberOfSequences,SetsPerSequence,trialnr,NumMatrixRows,NumMatrixColumns,char
vect,wind,states,extra);
lenflash=length(states.Flashing); ind=find(states.Flashing(1:lenflash-1)==0 &
states.Flashing(2:lenflash)==1)+1; % from the state information, make the following: % targets(len,MAXFLASH) showing the targets currently flashing % flashgroup(len,NumMatrixRows*NumMatrixColumns) showing target flash NumTargets=NumMatrixRows*NumMatrixColumns; MAXFLASH=19; THRESHRATIO=NumTargets/(NumMatrixRows+NumMatrixColumns); NumInFlash=states.Targ00(ind); len=length(NumInFlash); flashgroup=zeros(len,MAXFLASH); targets=zeros(len,NumTargets);
66 | P a g e
for t=1:MAXFLASH ts=num2str(t); if length(ts)<2 ts=['0',ts]; end flashgroup(:,t)=states.(['Targ',ts])(ind); flashgroup(NumInFlash<t,t)=0; for f=1:NumTargets targets(flashgroup(:,t)==f,f) = 1; end end
windowlen=wind(2)-wind(1); numresponse=size(Responses,1); numchannels=size(Responses,3);
MUD(:,2)=MUD(:,2)-wind(1)+(MUD(:,1)-1)*windowlen;
nsamp=size(MUD(:,2),1); response1=reshape(Responses,numresponse,numchannels*windowlen); pscore=response1(:,MUD(:,2)+1)*MUD(:,3); if exist('pscore_rec.mat', 'file') load pscore_rec; end
load guess
pscorex(:,guess) = pscore;
Typex(:,guess) = Type;
save pscore_rec pscorex Typex; fprintf(1, 'Classifying Responses...\n\n'); %savemscore;where-1 % test for finding peaks % avg responses by row/col choices=NumTargets; numletters=max(trialnr); NumTargFlashes=length(Type(Type>0))/numletters;
cflash=zeros(choices,NumTargFlashes); score=zeros(numletters,choices); lrv=zeros(1,choices);
cc=1;
for bb=1:numletters cflash=zeros(choices,NumTargFlashes); range=find(trialnr==bb); if ~isempty(range) code=targets(range,:); code(Type(range)==0,:)=[]; [junk,code]=max(sum(code));
67 | P a g e
if code>0 targ(cc)=code; for aa=1:choices rv=pscore(range(find(targets(range,aa))));
lrv(aa)=length(rv);
cflash(aa,1:NumTargFlashes)=rv;
cflash(aa,:)=cumsum(cflash(aa,:)); end
numseq(bb)=min(lrv);
score(cc,:)=cflash(:,NumTargFlashes);
[valt,maxtarg]=max(cflash); predictedtarg(cc,:)=maxtarg;
cc=cc+1; end end %------------------------------------------- % Dynamical Thresholding codes %------------------------------------------- for ii=1:NumTargFlashes auxtarg=cflash(:,ii); sorted=sort(auxtarg); m=mean(sorted(1:(end-1))); sd=std(sorted(1:(end-1))); ci(bb,ii)=(sorted(end)-m)/sd; end; %------------------------------------------------------- %savepscore; %cmd=0;errors; end
tar=charvect(targ); targ=repmat(targ',1,single(NumTargFlashes)); numcorrect=[predictedtarg==targ]; predicted=charvect(predictedtarg); [rrr,ccc]=size(predictedtarg); if rrr==1 predicted=predicted'; end
%cmd=1;errors; %----------------------- % Alain's modification %------------------------ numcorrectmat=numcorrect; %------------------------- if size(numcorrect,1)>1
68 | P a g e
numcorrect=sum(numcorrect); end
result=numcorrect/double(cc-1)*100;
guiout=fopen('gui_result.txt','a'); fprintf(guiout,'Timestamp-> %04d-%02d-%02d
%02d:%02d:%02d\r\n\r\n',fix(clock)); fprintf(guiout, ' Target: %s \r\n',tar); fprintf(guiout, '\r\n Flashes | %% Correct | Predicted Symbols' ); fprintf(guiout, '\r\n --------|-----------|------------------\r\n' ); fprintf(1, ' Target: %s \n',tar); fprintf(1, '\n Flashes | %% Correct | Predicted Symbols' ); fprintf(1, '\n --------|-----------|------------------\n' ); for kk=1:size(result,2); pred=predicted(:,kk)'; rind=find(numseq<kk); if ~isempty(rind) for uu=1:length(rind) pred(rind(uu))=' '; end end fprintf(guiout, ' %02d | %03d%% | %s
\r\n',kk,round(result(kk)),pred); fprintf(1, ' %02d | %03d%% | %s
\n',kk,round(result(kk)),pred); end fprintf(guiout, '\r\n\r\n') fprintf(1, '\n\n') fclose(guiout);
%----------------------------------- % George's modification %----------------------------------- fprintf('--------------------------------------------------------------------
-------------------------\n'); fprintf(' Dynamical Target Flashes \n'); fprintf('%s \t\t %s \t\t %s \t\t %s \t\t %s\n','Flash', 'ConfInt', 'WSR %max'
, 'No Feedback', 'Feedback (wrong=~,none=BLANK)'); fprintf('--------------------------------------------------------------------
-------------------------\n');
for ff=2:NumTargFlashes threshvec=sort(ci(:,ff)); threshvec=[0;(threshvec(1:end-
1)+threshvec(2:end))/2;max([ceil(ci(:,ff)*2);100.0])]; bestWSR=-1.0; for ii=1:length(threshvec) truepos=length(find(predictedtarg(:,ff)==targ(:,ff) &
ci(:,ff)>threshvec(ii))); falsepos=length(find(predictedtarg(:,ff)~=targ(:,ff) &
ci(:,ff)>threshvec(ii))); noreports=length(find(ci(:,ff)<=threshvec(ii))); if double(numletters) == noreports perr=1.0;
69 | P a g e
timefact=double(numletters); % NOT SURE ABOUT THIS factor
representing time wasted on no-reports else perr=(falsepos)/(double(numletters)-noreports); timefact=double(numletters)/(double(numletters)-noreports); %
factor representing time wasted on no-reports end if perr>=0.5 perr=0.499999; end selfact=1.0/(1-2*perr); % factor representing selections required to
make one good selection percentWSR=100.0/(selfact*timefact)/(double(ff)/2); %p=1-perr; %n=NumMatrixRows*NumMatrixColumns; %br=log2(n)+p.*log2(p)+(1-p).*log2((1-p)/(n-1)); %bx=br/log2(n); % symbols per trial %bx=bx/selfact; %percentWSR=100.0*bx/(double(ff)/2); if percentWSR>bestWSR; bestWSR=percentWSR; bestii=ii; noreps=noreports; str=charvect(predictedtarg(:,ff))'; str(predictedtarg(:,ff)~=targ(:,ff))='~'; str((ci(:,ff)<=threshvec(ii)))=' '; end end; if bestWSR<=0 bestWSR=0; end resultthresh(ff-1,:)=[double(ff),threshvec(bestii),bestWSR]; fprintf('%2d\t\t\t\t%4.2f\t\t\t%5.2f\t\t\t%2d\t\t\t<%s>\n', ff,
threshvec(bestii),bestWSR,noreps,str); end;
Classify_Driver.m
%script to determine the rankings of target letters in a target word %and produce graph and statistics on that data word = 'COMPUTER'; for seq=1:15 for letter=1:length(word) file = sprintf('%s_%s_%02d',word,word(letter),seq); [score,winner] = new_function(file); targ_letter = abs(word(letter)) - 64; [junk,I]=sort(-score); place=find(I==targ_letter);
winners(letter,seq) = winner; sequence_letter_places(letter,seq) = place; end end
70 | P a g e
for letter=1:length(word) min_flash(letter) = 30; for seq=15:-1:1 x = sequence_letter_places(letter,seq); if x == 1 min_flash(letter) = seq; else break; end end end
for letter=1:length(word) figure plot(sequence_letter_places(letter,:)) end hold off; figure
for seq=1:15 num_correct(seq) = 0; for letter=1:length(word) if sequence_letter_places(letter,seq)==1 num_correct(seq) = num_correct(seq) + 1; end end end
for seq=1:15 word_guess_build = ' '; for letter=1:length(word) letter_guess = char(winners(letter,seq)+64); word_guess_build = strcat(word_guess_build,letter_guess); end word_guess_seq(seq,:) = word_guess_build; end
word_guess_seq plot(num_correct)
new_function.m
%function which returns the normalized score for a target letter, %and the score of the winning letter function [normalized_cumulative_score, winner] = new_function(saveScore) load(saveScore); x = size(pscorex); if x(2) ~= 75 pscorex(:,25) = 0; Typex(:,25) = 0; end
for letter=1:25 norm_factor=mean(pscorex(pscorex(:,letter)>0,letter)); pscorex(:,letter)=pscorex(:,letter)/norm_factor;
71 | P a g e
normalized_cumulative_score(letter)=sum(pscorex(find(Typex(:,letter)),letter)
); end winner =
find(normalized_cumulative_score==max(normalized_cumulative_score(:))); end
xpre.m
%script which makes appropriate crops to sts.StimulusBegin in order to %remove data sequences load Computer2
for i=14:-1:1 fin = find(sts.PhaseInSequence(1:end-1)==3 &
sts.PhaseInSequence(2:end)~=3)+1; for j=1:8 beg = find(sts.StimulusBegin(1:end-1)< sts.StimulusBegin(2:end)); beg = beg(beg<fin(j)); beg = beg(end-14); sts.StimulusBegin(beg:fin(j))=0; sts.StimulusType(beg:fin(j))=0;
for k=0:6 cmd=sprintf('sts.Targ%02d(beg:fin(j))=0;',k); eval(cmd); end end save(sprintf('Computer2-%02d',i),'sig','sts','prm'); end