A NOVEL SYSTEM FOR CONTROL OF A ROBOTIC WHEELCHAIR BASED ONSSVEP-BCI FOR PEOPLE WITH LOCKED-IN SYNDROME

Richard M. G. Tello∗, Sandra Muller†, Andre Ferreira∗, Teodiano Bastos-Filho∗

∗Post-Graduate Program in Electrical Engineering, Federal University of Espirito Santo (UFES)Vitoria-ES, Brazil

†Electrical Engineering Department, Federal Institute of Espirito Santo (IFES)Vitoria-ES, Brazil

Emails: richard@ele.ufes.br, sandra.muller@ifes.edu.br, andrefer@ele.ufes.br,

teodiano.bastos@ufes.br

Abstract— In this work, we present the development of a novel approach for control of a robotic wheelchaircommanded by a SSVEP-BCI. Thus, it was developed a visual stimulator with aim of helping people with seriousphysical disabilities, as is the case of patients with locked-in syndrome (LIS). In this system here developed, thesubject can modulate the attention in flickering frequencies without requiring head neuromuscular control or eyemovements to control the navigation of a robotic wheelchair. Multivariate Synchronization Index (MSI) wasused as method to recognize the visual evoked potentials. Three subjects participated of the experiments andEEG signals were captured through a commercial wireless device named Emotiv Epoc Headset. Four frequencieswere used as stimuli and each target represents a specific action or class: 8.0 Hz (forward), 11.0 Hz (turn right),13.0 Hz (stop) and 15.0 Hz (turn left). The results were quite promising achieving high accuracy values withan average of 76.97 % using window length of 2s. Furthermore, a real path and a desired path were comparedfor the wheelchair navigation. Finally, our hypothesis about the visual selectivity, perception and power of theattention for stimuli in reduced space vision were confirmed obtaining promising results.

Keywords— Independent SSVEP-BCI, Robotic Wheelchair, attentional modulation, MSI.

Resumo— No presente trabalho, foi apresentado o desenvolvimento de um novo enfoque para o controle deuma cadeira de rodas robotica comandada atraves de uma SSVEP-ICC. Com esse intuito, foi desenvolvido umestimulador visual com o objetivo de ajudar pessoas com deficiencias severas motoras, como e o caso de pessoascom sındrome de bloqueio ou LIS. Neste sistema, o sujeito tem a capacidade de modular a atencao em frequenciaspiscantes sem precisar de movimentos dos globos oculares ou neuromusculares para o controle da navegacao deuma cadeira de rodas robotica. O Indice de Sincronizacao Multivariavel (MSI) foi utilizado como extrator decaracterısticas. Tres sujeitos participaram dos experimentos e os sinais cerebrais foram coletados atraves de umdispositivo comercial sem fio denominado Emotiv Epoc. Quatro frequencias foram utilizadas como estımulos ecada alvo representa uma acao especıfica ou classe: 8,0 Hz (ir para frente), 11,0 Hz (vire a direita), 13,0 Hz(de parada) e 15,0 Hz (vire a esquerda). Nossos resultados foram bastante promissorios, com altos valores deprecisao e uma media de 76,97 % analisando janelas de tempo de 2s. Por outro lado, uma rota percorridapela cadeira (percurso real) e um percurso desejado foram comparados. Finalmente, nossa hipotese relacionada aseletividade visual, a percepcao e o poder da atencao em espacos de visao reduzida foram confirmadas e resultadospromissorios foram obtidos.

Palavras-chave— SSVEP-BCI Independente, cadeira de rodas robotica, modulacao atencional, MSI.

1 Introduction

During the last years, several studies have at-tempted to construct communication techniquesindependent of peripheral neuromuscular activi-ties. A technology that promises to be a potentialalternative as well as an augmentative communi-cation (AAC) and control solution for people withsevere motor disabilities (Kelly et al., 2005a) arethe Brain Computer Interfaces (BCIs).

BCI is a technology which provides to hu-man a direct communication between the user’sbrain signals and a computer, generating an al-ternative channel of communication that does notinvolve the traditional way as muscles and nerves(Wolpaw J.R., 2000). Therefore, a BCI recordsbrain signals and extracts Electroencephalogra-phy (EEG) signal features and these features arethen translated into artificial outputs or com-mands that act in a real world. A subcategory ofBCI using Visual Evoked Potentials (VEPs) has

drawn attention for its high Information TransferRate (ITR), high accuracy and no previous train-ing to the subject.

When the eye retina is excited by a stimulus ata certain frequency, the brain generates an electri-cal activity of the same frequency with its multi-ples or harmonics. This stimulus produces a stableVEP of small amplitude termed as “Steady-State”Visually Evoked Potentials (SSVEPs) of the hu-man visual system.

In a typical SSVEP-BCI system, several stim-uli flickering at different frequencies are presentedto its user. The subject overtly directs attentionto one of the stimuli by changing his/her gaze at-tention (Zhang et al., 2010). This kind of SSVEP-BCI is commonly called as“dependent” since mus-cle activities, such as gaze shifting, are neces-sary. Patients with stroke trauma, poliomyeli-tis, amyotrophic lateral sclerosis (ALS), botulism,spinal cord injury (SCI), multiple sclerosis, andGuillain-Barre syndrome suffer from motor dis-

abilities, which can disrupt their communicationwith the external environment, resulting in the so-called locked-in syndrome (LIS). Therefore, “de-pendent” SSVEP-BCIs might not be applicablefor patients with traumatic brain injuries, pre-vented head movement or ocular motor impair-ments. Nonetheless, “independent” SSVEP-BCIscould be an alternative solution for these patients.

An Independent-SSVEP-BCI is controlled bysubject’s attentions without requiring head neuro-muscular control or eye movements. Also, regard-ing to Independent-BCI’s applicability in dailylife, its approach could be used in a portable sys-tem able to select a command on the screen of asmartphone using only of the attentional modula-tion.

Neurophysiological studies (Fries et al., 2001;McMains and Somers, 2004; Reynolds andChelazzi., 2004) have demonstrated an increasesin neural activity elicited by a visual stimuluswhen a human directs his/her attention to a re-gion of visual space containing a stimulus. Theseresults have led to suggest the hypothesis that at-tention improves the representation of a stimulusin the occipital areas of the cortex.

On the other hand, studies from (Hairstonet al., 2014; McDowell et al., 2013) reveal thatwireless EEG acquisition systems provide impor-tant advancements by pushing EEG technologytowards alternative approaches to establish con-nectivity between the electrode and the scalp.Their wireless un-tethered operation makes themmore applicable for use in every-day, less restric-tive scenario (Hairston et al., 2014). Thus, theyare better suited for studies where the goal is tomonitor neural activity within naturalistic scenar-ios (McDowell et al., 2013). However, the presenceof motion-and muscle-related signal artifacts stillremains a very important factor to take into ac-count for user applications in real-world.

This work presents a novel approach for con-trol a robotic wheelchair based on SSVEP-BCI.Thus, we have conducted an approach of at-tention modulation based on stimuli of reducedspaces. Four small targets represented in fourLight-Emitting Diodes (LEDs) that flicker in fourfrequencies established (8, 11, 13 and 15 Hz).Three subjects with average age of 28 years oldparticipated in the experiments and a path fornavigation was established in order to evaluatethe performance of the robotic wheelchair control.A commercial wireless device called Emotiv Epocheadset was used for the acquisition of EEG sig-nals. Multivariate Synchronization Index (MSI)was used as feature extractor in the recognitionof SSVEP components and the response time foreach commands was established in 2 seconds.

2 Methods

2.1 Wireless EEG Acquisition

Emotiv Epoc Headset is a wireless device usedfor acquiring signal. EEG signals from occipitalO1 and O2 channels with 128 samples per secondof sampling rate (fs) were collected. The sensorsused for reference are: (i) Common-Mode Sensing(CMS) and, (ii) Driven Right Leg (DRL) that arefixed for default and in parallel to P3/P4 channels(the two mastoids), respectively.

2.2 Visual Stimuli

A coupling structure of four small boxes (5cmx 5cm x 5cm) containing a LED in each oneand covered with thin white papers diffuser wasmounted in the top of a LCD screen. The vol-unteers sat aboard of the robotic wheelchair, infront of a stimulator system, 70 cm far from this.The timing of the four LEDs flickers was preciselycontrolled by a microcontroller (PIC18F4550, Mi-crochip Technology Inc., USA) with 50/50% on-off duties, and frequencies of 8.0 Hz (top), 11.0Hz (right), 13.0 Hz (bottom) and 15.0 Hz (left).These frequencies represent commands or classes:Class 1 (f1=8 Hz), Class 2 (f1=11 Hz), Class 3(f1=13 Hz) and Class 4 (f1=15 Hz), more de-tails are shown in Figure 1. These frequencieswere chosen due to: i) our previous studies (Telloet al., 2014a; Tello et al., 2014b; Tello et al., 2014d;Tello, Pant, Muller, Krishnan and Filho, 2015)have shown that these generate strongest SSVEPresponses; ii) safety recommendations specifiedin (Fisher et al., 2005) have shown that these;iii) studies conducted by (Pastor et al., 2003)about the relationship between visual stimuli andSSVEP-evoked amplitudes recommend these fre-quencies; iv) according to (Kelly et al., 2005b)is suggest that frequencies in alpha band im-proves the recognition of attentional modulationfor SSVEP stimuli. The amplitude and phase

Class 1 (Forward) 8 Hz

Class 2 (Right) 11 Hz

Class 3 (Stop) 13 Hz

Class 4 (Left)

15 Hz

5 cm

5 cm

Figure 1: Specifications of the visual stimulationunit based on LEDs.

of the SSVEP are highly sensitive to stimulusparameters, such as color, luminance, repetitionrate, contrast or modulation depth, and spatial

frequency. On the other hand, flicker stimulican elicit epileptic responses to certain luminanceor chromaticity. Higher luminance can induce ahigher risk of epilepsy (Vialatte et al., 2010) andthe chromaticity of a visual stimulus has a strongimpact on the human eye response in case of com-bination of colors (Drew et al., 2001). Red/blueand green/blue combinations have the strongesteffect on pupil constriction and they can seizureattacks (Drew et al., 2001). Regarding frequencydependency, repetitive visual stimuli modulatedat certain frequencies can also provoke epilepticseizures. According to (Drew et al., 2001), lowerfrequency flickers generally produce more power-ful constrictions, with color-dependence of flickersmost visible between 3 and 6 Hz.

Therefore, taking into account all aforemen-tioned recommendations and according to (Telloet al., 2014c; Tello, Muller, Ferreira and Bastos-Filho, 2015), we used the green and yellow col-ors for the stimulation with triangular and squareshapes where the subject will be able to modulateattention through of the perception in reduced vi-sual spaces. This attentional modulation was per-formed making a minimum visual angle (horizon-tal: <4.09o and vertical: <2.05o); these values areinsignificant and within of the systems that oper-ate in independent way of neuromuscular function,such as it is shown in Figure 2. Also, studies per-formed in (Tello et al., 2014c; Tello, Muller, Fer-reira and Bastos-Filho, 2015) demonstrated thatgreen and yellow colors obtained high values ofperformance and comfort for SSVEP stimulation.

4.09º

70 cm

70 cm

2.05º 5 cm

5 cm

15 Hz

11 Hz

8 Hz

13 Hz

(a)

(b)

Top view

Side view

Figure 2: Schematic diagram of the experimentalsetup. (a) Top view; (b) side view.

2.3 Subjects

Three male subjects were recruited to participatein this study (average age: 28.0; Standard Devia-tion (STD): 1.0). The research was carried out incompliance with Helsinki declaration, and the ex-periments were performed according to the rulesof the ethics committee of UFES/Brazil, underregistration number CEP-048/08. The volunteerswere labelled as: s1, s2 and s3. Previous selection

of volunteers was performed and topics relatedto the precaution as visual problems, headaches,family history with epilepsy and problems relatedto brain damage were taken. All volunteers re-ported not having any inconvenience for conduct-ing the tests and no one had previous experiencein using a BCI.

2.4 Experimental Procedure

A desired path was used to subjects move his/herwheelchair through attentional modulation. Theexperimental procedure is shown in an explana-tory Figure 3. Basically the procedure consistsof each forward movement which lasts twenty sec-onds, and each clockwise rotation or counterclock-wise 90 degrees lasts ten seconds. Each experi-ment lasts 110s (55 classified trials).

6 m

6 m

6 m

END START

6 m

Figure 3: The path of the navigation of the roboticwheelchair.

3 Feature Extraction and Classification

Brain signals were captured by the Emotiv Epocand then these signals were transmitted by wire-less and processed in an embedded computer byalgorithms developed in Matlab. This embeddedcomputer in the wheelchair has the following spec-ifications: Mini ITX motherboard, 3.40 GHz IntelCore i5 processor, and 4GB RAM. The EEG datawere segmented and windowed in window lengths(WL) of 2 s without overlapping. Then, a spatialfiltering is applied using a Common Average Ref-erence (CAR) filter and a band-pass filter between3-60 Hz for the twelve channels. A graphical ex-planation of the complete system is illustrated inFigure 4.

Several studies (Vialatte et al., 2010; Pastoret al., 2003) confirm that visual evoked potentialsare generated with greater intensity in the occipi-tal area of the cortex. Based on that fact, we haveevaluated the detection of SSVEPs located in thechannels O1 and O2, i. e., these two channels wereused as input vector for the feature extractor afterof filtering process aforementioned. MultivariateSynchronization Index (MSI) was used for featureextraction. A brief description of this technique isexplained below.

Wireless

EEG

UFES Robotic Wheelchair

Outputs

Navigation

Emotiv Epoc Headset

Visual stimulation (8, 11, 13 and 15 Hz)

Serial Port

Signal Processing

Personal Computer

Synchronizing signal

Acquisition

PIC18F4550 Flicker

generator

(a) (b)

Figure 4: (a) The schematic diagram of the SSVEP-BCI applied to robotic wheelchair control. (b) Rearview of the developed robotic wheelchair.

MSI is a novel method to estimate the syn-chronization between the actual mixed signals andthe reference signals as a potential index for recog-nizing the stimulus frequency. (Zhang et al., 2014)has proposed the use of a S-estimator as index,which is based on the entropy of the normal-ized eigenvalues of the correlation matrix of mul-tivariate signals. The reference matrix Yi of size2H × N is based on sines and cosines that simu-lates the signal contain the frequency of stimula-tion. In our case, we have considered the funda-mental frequency and one harmonic as the simu-lated frequency generator, i. e. H = 2. Autocor-relation matrices C11 and C22 for X and Yi, re-spectively, and crosscorrelation matrices C12 andC21 for X and Yi can be obtained as (Telloet al., 2014a), where i refers to the number of tar-gets.

C11 = (1/N).XXT (1)

C22 = (1/N).Yi.YiT (2)

C12 = (1/N).XYiT (3)

C21 = (1/N).Yi.XT (4)

A correlation matrix Ci can be constructed as

Ci =

[C11 C12

C21 C22

](5)

The internal correlation structure of X andYi contained in the matrices C11 and C22, re-spectively, is irrelevant for the detection of stim-ulus frequency (Carmeli et al., 2005). It can beremoved by constructing a linear transformationmatrix

U =

[C11

−1/2 0

0 C22−1/2

](6)

So that C111/2 C11

1/2=C11,C22

1/2C221/2=C22 and by applying the

transformation Ci = UCU which results ina transformed correlation matrix of size P × P ,where P = M + 2H (Carmeli et al., 2005). The

eigenvalues λi1, λi2,...,λiP of Ci, normalized as

λim = λim/∑P

m=1 λim for m = 1, 2, ..., P , can be

used to evaluate the synchronization index Si formatrix Yi as

Si = 1 +

∑Pm=1 λ

im log(λim)

log(P )(7)

see (Zhang et al., 2014). Using S1,S2,...,SK computed for the stimulus frequencies f1,f2,...,fK , the MSI can be estimated as

S = maximum1≤i≤K

Si (8)

In this case, the way to assess the perfor-mance of the SSVEP-BCI system was the Shan-non’s Information Transfer Rate (ITR), see detailsin (Vialatte et al., 2010).

4 Navigation Control

The kinematic model of our unicycle roboticwheelchair is described by a simple non-linearmodel (see Figure 5) as in Equation 9:xy

θ

=

v cos θv sin θω

, (9)

where M = (x,y,θ) is the wheelchair position andorientation in world reference frame, and the pair(v,ω) is the input control encompassing the linearand angular velocities. In our case, linear (v =0.3 m/s) and angular (w = 9o/s) velocities areconstant.

ݔ

(ݐ)ܯ

(ݐ)ݒ

ߠ ݕ

Figure 5: Wheelchair’s kinematics.

• If the wheelchair moves forward: v = 0.3 m/sand w = 0.

• Moves to right (clockwise): v = 0 and w < 0.

• Moves to left (counter-clockwise): v = 0 andw > 0.

• To stop: v = 0 and w = 0.

From these parameters, if the roboticwheelchair moves forward with a constant velocityof 0.3 m/s and a time established of 20s, this willmove it forward in 6 m. On the other hand, if therobotic wheelchair turns clockwise with angularvelocity of 9o per second and a time established of10s, this will have rotated 90o.

To calculate the current position of x, y and θof the robotic wheelchair, odometry is used by cal-culating the sum of the variations at each instantof time:

xyθ

=

∑t0 x · ∆t∑t0 y · ∆t∑t0 θ · ∆t

=

∑t0 v cos θ · ∆t∑t0 v sin θ · ∆t∑t

0 ω · ∆t

(10)

5 Experimental Results

A comparison of performance (accuracy) wastested for 3 subjects using WL = 2s aboard ofa robotic wheelchair in an online way. In ad-dition, these results were also compared withthe path and class desired. A summary of re-sults is shown in Table 1. Information transferrate (ITR) parameter was calculated according to(Tello et al., 2014a).

Table 1: Accuracy results from navigation of therobotic wheelchair using WL of 2 s.

Accuracy ITR TrialsSubjects [%] [bits/min] Classified

s1 80.00 57.67 55s2 78.18 53.85 55s3 72.73 43.56 55

Mean 76.97 51.63 -± STD ±3.78 ±7.41 -

6 Conclusion and Discussion

According to our results shown in Table 1, highvalues of accuracy were obtained with an aver-age of 76.97 % and STD of 3.78 %, taking intoaccount that the subjects didn’t perform neuro-muscular movements and four frequencies blinkingwere presented at the same time. In addition, thehighest value of ITR (57.67 bits/min) was foundin subject 1. Our studies showed good resultsand quite promising and even could be improved,taking into consideration that was the first timethat volunteers had used a BCI, and it is widelyknown that its continuous use can increase theaccuracy. Furthermore, our hypothesis about thevisual selectivity, colors perception and the powerof the attention to stimuli in spaces reduced vi-sion have been confirmed. On the other hand, asimilar path to the desired path between subjectswas performed (real path).

Acknowledgement

The authors thank to FAPES and CAPES for sup-porting this research.

References

Carmeli, C., Knyazeva, M. G., Innocenti, G. M.and Feo, O. D. (2005). Assessment of EEGsynchronization based on state-space analy-sis, NeuroImage 25: 339–354.

Drew, P., Sayres, R., Watanabe, K. and Shimojo,S. (2001). Pupillary response to chromaticflicker, Exp. Brain Res 136 (2): 256–262.

Fisher, R. S., Harding, G., Erba, G., Barkley,G. L. and Wilkins, A. (2005). Photic-and pattern-induced seizures: a review forthe Epilepsy Foundation of America Work-ing Group, Epilepsy Foundation of AmericaWorking Group. 46: 1426–1441.

Fries, P., Reynolds, J. H., Rorie, A. E. and Des-imone, R. (2001). Modulation of oscillatoryneuronal synchronization by selective visualattention, Science 3: 560.

Hairston, W. D., Whitaker, K. W., Ries, A. J.,Vettel, J. M., Bradford, J. C., Kerick, S. E.and McDowell, K. (2014). Usability of fourcommercially-oriented EEG systems, Journalof Neural Engineering .

Kelly, S., Lalor, E., Reilly, R. and Foxe, J.(2005a). Visual spatial attention tracking us-ing high-density SSVEP data for independentbrain-computer communication, Neural Sys-tems and Reh. Eng., IEEE Transactions on13(2): 172–178.

(a)

0 5 10 15-5

0

5

10

Y[m]

X[m]

Real TrajectoryDesired Trajectory

0 10 20 30 40 501

2

3

4

Class

Classified trial

Real ClassDesired Class

55

(b)

0 10 20 30 40 501

2

3

4

Class

Classified trial

Real ClassDesired Class

(c)55

0 5 10 15-5

0

5

10

Y[m]

X[m]

Real TrajectoryDesired Trajectory

(d) 0 5 10 15-5

0

5

10

Y[m]

X[m]

Real TrajectoryDesired Trajectory

0 10 20 30 40 501

2

3

4

Class

Classified trial

Real ClassDesired Class

55(e)

(f)

Start

End

Start

EndStartEnd

Figure 6: (a) Classified trials and (b) executed path for s1; (c) and (d) for s2; and (e) and (f) for s3,respectively.

Kelly, S. P., Lalor, E., Reilly, R. B. and Foxe, J. J.(2005b). Independent Brain Computer Inter-face Control using Visual Spatial Attention-Dependent Modulations of Parieto-occipitalAlpha, Neural Eng. IEEE EMBS .

McDowell, K., .Chin-Teng, L., Oie, K. S., Tzyy-Ping, J., Gordon, S., Whitaker, K. W., Shih-Yu, L., Shao-Wei, L. and Hairston, W. D.(2013). Real-World Neuroimaging Technolo-gies, Access, IEEE , vol. 1, pp.: 131,149.

McMains, S. A. and Somers, D. C. (2004). Multi-ple Spotlights of Attentional Selection in Hu-man Visual Cortex, Neuron pp. 677–686.

Pastor, M., Artieda, J., Arbizu, J., Valencia, M.and Masdeu, J. (2003). Human cerebral ac-tivation during steady-state visual evoked re-sponse, J. Neurosci. pp. 11621–11627.

Reynolds, J. H. and Chelazzi., L. (2004). Atten-tional modulation of visual processing, Annu.Rev. Neurosci. 27:: 611–647.

Tello, R., Muller, S., Bastos-Filho, T. and Fer-reira, A. (2014a). A comparison of tech-niques and technologies for SSVEP classifi-cation, (BRC), 5th ISSNIP-IEEE, pp. 1–6.

Tello, R., Muller, S., Bastos-Filho, T. and Fer-reira, A. (2014b). Comparison between wireand wireless EEG acquisition systems basedon SSVEP in an Independent-BCI, 36 thEMBC of the IEEE, pp. 22–25.

Tello, R., Muller, S., Bastos-Filho, T. and Fer-reira, A. (2014c). Evaluation of the influ-ence of stimuli color for a SSVEP-based BCI,XXIV (CBEB), Brazil .

Tello, R., Muller, S., Bastos-Filho, T. and Fer-reira, A. (2014d). Towards the portability ofan Independent-BCI based on SSVEP, XXIV(CBEB), Brazil .

Tello, R., Muller, S., Ferreira, A. and Bastos-Filho, T. (2015). Comparison of the Influ-ence of Stimuli Color on Steady-State VisualEvoked Potentials, Research on BiomedicalEngineering .

Tello, R., Pant, J., Muller, S., Krishnan, S. andFilho, T. B. (2015). An Evaluation of Per-formance for an Independent SSVEP-BCIBased on Compressive Sensing System, 13thProc. IUPESM World Congress on MedicalPhysics and Biom. Eng .

Vialatte, F. B., Mauriceb, M., Dauwelsc, J.and Cichocki, A. (2010). Steady-state vi-sually evoked potentials: Focus on essentialparadigms and future perspectives, Progressin Neurobiology 90: 418–438.

Wolpaw J.R., e. a. (2000). Brain-computer inter-face technology: a review of the first interna-tional meeting, Rehabilitation Engineering,IEEE Transactions on 8(2): 164–173.

Zhang, D., Maye, A., Gao, X., Hong, B., Engel,A. K. and Gao, S. (2010). An independentbrain-computer interface using covert non-spatial visual selective attention, Journal ofNeural Engineering 7(1): 016010.

Zhang, Y., Xu, P., Cheng, K. and Yao, D. (2014).Multivariate synchronization index for fre-quency recognition of SSVEP-based brain-computer interface, Journal of NeuroscienceMethods 221(0): 32 – 40.