Deep learning of dynamic functional connectivity states during

sleep and epilepsy using simultaneous EEG-fMRI

Joana Pernadas Carmonajoana.p.carmona@ist.utl.pt

Instituto Superior Tecnico, Universidade de Lisboa, Lisboa, Portugal

November 2018

Abstract

Functional Connectivity (FC) estimated from functional Magnetic Resonance Imaging (fMRI) datahas emerged as a powerful metric to investigate the brain’s intrinsic organization. Since it is modulatedby the level of arousal and by disease, FC-based classifiers targeting sleep staging and diagnosis of braindisorders have been proposed. Emerging trends in this context include the estimation of dynamic FC(dFC) features and the transition from conventional machine learning methods to deep learning, bothexplored in the present work. The starting point was the application of Convolutional Neural Net-works (CNNs) to classify sleep stages at individual level, provided the success of conventional classifiersreported by previous works. Subsequently, the method was extended to an exploratory scenario com-prising the classification of epileptic states at individual and group levels, as well as in a single-subjectclassification scenario. In the latter, dataset augmentation and transfer learning strategies were alsoinvestigated. Regarding the sleep staging problem, the implementation of different CNN architecturesresulted in balanced accuracies over 80%; however, the best performance was obtained using a shallowneural network. Similar results were obtained when applying this model to the epilepsy scenario at bothindividual and group levels. A systematic comparison with Support Vector Machines (SVMs) revealedan equivalent performance but presented some drawbacks in terms of computational cost and inter-pretation. In the single-subject classification scenario, however, the significant inter-subject variabilitycompromised the generalization ability of the tested classifiers. Further validation of the implementeddeep learning methods is largely dependent on an increased data availability.

Keywords: Sleep, Epilepsy, Functional Magnetic Resonance Imaging, Dynamic Functional Connec-tivity, Deep Learning, Convolutional Neural Networks

1. Introduction

The study of brain function at rest using BloodOxygen Level Dependent (BOLD)-fMRI led to theobservation of spontaneous low-frequency fluctua-tions that are correlated across networks typicallyactivated during task performance, the so-calledResting State Networks (RSNs) [40]. Given the re-markable spatial resolution of fMRI and its moder-ate temporal resolution, it is possible to investigatethe FC, which is defined as the correlation betweenthe time-series data of spatially distinct brain areas[35]. Although initially regarded as static over theentire duration of the fMRI acquisition, it has beendemonstrated that the FC exhibits fluctuations atdifferent temporal scales, hence being designateddynamic Functional Connectivity (dFC) [10]. It isnow well-established that the dFC is modulated bythe level of vigilance and also by disease, which hasinspired the development of dFC-based classifierstargeting sleep staging and diagnosis of brain disor-

ders [16, 34, 36, 45].

An emerging trend in this context is the choiceof deep learning methods over conventional ma-chine learning ones [16], which has been motivatedby their general ability to perform feature selec-tion in an automatic manner [27] and, in partic-ular, by their capacity to learn the complex anddiffuse patterns present in neuroimaging data dueto the hierarchical application of non-linear trans-formations [6, 49]. From the broad family of deeplearning methods, CNNs stood out for its efficienttraining and generalization capacity. CNNs havebroken benchmark records and became the state-of-the-art method for several problems within the fieldof computer vision [27]; however, their applicationto fMRI-derived FC data has only been pursued bya few studies [24, 32].

The starting point of the present work was theapplication of CNN architectures to the classifica-tion of sleep stages. The human sleep cycle can

1

be described as an interplay between two differentstages, the non-rapid eye movement (NREM) sleepand the rapid eye movement (REM) sleep, the for-mer being further divided into a number of stages(S1 - S4) related to sleep depth [37]. Several studieshave investigated stage-specific FC changes and es-tablished their neurobiological interpretation (com-prehensive reviews can be found in [34, 45]). More-over, simultaneously acquired electroencephalogra-phy (EEG) data allowed the labeling of those fea-tures due to the existence of electrophysiologicalevents that serve as biomarkers of different sleepstages [38]. Since it was demonstrated that sub-jects typically experience vigilance fluctuations dur-ing resting-state fMRI acquisitions [43], an objec-tive monitoring of the level of wakefulness throughthe development of dFC-based sleep-stage classifiershas been pursued by different studies [5, 21, 43, 46].However, most of them relied on SVMs, renderingthe application of deep learning methods in thiscontext largely unexplored.

Following the application of CNNs to the clas-sification of sleep stages using dFC features, theextension of this approach to the classification ofepileptic states was investigated. Epilepsy is a neu-rological disease characterized by the spontaneousoccurrence of seizures arising from an aberrant hy-persynchronization of large neuronal populations[17, 48]. The initial view of epilepsy as a disease in-volving epileptogenic zones in the brain evolved to aconceptualization of epilepsy as a network disorder[25]. Following this, FC impairments in epilepticpatients have been identified by a number of studies[8, 26, 29, 31]. The activity of these abnormal net-works comprises periods of steady-state brain activ-ity and periods marked by the occurrence of seizures[30, 42]. The alternation between them, and alsobetween periods of more or less intense inter-ictalactivity, is underlined by the dynamic transitionsbetween the epileptic (abnormal) brain state andnormal brain states. The existence of these statesis at the basis of the classification of epileptic statesperformed in the present work. We believe that thisdisease’s complex spatio-temporal dynamics can beadequately captured by dFC data; moreover, simul-taneously acquired EEG data allowed the identifica-tion of epileptiform activity (seizures and inter-ictalelectrical discharges, IEDs) and the attribution oflabels to dFC data.

FC features have been applied in the context ofclassification or prediction of several brain disor-ders, including schizophrenia, Attention Deficit Hy-peractivity Disorder, Alzheimer’s Disease, and MildCognitive Impairment (a comprehensive review canbe found in [16]). Regarding the type of classifier,most studies applied conventional machine learningmethods, in particular SVMs. One of the seminal

works in the application of CNNs to FC data waspublished in 2017 and targeted the classification ofpatients diagnosed with amnestic Mild CognitiveImpairment vs. healthy controls [32]. In that study,Meszlenyi and colleagues [32] proposed a CNN ar-chitecture specifically designed to receive FC matri-ces as input data named Connectome-ConvolutionalNeural Network (CCNN). This model outperformeddifferent classifiers (including SVMs, a simple neu-ral network, and a neural network with two hiddenlayers) and has proved to be more robust to addednoise and data modifications [32]. The key differ-ence of the present work is the extraction of dFCfeatures (rather than static ones), allowing the clas-sification of brain states over time.

The problems addressed in this work were sub-divided into three sequential studies: sleep stag-ing at individual level (Study I); classification ofepileptic states at individual level (Study II) andat group level (Study III). Additional objectives ofthis work include: i) to determine the features thatwere more decisive to the classification and to inter-pret these findings according to sleep stage-specificFC changes or to each patient’s clinical condition;ii) to investigate the applicability of dataset aug-mentation and transfer learning strategies; iii) andto compare the performance of CNNs with thatof more conventional machine learning methods, inparticular SVMs.

2. Data acquisition and preprocessing

2.1. Patients

A group of patients diagnosed with drug-refractory focal epilepsy underwent simultaneousEEG-fMRI acquisitions at the Imaging Center ofHospital da Luz in Lisbon. The EEG-fMRI studywas approved by the local ethics committee andwritten informed consent was provided by all pa-tients or by their legal representatives. A descrip-tion of the datasets obtained for all patients alongwith their clinical picture is provided in Table 1.Patient 2 was the only one that exhibited wakeful-ness fluctuations throughout the scanning time ac-cording to its EEG recordings, hence being eligiblefor Study I. All patients were selected for StudiesII and III, regardless of the detection of epilepticactivity on their EEG recordings.

2.2. Data acquisition

Image acquisition was performed on a 3 TeslaSiemens Verio scanner (Siemens, Erlangen). A3D, T1-weighted MPRAGE sequence was usedto acquire whole-brain structural images with anisotropic resolution of 1 mm. Functional data wasacquired using a 2D multi-slice gradient-echo EPIsequence with TR/TE = 2500/30 ms, 37 or 40axial slices yielding whole-brain coverage, and a

2

Table 1: Characterization of each patient’s clinical condition and description of the respective EEG-fMRI datasets.CAE = Childhood Absence Epilepsy ; CSWS = Continuous Spike Wave discharges in slow wave Sleep.

Patient Age Dataset Duration [min] # IEDs Clinical Condition

1 11 1 10 1 (seizure) CAE, with IEDs restricted to the left hemisphere.

2 91 10 596 CSWS, with IEDs over the left temporal lobe, verbal agnosia (Wernicke

type), and impaired ability to sustain attention.2 20 738

3 331 10 288 Continuous partial epilepsy, with large left-temporal cortical dysplasia,

accompanied by continuous myoclonias of the right hand.2 10 342

4 271 10 15 Refractory focal epilepsy, with IEDs over the posterior occipital-temporal

lobe and frontal propagation.2 5 7

5 51 20 754 CSWS, with right neonatal thalamic hemorrhage, IEDs over the posterior

right quadrant epileptogenic focus and frontal propagation.2 10 292

6 161 10 0 IEDs over the frontal lobe and a poorly characterized hyper-intense region

on structural MR images, compatible with an hypothalamic hamartoma.2 10 0

7 151 10 0

IEDs over the frontal lobe bilaterally, with a hypothesized hypothalamic hamartoma.2 10 0

8 151 5 0

Benign occipital epilepsy, with IEDs prominently over the left hemisphere.2 20 0

3 10 0

spatial resolution of 3.5 x 3.5 x 3.0 mm3. EEGdata was recorded simultaneously with the fMRIacquisition using a BrainCap MR model (Easy-Cap, Herrsching, Germany) combined with a MR-compatible 32-channel BrainAmp MR plus ampli-fier (Brain Products, Germany). The mentionedEEG cap contains 31 electrodes placed accordingto the standard 10-20 system, a reference electrode,and an additional one used for ECG recording.

2.3. Data pre-processing

Regarding EEG data, the gradient and pulse ar-tifacts resulting from the simultaneous acquisitionwith fMRI were removed through standard [4] andoptimized [2] correction procedures, respectively.Following this, EEG data was down-sampled to 250Hz and band-pass filtered in the range 1-45 Hz.

Resting-state fMRI datasets were pre-processedusing mainly the FMRIB’s Software Library (FSLv5.0, https://fsl.fmrib.ox.ac.uk/fsl). The main pre-processing steps included: non-brain tissue removalusing FSL’s brain extraction tool (BET, [39]); re-moval of non-neuronal BOLD signal fluctuationsof physiological origin following the procedure re-ported in [18] and [9]; slice timing and motion cor-rection using FSL’s MCFLIRT [22]. Functional im-ages were co-registered into each patient’s struc-tural space and into the MNI space through affinetransformations with 6 and 12 degrees of freedom,respectively, which were applied using FSL’s linearimage registration tool (FLIRT, [22, 23]). Lastly,fMRI data was spatially smoothed using a 5-mmFWHM Gaussian kernel.

2.4. Estimation of dFC features

The estimation of dFC features (schematicallyrepresented in Figure 1) was performed using FSL

and MATLAB R2016b (The MathWorks Inc., Nat-ick, MA, USA) in-house routines. Following brainparcellation into 90 cortical and subcortical re-gions defined by the AAL template [47], represen-tative time-series were obtained by averaging theBOLD signal within each of those Regions of Inter-est (ROIs). For visualization purposes, these re-gions were grouped into 7 broader anatomically-related brain areas: 1) frontal; 2) limbic; 3) oc-cipital; 4) parietal; 5) subcortical; 6) thalamus; 7)temporal. Another atlas (Craddock atlas, [14]) wasseparately used with the goal of obtaining brain par-cellations with different numbers of ROIs. Subse-quently, time-series data was temporally filtered inthe range 0.01-0.1 Hz, which allows the preserva-tion of low-frequency fluctuations of interest, whileremoving slow drifts and high-frequency fluctua-tions of physiological origin [19]. The connectiv-ity was estimated by selecting equal periods of timefrom every ROI’s timecourse (by means of a slidingwindow) and computing the pairwise Pearson cor-relation coefficient. The following sliding windowparameters were applied: window length of 37.5 s(15 TRs), which is within the recommended rangefound in the literature (30 - 60 s) [15, 20, 44], andstep size of 5 s (2 TRs). Considering that the brainhas been parcellated into N regions, the output ofthis analysis was a temporal sequence of N×N sym-metric dFC matrices. The final step in the estima-tion of each subject’s dFC data was the subtractionof the individual static FC matrix, which was com-puted by averaging the sliding window correlationsfor each pair of ROIs over time [28].

2.5. Image labelling

Sleep stages

For both EEG datasets of Patient 2, every 30-

3

T = 5 s

Pre-processed

fMRI data

Brain parcellation ROI time series extraction

Computation of windowed

correlations

W = 37.5 s

Estimation of dFC matrices

dFC data

Figure 1: Simplified pipeline for the estimation of dFC matrices from pre-processed fMRI data.

s segment was attributed one sleep stage label byan experienced neurophysiologist according to [37].Since the sliding window analysis applied a windowlength of 37.5 s, dFC matrices computed over pe-riods of time comprising different sleep stages werelabeled as transition states. In the dataset used forclassification, the number of dFC matrices per classwas the following: 99 for ‘S1’, 161 for ‘S2’, 21 for‘S1 → S2’, and 21 for ‘S2 → S1’.

Epileptic states

The identification of epileptiform activity in eachpatient’s EEG recordings was performed by anexperienced neurophysiologist. Since the windowlength used for dFC estimation could comprise pe-riods free of and characterized by epileptic activity,unique labels had to be based on the intensity ofthat activity rather than simple binary descriptors.

In Patients 1 to 5, a predictor of the ‘density’of epileptic events was obtained following the pro-cedure described in [1]. dFC matrices computedover periods of time with densities above patient-specific averages were assigned the label ‘epilepticstate’. In Patients 6 to 8, no clear epileptic activ-ity was identified on the EEG recordings. Never-theless, Abreu et al. [1] demonstrated that it waspossible to partially recover the epileptic networksof patients who did not exhibit IEDs on their EEGrecordings based on the Phase Synchronization In-dex (PSI) computed in a narrow frequency bandassociated with the occurrence of epileptic activity.Following the rationale in that study, it can be hy-pothesized that peaks in PSI values correspond to

periods of potentially epilepsy-related FC changes,as illustrated in Figure 2. dFC matrices computedover periods of time with PSI values above patient-specific thresholds were assigned the label ‘epilepticstate’.

Figure 2: Illustration of the labeling criterion for onerepresentative patient (Patient 2) on the temporal scaleof the sliding window analysis. The highlighted areasrepresent the periods of time whose dFC matrices wereassigned the ‘epileptic state’ label, based on the IEDs’density threshold. For comparison purposes, the PSIfor the same patient is also depicted.

3. Methods

3.1. Study I. Sleep staging at individual level

Study I was subdivided into two studies: a binaryproblem comprising sleep stages S1 and S2 and amulti-class problem including two additional tran-sition states (S1 → S2 and S2 → S1).

Model evaluation

The input data used for classification consistedof whole-brain dFC matrices. Model evaluation re-sorted to a 5-fold cross-validation (CV) approachwith balanced proportions of classes across folds.Given the pronounced intra-class variability, the

4

Table 2: Description of the dataset used in Study II andStudy III. NE = ‘non-epileptic’; E = ‘epileptic’.

Number of dFC matrices

Patient Total ‘NE’ class ‘E’ class

1 104 97 72 334 179 1553 214 145 694 142 107 355 334 229 1056 214 96 1187 214 139 758 385 178 207

Total 1941 1170 771

original sequence of dFC matrices was randomlyshuffled before data was partitioned into folds toassure representativity of the training data.

Architecture description

The architectures that were evaluated for bothproblems are schematically represented in Figure3. The filter design implemented in Architecture 1was inspired by the CCNN proposed by Meszlenyiet al. [32]. Unlike in natural images, in which simi-lar features are expected to be shared across neigh-bouring pixels, in FC matrices one should seek theconnection between each ROI and every other ROI,hence motivating the application of 1 × N filters.Architecture 2 was adapted from a CNN optimizedto solve a handwritten digit recognition task usingthe benchmark MNIST dataset. The implementa-tion of these two architectures aimed at comparingthe performance of a CNN specifically tailored forFC data with that of a conventional architectureapplied in computer vision tasks. Lastly, a simpleneural network consisting of single fully connectedlayer (Architecture 3) was also implemented. Giventhe symmetry that characterizes the obtained dFCmatrices, the possibility of using only their uppertriangular part as input for classification was inves-tigated by vectorizing the original matrices and ap-plying Architecture 3 with a dimensionally matchedfully connected layer.

Training

The CNN architectures were implementedand trained using the toolbox MatConvNet(http://www.vlfeat.org/matconvnet/) with the fol-lowing hyperparameters: learning rate of 0.001, mo-mentum term of 0.9, weight decay of 0.0005, andbatch size of 10 images. These values correspondto typical choices found in the literature, with theexception of the batch size, whose value was empir-ically set. All architectures were trained using theStochastic Gradient Descent (SGD) method withbackpropagation and the filter weights initializedas small random values (random values between 0and 1 scaled by a factor of 10−2) [41]. The training

process was monitored through the evolution of theobjective function (cross-entropy) and the classifi-cation error over the epochs. The number of epochsrequired for convergence of the method varied withthe architecture.

Control tests

Three control tests were conducted using theCNN architecture with the best performance in theoriginal classification task in order to assess whetherthe results were effectively determined by class-specific FC changes. In ‘Control test 1’, BOLD sig-nal intensity-based features (1×N vectors, where Nis the number of ROIs) were used as input for clas-sification. This test aimed at evaluating whetherthe use of dFC matrices, rather than BOLD sig-nal intensity-based features (which are commonlyapplied in brain decoding studies [7]), was deter-minant for the classification of sleep stages. Thegoal of control tests 2 and 3 was to assess whetherthe classification results were driven by meaning-ful FC changes rather than differences in its meanand variance across states, which was accomplishedthrough the generation of surrogate training databy means of a phase randomization. In ‘Controltest 2’, a Fourier transform was applied to the ROI-averaged BOLD signal timecourses and an indepen-dent random phase sequence was added to each sig-nal’s phase spectrum [20]. In ‘Control test 3’, ananalogous procedure was applied to dFC data byadding an independent random phase sequence tothe phase spectrum of each ROI’s FC timeseries [3].

Comparison with SVMs

A non-linear SVM with a radial basis func-tion (RBF) kernel was applied [46] using theMATLAB interface of the LIBSVM toolbox(https://www.csie.ntu.edu.tw/ cjlin/libsvm/, [11]).The regularization parameter C and the kernel pa-rameter γ were optimized by means of a grid-search in the exponential space comprising all pair-wise combinations of (C, γ) ∈

{2−20, 2−19..., 220

}.

Model selection and evaluation resorted to a nested5-fold CV strategy and the selected multi-class ap-proach was the one-versus-all.

3.2. Study II. Classification of epilepticstates at individual level

Some methodological options from Study I weremaintained in this study, namely data partitioningand model evaluation. Following the results of theprevious study, only the architecture that exhib-ited the best classification performance was appliedand the vectorized upper triangular part of the dFCmatrices was used as input. Regarding the train-ing protocol described for Study I, the only differ-ence was the number of epochs required for conver-gence of the method, which had to be adjusted ina patient-wise manner. The control tests described

5

CONV F = 1X90; N = 64

S = 1; P = 0 Re

LU CONV

F = 1x90; N = 128

S = 1; P = 0 Re

LU FC

F = 1x1; N = 96 Softmax

Re

LU FC

F = 1x1; N = K

CONV F = 7x7; N = 32

S = 1; P = 0 Re

LU Max POOL

F = 2x2

CONV F = 7x7; N = 64

S = 1; P = 0 Re

LU CONV

F = 7x7; N = 64

S = 1; P = 0 Re

LU

FC

F = 6x6;

N = 1024

Dropout

R = 0.4

FC

F = 1x1; N = K Softmax

FC

F = 90x90; N = K Softmax

Architecture 1

Architecture 2

Max POOLF = 2x2

Architecture 3

Max POOLF = 2x2

Figure 3: Schematic representation of the three architectures evaluated in Study I. CONV = Convolutional layer;FC = Fully Connected layer; POOL = Pooling layer; F = filter size; N = number of filters; S = stride; P =padding; R = dropout rate; K = number of classes (2 or 4). The dashed boxes represent layers without trainableweights.

for Study I were also applied in this study.For comparison purposes, a SVM was also im-

plemented following the description provided forStudy I. The only difference was the range of the hy-perparameter optimization, which was restricted toC ∈

{210, 211..., 220

}and γ ∈

{2−20, 2−19..., 2−10

}based on optimal range obtained for Study I.

3.3. Study III. Classification of epilepticstates at group level

3.3.1 Group-level classification

The dataset used in this problem consisted ofa combination of the individual datasets used inStudy II and the same model evaluation approachwas followed. Classification was performed usingthe architecture with the best performance in StudyI and training proceeded as described for that study.The implementation of the SVM followed the de-scription provided for Study II.

3.3.2 Single-subject classification

In this problem, model evaluation was performedthrough a Leave-One-Subject-Out Cross-Validation(LOSO-CV) approach. The first classification strat-egy consisted of applying the architecture with thebest performance in Study I and the same trainingprotocol.

For comparison purposes, a SVM was imple-mented following the approach described for StudyII, with the exception of the model selection andevaluation approach (nested LOSO-CV in thiscase).

Dataset augmentation

Provided the considerable inter-subject variabil-ity (as illustrated by the heterogeneity of clinicalconditions depicted in Table 1), the applicationof dataset augmentation strategies seemed to be apromising approach to achieve a greater coverage

of the epilepsy spectrum and, consequently, an im-proved generalization ability. To the best of ourknowledge, dataset augmentation applied to dFCfeatures has not been covered in the literature yet;therefore, two possible strategies were investigated.Both strategies were applied so that the size of theoriginal training set was scaled by a factor of 5.

The first one was based on the Sample Minor-ity Over-Sampling Technique (SMOTE) [13]. Thefollowing modifications were applied to the originalformulation: apply dataset augmentation to eachpatient’s entire training data (instead of only theminority class); generate synthetic data based onrandom samples from the same class (instead of itsnearest neighbors) to increase the variability intro-duced in the artificial data.

The second dataset augmentation approach im-plemented in this work was the addition of whitenoise with Gaussian distribution to the ROI-averaged BOLD signal timecourses prior to the es-timation of dFC matrices. The standard deviationwas empirically set to 0.7.

Transfer Learning

The increased complexity of the single-subjectclassification problem suggested the need to usedeeper CNN architectures; however, consideringthe relatively reduced size of the available dataset,training such CNN architectures from a randominitialization would potentially result in overfitting[41]. Therefore, a strategy evaluated in the contextof this problem was transfer learning through thefine-tuning of pre-trained networks.

From the pre-trained models available for Mat-ConvNet, the selected one was a version of the VGGnet (VGG-F) [12]. Given the size of the imagesused to pre-train the network, dFC matrices had tobe resized through an interpolation. To reduce po-

6

tentially adverse effects of artificial scaling the dFCmatrices, these were re-computed using a brain par-cellation defined by the Craddock atlas with 200ROIs [14], yielding 200× 200 matrices. Concerningthe training hyperparameters described for Study I,the only difference was the reduction of the learn-ing rate to 1× 10−6 to avoid abrupt changes of thelearned filter weights.

The following versions of the dataset were tested:i) original dataset; ii) augmented dataset using bothabovementioned strategies.

4. Results

4.1. Study I. Sleep staging at individual level

The evolution of the objective function and theclassification error over the training epochs for thethree architectures is presented in Figure 4 for thebinary problem (a similar pattern was observed forthe multi-class problem). The classification resultsobtained for both problems are summarized in Ta-ble 3.

Table 3: Balanced accuracy (%) obtained for the binaryand multi-class problems using the three architectures.A1 = Architecture 1; A2 = Architecture 2; A3 = Ar-chitecture 3.

A1 A2 A3

Binary problem 98.56 97.54 100.00Multi-class problem 83.22 84.28 93.38

Regarding the use of the upper triangular part ofthe dFC matrices as features for classification, theresults were identical to the ones presented in Table3 (for Architecture 3), hence validating the use ofthis data format.

The application of the three control tests resultedin near chance-level accuracy for both problems.

Since Architecture 3 comprises a single fully con-nected layer before the softmax one, it is possibleto directly analyze the weights attributed to eachROI’s FC values at the end of the training pro-cess. The absolute difference of the learned filterweights between the two classes (S1 vs. S2) is pre-sented in Figure 5. In the case of the multi-classproblem, the absolute difference of the learned filterweights between every class and the mean of the re-maining classes was computed. For both problems,the model with the best performance across the CVruns was selected.

The application of SVMs to the binary and multi-class problems resulted in mean balanced accuraciesof 100% and 92.94%, respectively.

4.2. Study II. Classification of epilepticstates at individual level

The classification performance obtained for allpatients using Architecture 3 and SVMs is sum-

marized in Table 4. The application of the threecontrol tests using Architecture 3 resulted in a nearchance-level accuracy for all patients.

Table 4: Balanced accuracy (%) obtained using Archi-tecture 3 and the SVM for all patients.

Patient Architecture 3 SVM

1 96.40 100.002 93.46 91.803 81.91 82.604 89.65 81.955 90.42 89.766 89.25 91.037 87.75 83.758 92.29 93.58

Following the approach described in Study I, theabsolute difference of the learned filter weights be-tween the two classes (‘non-epileptic’ vs. ‘epileptic’)are presented in Figure 6.

4.3. Study III. Classification of epilepticstates at group level

Group-level classification

The application of Architecture 3 to the group-level classification problem resulted in a mean bal-anced accuracy of 88.33%, whereas the applicationof a SVM to the same problem resulted in a meanbalanced accuracy of 89.98%.

Single-subject classification

Despite considerable efforts involving a combi-nation of different dataset augmentation strategiesand transfer learning, the attempts at performinga single-subject classification resulted in significantoverfitting of the training data with subsequent lackof generalization capacity. An identical result wasobtained when applying a SVM to the same datasetconfigurations.

5. Discussion

Classification using CNNs

Regarding Study I, a good classification perfor-mance was obtained for the three architectures,with balanced accuracies above 80%. Architectures1 and 2 had a similar performance for both prob-lems, suggesting that the architecture adapted fromthe CCNN proposed by Meszlenyi and colleagues[32] does not present significant advantages over theuse of conventional ones. However, a straightfor-ward comparison with the mentioned study is notpossible due to: the use of static rather than dFCfeatures, the availability of a dataset obtained froma larger cohort of subjects, and the metrics usedto estimate the FC. Architecture 3 was the onewith the best performance (highest accuracy andfewer training epochs) in both problems. Since thehierarchical application of convolutional layers in

7

Figure 4: Evolution of the objective function and classification error of the binary problem over the epochs (forone illustrative CV run) for the three architectures.

Figure 5: Visualization of the absolute difference be-tween the learned filter weights of the two classes forthe binary problem using the coarser brain parcellation.

CNNs allows the extraction of sequentially higher-level features, the obtained result indicates that nofurther feature extraction is required for the classifi-cation of sleep stages at individual level when usingdFC matrices. This outcome can be justified by thepresence of correlated data in the training and testsets as a result of methodological options regard-ing feature extraction and data partitioning, whichis known to result in optimistic performance esti-mates [33]. However, due to the limited data avail-ability, it was not possible to partition the datasetsuch that training and test data were uncorrelated.When comparing the performance of the binary andmulti-class problems, one can conclude that the ac-curacy obtained for the latter was consistently lowerfor every architecture. This result was expectedconsidering the higher complexity of the discrimi-nation task in the multi-class scenario. RegardingStudy II, a good performance was obtained by ap-plying Architecture 3 to all patients’ datasets, withbalanced accuracies above 80%. A comparison be-tween the classification performance of different pa-tients is not possible due to the heterogeneity oftheir clinical conditions. Lastly, in Study III, the

performance obtained for the group-level classifica-tion is within the range obtained in Study II, aswould be expected since the dataset used in thisproblem consists of a fold-wise combination of thedatasets from all patients.

Control tests

A near chance-level accuracy was obtained whenapplying the three control tests in the binary andmulti-class sleep staging problems, as well in theclassification of epileptic states at individual level.Regarding ‘Control test 1’, these results suggestthat the classification accuracy obtained with dFCmatrices is not driven by stage-specific BOLD per-cent signal changes. Moreover, the results of‘Control test 2’ and ‘Control test 3’ indicate thatthe original performance was effectively driven bymeaningful stage-specific dFC patterns, which de-pend on the preservation of the timing of the BOLDsignal fluctuations and of the temporal evolution ofeach ROI’s FC values.

Interpretation

In the binary problem, the highest difference ofweights was verified for FC changes involving thethalamus. The importance of this region for dis-criminating between sleep stages S1 and S2 is co-herent with the decrease in the FC between thethalamus and neocortical regions during sleep onset(stage S1) and its re-establishment in deeper stagesof NREM sleep (including S2) reported by differ-ent studies (and reviewed in [45]). In the multi-class problem, the differences in the learned filterweights between different classes (data not shown)were diffuse and not directly interpretable.

In Study II, it was possible to establish interest-

8

(a) Patient 1. (b) Patient 2. (c) Patient 3. (d) Patient 4.

(e) Patient 5. (f) Patient 6. (g) Patient 7. (h) Patient 8.

Figure 6: Visualization of the absolute difference of the learned filter weights for the ‘non-epileptic’ and ‘epileptic’classes for all patients using the coarser brain parcellation.

ing relationships between the most discriminativeFC features and each patient’s clinical conditions:in Patient 1, FC changes within subcortical regionsand the involvement of these structures (namely thecaudate nucleus) in the generation of spike-wavedischarges, the hallmarks of CAE; in Patient 2, FCchanges between parietal and temporal regions withsubcortical regions and the patient’s focal patternof CSWS; in Patient 3, FC changes between frontaland parietal regions and the thalamus, consistentwith the synchronization of thalamic neurons bythe patient’s cortical dysplasia; in Patient 5, FCchanges between parietal regions and the thalamusand the connection between the cortical sources ofCSWS and the patient’s thalamic lesion. Regard-ing Patients 6 to 8, it was not possible to establisha clear relationship between the difference in thelearned filter weights between the two classes andthe patients’ clinical conditions. One of the possiblereasons for this finding is the uncertainty inherentto the attribution of labels considering the absenceof clear manifestations of epileptic activity duringdata acquisition.

Comparison with SVMs

In general, the application of SVMs resulted in aclassification accuracy similar to the one obtainedwith CNNs. However, this method presented twodrawbacks in terms of computational cost (due tothe hyperparameter optimization) and direct inter-pretation of the features that contributed the mostto the classification. Although the classificationof sleep stages from dFC features using SVMs hasbeen implemented in previous works, a comparisonwith their results is not possible for two main rea-sons: the availability of larger cohorts of subjects[5, 43, 46] and the extraction of dFC features usingnon-overlapping windows [43, 46]. Regarding theclassification of epileptic states, it is not possible to

perform a comparison with existing literature sinceit had not been explored prior to the present work.

Single-subject classification

In spite of the different strategies implemented inthis part of Study III, it was not possible to achievesatisfactory results for the classification of epilepticstates from unseen patients. The subject-specificnature of dFC data suggested the need to use alter-native strategies targeting the removal of the staticFC. However, the implementation of different ap-proaches (the regression proposed by [50] and thePCA applied in [20]) did not have a positive impacton the results. On the other hand, the applicationof dataset augmentation strategies did not have theexpected effect on the classification results, possi-bly due to the pronounced inter-subject variabil-ity. In fact, the generation of synthetic dFC datafrom one patient may not increase the variability ofthe training data in a way that allows the classifierto perform predictions on new patients with signif-icantly distinct disease profiles. Lastly, the poorclassification results achieved when applying trans-fer learning can also be explained by the reduceddata availability associated with the considerableinter-subject variability.

6. Conclusions and Future Work

One of the main conclusions of this work is thatdeep models are not required when the classifica-tion of brain states using dFC matrices is formu-lated at the individual level. Nonetheless, its ex-tension to single-subject classification problems isexpected to largely benefit from the application ofdeeper models. In the epilepsy scenario, a relevantaccomplishment of the present study was the vali-dation of fMRI-derived dFC matrices as appropri-ate features for the classification of epileptic states.

9

In spite of the overall similar results obtained withCNNs and SVMs, the gap between their perfor-mance is expected to increase in favor of the formerin more complex problems, provided an adequateamount of data is available. Regarding the analysisof the learned filter weights, the generalization ofthis study’s findings is limited by the availability ofdata obtained from larger cohorts of patients withsimilar clinical profiles.

The main limitation of the present work was thesize of the available dataset. However, the recruit-ment of a larger group of subjects with a diagnosisof epilepsy for simultaneous EGG-fMRI acquisitionswould impose several challenges, not only regard-ing the recruitment itself, but also considering thehigh cost of those acquisitions and the technicallydemanding data pre-processing.

Apart from the increased data availability, severalinteresting work directions can be pointed. Focus-ing on epilepsy, the application of data-driven brainparcellation schemes, namely ICA, could be advan-tageous for two reasons: the possibility of describingthe brain states from a network perspective and theattenuation of the inter-subject variability presentin the dFC data arising from anatomic differences inthe epileptic networks. Another strategy with po-tential impact on a single-subject classification sce-nario would be the generation of artificial data com-bining datasets from different patients, while assur-ing the clinical significance of the synthetic data.

References[1] R. Abreu, A. Leal, F. Lopes da Silva, and

P. Figueiredo. EEG synchronization measures pre-dict epilepsy-related BOLD-fMRI fluctuations bet-ter than commonly used univariate metrics. Clini-cal Neurophysiology, 129(3):618–635, 2018.

[2] R. Abreu, M. Leite, J. Jorge, F. Grouiller,W. van der Zwaag, A. Leal, and P. Figueiredo.Ballistocardiogram artifact correction taking intoaccount physiological signal preservation in simul-taneous EEG-fMRI. NeuroImage, 135:45–63, 2016.

[3] E. A. Allen, E. Damaraju, S. M. Plis, E. B. Er-hardt, T. Eichele, and V. D. Calhoun. Trackingwhole-brain connectivity dynamics in the restingstate. Cerebral Cortex, 24(3):663–676, 2014.

[4] P. J. Allen, O. Josephs, and R. Turner. Amethod for removing imaging artifact from contin-uous EEG recorded during functional MRI. Neu-roImage, 12(2):230–239, 2000.

[5] A. Altmann, M. S. Schroter, V. I. Spoormaker,S. A. Kiem, D. Jordan, R. Ilg, E. T. Bullmore,M. D. Greicius, M. Czisch, and P. G. Samann.Validation of non-REM sleep stage decoding fromresting state fMRI using linear support vector ma-chines. NeuroImage, 125:544–555, 2016.

[6] M. R. Arbabshirani, S. Plis, J. Sui, and V. D. Cal-houn. Single subject prediction of brain disordersin neuroimaging : Promises and pitfalls. NeuroIm-age, 145:137–165, 2017.

[7] C. Cabral, M. Silveira, and P. Figueiredo. Decod-ing visual brain states from fMRI using an ensem-ble of classifiers. Pattern Recognition, 45(6), 2012.

[8] M. Cataldi, M. Avoli, and E. de Villers-Sidani.Resting state networks in temporal lobe epilepsy.Epilepsia, 54(12):2048–2059, 2013.

[9] C. Chang, J. P. Cunningham, and G. H. Glover.Influence of heart rate on the BOLD signal: Thecardiac response function. NeuroImage, 44(3):857–869, 2009.

[10] C. Chang and G. H. Glover. Time-frequency dy-namics of resting-state brain connectivity mea-sured with fMRI. NeuroImage, 50(1):81–98, 2010.

[11] C.-C. Chang and C.-J. Lin. LIBSVM: a libraryfor support vector machines. ACM Transactionson Intelligent Systems and Technology, 3(2):21–27,2011.

[12] K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zis-serman. Return of the devil in the details: Delvingdeep into convolutional nets. In British MachineVision Conference, 2014.

[13] N. V. Chawla, K. W. Bowyer, L. O. Hall, andW. P. Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelli-gence research, 16:321–357, 2002.

[14] R. C. Craddock, G. A. James, P. E. Holtzheimer,X. P. Hu, and H. S. Mayberg. A whole brain fMRIatlas generated via spatially constrained spectralclustering. Human Brain Mapping, 33(8):1914–1928, 2012.

[15] L. Deng, J. Sun, L. Cheng, and S. Tong. Char-acterizing dynamic local functional connectivity inthe human brain. Scientific Reports, 6:1–13, 2016.

[16] Y. Du, Z. Fu, and V. D. Calhoun. Classificationand Prediction of Brain Disorders Using FunctionalConnectivity: Promising but Challenging. Fron-tiers in Neuroscience, 12, 2018.

[17] R. S. Fisher, H. E. Scharfman, and M. DeCurtis.How can we identify ictal and interictal abnormalactivity? Advances in Experimental Medicine andBiology, 813:3–23, 2014.

[18] G. H. Glover, T. Q. Li, and D. Ress. Image-basedmethod for retrospective correction of physiologicalmotion effects in fMRI: RETROICOR. MagneticResonance in Medicine, 44(1), 2000.

[19] J. Gonzalez-Castillo and P. A. Bandettini. Task-based dynamic functional connectivity: Recentfindings and open questions. NeuroImage, 180:526–533, 2018.

[20] J. Gonzalez-Castillo, C. W. Hoy, D. A. Handw-erker, M. E. Robinson, L. C. Buchanan, Z. S. Saad,and P. A. Bandettini. Tracking ongoing cognitionin individuals using brief, whole-brain functionalconnectivity patterns. Proceedings of the NationalAcademy of Sciences, 112(28):8762–8767, 2015.

[21] A. Haimovici, E. Tagliazucchi, P. Balenzuela, andH. Laufs. On wakefulness fluctuations as a source ofBOLD functional connectivity dynamics. ScientificReports, 7(1):1–13, 2017.

[22] M. Jenkinson, P. Bannister, M. Brady, andS. Smith. Improved optimization for the robust and

10

accurate linear registration and motion correctionof brain images. NeuroImage, 2002.

[23] M. Jenkinson and S. Smith. A global optimisationmethod for robust affine registration of brain im-ages. Medical Image Analysis, 5(2):143–156, 2001.

[24] J. Kawahara, C. J. Brown, S. P. Miller, B. G.Booth, V. Chau, R. E. Grunau, J. G. Zwicker,and G. Hamarneh. BrainNetCNN: Convolutionalneural networks for brain networks; towards pre-dicting neurodevelopment. NeuroImage, 146:1038–1049, 2017.

[25] H. Laufs. Functional imaging of seizures andepilepsy: evolution from zones to networks. Cur-rent opinion in neurology, 25(2):194–200, 2012.

[26] H. Laufs, R. Rodionov, R. Thornton, J. S. Duncan,L. Lemieux, and E. Tagliazucchi. Altered fMRIconnectivity dynamics in temporal lobe epilepsymight explain seizure semiology. Frontiers in Neu-rology, 5:1–13, 2014.

[27] Y. Lecun, Y. Bengio, and G. Hinton. Deep learn-ing. Nature, 521(7553):436–444, 2015.

[28] N. Leonardi, W. R. Shirer, M. D. Greicius, andD. Van De Ville. Disentangling dynamic networks:Separated and joint expressions of functional con-nectivity patterns in time. Human Brain Mapping,35(12):5984–5995, 2014.

[29] R. Lopes, F. Moeller, P. Besson, F. Ogez,W. Szurhaj, X. Leclerc, M. Siniatchkin,M. Chipaux, P. Derambure, and L. Tyvaert.Study on the relationships between intrinsic func-tional connectivity of the default mode networkand transient epileptic activity. Frontiers inNeurology, 5:1–16, 2014.

[30] F. Lopes da Silva, W. Blanes, S. N. Kalitzin,J. Parra, P. Suffczynski, and D. N. Velis. Epilepsiesas dynamical diseases of brain systems: basic mod-els of the transition between normal and epilepticactivity. Epilepsia, 44(1):72–83, 2003.

[31] C. Luo, Q. Li, Y. Lai, Y. Xia, Y. Qin, W. Liao,S. Li, D. Zhou, D. Yao, and Q. Gong. Alteredfunctional connectivity in default mode network inabsence epilepsy: A resting-state fMRI study. Hu-man Brain Mapping, 32(3):438–449, 2011.

[32] R. J. Meszlenyi, K. Buza, and Z. Vidnyanszky.Resting State fMRI Functional Connectivity-BasedClassification Using a Convolutional Neural Net-work Architecture. Frontiers in Neuroinformatics,11:1–12, 2017.

[33] F. Pereira, T. Mitchell, and M. Botvinick. Machinelearning classifiers and fMRI: A tutorial overview.NeuroImage, 45:199–209, 2009.

[34] D. Picchioni, J. H. Duyn, and S. G. Horovitz.Sleep and the functional connectome. NeuroImage,80:387–396, 2013.

[35] R. Poldrack, J. Mumford, and T. Nichols. Hand-book of Functional MRI Data Analysis. CambridgeUniversity Press, 2011.

[36] M. G. Preti, T. A. Bolton, and D. Van De Ville.The dynamic functional connectome: State-of-the-art and perspectives. NeuroImage, 160:41–54,2017.

[37] A. Rechtschaffen. A manual for standardized ter-minology, techniques and scoring system for sleepstages in human subjects. Brain Research Institute.US Department of Health, Education, and Welfare,1968.

[38] D. Schomer and F. da Silva. Niedermeyer’s Elec-troencephalography: Basic Principles, Clinical Ap-plications, and Related Fields. Wolters KluwerHealth, 6th edition, 2012.

[39] S. M. Smith. Fast robust automated brain extrac-tion. Human Brain Mapping, 17(3):143–155, 2002.

[40] S. M. Smith, P. T. Fox, K. L. Miller, D. C.Glahn, P. M. Fox, C. E. Mackay, N. Filippini,K. E. Watkins, R. Toro, A. R. Laird, and C. F.Beckmann. Correspondence of the brain’s func-tional architecture during activation and rest. Pro-ceedings of the National Academy of Sciences,106(31):13040–13045, 2009.

[41] Stanford University. cs231n Convolutional NeuralNetworks for Visual Recognition. http://cs231.

github.io, 2018. Accessed: Feb. 18, 2018.[42] H. Stefan and F. H. L. da Silva. Epileptic neu-

ronal networks: Methods of identification and clin-ical relevance. Frontiers in Neurology, 4:1–15, 2013.

[43] E. Tagliazucchi and H. Laufs. Decoding Wakeful-ness Levels from Typical fMRI Resting-State DataReveals Reliable Drifts between Wakefulness andSleep. Neuron, 82(3):695–708, 2014.

[44] E. Tagliazucchi and H. Laufs. Multimodal imagingof dynamic functional connectivity. Frontiers inNeurology, 6:1–9, 2015.

[45] E. Tagliazucchi and E. J. van Someren. The large-scale functional connectivity correlates of con-sciousness and arousal during the healthy andpathological human sleep cycle. NeuroImage,160:55–72, 2017.

[46] E. Tagliazucchi, F. von Wegner, A. Morzelewski,S. Borisov, K. Jahnke, and H. Laufs. Automaticsleep staging using fMRI functional connectivitydata. NeuroImage, 63(1):63–72, 2012.

[47] N. Tzourio-Mazoyer, B. Landeau, D. Papathanas-siou, F. Crivello, O. Etard, N. Delcroix, B. Ma-zoyer, and M. Joliot. Automated anatomical la-beling of activations in SPM using a macroscopicanatomical parcellation of the MNI MRI single-subject brain. NeuroImage, 15(1):273–289, 2002.

[48] P. van Mierlo, M. Papadopoulou, E. Carrette,P. Boon, S. Vandenberghe, K. Vonck, and D. Mari-nazzo. Functional brain connectivity from EEG inepilepsy: Seizure prediction and epileptogenic fo-cus localization. Progress in Neurobiology, 121:19–35, 2014.

[49] S. Vieira, W. H. Pinaya, and A. Mechelli. Us-ing deep learning to investigate the neuroimag-ing correlates of psychiatric and neurological dis-orders: Methods and applications. Neuroscienceand Biobehavioral Reviews, 74:58–75, 2017.

[50] H. Xie, V. D. Calhoun, J. Gonzalez-Castillo,E. Damaraju, R. Miller, P. A. Bandettini, andS. Mitra. Whole-brain connectivity dynamics re-flect both task-specific and individual-specific mod-ulation: A multitask study. NeuroImage, 2017.

11