HAL Id: tel-00426852 https://tel.archives-ouvertes.fr/tel-00426852 Submitted on 28 Oct 2009 HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés. Mapping, timing and tracking cortical activations with MEG and EEG: Methods and application to human vision Alexandre Gramfort To cite this version: Alexandre Gramfort. Mapping, timing and tracking cortical activations with MEG and EEG: Meth- ods and application to human vision. Modeling and Simulation. Ecole nationale supérieure des telecommunications - ENST, 2009. English. tel-00426852
Transcript
HAL Id: tel-00426852https://tel.archives-ouvertes.fr/tel-00426852
Submitted on 28 Oct 2009
HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.
Mapping, timing and tracking cortical activations withMEG and EEG: Methods and application to human
visionAlexandre Gramfort
To cite this version:Alexandre Gramfort. Mapping, timing and tracking cortical activations with MEG and EEG: Meth-ods and application to human vision. Modeling and Simulation. Ecole nationale supérieure destelecommunications - ENST, 2009. English. tel-00426852
4.5 Running an inverse solver with two priors (one non differentiable and an ℓ2term) using proximity operators with EMBAL . . . . . . . . . . . . . . . . . . . . 129
4.6 Runing sparse inverse modeling with temporal data using proximity operators
40 % each column of lf_openmeeg is the forward field of a dipole
41 % in one direction of the coordinate system
Table 2.13: Computing an EEG leadfield with Fieldtrip and OpenMEEG.
88 CHAPTER 2. THE FORWARD PROBLEM
2.7 CONCLUSION
Brain functional imaging with M/EEG requires an efficient and accurate forward model.
In this section, we have presented the general framework to achieve good forward modeling.
It implied to introduce the physics with Maxwell equations, to make a set of hypothesis like
the quasi-static approximation and to use simplified head models. We have presented the
theory and discussed some implementation details before finally demonstrating that the for-
ward modeling that we contributed to develop and promote in the M/EEG community is the
most accurate BEM solver available.
Before closing this chapter, we would like to mention recent and promising work on for-
ward modeling with anisotropic conductivity models [170], i.e., models where the conductivity
can vary inside the same tissue. This method avoids the meshing step required by the above
FEM and BEM methods and provides quite accurate solutions in a very reasonable amount
of time.
We will now move on to the other fundamental aspect of brain functional imaging with
MEG: the inverse problem.
CHAPTER 3
THE INVERSE PROBLEM WITH
DISTRIBUTED SOURCE MODELS
In chapter 2, we have seen how the tiny electromagnetic fields produced by the neural activity
is modeled to understand what is measured with M/EEG devices. This aspect of the process-
ing of M/EEG data is called the forward problem. Its counterpart is the inverse problem. The
inverse problem of M/EEG is the procedure that consists in recovering the distribution of the
neural generators that have produced the measurements. Three main types of approaches
exist to solve this problem:
1. The parametric models usually referred to as dipole fitting approaches.
2. The beamforming or scanning techniques.
3. The image-based methods with distributed source models.
In this chapter, the first two approaches are briefly explained and commented. The last
one, the image-based method, is presented in more detail. The M/EEG inverse problem is pre-
sented in a classical framework where the solution is penalized with a regularization prior.
Standard priors are based on ℓ2 norms, leading to differentiable optimization problems and
closed-form solutions. In this chapter, focus is put on such priors, going from the simple
“Minimum-Norm” (MN) approach to spatiotemporal solvers and learning-based methods re-
cently presented in the literature.
This chapter covers the methodological aspects of multiple inverse solvers based on ℓ2priors. It presents for each of them the hypotheses made and the eventual limitations when
used with M/EEG data. The optimization strategies employed will also be commented and
discussed. For experimental results and simulation studies with the reviewed solvers, we
refer the reader to the original papers. We also provide the code snippets to use these solvers
using EMBAL , an open source toolbox that we wrote during this thesis.
130 CHAPTER 4. INVERSE MODELING WITH SPARSE PRIORS
the additive noise at the sensor level, the ℓ1 and ℓ0 priors create a bias for very focal source
distributions. To tackle this limitation, the Total Variation (TV) prior can be used.
The TV prior penalizes the solution with the ℓ1 norm of the gradient, in the present case
the surface gradient: TV(x) = ‖∇surfx‖1. It is a seminorm, since it can be equal to 0 even if x
is non-zero. Assuming x corresponds to a discretization with P1 elements on the tessellation,
∇surfx is a constant vector of R3 on each triangle. Let us denote it (∇x
px,∇ypx,∇z
px) ∈ R3,
where p indexes triangles. More details on how to compute the surface gradient with a P1
discretization can be found in [4]. With these notations, TV(x) can be written:
TV(x) =∑
p
√
(∇xpx)2 + (∇y
px)2 + (∇zpx)2
It can be proved that TV(x) is equal to the sum of the lengths of the isolevels of x [143]:
TV(x) =
∫
R
length(Ct)dt ,
where Ct = r ∈ mesh s.t. x(r) = t is the isolevel, or levelset, at level t. The consequence
is that penalizing the inverse problem with the Total Variation tends to produce piecewise
constant reconstructions with regular borders.
The TV norm has been widely used for image deconvolution and restoration [12, 30, 31].
It is related in functional analysis to the space of functions with bounded variations. For the
M/EEG inverse problem the TV prior was originally proposed in [5]. Here, we argue that the
TV regularization can be casted into the general framework of M/EEG inverse solvers with
sparsity inducing priors and that the iterative optimization schemes detailed above for the
standard ℓ1 norm can be directly adapted to invert with a TV prior.
The optimization problem considered is given by:
x∗ = arg minx
1
2‖m−Gx‖2F + λTV (x), λ > 0 . (4.13)
The TV is based on a ℓ1 norm, it is convex and lower semicontinuous. Hence, assuming
one knows how to compute the proximity operator associated to the TV norm, the Forward-
Backward iterations and Nesterov schemes can be used to optimize (4.13).
The proximity operator proxλ‖ ‖T Vcorresponds to following problem:
x∗ = arg minx
1
2‖y − x‖2F + λTV (x), λ > 0 (4.14)
This problem is known in the literature as the ROF (Rudin, Osher and Fatemi) problem
[189]. Various solvers have been proposed in the literature to solve this problem [31, 32].
Here, we detail the dual approach from Chambolle [30]. It consists in using a gradient-based
algorithm for solving the dual problem that interestingly is a smooth optimization problem
over a convex set.
The duality between the ℓ1 norm and the ℓ∞ norm reads
TV (x) = ‖∇x‖1= max
‖z‖∞≤1〈∇x, z〉 . (4.15)
We also need the adjoint relation between the gradient and the divergence operator:
〈∇x,y〉 = −〈x,div y〉 (4.16)
131
Minimization in equation (4.14) becomes:
minx
(
1
2‖y − x‖22 + λTV (x)
)
=λminx
(
1
2λ‖y − x‖22 + max
‖z‖∞≤1〈∇x, z〉
)
=λ max‖z‖∞≤1
(
minx
(
1
2λ‖y − x‖22 + 〈∇x, z〉
))
=λ max‖z‖∞≤1
(
minx
(
1
2λ‖y − x‖22 − 〈x,div z〉
))
(4.17)
The computation of the minimum and the maximum above can be exchanged because the
optimization over x is convex and the optimization over z is concave (see for example [188]).
By setting the derivative with respect to x to zero one gets:
x∗ = y + λ div z
Replacing x in previous expression leads to:
minx
1
2‖y − x‖22 + λTV (x)
=λ max‖z‖∞≤1
λ
2‖div z‖22 − 〈y,div z〉 − λ‖div z‖22
=λ max‖z‖∞≤1
−λ2‖div z‖22 − 〈y,div z〉
=− λ min‖z‖∞≤1
λ
2‖div z‖22 + 〈y,div z〉
=− 1
2min
‖z‖∞≤1λ2‖div z‖22 + 2λ〈y,div z〉
=− 1
2min
‖z‖∞≤1‖λ div z + y‖22 − ‖y‖22
=− λ2
2min
‖z‖∞≤1‖div z +
y
λ‖22 −
1
λ2‖y‖22
(4.18)
Hence z∗ is obtained by:
z∗ = arg min‖z‖∞≤1
‖div z +y
λ‖22
This provides the result from Chambolle [30].
This constrained problem can be solved by a projected gradient algorithm. The gradient
with respect to z is given by −∇(div z + y/λ) which gives the following iterative algorithm to
solve the ROF problem:
xn = y + λ div zn
zn+1i =
zni + (τ/λ)(∇xn)i
max(1, |zni + (τ/λ)(∇xn)i|)
(4.19)
Where τ is the gradient step. In order to guarantee the algorithm convergence one needs to
have:
τ ≤ 2
|||div∇|||where |||div∇||| stands for the spectral norm of the operator.
Note that Chambolle proposes an alternative strategy based on a fixed point method to
132 CHAPTER 4. INVERSE MODELING WITH SPARSE PRIORS
solve this constrained problem, but, our experience is that the fixed-point method does not
actually provide faster convergence rates than the simple projected gradient method. Like in
any standard gradient descent, the projected gradient just detailed can be improved with a
multistep approach [1].
The ROF problem for which we just described an optimization scheme corresponds to the
proximity operator associated with the Total Variation penalization. The real solution can
then be obtained using forward-backward proximal iterations (algorithm 4.3) or Nesterov
iterations (algorithm 4.4).
To our knowledge, Nesterov schemes have never been applied to the inverse problem of
M/EEG, in particular with a surface TV prior. In [4], the optimization procedure used suf-
fered from a very slow convergence rate. In this work, Adde et al. implemented the forward-
backward algorithm 4.3 proposed for TV optimization in [12]. Due to its slow convergence
rate, the author suggested to use a fixed step gradient scheme that requires to make the TV
prior differentiable by replacing the TV by:
TV (x) =∑
p
√
(∇xpx)2 + (∇y
px)2 + (∇zpx)2 + ǫ, ǫ > 0 .
Unfortunately, this scheme does not reach an optimal solution. However, in practice, it pro-
vides visually acceptable results in a small amount of time. The Nesterov scheme that we
propose here is a principled algorithm to solve (4.13) with guarantees of speed and optimal-
ity.
Figure 4.4 presents a result obtained with a TV prior with constrained orientations.
(a) Synthetic active region. (b) Reconstruction result obtained with a TV prior.
Figure 4.4: Simulation result using a TV prior. (a) The synthetic active region used to sim-
ulate MEG measurements. The measurements where corrupted with a small additive Gaus-
sian white noise (SNR=10). The activation pattern was designed to be piecewise constant
which is what is adapted for TV minimization. (b) The reconstruction result obtained by solv-
ing the inverse problem with a TV prior. The solution presents a clear “hot spot” at the correct
location and sets to 0 the majority of the remaining cortical surface.
The TV prior offers a principled to way to cope with spatially extended activations. How-
ever good care should be taken when applied with M/EEG data. Our experience shows that
when applied with too big values of λ, the TV prior tends to move the active regions towards
“flat” cortical regions. The complexe shape of the cortical mantle near the active region may
therefore in practice lead to localization errors. Also, to our knowledge, the bias towards
superficial sources has never been tackled with a TV prior. And finally, the case of dipolar
source spaces with unconstrained orientations has to our knowledge never been treated.
We are now done with the presentation of image-based inverse solvers that work on an
133
instant-by-instant basis. We will know present solvers that make use of the temporal infor-
mation in the data.
4.4 SPARSITY AND SPATIOTEMPORAL DATA
The reason for using temporal information in the inverse problem seems relatively
natural. Two neighboring time instants carry a very similar information since the under-
lying physiological phenomena have a low frequency compared to the sampling rate of the
recordings. The noise that corrupts the measurements is also highly correlated in time. And
our knowledge about physiology favors a vision of the sources as a set of sources, of limited
number, whose activity is stable in time.
4.4.1 VESTAL
The VESTAL method [108] addresses the main critic that is made about MCE. When using
MCE during a small time window, the set of active dipoles can vary significantly between
two neighboring instants, although one would expect a very similar current distribution. The
MCE solver leads to “spiky” estimated time courses.
In order to fix this problem, the VESTAL solver proposes to run an ℓ1 inverse problem at
each time instant, like MCE, but projects the sample-wise l1-norm estimates into the signal
subspace defined by a set of temporal basis functions. Let us write the SVD of the measure-
ments, M = USVT . The first columns of V are used as temporal basis functions.
The critiques that can be made about the VESTAL solver are the following. First, the
optimization scheme proposed in [108] can be largely improved, using the LARS algorithm
for example. Second, the authors do not clearly state what cost function is actually optimized
by their procedure. The following algorithm proposes a much better approach with the same
objective of mixing time and space with a sparsity inducing prior.
4.4.2 ℓ1 over space and ℓ2 over time
An ℓ1 prior brings sparsity, i.e., a limited number of active sources, while an ℓ2 prior brings
what we call diversity, i.e., no zero coefficients. The ℓ2 norm spreads the energy over all the
sources and therefore brings smoothness. By using an ℓ1 prior over space while keeping an
ℓ2 prior over time, the problem of “spiky” activation times series faced by the MCE method is
addressed. We recall that the MCE solver runs on each time instant independently with an
ℓ1 prior.
The approach described above consists in penalizing the inverse problem with a mixed
norm given by:
‖X‖21 =∑
i
√
∑
t
x2it , (4.20)
where i indexes space and t indexes time. One will denote this norm ℓ21.
It can also be modified to take into account some weighting coefficients:
‖X‖w;21 =∑
i
√
∑
t
wix2it (4.21)
in order to reduce the bias for superficial sources.
134 CHAPTER 4. INVERSE MODELING WITH SPARSE PRIORS
The optimization problem becomes:
X∗ = arg minX
1
2‖M−GX‖22 + λ‖X‖w;21, λ > 0 . (4.22)
This norm introduces an ℓ1 norm at a group level. Within a group the coefficients are
compared using an ℓ2 norm. In statistics and machine learning, it is known as the Group-
LASSO problem[235]. We also refer the reader to [141] for more details on mixed norms in
the signal processing context.
In the M/EEG community this norm was recently proposed in this exact form by Ou et
al. in [166]. Prior to this work, [74] proposed another grouping strategy this time between
orientations. In [166], the authors detail how to group both time and orientations. If the coef-
ficients of X are indexed by the position i, the orientation r, and the time t, the corresponding
prior is given:
‖X‖21 =∑
i
√
∑
r
∑
t
x2irt .
Integrating the orientations in the prior does not add any difficulty in the optimization.
In order to speed up the computation, it is proposed in [166] to reduce the rank of the
matrix M with an SVD and to work on the SVD components rather than on the raw temporal
data. When considering K SVD components, the size of the matrix to invert is dm ×K rather
than dm × dt. Due to the ℓ2 norm within the groups, this is perfectly justified. If we use all
the temporal components of the SVD, we project the data using a basis of orthogonal vectors
and the temporal ℓ2 norm is not changed.
However, in [166], the authors use an interior point methods to solve the optimization
problem. Interior point method, also referred to as barrier methods, are a certain class of
algorithms to solve linear and nonlinear convex optimization problems. They guarantee the
optimality of the solution in polynomial time but do not scale very well to large problems, i.e.,
source spaces with a high number of dipoles and a lot of time instants. The complexity of the
algorithm proposed in [166] is in fact cubic in the number of variables.
As for the ℓ1 problem, various methods exist to compute the inverse problem with an ℓ21norm. The particular method that does not apply is the LARS. One can use a coordinate
descent algorithm [77], an IRLS solver similar to the one proposed for the LASSO or an
iterative method based on proximity operators similar to what has been presented for the ℓ1and TV priors [55, 224].
To apply the forward-backward iterations or the optimal scheme from Nesterov, one needs
to compute the proximity operator associated to a ‖ · ‖21 prior.
Let us denote by Xi the ith row of X. By definition, the proximity operator associated to
the ℓ21 norm is given:
X∗ = proxλ‖ ‖21(Y)
= arg minX
1
2‖Y −X‖2F + λ‖X‖21
(4.23)
The solution is obtained using the following proposition:
Proposition 4.4 (Group-LASSO proximity operator). The solution X∗ = proxλ‖ ‖21(Y) of the
proximity operator associated to the Group-LASSO is given group by group (here row by row)
by:
Xi∗ = Yi
(
1− λ
‖Yi‖2
)+
. (4.24)
Using the algorithms 4.4 and 4.3 detailed for the ℓ1 prior, one can compute the optimiza-
tion in (4.22). The only difference comes from the modification of the proximity operator. If
order to run this algorithm with EMBAL , the code snippet in table 4.6 can be used.
A ℓ1 prior is set over the index k corresponding to the condition, while an ℓ2 prior is used
over space and time. By doing so, each dipole has an incentive to explain a small number
of conditions. The conditions are not supposed to change during the time window. Note that
‖X‖w;222 = ‖X‖w;F and that if dk = 1, i.e., only one condition, ‖X‖w;212 = ‖X‖w;F . This means
that in the case where only one condition is considered, the ℓw;212 solution corresponds to the
widely used MN and WMN (see section 3.2).
Here also, solving (4.25) is based on the computation of the proximity operator associated
to the ℓw;212 norm.
The proximity operator associated with the mixed norm ‖.‖2w;212 is analytically given by
the following proposition. We denote yi,k,• = (yi,k,1, yi,k,2, . . . , yi,k,T ).
Proposition 4.5. Let y ∈ Cdxdkdt be indexed by a triple index (i, k, t). Let w a sequence of
strictly positive weights such that ∀t, wi,k,t = wi,k. Let ri,k be defined as ri,kdef= ‖yi,k,•‖w,2/wi,k
, where ‖yi,k,•‖w,2def=√
wi,k
∑
t |yi,k,t|2. For each i, let the indexing denoted by k′i be defined
such that ∀k′i, ri,k′i+1 ≤ ri,k′
i. Let the index Ki and the quantity Kwi
def=∑Ki
ki=1 wi,kibe defined
such that
λ
Ki∑
k′i=1
wi,k′i
(
ri,k′i− ri,Ki
)
< ri,Ki≤ λ
Ki+1∑
k′i=1
wi,k′i
(
ri,k′i− ri,Ki
)
.
Then the solution z = proxλ2 ‖.‖2
w;212(y) is given for each coordinate (i, k, t) by
zi,k,t = yi,k,t
1− λ√wi,k
1 + λKwi
Ki∑
k′i=1
‖yi,k′i,•‖w,2
‖yi,k,•‖2
+
.
Proof. To simplify the notations in the demonstration, we remove the 12 in the proximity
137
operator. We address here the equivalent problem:
x∗ = arg minx
‖y − x‖22 + λ‖x‖2w;212 , (4.26)
with
‖x‖2w;212 =∑
i
∑
j
(
∑
k
wi,j |xi,j,k|2)1/2
2
.
To simplify the notations we write ‖yi,k,•‖2 and ‖yi,k,•‖w,2 respectively as ‖yi,k‖2 and ‖yi,k‖w,2.
Let us derive the functional in (4.26) with respect to xi,j,k. It leads to the following system
of variational equations:
|xi,j,k| = |yi,j,k| − λ√wi,j |xi,j,k|‖xi,j‖−1
2 ‖xi‖w;21
arg(xi,j,k) = arg(yi,j,k)
which gives:
|xi,j,k|(
1 + λ√wi,j‖xi,j‖−1
2 ‖xi‖w;21
)
= |yi,j,k| (4.27)
⇒ |xi,j,k|2(
1 + λ√wi,j‖xi,j‖−1
2 ‖xi‖w;21
)2= |yi,j,k|2
By summing over k, we get:
‖xi,j‖2(
1 + λ√wi,j‖xi,j‖−1
2 ‖xi‖w;21
)
= ‖yi,j‖2⇒ ‖xi,j‖2 + λ
√wi,j‖xi‖w;21 = ‖yi,j‖2 . (4.28)
We have that:
‖xi‖w;21 =∑
l/‖xi,l‖2>0
√wi,l‖xi,l‖2 .
which implies that:
‖xi,j‖2 = ‖yi,j‖2 − λ√wi,j
∑
k/‖xi,k‖2>0
√wi,k‖xi,k‖2 . (4.29)
Using (4.28), we have, if j and k satisfy ‖xi,j‖2 > 0 and ‖xi,k‖2 > 0, that:
‖xi,k‖2√wi,k
=‖xi,j‖2√wi,j
+‖yi,k‖2√wi,k
− ‖yi,j‖2√wi,j
.
By injecting it in (4.29) we get:
‖xi,j‖2 = ‖yi,j‖2 − λ√wi,j
∑
k/‖xi,k‖2>0
wi,k
(‖xij‖2√wi,j
+‖yi,k‖2√wi,k
− ‖yi,j‖2√wi,j
)
⇔ ‖xi,j‖2 = ‖yi,j‖2 − λKwi‖xi,j‖2 + λKwi
‖yi,j‖2 − λ√wi,j
∑
k/‖xi,k‖2>0
√wi,k‖yi,k‖2
⇔ ‖xi,j‖2 = ‖yi,j‖2 −λ√wi,j
1 + λKwi
∑
k/‖xi,k‖2>0
‖yi,k‖w;2
,
where Kwi=∑
k/‖xi,k‖2>0 wi,k. This provides the solution for ‖xi,j‖2 > 0. When
‖yi,j‖2 −λ√wi,j
1 + λKwi
∑
k/‖xi,k‖2>0
‖yi,k‖w;2 ≤ 0
138 CHAPTER 4. INVERSE MODELING WITH SPARSE PRIORS
it implies that ‖xi,j‖2 = 0.
We therefore have for all i:
‖xi,j‖2 =
‖yi,j‖2 −λ√wi,j
1 + λKwi
∑
k/‖xi,k‖2>0
‖yi,k‖w;2
+
.
Let us introduce the indexing k′i such that ‖xi,k′i+1‖2 ≤ ‖xi,k′
i‖2 and the index Ki such that
‖xi,Ki‖2 > 0 and ‖xi,Ki+1‖2 = 0. After reordering, we have that:
‖yi,Ki‖2 −
λ√wi,Ki
1 + λKwi
Ki∑
k′i=1
‖yi,k′i‖w;2 > 0 ,
which leads to:
λ
Ki∑
k′i=1
wi,k′i
(
ri,k′i− ri,Ki
)
< ri,Ki.
This provides the reordering in the proposition and the equation:
‖xi,j‖2 =
‖yi,j‖2 −λ√wi,j
1 + λKwi
Ki∑
k′i=1
‖yi,k′i‖w,2
+
. (4.30)
Let us rewrite (4.27):
|xi,j,k| =|yi,j,k|
1 + λ√wi,j‖xi,j‖−1
2 ‖xi‖w;21
=|yi,j,k|‖xi,j‖2
‖xi,j‖2 + λ√wi,j‖xi‖w;21
Using (4.28), we get:
|xi,j,k| =|yi,j,k|‖xi,j‖2‖yi,j‖2
.
By injecting the result in (4.30) in this equation we get:
|x∗i,j,k| =|yi,j,k|
(
‖yi,j‖2 − λ√
wi,j
1+λKwi
∑Ki
j=1 ‖yi,j‖w,2
)+
‖yi,j‖2
= |yi,j,k|(
1− λ√wi,j
1 + λKwi
∑Ki
j=1 ‖yi,j‖w,2
‖yi,j‖2
)+
.
Remarks.
1. If T = 1, then proxλ‖.‖2w;212
(y) = proxλ‖.‖2√w;12
(y) which corresponds to the Elitist-Lasso
problem [129].
2. This proposition also provides the proximity operator for the ℓw;21 norm introduced in
section 4.4.2.
3. The proximity operator is known analytically. It is simply a shrinkage operator after a
sorting operation. It implies that the solution is exact and relatively fast to compute.
139
Columns (G·i)i of M/EEG forward operators are not normalized. The closer is the dipole i
from the head surface, the bigger is ‖G·i‖2. This implies that a naive inverse procedure would
favor dipoles close to the head surface. Using a weighted norm is an alternative to cope with
this problem. With the mixed norm ‖.‖w,212, it is done by setting wi,k = wi = ‖G·i‖2.
ℓw;212 with unconstrained orientations
Up to here, the ℓw;212 prior was presented assuming a source space with constrained orien-
tations. It is however easy to integrate into the framework the unconstrained case. Let us
denote dr the number of oriented dipoles set at each brain location. The number of brain loca-
tions is denoted dx. Let X ∈ Rdxdr×dkdt have its elements now indexed by (i, r, k, t), i indexes
the position, r the orientation, k the condition and t the time. The prior is, in this case, given
by:
‖X‖w;212 =
dx∑
i=1
dk∑
k=1
√
√
√
√
dt∑
t=1
dr∑
r=1
wi|xi,r,k,t|2
2
1/2
.
This norm is still of the form of ℓw;212 and leads therefore to the same optimization strategy.
Like in the constrained case, bias for superficial sources can be corrected by setting:
wi =
√
∑
r
‖G·ir‖22 ,
where G·ir stands for the forward field of the dipole with position i and orientation r.
4.5.2 Simulations
By setting a ℓ1 prior between conditions, the mixed norm proposed penalizes overlap be-
tween active cortical regions. In order to illustrate this, we generated two synthetic datasets.
The first reproduces part of the organization of the primary somatosensory cortex (S1) [174].
Three, non overlapping, cortical regions with a similar area (cf. Fig. 4.6a), that could corre-
spond to the localization of 3 right hand fingers, have been computed and used to generate
synthetic measurements corrupted with an additive Gaussian random noise. The amplitude
of activation for the most temporal region (colored in red in Fig. 4.6), that could correspond
to the thumb, was set twice bigger than the amplitudes of the two other regions. This situa-
tion, where the source amplitudes differ between conditions, is relatively common with real
M/EEG data. The inverse problem was then computed with a standard ‖.‖w;F norm and the
‖.‖w;212 mixed norm. Within the 3 neighboring active regions, a label corresponding to the
condition giving the maximum of amplitude in each of the three conditions was assigned to
each dipole. Quantification of performance was done for multiple values of signal-to-noise
ratio (SNR) by counting the percentage of dipoles that have been incorrectly labeled. The
SNR is defined here as 20 times the log of the ratio between the norm of the signal and the
norm of the added noise. Results are also presented in Fig. 4.5. Results with an ℓ1 prior,
has also been added. It can be observed that the ‖.‖w;212 produces systematically the best
result. The ℓ1 is very rapidly affected by the decrease of SNR, which is known in the M/EEG
community. In order to have a fair comparison between all methods, λ was set in each case to
have ‖M−GX∗‖F equal to the norm of the added noise, known in the simulations.
Results are illustrated in Fig. 4.6b and 4.6c on a region of interest (ROI) around the left
primary somatosensory cortex. It can be observed that the extent of the most lateral region,
obtained with ‖.‖w;F , is overestimated while the result obtained with the ‖.‖w;212 mixed norm
is relatively accurate. Similar simulations have been performed in the primary visual cortex
(V1), reproducing the well known retinotopic organization of V1. Results are presented in
140 CHAPTER 4. INVERSE MODELING WITH SPARSE PRIORS
Fig. 4.7. Simulations lead to the same conclusion about the superiority of the ‖.‖w;212 mixed
norm for the mapping of such brain functional organizations.
0 3 6 9 200
20
40
60
80
100
SNR (dB)
Err
or
(%)
|| ||w;F
|| ||w;111
|| ||w;212
Figure 4.5: Evaluation of ‖.‖w;F vs. ‖.‖w;212 vs. ‖.‖w;111 estimates on synthetic somatosensory
data. The error represents the percentage of wrongly labeled dipoles.
(a) Simulation data
(b) ‖.‖w;F result (ROI) (c) ‖.‖w;212 result (ROI)
Figure 4.6: Illustration of result on the primary somatosensory cortex (S1) (SNR = 20dB).
Neighboring active regions reproduce the organization of S1.
141
(a) Simulation data
(b) ‖.‖w;F result (ROI) (c) ‖.‖w;212 result (ROI)
Figure 4.7: Illustration of result on the primary visual cortex (V1) with SNR = 20dB. Neigh-
boring active regions reproduce the retinotopic organization of V1. When comparing the two
results in (b) and (c) with the simulation data in (a), it can be observed that the result in (c)
obtained with the ‖.‖w;212 prior provides the most accurate result.
4.5.3 MEG study
Results of the proposed algorithm using MEG data from a somatosensory experiment are
now presented. The data acquisition was done using a CTF Systems Inc. Omega 151 system
with a 1250 Hz sampling rate. The somatosensory stimulation was an electrical square-wave
pulse delivered randomly to the thumb, index, middle and little finger of each hand of a
healthy right-handed subject. Evoked data were computed by averaging 400 repetitions of
the stimulation of each finger. To produce precise localization results, the triangulation over
which cortical activations have been estimated was sampled with a very high number of
vertices (about 55 000). The forward modeling was performed with a spherical head model1
using dipoles with fixed orientations given by the normals to the cortex [50].
Prior to the current estimation, data were whitened using the noise covariance matrix Σ,
estimated on the period before stimulation. Let Σ = LT L the Cholesky factorization of Σ.
Whitening consists in replacing G by L−1G and M by L−1M. With an additive Gaussian
noise model this implies that the noise, given by M − GX ∈ Rdx×dkdt , is assumed to have
a standard normal distribution. This implies that a good estimate of ‖M −GX∗‖F is given
by√dxdkdt. Therefore, the regularization parameter λ was set in order for X∗ to be also the
solution of the constrained problem: X∗ = arg minX ‖X‖ subject to ‖M −GX‖F ≤√dxdkdt.
The optimization is done with the algorithm 4.5.
Results obtained with the right hand fingers during the period between 42 and 46 ms are
presented in figure 4.8. Knowing that for this somatosensory dataset active parcels should
1http://neuroimage.usc.edu/brainstorm/
142 CHAPTER 4. INVERSE MODELING WITH SPARSE PRIORS
have negative activations around 45 ms, regions with positive activations were first removed.
Within the remaining regions, a label was assigned to each dipole based on its maximum am-
plitude across conditions. For each condition, equivalently each label, the biggest connected
component was kept. Each of the 4 estimated components, corresponding to the 4 right hand
fingers are presented in Fig. 4.8. Solutions using both norms ‖.‖w;F and ‖.‖w;212 are detailed.
With ‖.‖w;212 the well known organization of the primary somatosensory cortex [174] is suc-
cessfully recovered, while with ‖.‖w;F , the component corresponding to the index finger is
overestimated leading to an incorrect localization of the area corresponding to the thumb.
(a) Fingers color coded (a) ‖.‖w,212 result
(c) ‖.‖w,F result (ROI) (b) ‖.‖w,212 result (ROI)
Figure 4.8: Labeling results of the left primary somatosensory cortex in MEG.
143
4.6 CONCLUSION
In this chapter we presented various approaches and algorithmic details necessary
to find the solution of the M/EEG inverse problem with a sparsity inducing prior. State-
of-the-art optimization techniques are presented and discussed in order to provide methods
tractable on big datasets. We have presented simple algorithms, which is particularly im-
portant to facilitate their adoption by the M/EEG community. And finally, we detailed fast
algorithms which are of major interest since real studies require to compute statistics in order
to validate neuroscientific hypotheses. Indeed, when using robust non-parametric statistics,
the inverse solver needs to be run thousands of times. This makes the speed of convergence
of the solver a very critical issue.
Our last contribution describes an inter-condition prior that improves the localization of
cortical activations by offering the possibility to use a prior between different experimental
conditions. By proposing to perform the inverse problem on multiple conditions simultane-
ously and to use a mixed norm that sets an ℓ1 prior between each condition, the method
penalizes current estimates with an overlap between the corresponding active regions. When
such an hypothesis holds anatomically, the more conditions are recorded and used in the in-
verse problem, the better is the localization of neuronal activity. By keeping an ℓ2 prior over
space and time, the proposed method guarantees a good robustness to noise, like standard ℓ2based methods. This approach also improves over the smeared reconstructions observed with
standard ℓ2 inverse solutions. This is confirmed by the simulations and the MEG somatosen-
sory data, with which the method is successfully illustrated.
As for the previous chapter, all the algorithms detailed in this chapter have been imple-
mented and tested with synthetic and real MEG data. The source code of the solvers and the
demo scripts with synthetic and real data are available is a Matlab toolbox called EMBAL
(Electro-Magnetic Brain Activity Localization):
https://gforge.inria.fr/projects/embal
We refer the reader to the demo scripts running on synthetic MEG data:
• demo inverse l1.m: contains a comparison of LARS, IRLS, Landweber and Nesterov for
the ℓ1 problem.
• demo inverse l21.m: contains a comparison of IRLS, Landweber and Nesterov for the
ℓw,21 problem.
• demo inverse TV.m: contains a comparison of IRLS, simple gradient descent (with adap-
tive step size obtained with line search), Landweber and Nesterov for the TV problem.
Nothing was said in this chapter about the use of IRLS and simple gradient descent for
solving the TV problem as it only solves a “smoothed” version of the TV problem. Also, for the
IRLS solver, the presence of the gradient makes the solver even more numerically unstable
than in the ℓp case.
We finish the chapter by listing factors that could be investigated in future studies. Among
these is the ability to use a TV prior with dipoles having unconstrained orientations. This
can be related to an image deconvolution problem when considering a colored image with
3 channels (red, green and blue). Furthermore, the algorithms detailed in this chapter can
be directly applied to multichannel deconvolution problems. Also, we plan to investigate
recently proposed iterative algorithms with the same convergence rate as Nesterov’s scheme
[1] but with a smaller computational complexity. Finally, another major topic we would like to
explore is the case where the prior contains overlapping groups of variables. This would allow
to deal with temporal data where the active sources might not be flagged as active during the
The ℓ2-norm vs. the ℓ212-norm. As it has been observed above, a standard ℓ2 prior, a.k.a.
Minimum-Norm, tends to overestimate the extent of active regions. In order to reduce this
problematic behaviour of standard ℓ2 inverse solvers, we have proposed to invert all the ex-
perimental conditions simultaneously and to promote non overlapping activations by using
what we called in chapter 4 an inter-condition sparse prior. This prior described in detail
in section 4.5 is based on a mixed norm with 3 levels where sparsity is induced between
conditions using an ℓ1 norm. The inverse solver is not linear any more but the problem is
still convex, which offers the possibility to perform the current estimation with very efficient
algorithms. In the following results, the optimization with the ℓ212 prior is performed with
Nesterov’s iterative scheme (cf. algorithm 4.4 in chapter 4).
When working with an ℓ212 prior, the measurements are indexed with a triple index. Here
we concatenate the Fourier coefficients estimated on all trials. Each coefficient is indexed by
the sensor, the condition and the trial. Compared to what is presented in chapter 4, the last
index used here do not correspond to the time but to the trial. With 6 experimental conditions,
the matrix to invert is in Rdm×6dl . After running the inverse solver, the estimated Fourier
coefficients on the source space form a matrix in Rdx×6dl . The same statistical procedure as
for the MN is then used to threshold the activation maps.
A comparison of mapping results obtained with a simple MN and an ℓ212 prior is presented
in figure 5.26. It can be observed that the extent of the activation for the right horizontal
meridian is significantly improved by using the ℓ212 prior. The active region now clearly lies
only in the calcarine sulcus which is consistent with our knowledge about the organization
of the primary visual cortex. However the activation for the upper right quadrant is still not
correctly localized.
The full retinotopic maps obtained with the ℓ212 prior are presented in figure 5.27.
With the MiMS solver. The MiMS inverse solver (cf. section 3.3.1 ) was also experi-
mented on the same data by Benoit Cottereau during his thesis. Results obtained with his
method can be found in [6].
The MiMS solver belongs to the class of Bayesian inverse solvers in the sense that the
weights of an ℓ2 prior are learned. These weights are learned with a multi-resolution ap-
proach based on multipolar modeling of spatially extended cortical parcels (cf. section 3.3.1).
In the approach exposed in [6], the weights are learned on the full temporal data during the
stimulation period. Then the WMN linear inverse solver obtained with the learned weights is
used to get the source estimates. Cottereau then estimates the PSD at 15 Hz for each source
in each trial using a multitaper approach. The PSD estimator is based on Welch’s method (cf.
section 5.5.1 ). Finally, he performs on the PSDs a non-parametric permutation test similar
to the one we exposed above.
Our approach and his approach differ in a number of points. First, Cottereau learns
on the full data even though he is using only the Fourier coefficient at 15 Hz in his non-
parametric statistical test procedure. This means that he may exploit in his localization
pipeline information in a wider region of the spectrum than solely 15 Hz. In order to test
this hypothesis, it would be interesting to band pass filter the data around 15 Hz to see if the
MiMS inverse solver keeps providing the same localization results. Second, by learning the
weights with multipolar expansions, Cottereau limits the influence of the dipole orientations
on the results. Indeed, the multiresolution approach can be seen as a way to provide regions
of interest without really considering the orientation of each individual dipole. Once the
matrix used for linear inversion is computed, his strategy is very similar to one we used in
this chapter. However, it is necessarily slower since the PSDs are actually estimated in the
source space using the reconstructed time series.
174 CHAPTER 5. FAST RETINOTOPIC MAPPING WITH MEG
(a) Map obtained with MN.
(b) Map obtained with the ℓ212 prior.
Figure 5.26: Comparison of retinotopic mapping results obtained with a MN and with the ℓ212prior. Results are presented on left hemisphere using the GM-CSF interface as source space.
Statistical tests were run on the Fourier coefficients (p=0.05 with 15000 permutations). Data
correspond to the stimulation at a frequency of 7.5 Hz.
175
(a) Left hemisphere (Medial view) (b) Inflated left hemisphere (Medial view)
(c) Right hemisphere (Medial view). (d) Inflated right hemisphere (Medial view).
Figure 5.27: Retinotopic map result obtained with an inverse solver based on an ℓw;212 prior.
Statistical tests were run on the Fourier coefficients (p=0.05 with 15000 permutations). Data
correspond to the stimulation at a frequency of 7.5 Hz. Results are represented on the inter-
face between the gray matter and the cerebro-spinal fluid.
176 CHAPTER 5. FAST RETINOTOPIC MAPPING WITH MEG
To our knowledge, Cottereau uses an FFT to extract the PSDs. This is not required if
the interest is only in the spectral coefficient at 15 Hz. A simple correlation with a complex
sinusoid is enough. It also avoids the problems at the border of the time interval by not
requiring to compute a circular convolution like with a discrete Fourier transform.
Taking the best of both approaches would consist in plugging into our pipeline the mul-
tiresolution approach in order to improve the MN results with the weights learned by the
MiMS procedure. This would favor more spatially regular activation patterns and certainly
improve the mapping results while keeping a computationally efficient procedure.
5.5.3.5 Effect of the orientation constraint
As mentioned when discussing the mapping strategy based on MiMS, a possible limitation of
the solvers we experienced is their strong dependence on the orientation of the dipoles sam-
pled over the triangulated source space. A solution to circumvent this problem is to work with
unconstrained orientations. At each location of the source space lie 3 dipoles, each oriented in
the direction of a coordinate axis. This provides 3 coefficients at each location. The amplitude
of the activation at each location is then obtained by computing the norm of the vector formed
by these 3 coefficients. This provides a straightforward way to estimate the PSD at a given
location when no orientation contraints are used. The statistical test procedure can then be
identically computed on the PSDs estimated separately on the stimulation and the baseline
periods.
A result obtained with unconstrained orientation on the same data as in section 5.5.3.1
is presented in figure 5.28. It can be observed that the absence of orientation constraints
produces spatially smoother active regions. The border of the active region is less influenced
by the change of curvature in the source space. Results are consequently more robust to the
intricate structure of the cortical region neighboring the calcarine fissure. However, ignoring
the orientations tends to produce even wider active regions by spreading the currents esti-
mates over the banks of the calcarine fissure. As a result, retinotopic mapping with an ℓ2 prior
also appeared to be very challenging when using no orientation constraints. Improvement on
the robustness of the method to the complex anatomic structure of the occipital cortex has the
drawback of an increased tendency of the minimum-norm to smear the reconstructions over
wide cortical areas when the orientations are ignored.
5.6 TIMING VISUAL DYNAMICS WITH MEG
The primary reason for using M/EEG is to exploit its very good temporal resolution.
Up to here, we focused on the spatial precision of M/EEG source estimates and we exploited
the excellent temporal resolution of M/EEG by using a frequency tagged stimuli. By doing
so the SNR is improved which facilitates the localization. However, the ultimate objective
concerns the measurement of delays between cortical activations and particularly between
different visual areas.
5.6.1 Estimating timings in the visual cortex with M/EEG: Literature
review
Using EEG in [213], the authors address this issue with fitted dipoles whose positions were
constrained with fMRI localization results. Dipoles are located in V1, jointly V2v and V3v,
jointly V2d and V3d, left and right LOV5 (Lateral occipital V5). Once the amplitude time
series are estimated for each dipole, the peaks of the waveform can be compared. The delays
are estimated by measuring peak to peak time intervals. In [162, 190], delays of activation
177
(a) Right hemisphere (Medial view).
(b) Right hemisphere (Medial view) on inflated cortex.
Figure 5.28: Example of localization obtained with no orientation constraint using a MN.
Statistical tests were run on the PSDs (p=0.05 with 15000 permutations). Data correspond
to the stimulation at a frequency of 7.5 Hz in the left lower quadrant of the visual field.
178 CHAPTER 5. FAST RETINOTOPIC MAPPING WITH MEG
in visual system are also obtained with fMRI localizers and dipole fitting. To our knowledge,
timing of activations in the visual cortex with MEG has always been done in conjunction with
fMRI data.
In all these studies, the delays observed correspond to a few milliseconds. This observa-
tion is of major interest with respect to the frequency used in our steady-state stimulations.
Measurements based on the phase are limited to the interval between 0 and 2π, or equiva-
lently −π and π. It corresponds to a full cycle which lasts, at a frequency of 15 Hz, 66.6 ms.
A measure of phase difference can therefore be related to a time delay as long as the delay
of interest is smaller than 66.6 ms. This is the case, in particular, for the delays between
activations in the visual cortex as it is reported in various studies [28, 162, 190, 213]. This
latter remark suggests that time delays could be extracted from the phase estimated on the
sources. A recent study from Di Russo et al. [191] tends to confirm this point.
5.6.2 Extracting information from the phase
The Fourier coefficients at 15 Hz at the source level are denoted Xstim. The coefficient, de-
noted xstimil , on line i and column l in Xstim corresponds to the Fourier coefficient at 15 Hz for
the dipole at position i during trial l. In order to evaluate the quality of the phase information,
a quantity referred to in the M/EEG literature as phase lock can be computed [133].
We define in our case the phase lock, or phase locking value (PLV), for dipole at position i
by:
PLV(i) =1
dl
∣
∣
∣
∣
∣
dl∑
l=1
xstimil
|xstimil |
∣
∣
∣
∣
∣
.
It can be observed that PLV(i) ∈ [0, 1] and that PLV(i) is equal to 1 if all the angles, i.e., the
arguments, of the complex values xstimil are the same. This means that, the bigger is PLV(i),
the more the phase is stable across trials. This is illustrated in figure 5.29.
PLV APLV
Figure 5.29: Schematic representation to illustrate the computation of the phase locking
value (PLV) and the angular information called APLV (see text).
A PLV map computed on the dataset that helped to illustrate section 5.5.3.1 is presented
in figure 5.30. A clear “hot spot” can be observed on the upper bank of the calcarine sulcus
which is consistent with the mapping results obtained above on the same dataset.
Once the PLV is computed in order to assert that the phase contains information stable
across trial, we can investigate the angular part of the average vector. Please note that if a
quantity is stable across trials, it means that it is related to the stimulation. While the abso-
lute value of the average vector provides the PLV, the angle contains the phase information
179
Figure 5.30: Example of phase lock value (PLV) map. The closer is the PLV to 1, the more
the phase of the estimated Fourier coefficients is stable across trials. One can observe a clear
“hot spot” on the upper bank of the calcarine sulcus. This agrees with our knowledge on V1
for a stimulation in the lower left quadrant in the visual field.
which can provide information about delays. We call this quantity APLV and we define it by:
APLV(i) = ang
(
1
dl
dl∑
l=1
xstimil
|xstimil |
)
∈ [−π, π] .
The computation of the APLV is illustrated in figure 5.29.
5.6.3 Preliminary results
In previous sections, we detailed our strategy for the analysis of the phase information at the
cortical level in order to investigate delays. The PLV provides a principled way to assert if the
phase is stable across trials and therefore if it contains information related to the stimulation.
If the PLV is close to 1, as it is the case with our example in figure 5.30, the APLV provides
an angular information also related to the stimulation which might lead to new insights on
the delays. A map of APLV restricted to the regions flagged as active by permutation tests on
the Fourier coefficients is provided in figure 5.31.
Such a map shows the presence of distinct values of angle in the occipital region. However,
it appears to be hard to directly interpret the differences of angle between two regions as a
delay of propagation. Due to the numerous steps to complete in order to obtain such an im-
age, we acknowledge that such an interpretation requires much more validation possibly by
testing with multiple stimulation frequencies. Our results on the phase are very preliminary
and are presented here just to motivate the study of delay estimation with MEG data and
steady-state visual stimulation. Also, we should recall that phase or delay estimation comes
after the mapping which means that the mapping procedure should first be considered as a
solved problem.
5.7 DISCUSSION
In this chapter, we demonstrated that fast retinotopic mapping of V1 with MEG
could be achieved using relatively simple mathematical and algorithmic tools. This was
made possible thanks to a set of technical decisions from the design of the protocol to the
180 CHAPTER 5. FAST RETINOTOPIC MAPPING WITH MEG
(a) Full right hemisphere. (b) Zoom on occipital cortex.
Figure 5.31: Sample phase map used for delay estimation. The quantity represented is the
APLV (see text) restricted to the active region delineated by permutation test on the Fourier
coefficients. Data correspond to the stimulation at a frequency of 7.5 Hz in the lower left
quadrant of the visual field. Results are represented on the inflated interface between the
white matter and the gray matter.
data analysis. The first decision was to use a steady-state stimulation protocol with a flicker-
ing pattern. With a steady-state stimulation and a frequency tagged stimulus, it is possible
to automatically extract from the spectrum of the signal the relevant information. It is not
the case with a standard study based on event related potentials (ERPs). With an ERP based
protocol, the information is extracted at the peak of the waveform in the time domain. The
peak of activation is not stable across conditions and subjects, therefore the extraction re-
quires a manual intervention. The second advantage of the frequency tagged stimulus is that
it allowed us to extract the signal of interest from raw data. All the results presented in this
chapter were obtained without any data cleaning. We used for the stimulation a contrasted
pattern with multiple orientations in order to increase the amplitude of the neural response
and consequently improve the SNR. Finally, we decided to solve the inverse problem with a
distributed source model in order to be able to reconstruct the activation pattern produced
by an unknown number of active sources but also to obtain mapping results with spatially
extended active regions. These concomitant choices of protocol and data analysis strategy
allowed us to obtain a fully automatic procedure for the retinotopic mapping of V1 with MEG.
Our understanding of the different steps in the pipeline enabled us to significantly speed
up the computation by limiting the computation of the solution of the inverse problem to
the Fourier coefficients of interest and by performing the non-parametric tests with simple
linear algebra. In this chapter, we argued that to actually achieve retinotopic mapping it is
necessary to have a pipeline where all the experimental conditions are processed with the
same parameters. In our analysis of the data, we tried to find solutions where manual tuning
was not required for each experimental condition independently. This led us to the conclusion
that a possible solution was to invert the measurements, or more precisely here the Fourier
coefficients, for all the conditions simultaneously.
Such a multi-condition analysis was presented above using a regularization prior based
on a ℓw;212 mixed norm. In section 5.5.3.4, we demonstrated that this prior can actually
improve the mapping results. Our experience with the ℓw;212 norm shows that it clearly helps
181
to delineate the active regions for each condition. It sets the reconstructions to 0 over regions
where other conditions are more likely to be active and it enhances the amplitudes of the
reconstructions in other regions. By doing so it helps to control the spatial extent of active
patterns and it improves the mapping.
Using the ℓw;212 prior implies an increase in computation time. However, the efficient
algorithms detailed in chapter 4 allowed us to get results with a ℓw;212 prior in a few minutes
with highly sampled cortical meshes. However, our experience shows that the ℓw;212 prior
does not help to obtain significantly active dipoles for conditions where the MN has failed to
see something significant.
During our exploration of these data, we met a set of difficulties. Among these is the
approximative estimation of the dipole orientations when they are fixed by the normals to the
mesh. The cortical region neighboring the calcarine fissure can be very intricate which makes
the estimates of the normals to the gray matter quite noisy in this brain region. Sensitivity
of the results to the source space is clearly illustrated above, where reconstruction on the
WM/GM and GM/CSF interfaces are compared. However, using unconstrained orientations
creates other problems already discussed.
Our interest for other solvers than MN was largely motivated by the difficulties we met
when addressing the problem of retinotopic mapping with MEG. In our investigations on
these data, we tried to use the ℓ21 prior (cf. section 4.4.2) but as we could have expected it
did not provide very nice results as this solver by construction sets an ℓ1 norm over space
and consequently cannot really reconstruct spatially extended activations. We also tried to
use the Gamma-MAP inverse solver (cf. section 3.3.2) but it appeared quite difficult to have
good source covariance templates and a good noise covariance estimate to obtain good re-
sults. This is probably due to our limited expertise with this solver on real data. In order to
promote spatially smooth reconstructions, we also tried a prior based on the ℓ2 norm of the
surface gradient. We called this solver HEAT in chapter 3. However, results with this solver
show that such a prior can significantly change and degrade the localization results when the
cortical region of interest is particularly intricate.
In our investigations, we also tried to use other criteria than the 10% rule of thumb from
Brainstorm to estimate the lambda in the MN inverse solver. We present in figure 5.32 and
figure 5.33 a retinotopic mapping obtained using the GCV and the L-curve methods. The
lambda parameter was estimated independently with the Fourier coefficients obtained in
each trial (baseline and stimulation). This result presents smaller active regions in compari-
son to the results in figure 5.24(c). With the GCV, the active region for the upper left quadrant
of the visual field (lower bank of the calcarine sulcus on the right hemisphere) almost disap-
pears while with the L-curve it is completely removed. This suggests that the GCV and the
L-curve tend both to estimate a regularization parameter smaller than the one obtained with
Brainstorm’s 10% rule. Also, we can conclude from this example that the L-curve tends to
provide a value of lambda smaller than the GCV. This agrees with the recurrent claim that
the L-curve approach tends to “under regularize” the inverse problem of M/EEG.
From the investigations and preliminary results presented in this chapter we can draw
some conclusions. The first one in that using the phase of the Fourier coefficients when
performing statistics can be very useful to improve the mapping results. It actually increases
the power of the test. However, our experience tends to prove that estimates of the phase
require long periods of stimulation to be estimated robustly. In the results presented above,
the phase was estimated on a period of stimulation during 5 s. With the same dataset, we
tried our mapping pipeline after artificially shortening the period of stimulation. By doing so,
we observed that the mapping quality starts rapidly to degrade when the phase is estimated
on less than 3.5 s of MEG signal. We have also run the same computations when working with
only the PSDs. It appeared that the mapping is still relatively stable even when the periods
of stimulation last 3 s in each trial. This suggests that the phase of the Fourier coefficients
182 CHAPTER 5. FAST RETINOTOPIC MAPPING WITH MEG
(a) Left hemisphere with GCV.
(b) Left hemisphere with L-curve.
Figure 5.32: Comparison of mapping results obtained using the GCV and the L-curve meth-
ods to estimate the regularization parameter. Statistics are performed on the Fourier coeffi-
cients. Data correspond to the stimulation at a frequency of 7.5 Hz in the right visual field.
Results are represented on the WM/GM interface.
183
(a) Right hemisphere with GCV.
(b) Right hemisphere with L-curve.
Figure 5.33: Comparison of mapping results obtained using the GCV and the L-curve meth-
ods to estimate the regularization parameter. Statistics are performed on the Fourier coef-
ficients. Data correspond to the stimulation at a frequency of 7.5 Hz in the left visual field.
Results are represented on the WM/GM interface.
184 CHAPTER 5. FAST RETINOTOPIC MAPPING WITH MEG
can be easily estimated incorrectly.
185
5.8 CONCLUSION
While in the literature many studies involve both fMRI and M/EEG data when investigat-
ing the human visual cortex, we demonstrated in this chapter that basic retinotopic mapping
of V1 was possible with MEG data only. Our review of the literature showed that our ap-
proach based on MEG measurements, steady-state stimuli and distributed inverse solvers,
had never been done. The steady-state stimuli allowed us to have an automatic way to ex-
tract the signal of interest from raw data without any artifact rejection method or filtering.
Our expertise on inverse solvers allowed us to design a very efficient pipeline to estimate the
Fourier coefficients of interests at the source level. Finally, the non-parametric statistical
method we detailed above allowed us to extract significantly active regions very efficiently.
During this study, we faced a set of difficulties. The most important appeared when map-
ping multiple conditions. What was particularly challenging was the fact that the different
conditions i.e., position of the flickering pattern in the visual field, were producing data with
differences in quality and SNR. The amplitudes of the signal of interest were different at the
sensor level depending on the depth of the source, its orientation and its position (ventral or
dorsal). This issue motivated our methodological contribution as a solver where all the exper-
imental conditions are inverted simultaneously. This solver is based on a mixed norm where
the overlap of active regions is penalized using an ℓ1 norm. We called this regularization, an
inter-condition sparse prior.
To conclude this chapter, we would like to mention that this study was at the center of our
work during this thesis. It motivated most of our methodological investigations and contri-
butions. It was also our first occasion to participate in data acquisitions and the first time
we were confronted to the problem of controlling an experimental protocol. We can now fully
appreciate the fact that this step of the study is non trivial and particularly important when
dealing with brain functional imaging data.
186 CHAPTER 5. FAST RETINOTOPIC MAPPING WITH MEG
CHAPTER 6
TRACKING CORTICAL
ACTIVATIONS WITH
SPATIO-TEMPORAL CONSTRAINTS
The work presented in this chapter goes one step beyond inverse modeling and source local-
ization. It aims at providing a way to sketch the evolution of the cortical activations after
stimulus onset. The proposed method builds on top of standard and widely used linear in-
verse solvers and proposes a strategy to extract spatiotemporally consistent, i.e., physiolog-
ically relevant, active patterns. The limitations of classical linear inverse solvers are well
known. Indeed, as it is discussed in chapter 3, they tend to smear the estimated distributions
of currents over the cortex, and they do not achieve a selection of active regions by setting
coefficients to zero, as it could be done with a sparsity inducing prior. This work gives a prin-
cipled method to achieve such a selection and to remove spurious activations that might be
introduced by basic instant by instant linear inverse solvers.
Exploiting the graph structure of the triangulated cortical surface and the high time sam-
pling of M/EEG recordings, neural activations are tracked over time using a very efficient
graphcut-based algorithm. Such an approach computes a minimum cut on a particularly
designed weighted graph, imposing spatiotemporal regularity constraints on the activation
patterns. Labels are assigned to each node of the graph, distinguishing between active vs.
non-active conditions. The method works globally on the full time period of interest, can
cope with spatially extended active regions and allows the active domain to exhibit topology
changes over time. The algorithm is illustrated and validated on synthetic data. Application
of the method to two MEG datasets demonstrate the ability of the algorithm to track cortical
activations in the primary visual cortex and the somatosensory cortex.
By providing instantaneous measurements of the weak electromagnetic fields generated
by neural activations, M/EEG offer a way to estimate neural currents with a millisecond
temporal resolution. Given these current estimates, which can be seen as images of the active
brain, a new challenge consists in studying the spatiotemporal evolution of neural activities
rather than only localizing specific brain areas involved in experimental tasks [136].
A parallel can be drawn between the challenge proposed here and the problem referred
to as tracking in image processing (see [233] for a recent survey). Both applications have
some similarities and some differences. With neuroimaging data, activations can rapidly
move from one part of the brain to another one via white matter tracks; the signals in the
connecting axons are not captured by MEG or EEG. This phenomenon can be compared to
the occlusion problem in video sequences. Brain activations can move and appear while their
intensities and contrasts change over time, which has some similarity with the illumination
problem. On the other hand, there are also some fundamental differences. Brain activations
are not rigid objects making irrelevant constraints such as point-to-point correspondence be-
tween time instants [126], common motion constraints or shape priors [138]. The topology
and the shapes of the active brain regions can evolve over time. This suggests that the track-
ing methods developed in the computer vision community can provide principled methods to
capture the dynamics of active brain regions, but cannot be applied directly. In the context of
brain functional imaging with M/EEG where the activations are defined over a triangulated
cortex with a natural graph structure, combinatorial optimization techniques based on graph
cuts are the most relevant. Graph cut based techniques in the computer vision community
were first applied for image restoration [96] and segmentation [21, 231]. Examples of video
segmentation via graph cuts are presented in [21], where a complete set of frames is provided
as input to the algorithm that treats the entire sequence as a 3D grid of pixels. Related work
such as [122, 232] present results of object tracking using graph cuts by solving the prob-
lem frame by frame. In this chapter, a graph cut based approach is designed to achieve the
tracking of brain activations and a global optimization on the full temporal data is advocated.
As discussed in chapters 3 and 4, when considering distributed source models, the inverse
problem is ill-posed and requires therefore to set priors on the solution. These priors can,
for example, be based on the subject’s anatomy when constraining the sources to lie on the
triangulated cortex, or on results from other imaging modalities such as fMRI. Schematically,
setting a prior on the solution consists in defining a norm and finding the solution that has
the smallest norm among the ones that explain the measured data sufficiently well.
Commonly, this norm is a ℓ2 norm. Solutions obtained by such constraints lead to linear
solutions (cf. chapter 3). Even if these methods are known to smear the estimated distri-
butions of cortical currents, often leading to solutions that are too widely extended, they are
still considered as the standard methods. Technical reasons for the success of such inverse
procedures are that they are easy to implement, make very few assumptions on the solutions,
are very fast to compute and relatively robust considering the level of noise present in real
M/EEG datasets. More importantly, they are used in the M/EEG community because they
provide localization results that are sufficiently accurate. There are some cognitive neuro-
science studies where the precision of the localization is not critical. On the contrary extreme
precision should be required for pre-surgical recordings, e.g., for epileptic patients.
Basic inverse solvers do not integrate constraints on the spatial or temporal regularity of
the activations, although physiology imposes such regularity on cortical activations. Various
contributions have proposed methods to integrate such smoothness priors in an ℓ2 inverse
problem by adding for example spatial or temporal smoothing operators, like a Laplacian,
in the regularization (see section 3.2.2.4). Such Laplacian based methods appear to be diffi-
189
ΩΩc
Time
Ωt
Figure 6.1: Schematic illustration of spatiotemporal active cortical regions. Ω (resp. Ωc)
indicates the active (resp. non-active region). Ωt is the restriction of Ω to time t.
cult to use with real data as it is unclear how the smoothness prior affects the localization.
Depending on the complexity of the cortical sheet structure around the active brain region,
a bad spatial smoothness prior can strongly bias the localization result. More recently, the
use of a mixed norm has been proposed to better constrain the M/EEG inverse problem (cf.
section section 4.4.2 ). However, such a method, based on an ℓ1 prior over space and an ℓ2prior over time has difficulties to cope with spatially extended activations. With an ℓ1 prior
over space, the lower the SNR, the more the inverse problem is penalized and the more focal
is the active region. Also, no matter how good is the SNR, the ℓ1 prior implies that an optimal
solution has fewer active focal sources than the number of sensors. This implies that with a
detailed source space an extended region cannot be reconstructed using such a prior. In order
to cope with this problem, one could introduce a TV prior with temporal regularization, but
such a solver would not be very tractable on real datasets. Since [9], other techniques based
on Bayesian estimation have also been applied to the M/EEG inverse problem (cf. section 3.3).
Such methods aim at estimating the unknown source covariance matrix in order to learn the
good prior used to regularize the inverse problem. Localization precision obtained in simula-
tion studies using these methods is very promising but still these approaches are not flawless.
First, these methods rely on the assumption that the source covariance is stable in the time
window during which the learning step is achieved. This is not true with real data, especially
for long time windows that are considered when looking at late brain responses. Second, the
results obtained with these methods depend on the source covariance templates defined as
input. Even if the Bayesian estimators developed within these frameworks aim at selecting
the good templates, the choice of these templates can have a strong impact on the localization
results. We refer the reader to the end of chapter 3 for a discussion on this topic.
Such an observation leads to the conclusion that, even if various recent contributions
offer very promising methods to solve the M/EEG inverse problem, standard linear inverse
methods, and more specifically simple ℓ2 priors, provide sufficiently good neural currents
estimates, in order to track the dynamics of distributed and rapidly-evolving cortical current
patterns in a principled manner. The method presented in this contribution provides a way
to follow over time the “hot spots”, i.e., the active regions (cf. figure 6.1), while preserving
spatiotemporal regularity. In the framework detailed in this chapter, the topology of active
regions can change over time. They can appear, split, merge, and disappear. This makes the
method able to handle spatially extended active regions while allowing the active domain to
evolve during the time window of interest. To our knowledge no other existing method offers
such possibilities. Thanks to recent implementations of graph-based algorithms, the method
detailed here is tractable on real datasets and offers a very efficient tool to capture the brain
dynamics.
The rest of this chapter consists of two parts. Section 6.2 presents the optimization frame-
work that is used to select coherent spatiotemporal activations defined over a triangulated
mesh. A variational formulation of the tracking problem is introduced with its discretization
over a triangulation, leading to an optimization problem that can be very efficiently solved
using a graph cut algorithm. Section 6.2 concludes with a validation on synthetic datasets.
190 CHAPTER 6. TRACKING CORTICAL ACTIVATIONS
∆Ωc
Ω
(a) Thresholding
∆Ωc
Ω
(b) Regularized Thresholding
∆
ΩcΩ T
ime
(c) Tracking on spatiotemporal do-main
Figure 6.2: From thresholding to tracking.
Section 6.3 presents the application of the algorithm to MEG data with two different datasets
exhibiting activations in the primary visual cortex and the somatosensory cortex. Even if the
results presented are obtained with MEG data, the method can be directly applied to neural
currents estimated from EEG data as well.
6.2 TRACKING WITH GRAPH CUTS ON A TRIANGULATEDSURFACE
6.2.1 From Thresholding to Tracking
Let f be a real valued function, defined over a domain ∆
f : ∆→ R
When ∆ contains a temporal dimension, finding an “active” region, denoted Ω, vs. a “non
active” region of ∆, denoted Ωc can be viewed as detecting activity over time. The regions Ω
and Ωc form a partition of ∆, i.e., Ω ∩ Ωc = ∅ and Ω ∪ Ωc = ∆. The function f encodes the
likelihood for an element of ∆ to be inactive, and is thus assumed to take small values in
active regions.
A coarse tracking result can be obtained by simple thresholding, i.e., Ω∗ = x ∈ ∆ s.t. f(x) ≤T where T ∈ R is the thresholding value. However, results obtained by thresholding can be
very noisy when f is corrupted by noise. Results are considered to be noisy if the border of
the active region is irregular or if Ω consists of very small active regions. This is illustrated
in figure 6.2(a). It can be shown that the result obtained by thresholding is the solution of the
following variational problem:
Ω∗ = arg minΩ
∫
Ω
f(x)dx+
∫
Ωc
Tdx = arg minΩD(Ω) . (6.1)
D(Ω) is a data fidelity term. One can improve the thresholding by forcing the solution to be
regular. This is done using a Lagrangian approach that adds a term to (6.1) that penalizes
solutions Ω based on a measure of their regularity R(Ω) ∈ R+. Equation (6.1) becomes
Ω∗ = arg minΩD(Ω) + λR(Ω) , λ ∈ R+ . (6.2)
To improve robustness to noise, the regularity measure should prevent the occurrence of
small isolated regions. If the domain ∆ is only spatial, such a regularity can be enforced by
191
penalizing the solution Ω∗ by the length of its border ∂Ω∗ [158]. Figure 6.2(b) illustrates a
result obtained with such a regularization.
If ∆ is a spatiotemporal domain, imposing the regularity of ∂Ω can be achieved by en-
forcing the restriction of the domain at each time instant to have a small perimeter but
also by enforcing an overlap between neighboring time restrictions of Ω. The regularisa-
tion measure R(Ω) can be separated in two parts: a spatial regularisation measure, denoted
Rspace(Ω), and a temporal measure, denoted Rtime(Ω). A solution obtained using the penalty
term R(Ω) = Rspace(Ω) + Rtime(Ω) is illustrated in figure 6.2(c). By imposing to the active
region, Ω∗, to be regular over space and time, one creates the tubular structures that appear
in figure 6.2(c). Each of the tubular structures can be seen as an active region evolving over
time. Being able to exhibit such tubular structures on a triangulated domain, and therefore
to track the activations over time, is the objective of the algorithm proposed in this chapter.
6.2.2 Discretization on a Triangulation
Let us consider T a triangulation consisting of vertices xi and triangles np, and f a function
defined over the vertices of T and over time:
f : (xi)i × (tk)k=1,...,K → R (6.3)
The set of pairs of adjacent triangles is denoted by E , i.e., “np and nq are adjacent triangles” is
equivalent to “(p, q) ∈ E and (q, p) ∈ E”. The restriction of f to an instant tk is denoted by ftk.
To clarify the presentation, K is first supposed to be equal to 1, f = ft1 . It corresponds to the
case where ∆ is only spatial, i.e., no temporal dimension. On a triangulation, partitioning ∆
in Ωc and Ω consists in assigning to each triangle np a label 0 or 1. The label 0 corresponds to
Ωc and 1 to Ω. The integrals in (6.2) can be rewritten:
∫
Ω
f(x)dx =
∫
Ω
ft1(x)dx =∑
np∈Ω
∫
np
ft1(x)dx and
∫
Ωc
Tdx =∑
nq∈Ωc
∫
nq
Tdx
The perimeter of the active region is obtained by the discretization of ∂Ω using the edges of
the triangulation. The regularization term is here given by the sum of the lengths of the
edges separating Ω from Ωc:
∫
∂Ω
dl =∑
(p,q)∈E/np∈Ω, nq∈Ωc
lpq
where lpq stands for the length of the edge between triangle np and triangle nq. Furthermore,
this regularization term is weighted by a constant defined as λspace. The energy in (6.2)
becomes:
Ω∗ = arg minΩ
∑
np∈Ω
Dp(1) +∑
nq∈Ωc
Dq(0) + λspace
∑
(p,q)∈E/np∈Ω, nq∈Ωc
lpq (6.4)
where∫
npft1(x)dx = Dp(1), and
∫
nqTdx = Dq(0) (0 and 1 refer to the 2 labels). If f is assumed
to be affine on each triangle, i.e, f is discretized with P1 elements, Dp(1) = ap(f(xp1)+f(xp2)+
f(xp3))/3 where p1, p2 and p3 are the indices of the vertices of the triangle np, and ap stands
for its area. Similarly, Dq(0) = aqT .
By rewriting (6.2) in this discrete form, the energy to minimize has been cast into a
Markov Random Field optimization framework [84] that can be very efficiently solved us-
ing graph-based methods [179]. These methods, that have recently been extensively used in
Computer Vision [23], establish the equivalence between energy minimization and finding
192 CHAPTER 6. TRACKING CORTICAL ACTIVATIONS
Figure 6.3: Energy discretization on a triangulated mesh.
Table 6.1: Edge weights, i.e., link capacities, of the graph for tracking on a triangulated
mesh. Graph nodes np are indexed by a space index p. N-Links of type “Spatial” control
spatial regularization.T-Links Weight
S → np Dp(0)np → T Dp(1)
N-Links Weight Type
np ↔ nq λspacelpq Spatial
the minimal cut of a specially designed graph. They are commonly known as “graph-cut”
methods. The difficulty is to design a weighted graph providing a natural correspondence
between the partitioning of the graph and the energy that is minimized.
See Appendix B for a short introduction to graph cuts.
An illustration of the graph constructed for the current optimization is presented in fig-
ure 6.3. Such a construction is inspired by [96], where a similar graph is used for binary image
restoration. Contrarily to [96], or more recently to [21], where the graph is constructed on nD
grids, the current application imposes to work on temporal data defined on 2D triangulated
surfaces embedded in 3D.
Each triangle of the cortex mesh corresponds to one node of the graph. These nodes are
thus indexed like the triangles, i.e., using the notation np. There are two supplementary
terminal nodes : the “Source” S and “Sink” T that represent respectively the domains Ω
and Ωc. Each node np is linked to both S and T (these edges referred as T-links imply that
the triangle np can belong to one of the two domains represented by S or T ). Furthermore,
the graph contains edges between each of the nodes that correspond to adjacent triangles.
These edges are referred as N-links. Cutting the graph in two consists in separating the
“Source” from the “Sink” by removing some edges. With a minimal cut, each node will remain
connected to only one of the terminal nodes so that the remaining graph directly corresponds
to a partitioning of the mesh into two domains Ω and Ωc. Table 6.1 details the edge weights
corresponding to the energy (6.4). In practice, edge weights, i.e., link capacities, must be
positive. Therefore, prior to the computation of the minimum cut, edge weights are translated
to guarantee that they all satisfy this computational constraint.
One can notice that the cost associated to a cut, defined as the sum of the edge weights
along the path of the cut, is equal to an energy value. The minimum cut thus provides the
optimum. Graph partitioning via minimum cut is, in turn, known to be equivalent to a poly-
nomial problem: the max flow problem [73]. The fact that an exact solution is obtained by a
single binary cut guarantees the algorithm to be extremely fast (a few seconds for the prob-
lems addressed in this contribution) and also globally optimal [179]. In practice, the min-cut
193
Table 6.2: Edge weights, i.e., link capacities, of the graph for tracking on a triangulated
mesh. Graph nodes np,k are indexed by a space index p, and a time index k. “Spatial” N-
Links control spatial regularization and “Temporal” ones control temporal overlap between
neighboring time instants thus temporal regularization. ap is the area of triangle p and lpq is
the length of the edge separating triangle p and triangle q.T-Links Weight
S → np,k Dp,k(0)np,k → T Dp,k(1)
N-Links Weight Type
np,k ↔ nq,k λspacelpq Spatial
np,k ↔ np,k+1 λtimeap Temporal
is therefore obtained via the computation of the max-flow using an open source implementa-
tion1 [22].
When considering multiple time instants, a similar approach is used. The nodes (np,k)p,k
are now indexed by the triangle index p and the time k. The full graph is obtained by stacking
spatial graphs obtained for each tk and adding N-links between triangles in neighboring time
instants. The number of nodes in the graph is now equal to the number of time instants times
the number of triangles. Both terminal nodes S and T are still unique. Edges weights can
now integrate temporal smoothness (See Table 6.2).
In the optimization framework detailed above, the smoothness term becomes λspaceRspace(Ω)+
λtimeRtime(Ω), where:
Rspace(Ω) =
K∑
k=1
R(Ωtk) =
K∑
k=1
∑
(p,q)∈E/np,k∈Ωtk, nq,k∈Ωc
tk
lpq
Rtime(Ω) =K−1∑
k=1
∑
p/np,k∈Ωtk,np,k+1∈Ωc
tk+1
ap +∑
p/np,k∈Ωctk
,np,k+1∈Ωtk+1
ap
.
(6.5)
In this formulation the regularization parameters λspace and λtime do not depend on the
position in space or in time. However they could be tuned independently for each edge (p, q).
One could think of using for example the curvature of the cortex to promote the cuts on
regions of high curvature. However, without such a priori and for the sake of simplicity λspace
and λtime were kept constant over space and over time.
The weights are detailed in Table 6.2. N-Links of type “Temporal” promote overlap be-
tween neighboring time instants and thus enforce temporal smoothness.
The complexity of the graph cut algorithm is O(N3) where N stands for the number of
nodes in the graph. However in practice, like observed in [22] with nD grids, computation
time appears to increase linearly with the number of nodes (cf. figure 6.4). More than the
computation time, the limiting factor when dealing with large graphs is the memory con-
sumption of the implementation used in this contribution.
6.2.3 Tracking Results with Synthetic Data
The tracking algorithm is now illustrated on two synthetic datasets. The first simulation on
a randomly triangulated sphere is designed to be simple and to demonstrate the influence
of the regularization parameters. The algorithm is then applied to a more realistic dataset
exhibiting three simultaneous moving “hot spots” on a Bunny triangulation.
For the first dataset, the domain ∆ consists of a triangulated sphere with 3 time instants.
About 30 000 vertices are randomly sampled over the sphere and the triangles are obtained
with a Delaunay triangulation. The f function was generated to simulate the displacement
Figure 6.5: Result of tracking using the graph cut algorithm on synthetic dataset defined on
a randomly triangulated sphere with 30 000 vertices. Colored lines correspond to the border
of the active regions and the color codes for the time instant. Initial data f represented in (a)
is equal to 0 in active regions and 1 outside. Prior to the tracking, Gaussian white noise with
a standard deviation equal to 1 was added to f .
for which the definition of regularization parameters never actually required a very fine tun-
ing.
In a second dataset, three “hot spots” are moving simultaneously during 100 time frames
over a “Bunny” triangulation with about 8000 vertices. These data, also used as a validation
set in [135], are presented at 5 different time instants in figure 6.7(a). Such data were de-
signed to provide a more complex and realistic synthetic dataset according to the geometry
but also to the time scales of the brain activations measured by M/EEG. In order to respect
the convention that f should take small values in active region, f was set to the opposite of
the actual activations defined over the mesh. According to the signal amplitude, the param-
eter T was set to −4 ∗ 10−3. Prior to the tracking, an additive Gaussian white noise with
standard deviation equal to 6 ∗ 10−3 was added to the synthetic data. Results of thresholding
and tracking are presented in figure 6.7(b) and figure 6.7(c). It can be noticed that the method
can actually cope with topology changes since figure 6.7(c) presents the merging of two active
regions. Here also, it can be observed that the tracking algorithm provides a clear view of the
dynamics of the activation defined over the triangulation.
6.3 APPLICATION TO M/EEG DATA
196 CHAPTER 6. TRACKING CORTICAL ACTIVATIONS
λspace
λtim
e
0 2 4 60
0.2
0.4
0.6
−2
−1.8
−1.6
−1.4
−1.2
−1
(a) Labeling errors in logarithmic scale
λspace
λtim
e
0 2 4 60
0.2
0.4
0.6
(b) Labeling errors smaller than 1.5%
λspace
λtim
e
0 2 4 60
0.2
0.4
0.6
(c) Region where the estimated number ofcomponents is equal to 3
λspace
λtim
e
0 2 4 60
0.2
0.4
0.6
0
0.2
0.4
0.6
0.8
1
(d) Dice’s coefficient
Figure 6.6: Labeling errors obtained by the tracking algorithm for various pairs of regular-
ization parameters (λspace, λtime). In (a) Error was quantified by the ratio between the number
of mislabeled vertices and the total number of vertices. The color-coded errors are presented
in logarithmic scale. The best performance is obtained with λspace = 2.3 and λtime = 0.04, but
the performances remain very acceptable with parameters in a wide interval around these
values. This is illustrated in (b) where is represented the region in which errors are smaller
than 1.5%. In (c) is represented the region where the number of components is correctly esti-
mated to 3. In (d) the performance is measured using the Dice’s coefficient (6.6). The closer it
is to 1, the better it is. All performance measures confirm that the result is relatively robust
to the definition of the parameters λspace and λtime.
197
Frame 1 Frame 25 Frame 50 Frame 75 Frame 100(a) Synthetic dataset on “Bunny” triangulation presented at multiple time instants.
(b) Tracking result with no regularization :λspace = 0 and λtime = 0
(c) Tracking result with spatiotemporal regular-ization : λspace > 0 and λtime > 0
Figure 6.7: Result of tracking using the graph cut algorithm on a synthetic dataset defined
on the “Bunny” triangulation with 8000 vertices and 100 time instants. The original data
consists of three moving “hot spots” illustrated in (a). Data were corrupted by an additive
Gaussian white noise with a standard deviation equal to 6 ∗ 10−3. Figure (c) demonstrates
the ability of the method to cope with topology changes. Between frame 0 and approximately
frame 30 the 2 activations on the head of the bunny merge.
198 CHAPTER 6. TRACKING CORTICAL ACTIVATIONS
The tracking method presented above is now applied on two MEG datasets. The
first one is obtained with a visual stimulation paradigm that consists of a series of expanding
checkerboard rings. Such a stimulation creates a propagation of activation along the pri-
mary visual cortex (V1) that enables direct application of the tracking algorithm. The second
dataset consists of a somatosensory finger stimulation. For this dataset, due to the different
amplitude levels of activations within the various brain regions involved in the processing of
the task, a particular data fidelity cost is designed prior to the tracking algorithm.
6.3.1 Results on visual stimulation
In this experiment, expanding checkerboard rings extended radially from 0 to 4 degrees of vi-
sual eccentricity are presented periodically with a frequency of 5 Hz (see figure 6.8). Because
of the retinotopic organization along the calcarine fissure and V1, the optical flow generated
by the expanding checkerboard rings produces a posterior - anterior wave of activation as
illustrated by figure 6.9. It is this propagating wave within V1 that we propose to track.
Frame 1 Frame 2 Frame 3 Frame 4
Figure 6.8: One block of successive frames used to produce expanding checkerboard rings.
The block of 4 frames is projected periodically with a frequency of 5 Hz.
L I N G U A L
C U N E U S
P R A
E C U
N E
U S
Infr
C a
l c
a r i
n e
f i s s u r e
Parie
to-o
ccipita
lfissure
I S T
G Y R U S
Propagation
Figure 6.9: Schematic representation of the cortical activation propagation produced by the
expanding checkerboard rings. The propagating wave covers the primary visual cortex (V1)
on both sides (superior and inferior) of the calcarine fissure, from the posterior to the anterior
part of the cortex (Adapted from 20th U.S. edition of Gray’s Anatomy of the Human Body,
1918, public domain).
199
Data acquisition and analysis
The MEG data were acquired at 1250 samples/sec with a 151-SQUID sensor CTF MEG (CTF
System Inc.). In each trial, rings were presented for 2.5 s after a prestimulation period of
1.4 s. The first 500 ms of stimulation correspond to a transitory period before 2 s of steady
state period (cf. figure 6.10). In order to improve the signal-to-noise ratio, 33 trials were
averaged and the data were band pass filtered between 2.5 and 7.5 Hz. These data come
originally from the study presented by D. Cosmelli et al. in [41].
Steady state
period
Prestimulation
period
Transitory
period
Time
1.4 s 0.5 s 2 s
...
t=0
Figure 6.10: Experimental protocol for visual stimulation with the expanding checkerboard
rings. Prestimulation period lasts for 1.4 s before 2.5 s of stimulation. Stimulation period is
divided in two: the transitory period estimated around 0.5 s and the steady state period that
lasts for 2 s. Tracking is performed during the steady state period.
In order to estimate the source amplitudes, the forward problem was computed with a
spherical head model and a distributed source space consisting of a cortex triangulation with
about 50 000 vertices. An inverse solution was computed with an ℓ2 penalization term using
dipoles with unconstrained orientations. The source amplitudes, denoted sit (i indexes space
and t time), were computed by computing the norm of the equivalent current dipole at each
location. Using prestimulation recordings sit was then normalized. Assuming stimulation
starts at t = 0, this corresponds to: zit = sit/σi where σi is the standard deviation estimated
on (sit)t<0. By doing so, the zit are all positive and have large values in active brain regions.
Tracking
The tracking algorithm was run using as data term the computed zit. To follow previously
exposed conventions it leads to ft(i) = −zit. The threshold T was set manually in order to
obtain active regions lying approximately within V1. To limit computation time and memory
consumption during processing, the tracking was performed independently on each period
of stimulation during the steady state period. With the 5 Hz stimulation frequency, it corre-
sponds to a period of 200 ms. The results of the tracking algorithm are presented in figure 6.11
while a comparison between thresholding and tracking results is presented in figure 6.12. It
can be observed in figure 6.11 a clear propagation of the activation along the calcarine fissure
and V1, from the posterior part of the cortex towards its anterior part. By observing the de-
tailed comparison in figure 6.12, it appears clearly that the graph cut based spatiotemporal
regularization provides more consistent activation patterns by regularizing the propagation
front and removing false activations outside of the primary visual cortex.
200 CHAPTER 6. TRACKING CORTICAL ACTIVATIONS
(a) Medial view (b) Zoom of the calcarine fissure
Figure 6.11: Tracking results obtained with visual stimulation of expanding checkerboard
rings (λspace = 3 , λtime = 1). Color codes for the first time of activation during the time
window considered, between 2310 and 2367 ms after the beginning of the stimulation. Col-
ormap was reduced to 6 colors to present a clearer representation of the propagation. Source
estimates were obtained with a spherical forward model and a minimum-norm inversion us-
ing unconstrained orientations. Source amplitudes at each position were normalized using
prestimulation recordings (see text). Triangulation has about 50 000 vertices. The graph cut
based spatiotemporal regularization provides consistent activation patterns by regularizing
the propagation front and exhibiting active regions in the primary visual cortex.
6.3.2 Results on somatosensory data
Data acquisition and analysis
Acquisition of the somatosensory data was done with the same MEG device that was used for
the visual stimulation paradigm. The somatosensory stimulation was an electrical square-
wave pulse delivered randomly to the thumb, index, middle, and little finger of each hand of
a healthy right-handed subject. The stimulus intensity was below the motor threshold. In
order to improve the SNR, 400 recordings were averaged for each finger. These data come
originally from the study presented by S. Meunier et al. in [150]. To produce precise tracking
results, the triangulation over which cortical activations have been estimated was sampled
with a very high number of vertices (about 55 000). The forward modeling was performed
with a spherical head model. The source activations were computed with an ℓ2 prior using
constrained orientations. The reason for using constrained orientations with this dataset, is
that the shape of the cortical mantle around the brain regions activated by such a stimulation
is much simpler than in the occipital region around the calcarine fissure. Dipole orientations
provided by the normals to the mesh obtained by segmentation of the gray matter are in this
case better estimated.
Somatosensory stimulation is commonly used to validate M/EEG methods since it is known
that somesthetic inputs project in precise brain areas [104]. Among these areas are the pri-
mary motor cortex (S1) and the secondary motor cortex (S2). While the amplitude of the
first activation in S1 is high, the activation that appears later after stimulation in S2 is
much weaker but still present. This leads to the conclusion that a same threshold for both
activations in S1 and S2 is bound to fail. This issue can be compared to the illumination
problem when tracking objects in video sequences. The object keeps moving but its intensity
and contrast can change over time. To tackle this problem, the tracking algorithm on the
somatosensory dataset requires as preprocessing to construct a particular data cost function
201
Time (a) 2310 ms (b) 2344 ms
Th
resh
old
ing
Tra
ckin
g
Time (c) 2368 ms (d) 2374 ms
Th
resh
old
ing
Tra
ckin
g
Figure 6.12: Comparison between naive thresholding and tracking with spatiotemporal reg-
ularization. Tracking and thresholding results are presented at multiple time instants dur-
ing visual presentation of the expanding checkerboard rings. Thresholding corresponds to
λspace = 0 and λtime = 0, i.e., no regularization, while the tracking is performed with λspace = 3and λtime = 1. (a) illustrates how the tracking manages to remove the spatially inconsistent
activation on the lower part of V1. (b) illustrates how the tracking makes the incorrect acti-
vation on the anterior part of the parieto-occipital fissure (cf. figure 6.9) disappear. (c) shows
how the tracking fills the hole in the active region. This is consistent with the retinotopic
organization of V1 and the rings used in the visual stimulation. (d) is another illustration of
the regularization, here during the half period when the activation leaves V1.
202 CHAPTER 6. TRACKING CORTICAL ACTIVATIONS
that enables to define a common threshold at any time after stimulation while still using the
essential information that is the source amplitude.
Designing f with heterogeneous activation levels
M/EEG data have typically a sampling rate around 1000 Hz and the characteristic times of
the phenomena that are being recorded are about a few milliseconds. Although this time of
course depends on the type of neural activations, it suggests that an activity is significant if
it lasts for a few consecutive time instants. Hence, the construction of a function f based on
small time windows rather than each time instant is still relevant. From now on, k indexes
time windows rather than time instants. These windows are denoted by wk.
The natural idea behind the choice of f is that a vertex is very likely to be active during a
time window wk if its activity is close, in relative distance, to the activation of a source that
captures a significant amount of energy.
Let a0i (k) denote the activation time series of vertex i during window wk (for the time being
the superscript 0 can just be ignored). The template a0i0
(k) is defined as the time series that
captures the highest amount of energy (i0 = arg maxi ‖a0i (k)‖2). The function f can now be
defined over window wk by:
fk(i) =‖a0
i (k)− a0i0
(k)‖22‖a0
i0(k)‖2
∈ [0, 1] (6.7)
The function f is designed to take its values between 0 and 1 in order to facilitate the choice
of the threshold T . The influence of the activation level does not appear any more directly in
f but only in the choice of the template.
It is however possible to use multiple templates since activations are very likely to be
simultaneously localized in different regions having different temporal types of activations.
This is done using a greedy algorithm similar to standard matching pursuit algorithms. Note
that such greedy approaches have been successfully used in the field of M/EEG with the
RAP MUSIC inverse problem solver [153]. In both procedures, the most significant source
is first estimated. Its contribution to the data is then removed before looking for the next
significantly active dipole. This continues until the data have been sufficiently well explained.
Although the RAP MUSIC algorithm uses signal subspaces, the idea developed here using
temporal templates is fairly similar.
Let A0k be the matrix of all activations during window k, A0
k = (a0i (k))i. Each column
of A0k contains the activation time series of one dipole. The objective is to select the best L
templates that capture most of the activity during window wk. The strategy to do so consists
in selecting them iteratively. Template l is obtained after template l−1 by finding the column
of Alk = (al
i(k))i that has the biggest ℓ2 norm, il = arg maxi ‖ali(k)‖2. The matrix Al+1
k is
obtained by projecting the columns of Alk orthogonally to the vector al
il.
Al+1k = Πl
kAlk =
(
I− alilalT
il
‖alil‖22
)
Alk (6.8)
The process ends when ‖Alk‖2 becomes smaller than a certain percentage P ∈ [0, 1] of ‖A0
k‖2(‖ ‖2 stands for the Frobenius norm). Note that since Πl
k is a projector, necessarily ‖Al+1k ‖2 <
‖Alk‖2 must hold. When considering multiple templates the function f is defined by:
fk(i) = maxl=0,...,L−1
‖ali(k)− al
il(k)‖2
2‖alil(k)‖2
∈ [0, 1] (6.9)
The procedure has the advantage that the time series a0i0, . . . ,a0
iL−1correspond to particular
203
brain locations. More fundamentally, this construction of f has the advantage of making the
threshold easy to set. However, during this work several other attempts were also made to
define f differently using statistical quantities. Using “noise normalized” inverse methods
like dSPM [49] and sLORETA [171], it appeared to be impossible to define a common thresh-
old leading to well localized activations both for the activation peak around 45 ms as well
as for later responses. Permutation tests [169] were also investigated using time dependent
thresholds computed based on a given p-value. Thresholds obtained using False Discovery
Rates (FDR) [85] were also experimented. Both of these approaches, particularly computa-
tionally time demanding, did not produce very satisfactory results. The proposed approach
has the advantage of speed and simplicity when it comes to selecting the parameters, making
the tool quite easy to use.
Tracking
The window size was set equal to 20 ms with an overlap between neighboring windows of
75%, i.e., 15 ms. For each window, f was computed using multiple templates with P = 0.2.
In practice, a maximum of 3 templates were chosen within each time window. The threshold
was set equal to T = 0.2. Results with the right index finger are presented in figure 6.13.
Color codes for time, or equivalently a window index, and indicates the border of the active
region in each time window. This result provides a representation of the brain dynamics
after stimulation of the right index finger. Cortical activations are successfully tracked over
time, as early as 30 ms after stimulus onset in left primary somatosensory cortex (S1), during
the displacement of neural activity along the postcentral gyrus all the way to the secondary
somatosensory cortex (left and right) and the left Brodmann area 5. This confirms what is
reported in [104] about the processing of such somatosensory tasks by the human brain. In
order to illustrate the influence of the spatiotemporal regularization on the solution, results
with no regularization at all, and no temporal regularization are provided in figure 6.14.
It can be observed that the spatiotemporal regularization is actually required to prune the
spurious activations.
204 CHAPTER 6. TRACKING CORTICAL ACTIVATIONS
(a) Result on the partially inflated cortex. The green dot corresponds to the location of the equivalentcurrent dipole located at 44 ms after stimulation. Colored lines correspond to the border of the activeregion at different time instants (1 to 3).
(b) Zoom on the tracking result in S1.
Figure 6.13: Result of tracking using the graph cut algorithm on somatosensory dataset.
Source estimates were obtained with a spherical forward model and a minimum-norm inver-
sion on MEG data obtained from a somatosensory evoked response study. Triangulation has
about 55 000 vertices. Data presented here are for the stimulation of the right index finger.
Cortical activations are tracked over time (λspace = 0.05 and λtime = 0.05), as early as 30 ms
after stimulus onset in left primary somatosensory cortex (S1), during the displacement of
neural activity along the postcentral gyrus all the way to the secondary somatosensory cortex
(left and right) and left Brodmann area 5 [104].
205
(a) Tracking result on inflated cortex: Left hemi-sphere
(b) Tracking result on inflated cortex: Right hemi-sphere
(c) Result with no regularization (λspace = 0 , λtime =0)
(d) Result with no temporal regularization (λspace > 0, λtime = 0)
Figure 6.14: Influence on the regularization on the tracking result on the somatosensory
dataset. Source estimates were obtained with a spherical forward model and a minimum-
norm inversion on MEG data obtained from a somatosensory evoked response study. Trian-
gulation has about 55 000 vertices. Data presented here are for the right index finger. It can
be observed that the spatiotemporal regularization is actually required to prune the spurious
activations.
206 CHAPTER 6. TRACKING CORTICAL ACTIVATIONS
6.4 CONLUSION
In this chapter a method has been proposed to address the challenging problem
of robustly estimating the spatiotemporal evolution of neural activities rather than just lo-
calizing specific brain areas involved in experimental tasks. The approach is based on an
optimization over the active domain that penalizes spurious activations and extracts activity
with spatiotemporal regularity. A Lagrangian formulation is derived and it is shown how the
functional obtained can be discretized over a triangulation and efficiently optimized with a
graph cut based procedure.
The principled method proposed here allows the active domain to have topology changes.
It implies that active regions can appear, split, merge and disappear during the time window
of interest. A source is not necessarily flagged as active during the full time period of interest.
Thanks to the use of a very efficient graph cut implementation, with a linear complexity
observed in practice, the optimization can be run in a few seconds on real M/EEG datasets
with highly discretized cortical meshes.
A possible perspective for this work is to integrate into the temporal regularisation term
a prior coming from diffusion MRI data. By measuring white matter anisotropy such data
provide insights on subcortical neural pathways. With the latter graph-based formulation
such an information could be easily integrated by adding temporal edges between triangles
strongly linked by the subcortical fiber bundles present in the white matter. Possibly, such
links could be defined between non neighboring time instants in order to model delays in the
propagation of the neural activations. Moreover, using the residual capacities of the edges
after computing the min-cut, it may be possible to quantify which subcortical edges have
influenced the solution, leading to fruitful insight about how the processing of the information
has been distributed across the different functional brain regions involved in the task.
The source code of the tracking algorithm and the demo scripts necessary to reproduce
the figures of this chapter are available in a Matlab toolbox called EMBAL (Electro-Magnetic
Stimulus-locked averaging is applicable for very early (primary) responses whose charac-
teristics are very stable, such as early somatosensory evoked potentials. However, for later
responses, corresponding to a more complex treatment of information by the brain, charac-
teristics such as latencies and amplitudes are usually variable across trials, typically because
of habituation effects, anticipation strategies, or fatigue [130]. For responses with latency
variability, signals time-locked to the stimulus onset are not properly aligned with respect to
the timing of the single-trial brain responses, and this leads to considerable blurring when
the averaged evoked response is computed. The event related brain response is hence inaccu-
rately estimated, and this can lead to wrong conclusions about the latencies between neural
activations [123]. This chapter provides methodological tools to estimate the variability, e.g.,
the latency, of brain responses across trials, and thus to correct for the averaging bias.
Analysis of single-trial EEG was pioneered by Woody’s cross-correlation averaging [230].
In this work, the latency of single-trial event-related activity is estimated by using a tem-
plate, supposed to model accurately the evoked response. The algorithm alternately esti-
mates the template and corrects for the latency bias in each trial by finding the maximum
of a cross-correlation. In each iteration, the template is updated with the average of the
corrected data. Convergence is observed in practice, but not guaranteed. The obtained so-
lution may not be globally optimal since it depends on the time series used to initialize the
algorithm. Subsequent work has extended Woody’s idea by including amplitude variability
and by placing the estimation of single-trial parameters in a maximum likelihood frame-
work [114, 134, 147, 206, 207]. Direct denoising of EEG single-trial data with a time-scale
decomposition has been proposed [181], which designs a wavelet template from the aver-
age signal across trials. As in [230], the average signal is explicitly used as a template in
the computation. Several other methods, based on linear decompositions or wavelet analy-
sis [15, 16, 185, 221], make the assumption that the evoked response can be well represented
by a linear combination of functions within a dictionary. The dictionary is provided as a prior
to the algorithm or learned from the data, e.g., with a singular value decomposition (SVD).
All methods described above suffer from various pitfalls: the lack of proof of convergence
of the procedure, the dependence of the results on the initialization, the use of the average
data in the computation, and the necessity to define an a priori model for the waveform of the
response.
This chapter approaches cross-trial variability from a different perspective, in a data-
driven way, free of the pitfalls just enumerated. The problem is cast into an optimization
framework in which global optimality can be proved, and where initialization is not an issue.
Moreover, the solution can be found very efficiently by using associated fast algorithms. The
method proceeds in two stages. First, we propose to sort the trials automatically according to
the variability of the evoked potentials without estimating it explicitly. The estimation of the
variability is performed in a second stage.
As an illustration, figure 7.1 presents raster plots of a raw dataset (a) and a reordered
dataset (b): in these images, each line represents a trial. In figure 7.1(b), the trials have been
reordered according to the measured motor response of the subject. A structure thus becomes
visible and can be used to interpret the single-trial ERPs. Such representations of multi-trial
ERP recordings, called ERP images were pioneered in [123] and made generally available
through EEGLAB [59]. EEGLAB proposes several ways of reordering ERP images, according
to event triggers, or to the phase of time-frequency decompositions.
In this chapter, we propose a reordering, based only on the evoked response, without re-
lying on any external information. In some multi-trial ERP recordings, it is reasonable to
assume that a similar neural activation occurs in each repetition of the experiment, but with
210 CHAPTER 7. SINGLE-TRIAL ANALYSIS WITH GRAPHS
a latency between stimulation and response that is variable across trials. This leads to the
intuition that the latency of the response is the main “degree of freedom” within the data.
The single-trial time series then lie on a noisy one-dimensional manifold which can be pa-
rameterized by this latency. In order to capture this “degree of freedom”, manifold-learning
techniques are very well-suited: by providing low dimensional representation of the data,
they offer an efficient way of revealing the structure present in a dataset. We propose to use
methods based on eigendecompositions of graph Laplacians [14, 37]. Nonlinear dimensional-
ity reduction methods have been applied before to functional MRI data [7]. In [8] and [198],
low dimensional representations of fMRI datasets have been used to identify and classify
brain functional regions. For EEG data, our previous work [94] has shown how Laplacian
Eigenmaps [14] can be used to reveal the structure of an EEG dataset.
In this chapter, the nodes of the constructed graphs correspond to single-trial EEG time
series and it is shown on an EEG sample dataset that the “best” one dimensional represen-
tation of the data obtained with graph Laplacian procedures is monotonically related to the
response latency. Equivalently, it is shown that the first coordinate in the low dimensional
space can be used to reorder the dataset as in figure 7.1(b). We will refer to this step of the
method as the spectral reordering of the raster plot.
Trial
Time (ms)
0 199 398 597
50
100
150
200
−20
−10
0
10
20
(a) Original raster plot
Trial
Time (ms)
0 199 398 597
50
100
150
200
−20
−10
0
10
20
(b) Raster plot reordered with themeasured motor response of the sub-ject.
Figure 7.1: Illustation of raster plot reordering on real EEG recordings. Each line of the
image represents a time series and the color codes for the signal amplitude in µV . In this
dataset the measured latencies of the motor responses, represented by the dark semi-vertical
line in figure 7.1(b), are used to sort single-trial time series.
Spectral reordering reveals the structure present in the raster plot, but it does not ex-
plicitly estimate the trial-dependent parameters in each trial. This problem is tackled in the
second step of the procedure, eventually leading to optimized event-related averaging. Given
a reordered raster plot, estimating the latencies corresponds to finding an increasing function
similar to the one defined with the dark semi-vertical line in figure 7.1(b). This function takes
its values on the grid defined by the image, and can only assume a finite number of discrete
values. The problem of extracting the latency information from the reordered raster plots can
thus be viewed as a combinatorial problem, which can be very efficiently optimized using a
graph cut algorithm. This step is performed independently from the spectral reordering step,
and only requires a sorted raster plot as input.
The chapter is organized as follows. In section 7.2 we introduce Manifold Learning and
present the graph Laplacian method. In section 7.3, we apply it to the reordering of multi
trial EEG data and show how the proposed approach compares to the more classical Principal
Components Analysis (PCA). Section 7.4 focusses on latency estimation and formulates it
as a combinatorial optimization problem. An efficient solution to this problem is proposed
by computing a minimal cut on a specially designed graph. Finally, a strategy to estimate
the different parameters is detailed and the robustness of the procedure is investigated by
211
numerical simulations. Results on synthetic and real data accompany each of the methods
presented.
7.2 MANIFOLD LEARNING
Let (xi)i=1,...,N be N elements of a metric space (X , dX ), which are distributed with a
probability distribution p on a low-dimensional smooth sub-manifoldM of X . In this chapter,
the (xi)i=1,...,N are time series and N the number of trials.
This section deals with manifold learning techniques: Section 7.2.1 presents a Principal
Components approach to variability analysis. Sections 7.2.2 and 7.2.3 recall notions on non-
linear embedding, leading to the Laplacian embedding algorithm. Differences between the
linear and non-linear approaches are emphasized on synthetic datasets in section 7.2.1 and
will also be illustrated in section 7.3.1.
7.2.1 Principal Component Analysis
Principal Component Analysis represents the data in a new coordinate system, obtained
through a rotation that diagonalizes the empirical covariance matrix. Although PCA per
se does not modify the dimensionality of the representation, by ordering the eigenvalues of
the empirical covariance matrix, one can represent the data in the leading PCA directions.
The PCA representation is a valuable tool for exploratory analysis as it can provide a repre-
sentation of the structure present in the data.
To take the example of latency variability, consider a dataset of translated versions of a
reference template xi(t) = x(t− τi). The data is wide-sense stationary, and its covariance ma-
trix Cx is diagonalized in the discrete Fourier basis. The Fourier transform of the translated
signal xi(t) = x(t− τi) is phase modulated: xi(ω) = eiτi ω x(ω), where ω denotes the frequency.
Principal Components correspond to the frequencies that dominate the power spectrum of
the signal Px(ω) (the Fourier transform of the covariance matrix Cx). Consider the dominant
frequency ω1: the coordinates of each trial xi in the two first PCA directions are cos(τiω1 +
φ)|x(ω1)| and sin(τiω1 + φ)|x(ω1)|, where φ is the phase of x(ω1). The data thus organize along
a one-dimensional manifold parameterized by latency τi.
Time (ms)
Tria
l
100 200 300 400 500
100
200
300
400
500
(a) Original raster plot (a setof jittered Gabor wavelets)
−40 −20 0 20−30
−20
−10
0
10
20
(b) 2D PCA projection of theset of translated time-courses
−40
−20
0
20
−50
0
50−20
0
20
40
(c) 3D PCA projection of the set oftranslated time-courses
Figure 7.2: PCA analysis of a set of 500 jittered time series of 512 time samples.
To illustrate this, a set of 500 jittered time series, each with 512 time samples is displayed
as a raster plot in Figure 7.2(a). The projections of the data in the leading two and three
PCA dimensions, in Figure 7.2, cluster along curves, indicating the 1D structure of the data.
Reordering the time-courses with respect to latency is equivalent to finding a parameteriza-
212 CHAPTER 7. SINGLE-TRIAL ANALYSIS WITH GRAPHS
tion of the curves in Figure 7.2(b) and 7.2(c). In the next subsections, we present methods to
automatically reorder time series according to their 1D variability.
7.2.2 Nonlinear embedding
X
M
. ... .. . .. ..
. . .. xi.. .. .R
n
...
.
.
..
..
..
.. .
......i
f(x )
f
Figure 7.3: Non-linear embedding into a low-dimensional Euclidian space
Given the distance dX and the points (xi), the aim is to recover the structure of M via
an embedding function f that maps the (xi) into a low-dimensional Euclidian space Rn (cf.
figure 7.3). The embedding f provides a low-dimensional representation of the dataset and
also a parameterization of the manifold. When M has a 1-d structure, the first coordinate
of f can be used to order the points (i.e., in our context, the time series), provided that the
function f satisfies a “regularity” constraint: if two points x and z are close in M, then so
must f(x) and f(z) in Rn. This is sometimes referred to as a minimal distortion property.
For n = 1, a Taylor expansion provides the following inequality [14]:
|f(z)− f(x)| ≤ dM(x, z)‖∇f(x)‖+ o(dM(x, z)) (7.1)
where ∇f stands for the gradient of f and dM is the geodesic distance on the manifold be-
tween points x and z. The notation g(z) = o(dM(x, z)) means thatg(z)
dM(x,z) tends to 0 as z tends
towards x. The inequality in equation (7.1) means that dM(x, z)‖∇f(x)‖ is a good first order
upper bound for |f(z)− f(x)|.In order to obtain an embedding that satisfies the “regularity” constraint, Laplacian-based
methods try to control the smoothness of f globally by minimising
∫
M||∇f(x)||2p(x)sdx ,
provided that∫
M||f(x)||2p(x)sdx = 1 (i).
The latter condition removes the scaling indetermination for the function f . The parameter
s controls the influence of the probability density on the solution. If s is strictly positive, the
regularity of f will be more constrained in high density regions.
The optimization problem under constraint (i) can be formulated as a saddle-point problem
for the Lagrangian L(f, λ):
L(f, λ) =
∫
M||∇f(x)||2p(x)sdx+ λ
(
1−∫
M||f(x)||2p(x)sdx
)
. (7.2)
Introducing the s-weighted Laplacian operator defined as ∆sf , − 1ps div(ps∇f) and 〈f, g〉M ,
213
∫
M 〈f(x), g(x)〉p(x)sdx, L(f, λ) defined in equation (7.2) can be rewritten:
L(f, λ) =
∫
M||∇f(x)||2p(x)sdx+ λ
(
1−∫
M||f(x)||2p(x)sdx
)
=
∫
M〈∇f(x),∇f(x)〉p(x)sdx+ λ
(
1−∫
M〈f(x), f(x)〉p(x)sdx
)
=
∫
M〈p(x)s∇f(x),∇f(x)〉dx+ λ
(
1−∫
M〈f(x), f(x)〉p(x)sdx
)
=
∫
M−〈div(p(x)s∇f(x)), f(x)〉dx+ λ
(
1−∫
M〈f(x), f(x)〉p(x)sdx
)(*)
=
∫
M〈∆sf(x), f(x)〉p(x)sdx+ λ
(
1−∫
M〈f(x), f(x)〉p(x)sdx
)
= 〈∆sf, f〉M + λ (1− 〈f, f〉M)
Step (*) comes from the fact that the gradient operator is the negative adjoint of the diver-
gence.
Differentiating L(f, λ) with respect to f leads to:
∂L(f, λ)
∂f= ∆sf − λf
Therefore setting this derivative to zero imposes f to satisfy:
∆sf = λf
thus f to be an eigenvector of the operator ∆s associated to the eigenvalue λ.
Notice that if ∆sfi = λifi,
L(fi, λi) = λi
This implies that optimizing the constrained problem whose Lagrangian is given by L(f, λ)
requires to find the eigenvectors of ∆s with the smallest eigenvalues.
The constant function fcst equal to 1 everywhere is an eigenfunction of ∆s for the eigen-
value 0. To avoid this trivial embedding, the solution f is constrained to be orthogonal to the
function fcst, i.e.,∫
M f(x)p(x)sdx = 0 (ii).
The optimal embedding f under constraints (i) and (ii) is the eigenfunction of ∆s corre-
sponding to the smallest non-zero eigenvalue.
In order to compute f from a manifold sampled with a limited number of points, the
operator ∆s needs to be approximated. Graph Laplacian methods approximate ∆s by the
Laplacian of a particularly designed graph. Let G = (V, E) be an undirected graph where
V are the nodes (xi)i=1,...,N and E are the edges. A weight wij is associated to every edge
(i, j) ∈ E , leading to a weighted graph G. The Laplacian L of the graph is a matrix defined by
L = D −W where W = (wij)ij and D is diagonal with Dii =∑
j wij .
The random-walk Laplacian is a normalized version of L defined by Lrw = D−1L = I −D−1W where I is the identity.
A weighting matrix W yielding a good approximation of ∆s must now be defined. This is
done with the help of a similarity measure k which is a non-increasing function of the distance
dX . Here, k is a Gaussian kernel with standard deviation σ:
k(xi, xj) = e−dX (xi,xj)2
σ2 .
Results of [107] show that, for a given k, the random-walk Laplacian of G converges almost
214 CHAPTER 7. SINGLE-TRIAL ANALYSIS WITH GRAPHS
surely to ∆s when the sample size N goes to infinity, if
wij =k(xi, xj)
(d(xi)d(xj))1−s/2(7.3)
where d(x) =∑N
i=1 k(x, xi) is an estimator of the probability density function over M, The
embedding function can therefore be obtained by computing the eigenvectors fq of Lrw satis-
fying:
(I −D−1W )fq = λqfq
(D −W )fq = λqDfq
Lfq = λqDfq
(7.4)
It can easily be proved that 0 is a trivial eigenvalue and also that L is a symmetric positive
matrix, which implies, since d(xi) ≥ 0 for all i, that the generalized eigenvalues λq are all
positive and can be ordered:
0 = λ0 ≤ λ1 ≤ · · · ≤ λq ≤ λq+1 ≤ · · · ≤ λN−1.
The embedding f into Rn is then given by:
f(xi) = (f1(i), f2(i), ..., fn(i)).
where fq(i) is the ith component of fq.
7.2.3 Laplacian embedding algorithm
Observe that, in the case of a uniform sampling over the manifold, the parameter s has no
influence. Numerical experiments on synthetic and real EEG data showed that s did not have
much influence on the results, and as a consequence s was set to 1. This corresponds to the
Diffusion Map algorithm [37]. The following algorithm details the different steps to compute.
Laplacian-based n-dimensional embedding:
• Set dX , σ and n the dimension of the embedding.
• Compute K with K(i, j) = e−dX (xi,xj)
σ2 .
• Compute DK with DK(i, i) =∑N
j=1K(i, j) and DK(i, j) = 0 if i 6= j.
• Compute W = D−1/2K KD
−1/2K (equation (7.3) with s = 1)
• Compute D with D(i, i) =∑N
j=1W (i, j) and D(i, j) = 0 if i 6= j.
• Find the n + 1 first generalized eigenvectors fk solution of (D − W )fk = λkDfk,
k = 0, . . . , n.
• The coordinates of point xi in Rn are (f1(i), f2(i), . . . , fn(i)).
Since L is symmetric, its first eigenvectors can be computed efficiently with an iterative
method, for example an Implicitly Restarted Arnoldi Method [137]. For computational effi-
ciency, it is possible to set to 0 the wij below a threshold, leading to sparse matrices, and re-
ducing the computational cost of matrix-vector multiplications at each iteration. Note should
be taken that too high a threshold leads to a very coarse approximation of the solution.
215
In order to provide more insight on the discrete solution, it can be observed from equa-
tion (7.4) that the solution f1 also solves the following optimization problem:
arg minfT Df=1, fT D1=0
fTLf
and that expanding fTLf gives:
fTLf =1
2
∑
(i,j)∈Nwij(f(i)− f(j))2
Since each term of the sum is positive, minimizing fTLf requires to minimize each wij(f(i)−f(j))2 which implies that if wij is big, i.e., dX (xi, xj) is small, (f(i) − f(j))2 should be small.
This can be directly related to the “regularity” constraint mentioned above. Although this
comment helps to bridge the gap between the discrete and the continuous formulations of the
problem, it can be noticed that, contrary to the continuous formulation, the discrete problem
does not provide much insight into the influence of the density p.
Manifold learning methods are “data driven”. They capture the structure of the dataset,
provided that the chosen distance dX is appropriate. When dealing with time series, many
distance functions can be used. In practice, dX does not need to be an actual distance, and
instead it can measure the difference between features of interest for two elements of the
dataset. One may even design dX to be blind to some features of the data, which are irrelevant
for the application at hand.
7.3 SPECTRAL REORDERING OF EEG TIMES SERIES
In this context, each xi is a time series, and X = RT where T is the number of time
samples. The same stimulus has been delivered N times leading to N time series.
7.3.1 Toy examples
Once computed, the Laplacian embedding f provides a parameterization of the manifoldM,
which means that ifM has a noisy 1D structure, the first coordinate, f1, orders the elements
along the manifold. This is now illustrated with the noiseless synthetic dataset already pre-
sented in Section 7.2.1. It is simulated with T = 512 and N = 500 points. The embedding
was performed using the Euclidian distance and a Gaussian kernel. The embedded points
are represented in figure 7.4(c). It can be observed that the embedding unfolds the mani-
fold structure. The ordering provided by f1 can be encoded with a color, hence each point
of the PCA representation can be colored. The 2D and 3D PCA point clouds are presented
in figure 7.4(a) and figure 7.4(b): observe that the color changes continuously along the one
dimensional structure. Figure 7.4(d) presents the reordering of the raster plot already pre-
sented in figure 7.2(a).
To illustrate that the manifold learning method can capture more than a variability in
latency, a sample dataset with a variability in latency and scale has been designed (time series
are “stretched”). The reordered dataset and the 3D PCA colored embedding are presented in
figure 7.5.
The EEGLAB toolbox offers an alternative way of ordering signals, based on the phase
of a time-frequency decomposition using Gabor wavelets. The user is asked to provide the
latency of the response, the number of oscillations and the frequency of interest (the frequency
216 CHAPTER 7. SINGLE-TRIAL ANALYSIS WITH GRAPHS
(a) Points of the 2D PCA pro-jection, colored with respect totheir first Laplacian embed-ding coordinate
(b) Points of the 3D PCA projection,colored with respect to their firstLaplacian embedding coordinate
(c) Embedded points colored with re-spect to the first Laplacian embed-ding coordinate.
Time (ms)
Tria
l
100 200 300 400 500
100
200
300
400
500
(d) Reordered raster plot
Figure 7.4: Illustration of manifold learning using graph Laplacian on the synthetic dataset
of Figure 7.2.
(a) Points of the 3D PCA pro-jection, colored with respect totheir first Laplacian embed-ding coordinate
(b) Embedded points colored withrespect to the first Laplacian embed-ding coordinate.
Time (ms)
Tria
l
100 200 300 400 500
100
200
300
400
500
(c) Reordered raster plot
Figure 7.5: Illustration of manifold learning using graph Laplacian on a synthetic dataset
with latency and scale variability (time series are “stretched” in time with the increase of the
latency).
217
Time (ms)T
ria
l
100 200 300 400 500
100
200
300
400
500
(a) Phase reordering on thedataset in figure 7.2
Time (ms)
Tria
l
100 200 300 400 500
100
200
300
400
500
(b) Phase reordering on thedataset in figure 7.5
Figure 7.6: Reordering results obtained on the datasets in figure 7.2 and figure 7.5 with the
EEGLAB reordering method based on the phase of a Gabor wavelet, with a priori defined
latency and frequency.
can also be automatically set as maximizing the Fourier spectrum). Once these parameters
are set, the phase for the corresponding Gabor function is computed and the raster plot is
reordered accordingly. The two datasets presented in figure 7.2 and figure 7.5 are shown,
reordered with EEGLAB, in figure 7.6. It can be noticed that the method fails to accurately
reorder the raster plots, for different reasons in the two examples. In the first one, the Gabor
waveform that has been translated has multiple cycles which implies that the phase in [0, 2π]
cannot capture the variability. In the second one, due to the temporal scaling, the frequency
slightly varies across trials. The EEGLAB procedure assumes a constant frequency across
trials, and estimating the phase when the frequency is not well defined leads to errors.
7.3.2 Spectral reordering with realistic time series
The manifold learning method has just been illustrated on synthetic toy examples. We now
apply it to a more challenging synthetic problem where the time series are corrupted by an
additive noise, and to real ERP recordings.
As a preprocessing, the time series are centered and normalized so that ‖xi‖ = 1. After
this normalization, the half Euclidian distance is given by ‖xi − xj‖2/2 =√
(1− 〈xi, xj〉)/2,
which implies that it is equivalent to considering the correlation between time series.
As mentioned earlier dX may not be a real distance but just a similarity measure. To
investigate the use of different similarity measures, we propose to use:
dX ;r(xi, xj) =
(
1− 〈xi, xj〉2
)r/2
(7.5)
where r is a power exponent applied to the classical Euclidian distance.
Such a similarity measure is sensitive to time shifting, and is therefore appropriate for
capturing the latency information. It also has the advantage that dX ;r(xi, xj) ∈ [0, 1] which
constrains the choice of σ (see Section 7.5).
The parameters used by the spectral embedding are s (fixed to 1 in practice), r and σ.
Section 7.5.1 provides a strategy to set these parameters.
The synthetic data consists in 100 time series (N = 100 and T = 500) computed from a
template (cf. figure 7.11(e), green curve) by translating the positive deflection with a random
time lag (with a Gaussian probability distribution of standard deviation σlag), and adding
noise with a SNR measured in variance ratio (variance of signal divided by variance of the
noise). Figure 7.7(a), presents a dataset with σlag = 50 ms and an additive autoregressive (AR)
noise (SNR=1.5=1.76dB). The AR model of order 8 was fitted on spontaneous EEG activity.
The second application of the method is to auditory oddball EEG data. This paradigm
218 CHAPTER 7. SINGLE-TRIAL ANALYSIS WITH GRAPHS
consists of alternating frequent tones and rare (“target”) tones. It is known to elicit a positive
EEG deflection to the rare tones, referred to as the “P300” or “P3” wave, more prominent on
the midline electrodes and occurring at a latency around 300 ms [17]. The data is recorded
from the central electrode Cz (cf. figure 1.22 in chapter 1), sampled at a rate of 256 Hz and
processed with a high-pass filter at 0.5 Hz (Butterworth Zero Phase Filter, Time constant
0.3183 s, 12 dB/oct) and a low-pass filter at 8 Hz (Butterworth Zero Phase Filter, 48 dB/oct).
The positive deflection of the P300 wave, in the 3-5 Hz range, is preserved. Figures 7.7(a) and
7.8(a) present raster plots of both data sets. The random nature of the time latency of the
P300, as first observed in [130], is obvious in the oddball raster plot.
Time (ms)
Tria
l
100 200 300 400 500
20
40
60
80
100
(a) Raster plot of raw time series.
0 100 200 300 400 500
−0.5
0
0.5
1
Time (ms)
(b) Two time series belonging to thedataset.
−0.1 −0.05 0 0.05 0.1−0.2
−0.15
−0.1
−0.05
0
0.05
f 1
f2
(c) 2D embedding of the time series.
Time (ms)
Tria
l
100 200 300 400 500
20
40
60
80
100
(d) Reordered raster plot of time se-ries using of the first coordinate ofthe embedding f .
Figure 7.7: Spectral reordering results on synthetic data (σlag = 50 ms). Embedding was
performed with r = 2 and σ = 0.1.
In both cases, the time series were embedded into a two dimensional space (cf. figure 7.7(c)
and figure 7.8(c)). In both figures, it can be noticed that the points are clustered along an
elongated 1D structure, as it was the case with the toy example in figure 7.4. The first
coordinate can therefore be used to correctly parameterize the manifold and to order the
time series. By observing the reordered raster plots in figure 7.7(d) and figure 7.8(d), and
comparing them to a raster plot reordered using the externally measured motor response, it
appears that the first coordinate of f has correctly captured the latency information.
7.4 ROBUST LATENCY ESTIMATION VIA DISCRETE OPTI-MIZATION
Once the raster plot has been correctly reordered, estimating the latency of the response
consists in tracing a non decreasing line through the extrema of the raster plot, like the dark
semi-vertical line in figure 7.1(b). Though this may appear a simple task, it is non trivial to
219
Time (ms)
Trial
−7 187 382 578 773
50
100
150
200
(a) Raster plot of raw time series
0 200 400 600
−10
−5
0
5
10
Time (ms)
(b) Example of time series belongingto the dataset.
−0.15 −0.1 −0.05 0 0.05 0.1−0.25
−0.2
−0.15
−0.1
−0.05
0
0.05
0.1
f 1
f2
(c) 2D embedding of the time series.
Time (ms)
Trial
−7 187 382 578 773
50
100
150
200
(d) Reordered raster plot of time se-ries using the first coordinate of theembedding f .
Figure 7.8: Spectral reordering results on EEG oddball time series. The solid vertical line
in the raster plots corresponds to the stimulus onset. The vertical dashed lines provide the
limits on the time window used to reorder the data. Embedding was performed with r = 2and σ = 0.05.
220 CHAPTER 7. SINGLE-TRIAL ANALYSIS WITH GRAPHS
automatize.
In order to automatically extract the latency corresponding to each time series from a
reordered plot, one should exploit the global information in the data and take into account the
a priori information that latencies are monotonically related in the reordered trials. Finding
an increasing function such as the one defined by the dark semi-vertical line in figure 7.1(b),
is equivalent to partitioning the raster plot in two, i.e., achieving a two class segmentation of
the ERP image, while taking into account the monotony constraint (cf. figure 7.9(b)).
7.4.1 Optimization framework
After reordering, it can be assumed that the latencies of brain responses of two neighboring
trials xi and xi+1 are close. Let us denote li the latency of the response for trial i. With the
Markov Random Field (MRF) optimization framework pioneered in [84], the robust estima-
tion of the (li)i=1,...,N amounts to solving:
(li)∗i=1,...,N = arg min
(li)i=1,...,N
E(li) with E(li) =
N∑
i=1
Di(li) + α
N−1∑
i=1
Vi(li, li+1) (7.6)
Each term Di(li), usually called a data term, tends to set li to the best latency value for
time series xi, regardless of the other series. When considering EEG evoked potentials, high
deflections of the signal from the baseline are of particular interest. Thus a reasonable choice
is to set li so that xi(li) is maximal (or minimal). In our optimization formulation, this leads
to Di(li) = φ(xi(li)), where φ is a positive decreasing function1. In practice, when looking for
positive deflections φ is defined by φ(xi(j)) = M − xi(j) ≥ 0 where M = maxi,jxi(j).
Since single-trial EEG recordings are heavily corrupted by noise, robust estimation of
lags is obtained by adding a second term to the energy E. This has been made possible by the
spectral reordering, that guarantees that for any i, the time series xi and xi+1 are similar, and
thus, neighboring latencies l∗i and l∗i+1 should be close. The smoothing term Vi(li, li+1) must
increase when the distance between li and li+1 increases. Without any prior information, it is
natural to set all the Vi to a unique function V that does not depend on i.
In our case, we choose V (li, li+1) = |li+1−li|while strongly imposing li ≤ li+1,∀i. Parameter
α penalizes the difference of latencies between two neighboring time series.
7.4.2 Graph Cuts algorithm
Time (ms)
Trial
−7 187 382 578 773
50
100
150
200
(a) Automatic lag extraction result
50
100
150
200
(b) Corresponding binary seg-mentation
Figure 7.9: Result of binary partitioning using the graph cut algorithm applied on the raster
plot reordered with motor response (cf. figure 7.1(b))
1Di(li) should be positive for technical reasons imposed by the Graph Cuts method – see Section 7.4.2.
221
The unknown variables li take their values on the same time samples as the time series
xi, and can thus only take a finite number of discrete values. This implies that the problem
necessarily has a solution, which may however not be unique. In order to find a fast solu-
tion of such discrete combinatorial optimization problems, graph-based approaches are being
extensively used by the Computer Vision community [23].
MRF (cf. equation (7.6)) optimization with graph cuts is applicable depending both on the
discrete set of possible values for li and on the choice of the smoothing terms V . They can
yield global or local minima. When li can take only two values, i.e., labels, global optimal-
ity is guaranteed [96]. With multiple labels, global optimality is possible when the li can be
linearly ordered [23]. This is the case with the one-dimensional problem addressed in this
contribution. The optimal solution is here obtained by computing only one minimal cut of a
specially designed graph. More generally graph cuts methods are adapted for MRF optimiza-
tion when the smoothing terms lead to a sub-modular energy [127]. See Appendix B for a
short introduction to graph cuts.
To construct the graph in our case, it can be observed that a set of latencies satisfying the
constraints can be defined by partitioning the raster plot into two classes. This is illustrated
in figure 7.9(b) where the optimization has been performed on the raster plot reordered by
the motor response figure 7.9(a). The border between the two regions provides an estimation
of the latency. Our problem thus amounts to designing a weighted graph that provides a
natural correspondence between the partitioning and the energy detailed in equation (7.6).
With the previously defined V , a graph can be constructed as in figure 7.10. Each node of
the graph ni,j is indexed by trial number and by time, i.e., a line index i and a column index j,
except for the terminal nodes (“source” S and “sink” T ). Cutting the graph in two consists in
separating the “source” from the “sink”. Edge weights are detailed in table 7.1. Edges with∞weights are used to guarantee the constraints: horizontal ∞ weighted edges guarantee that
the cut goes through each line once, i.e., li is unique for each time series, while vertical ∞weighted edges guarantee that the solution is increasing.
One can notice that the cost associated to a cut, defined as the sum of the edge weights
along the path of the cut, is equal to an energy value. The minimum cut thus provides the op-
timum. The solution is obtained by a single binary cut, which makes the algorithm extremely
fast (a few milliseconds) and also globally optimal. Graph partitioning via minimum cut is, in
turn, known to be equivalent to a polynomial problem: the max flow problem [73]. We solve
the minimum cut problem with the max flow algorithm described in [22]2. The complexity
observed in practice is linear in the number of nodes in the graph, i.e., O(NT ).
The spectral reordering may provide a raster plot ordered from large latencies to small
latencies, in which case one should seek a non increasing partioning of the raster plot. This
is why, in practice, the graph cut algorithm is run twice on each dataset, once with the order
provided by the manifold learning algorithm and once with this order inverted. The order
that leads to the smallest energy is kept.
7.4.3 Result of single-trial latency extraction
The optimization procedure previously described was applied to the reordered datasets dis-
played in figure 7.7(d) and figure 7.8(d). Results are presented in figure 7.11(a) and fig-
ure 7.11(b). Such a lag extraction technique would have been inapplicable on the unordered
time series displayed in the raster plots of figure 7.7(a) and figure 7.8(a), and is only made
possible after reordering by the non linear embedding.
Figures 7.11(c) and 7.11(d) present the synthetic and the oddball datasets after realign-
ment, i.e., after raster plot reordering and lag correction. The evoked potentials were com-
puted by standard averaging with and without latency correction cf. figure 7.11(e) and fig-
2using the open source implementation http://www.adastral.ucl.ac.uk/˜vladkolm/software.html
222 CHAPTER 7. SINGLE-TRIAL ANALYSIS WITH GRAPHS
SSource
n11
n21
n31
n12
n22
n32
n13
n23
n33
n14
n24
n34
n15
n25
n35
T Sink
∞
∞
∞
∞
∞
∞
∞α
∞α
∞α
∞α
∞α
∞α
∞α
∞α
∞α
∞α
D1(1)
∞
D2(1)
∞
D3(1)
∞
D1(2)
∞
D2(2)
∞
D3(2)
∞
D1(3)
∞
D2(3)
∞
D3(3)
∞
D1(4)
∞
D2(4)
∞
D3(4)
∞
Figure 7.10: Graph illustration for an image N × T (N = 3 time series of length T = 4) with
an example of minimal cut in red. Time instants index the horizontal edges. In blue are the
nodes linked to the “source” node and in gray the nodes linked to the “sink” node after cutting
the graph. The corresponding energy is the sum of the weights on the edges (in red) crossed
traction on reordered raster plot of synthetic time series (a) and oddball time series (b). Lags
correction on raster plot of synthetic time series (c) and oddball time series (d). (e) evoked po-
tential computed after reordering and lags correction on synthetic dataset (the green curve is
the template used to generate the data, the red curve is the evoked potential without reorder-
ing and the blue curve is the evoked potential after reordering). We observe a good fit between
the template and the evoked potential after reordering. (f) evoked potential computed after
reordering and lags correction on the auditory oddball dataset (the red curve is the evoked
potential without reordering and the blue curve is the evoked potential after reordering).
7.5 PARAMETER ESTIMATION AND ROBUSTNESS
The output of the spectral reordering depends on the definition of the distance dX and on
224 CHAPTER 7. SINGLE-TRIAL ANALYSIS WITH GRAPHS
the σ used in the Gaussian kernel. It also depends on the parameter s used to weight the
Laplacian with the density, although our experience is that the influence of s on the results
is negligible. In order to limit the computation time of the following procedure, s was set to
1 by default, corresponding to the Diffusion Map algorithm proposed in [37]. According to
our definition (7.5), dX depends on the exponent r. Once the reordering is done, the graph
cut method requires the setting of α used to control the regularity of the cut. We propose an
automatic way to set these parameters. The robustness of the procedure is evaluated with
numerical simulations.
7.5.1 Parameter estimation
The spectral reordering is good if it succeeds in exhibiting the monotonic structure observed
for example in figure 7.11(a) and figure 7.11(b). With a proper reordering, the graph cut pro-
cedure, constrained to provide non decreasing cuts, reaches lower energy levels. This obser-
vation motivates the following strategy to estimate the parameters of the spectral reordering.
Estimating the parameters of the spectral reordering
Let us denote the parameters of the spectral reordering θ = (r, σ) and E∗α(θ) the value of the
energy reached by the graph cut algorithm with α fixed. The best θ∗ is simply obtained by:
θ∗ = arg minθ
E∗α(θ) .
In practice, the xi are normalized and dX ;r is designed to take its values in [0, 1]. For r and
s fixed, the tested values for σ were (0.1 + 0.1k)k with k = 1, . . . , 20. Using a fixed range of
values for σ is made possible by the constraint dX ;r ∈ [0, 1].
For a standard M/EEG dataset with a few hundreds of trials, testing all the different sets
of θ takes a few seconds. Lag extraction results for the oddball data are presented in fig-
ure 7.13 for different values of α. For α set to 0.1, Eα is displayed in figure 7.12 as a function
of σ with r = 1 and r = 2. The optimum is reached for r = 2 and α = 0.05.
0 0.05 0.1 0.15 0.234
36
38
40
42
44
46
Sigma
E lags
r=1r=2
Figure 7.12: E∗α as a function of r and σ. Computation is done on the oddball dataset with
α = 0.1.
Estimating α
For α fixed, the above procedure provides a way to estimate the parameters of the spectral
reordering. In order to find the α, we propose a method based on K-fold cross-validation.
In figure 7.11(e), it can be observed that the new evoked potential obtained after lag cor-
rection matches the template used in the simulation. This validates the procedure in the
225
Time (ms)
Trial
−7 187 382 578 773
50
100
150
200
(a) α = 0.01, σ = 0.05
Time (ms)
Trial
−7 187 382 578 773
50
100
150
200
(b) α = 0.1, σ = 0.05
Time (ms)
Trial
−7 187 382 578 773
50
100
150
200
(c) α = 1, σ = 0.15
Figure 7.13: Reordered raster plots with lags estimate for different values of α. The σ are
automatically defined for α fixed as described in section 7.5.1. Results are obtained with
r = 2.
synthetic case. Unfortunately, in practice there is no “ground truth”. To circumvent this ob-
stacle, the proposed strategy consists in estimating the new evoked potential e(t) on a portion
of the data, called the learning set, and checking if it correlates well with the rest of the data
called the test set. The dataset is in practice partitioned into K disjoint subsets (Ck)k. For
k ∈ [1,K], the test set is the kth subset and the learning set is the rest. With K = 10, the
evoked potential ek(t) is estimated on 90% of the data and tested on the remaining 10%.
Since the lag is unknown for the test set, the correlation between the evoked potential
and each time series is given by the maximum of the cross correlation with arbitrary lag. The
score of generalization Sk is given by:
Sk =∑
i∈Ck
maxτ〈ek(·), xi(· − τ)〉
The procedure is run K times leading to a score S =∑
k Sk.
The α that achieves the maximum score S is selected.
In practice the tested values for α were 0, 0.001, 0.01 and 0.1. As expected, with no noise,
the procedure automatically sets α to 0. On the oddball EEG dataset, the procedure leads
to α = 0.1. This also confirms that using smoothing terms Vi(li, li+1) is mandatory in the
presence noise to obtain a proper estimate of the latencies and subsequently of the evoked
response.
With 4 values for α and 10 values for σ, the computation time on the oddball dataset is
around 5 minutes per value of r. About half of the time is spent in the computation of the
eigenvectors. Note that for each set of parameters the computation is independent. This
allows easy parallel computing which could speed up the parameter estimation procedure.
7.5.2 Validation
In order to validate the procedure and also investigate its robustness to noise, various numer-
ical experiments were performed on simulated datasets. For each dataset, the parameters
were automatically set using the procedure described above. The resulting evoked potential
is denoted e∗(t). Assuming the latencies are known, by realigning the data and averaging,
one obtains the best evoked potential given the SNR of the dataset. This evoked potential,
obtained with the known real latencies, is denoted erl(t). The real template used to generate
the synthetic dataset is denoted eref (t). The error on the solution was then computed as:
Error =
∣
∣
∣
∣
〈 e∗
‖e∗‖ ,eref
‖eref‖〉 − 〈 erl
‖erl‖,eref
‖eref‖〉∣
∣
∣
∣
∈ [0, 2] (7.7)
226 CHAPTER 7. SINGLE-TRIAL ANALYSIS WITH GRAPHS
The smaller the error, the better the estimation of the lags and of the evoked potential.
Results with 3 different templates (cf. figure 7.14(a)) are presented in figure 7.14(b). Sim-
ulations were performed with two different types of noise (uncorrelated white noise and noise
computed with an autoregressive filter whose coefficients were fitted on spontaneous EEG
activity.
First it can be observed that in the noiseless case, the error is equal to 0. This validates
the method and demonstrates that the estimation procedure is unbiased. Second it can be
noticed that the best performance is obtained with uncorrelated white noise and that the
errors remain small even with low SNR.
0 100 200 300 400 500−0.2
−0.1
0
0.1
0.2
Time (ms)
Template 1
Template 2
Template 3
(a) The 3 different templates used in thesimulations.
−2 0 2 4 60
0.02
0.04
0.06
0.08
SNR (dB)
Err
or
AR noise
White noise
(b) Errors (see (7.7)) obtained with the 3templates in (a) and 2 different types ofnoise (uncorrelated white noise and noisecomputed with an eighth-order autore-gressive filter whose coefficients were fit-ted on spontaneous EEG activity).
Figure 7.14: Simulation results and errors estimates with different types of evoked responses
(σlag = 50 ms). These values are the average errors obtained out of 10 repetitions of the
experiment. It can be observed that errors are equal to 0 with no noise. The worst error 0.1
is observed with SNR = −3dB on the template used in previous simulations. A correlation
error below 0.1 can be considered as satisfactory.
7.6 DISCUSSION
By making use of advanced graph-based methods, we have proposed a robust and very fast
two-step procedure for estimating the variability of evoked neural responses on single-trial
MEG or EEG data: spectral reordering by Laplacian embedding followed by an estimation
procedure by graph cuts. The whole process runs in a few seconds on real datasets of several
hundred trials covering several hundred time points. The approach is a model-free, “data
driven” algorithm, offering guarantees of global optimality for both of the steps. It does not
suffer from initialization problems and does not assume a model, e.g., imposing that the data
can be well represented in an a priori dictionary of waveforms [16]. The procedure has several
parameters that can be set automatically, as explained. Finally, numerical experiments on
synthetic datasets confirm the robustness to noise of the full procedure.
This contribution puts an emphasis on the latency estimation problem. However, as il-
lustrated in figure 7.5(c), the manifold learning method can handle other types of variability,
for whose estimation a graph cut procedure could be designed. As long as the model of a
1D manifold holds, i.e., that the variability can be parameterized by a single parameter, the
methodology of this chapter can be applied.
227
Quantifying the variability of brain response delays, non invasively, in humans, can help
to improve our understanding of the cognitive treatment of information. As presented in this
chapter, once the delay has been corrected on each trial, the data can be realigned to the
neural response, improving the quality of the estimated evoked response. A better control of
the temporal aspects of the signal can moreover improve the spatial precision of the source
reconstructions when solving the inverse problem [11].
Another application of single-trial estimation is related to the correlations between dif-
ferent channels, coming from the correlations between latencies of different functional brain
regions. By computing delays of response independently on different channels, interactions
between delays can be investigated. We can hypothesize that two neural processes that are
independent exhibit uncorrelated delays while two sequential neural tasks have highly cor-
related delays. To quantify such interactions between various brain functional areas, simple
correlations can be computed. It would be interesting to investigate what difference the delay
correction, based on the data of one channel, makes to the amplitude of evoked potential on
other channels. An augmentation of this amplitude on an other channel could be interpreted
as a strong correlation between them, while an absence or very weak modification of this
amplitude would suggest on the contrary an absence of correlation or a very weak one. Such
questions, raised in the cognitive neurosciences community, can be addressed with the help
of the method proposed in this chapter.
Although our method was applied to time series coming from a single EEG channel, it
is also possible to estimate the delay of activations of an ICA component or more generally
any source configuration. By employing signal-space projectors [208], or by projecting the full
EEG or MEG recordings onto the forward field of a source configuration, we obtain a time
series for each repetition of the experiment, which is what our algorithm requires as input.
The method detailed in this chapter has been implemented as an EEGLAB plugin. The
code snippet in table 7.2 details how the plugin can be used from a Matlab script.
228 CHAPTER 7. SINGLE-TRIAL ANALYSIS WITH GRAPHS
1 % Example Matlab code for single-trial latency estimation
