J.Y. Tourneret
Post-nonlinear Mixing Models for the Analysis of
Hyperspectral Images
JEAN-YVES TOURNERET
Institut de recherche en informatique de Toulouse (IRIT)
University of Toulouse, France
Joint work with
YOANN ALTMANN AND NICOLAS DOBIGEON
Grenoble 2013 – p. 1/45
J.Y. Tourneret
Outline
Part 1: Linear/ Nonlinear unmixing
Part 2: Polynomial Post-Nonlinear Mixing Model
(PPNMM) for spectral unmixing
Part 3: Nonlinearity detection
Grenoble 2013 – p. 2/45
J.Y. Tourneret
Part 1: Linear/ Nonlinear Unmixing
Hyperspectral Imagery
Grenoble 2013 – p. 3/45
J.Y. Tourneret
Linear Unmixing
Linear mixing model
Reference: Keshava and Mustard, “Spectral unmixing", IEEE Signal Proc.
Magazine, Jan. 2002.Grenoble 2013 – p. 4/45
J.Y. Tourneret
Assumptions for Linear Unmixing
Pure materials sitting side-by-side in the scene
Observation: sum of individual contributions associated
with each component
Single path for the different photons
Valid as a first approximation
Grenoble 2013 – p. 5/45
J.Y. Tourneret
Linear mixing model
y =
R∑
r=1
armr + n = Ma+ n
a = [a1, . . . , aR]T ,M = [m1, ...,mR]
y observed pixel in L bands
R number of pure materials or endmembers (most
spectrally pure vectors)
mr spectrum of the rth endmember
ar abundance of the rth endmember in the pixel
n noise vector n ∼ N (0L, σ2IL)
Grenoble 2013 – p. 6/45
J.Y. Tourneret
Linear mixing model
Physical constraints
Positivity : ar ≥ 0,∀r ∈ 1, ..., R.
Sum-to-one :R∑
r=1
ar = 1.
Grenoble 2013 – p. 7/45
J.Y. Tourneret
Linear mixing model
Unmixing stepsa
Endmember extraction: estimating the spectral
signatures, or endmembers, ( N-FINDRb, VCAc...)
Inversion: estimating the abundances (FCLSd, Bayesian
algo.e...)
aBioucas-Dias et al., “Hyperspectral unmixing overview: geometrical, statistical, and sparse regression-based
approaches,” in IEEE JSTARS, April 2012.
bWinter, “An algorithm for fast autonomous spectral end-member determination in hyperspectral data ,” in
Proc. SPIE, 1999.c
Nascimento et al., “Vertex Component Analysis: A fast algorithm to unmix hyperspectral data,” IEEE TGRS,
Apr. 2005.
dHeinz et al., “Fully constrained least-squares linear spectral mixture analysis method for material quantifica-
tion in hyperspectral imagery,” IEEE TGRS, Mar. 2001.e
Dobigeon et al., “Semi-supervised linear spectral unmixing using a hierarchical Bayesian model for hyper-
spectral imagery,” IEEE Trans. Signal Process., Jul. 2008.
Grenoble 2013 – p. 8/45
J.Y. Tourneret
Linear mixing model
Unmixing stepsa
Joint estimation of endmembers and abundances
(DECAb, BLUc...)
aBioucas-Dias et al., “Hyperspectral unmixing overview: geometrical, statistical, and sparse regression-based
approaches,” in IEEE JSTARS, April 2012.
bNascimento et al., “Dependent Component Analysis: A Hyperspectral Unmixing Algorithm,” in Proc IGARSS.,
Jul. 2007.c
Dobigeon et al., “Joint Bayesian endmember extraction and linear unmixing for hyperspectral imagery,” IEEE
Trans. Sig. Process., Nov. 2009.
Grenoble 2013 – p. 9/45
J.Y. Tourneret
Nonlinear unmixing
Keshava and Mustard, “Spectral unmixing", IEEE Sig. Proc. Mag., 2002.
Grenoble 2013 – p. 10/45
J.Y. Tourneret
The Generalized Bilinear model
y =R∑
r=1
armr +R−1∑
i=1
R∑
j=i+1
aiajγi,jmi ⊙mj + n
A. Halimi et al, “Nonlinear unmixing of hyperspectral images using a
generalized bilinear model", IEEE Trans. Geo. and Remote Sens., 2011.
Grenoble 2013 – p. 11/45
J.Y. Tourneret
Nonlinear unmixing
Nonlinear mixing models
Possible interactions between the components of the
scene
Nonlinear terms included in the mixing model
Several models depending on the nature of the scene
(intimate mixturesa, bilinear modelsbc,kernel modelsd,...)
aHapke, “Bidirectional reflectance spectroscopy,” J. Geophys. Res., 1981.
bFan and al., “Comparative study between a new nonlinear model and common linear model for analysing
laboratory simulated-forest hyperspectral data,” Remote Sensing of Environment, 2009.c
Halimi and al., “Nonlinear unmixing of hyperspectral images using a generalized bilinear model,” IEEE TGRS,
2011.
dChen and al., “A novel kernel-based nonlinear unmixing scheme of hyperspectral images,” ASILOMAR 2011.,
Nov. 2011.
Grenoble 2013 – p. 12/45
J.Y. Tourneret
Part 2: PPNMM
Post-Nonlinear mixing model
y = g
(R∑
r=1
armr
)+ n = g (Ma) + n
General class of nonlinear models studied for source
separationab
g: unknown linear/nonlinear application
aJutten and Karhunen, “Advances in nonlinear blind source separation,” Proc. 4th Int. Symp. ICA, Apr. 2003.
bBabaie-Zadeh and al., “Separating convolutive post non-linear mixtures,” Proc. 3rd ICA Workshop, Jun.
2001.
Grenoble 2013 – p. 13/45
J.Y. Tourneret
PPNMM
Grenoble 2013 – p. 14/45
J.Y. Tourneret
PPNMM
Mathematical formulation
y = Ma+ b (Ma)⊙ (Ma) + n
n: additive Gaussian noise
b: nonlinearity parameter
M: known matrix of endmembers
a: abundance vector
Remarks
b = 0 : LMM, b 6= 0 : nonlinear model
Weierstrass theorem: gb(s) = s+ bs2
Grenoble 2013 – p. 15/45
J.Y. Tourneret
Parameter Estimation (I)
Bayesian Inferencea
f(a,b, σ2|y) ∝ f(y|a,b, σ2)f(a,b, σ2)
Gaussian likelihood
Appropriate priors to handle the constraints
Estimation algorithm based on MCMC methods
(significant computational complexity)
aAltmann and al., “Supervised nonlinear spectral unmixing using a post-nonlinear mixing model for hyper-
spectral imagery,” IEEE Trans. Image Process., to appear.
Grenoble 2013 – p. 16/45
J.Y. Tourneret
Bayesian Estimation (I)
Likelihood
y|a, b, σ2 ∼ N(Ma+ b (Ma)⊙ (Ma) , σ2IL
)
Priors
Abundances
A uniform prior is chosen for a\R = [a1, . . . , aR−1]T
in
the simplex
S =
{a\R
∣∣ar ≥ 0, r ∈ 1, . . . , R − 1 and
R−1∑
r=1
ar ≤ 1
}
withaR = 1−
R−1∑
r=1
ar
Grenoble 2013 – p. 17/45
J.Y. Tourneret
Bayesian Estimation (II)
Priors
Nonlinear coefficient
Conjugate Gaussian distribution
b∣∣σ2
b ∼ N(0, σ2
b
)
Here, σ2b is assigned a conjugate inverse-gamma
distribution σ2b ∼ IG (γ, ν) (where (γ, ν) are fixed real
parameters, i.e., (γ, ν) = (1, 10−2).
Noise variance
Non-informative Jeffreys’ prior
f(σ2) ∝ 1
σ2IR+(σ2)
Grenoble 2013 – p. 18/45
J.Y. Tourneret
Bayesian Model
Joint posteriori distribution of θ ={a\R, b, σ
2, σ2b
}
f(θ|y) ∝ 1
σ2
(1
σ2b
) 32+γ
f(y|a\R, σ2, b) exp
(−b2 + 2ν
2σ2b
)1S(a\R)
Distribution too complex to compute θMMSE ou θMAP.
Simulation Method
A Markov Chain Monte Carlo (MCMC) method is
proposed to sample according to f(θ|y) and to compute
the Bayesian (MMSE or MAP) estimators.
Grenoble 2013 – p. 19/45
J.Y. Tourneret
Principles
Generation of samples θ(1), . . . ,θ(Nr) according to f(θ|y)thanks to a Metropolis-Within-Gibbs sampler
Computation of the MMSE or MAP estimators
θMMSE =1
Nr
Nr∑
i=1
θ(i+Nbi)
θMAP = argθ(i)
max f(θ|y) i = Nbi + 1, . . . , Nbi +Nr
Nbi is the burn-in period and Nr is the number of samples
used for the estimation
Grenoble 2013 – p. 20/45
J.Y. Tourneret
A Metropolis-within-Gibbs Sampler
Four sampling steps
Sampling according to f(a\R|b, σ2, σ2b ,y) using a random
walk (Metropolis-Hastings) truncated in the simplex
Sampling according to f(σ2|a, b, σ2b ,y) using an
inverse-gamma distribution
Sampling according to f(b|a, σ2b , σ
2,y) using a Gaussian
distribution
Sampling according to f(σ2b |a, b, σ2,y) using an
inverse-gamma distribution
Grenoble 2013 – p. 21/45
J.Y. Tourneret
Parameter estimation (II)
Observation model
y = φ (θ) + n
with
φ (θ) = Ma+ b (Ma)⊙ (Ma) and θ = [a, b]T
Maximum likelihood estimator
θ = argminθ
‖y − φ (θ)‖2
subject to the positivity and sum-to-one constraints for a
Non-linear optimization problem with linear constraints!!
Grenoble 2013 – p. 22/45
J.Y. Tourneret
Linearization
Iterative algorithma
Taylor Expansion (linearization) of φ at θ(t)
φ(θ) = φ(θ(t))+∇φ
(θ(t))(
θ − θ(t))+ ǫ
Recursive minimization
θ(t+1) = argminθ
∥∥∥y − φ(θ(t))−∇φ
(θ(t))(
θ − θ(t))∥∥∥
2
subject to the positivity and sum-to-one constraints for a
Convergence difficult to prove
aAltmann and al., “Supervised nonlinear spectral unmixing using a post-nonlinear mixing model for hyper-
spectral imagery,” IEEE Trans. Image Process., vol. 21, no. 6, pp. 3017-3025, June 2012.
Grenoble 2013 – p. 23/45
J.Y. Tourneret
Steepest descent
Principlesa
Reparametrization
φ (θR) = Ma+ b (Ma)⊙ (Ma) with θR = [a−R, b]T
Steepest descent algorithm
θ(t+1)R = θ
(t)R − λ∇J
(θ(t)R
)and J (θR) =
1
2‖y − φ (θR)‖2
where λ is chosen to satisfy the constraints (line search)
Convergence toward a local minimum
aAltmann and al., “Supervised nonlinear spectral unmixing using a post-nonlinear mixing model for hyper-
spectral imagery,” IEEE Trans. Image Process., vol. 21, no. 6, pp. 3017-3025, June 2012.
Grenoble 2013 – p. 24/45
J.Y. Tourneret
A Toy Example
Pure materials: Galvanized Steel Metal (red), Green Grass(blue) and Olive Green Paint (green)
Grenoble 2013 – p. 25/45
J.Y. Tourneret
A Toy Example
Parameters: (a1, a2, a3) = (0.3, 0.6, 0.1), b = 0.3, SNR =15dB.
Grenoble 2013 – p. 26/45
J.Y. Tourneret
MMSE Estimators with Confidence Intervals
Grenoble 2013 – p. 27/45
J.Y. Tourneret
PPNMM Validation
I1 I2 I3 I4
(LMM) (FM) (GBM) (PPNMM)
FCLS (LMM) 1.58 24.72 9.49 16.87
Taylor (FM) 22.67 1.49 12.61 26.33
Taylor (GBM) 6.32 14.67 7.07 15.61
Taylor (PPNMM) 2.70 3.83 3.26 3.33
Abundance RMSEs (×10−2): synthetic imagesa
aAltmann and al., “Supervised nonlinear spectral unmixing using a post-nonlinear mixing model for hyper-
spectral imagery,” IEEE Trans. Image Process., vol. 21, no. 6, pp. 3017-3025, June 2012.
Grenoble 2013 – p. 28/45
J.Y. Tourneret
PPNMM Validation
I1 I2 I3 I4
(LMM) (FM) (GBM) (PPNMM)
FCLS (LMM) 5.28 5.74 5.42 5.48
Taylor (FM) 5.61 5.28 5.38 5.75
Taylor (GBM) 5.31 5.40 5.30 5.42
Taylor (PPNMM) 5.29 5.29 5.28 5.28
Reconstruction errors (×10−2): synthetic imagesa
aAltmann and al., “Supervised nonlinear spectral unmixing using a post-nonlinear mixing model for hyper-
spectral imagery,” IEEE Trans. Image Process., vol. 21, no. 6, pp. 3017-3025, June 2012.
Grenoble 2013 – p. 29/45
J.Y. Tourneret
Estimated Nonlinearity Parameter
-0.2 -0.1 0 0.1 0.20
5
10
15
20
I1
b-0.2 0 0.2 0.40
2
4
6
8
I2
b
-0.2 0 0.2 0.40
2
4
6
I3
b-0.4 -0.2 0 0.2 0.40
0.5
1
1.5
2
I4
b
Grenoble 2013 – p. 30/45
J.Y. Tourneret
Real data: Cuprite Image
AVIRIS Cuprite image of 190× 250 pixels (composite natural colors).
R = 14 endmembers, L = 189 spectral bands.Grenoble 2013 – p. 31/45
J.Y. Tourneret
Nonlinearity Parameter (Cuprite)
Grenoble 2013 – p. 32/45
J.Y. Tourneret
Reconstruction Errors (Cuprite)
Table 1: Reconstruction error for the Cuprite image.
RE (×10−2)
LMM FM GBM PPNMM
2.11 3.03 2.02 1.19
Grenoble 2013 – p. 33/45
J.Y. Tourneret
Part 3: Nonlinearity detection
Binary hypothesis testinga
H0 : y is distributed according to the LMM (b = 0)
H1 : y is distributed according to the PPNMM (b 6= 0)
Statistical properties of the nonlinearity parameter estimator
H0 : b ∼ N (0, s20)
H1 : b ∼ N (b, s21)
where s20 is a function of (a, σ2) and s21 depends on (a, b, σ2).
aAltmann and al., “Supervised nonlinear spectral unmixing using a post-nonlinear mixing model for hyper-
spectral imagery,” IEEE Trans. Image Process., vol. 22, no. 4, pp. 1267-1276, April 2013.
Grenoble 2013 – p. 34/45
J.Y. Tourneret
Known parameters a and σ2
Generalized Likelihood Ratio Test (GLRT)
Definition
supb
p(b|H1)
p(b|H0)
H1
≷H0
ν ⇔ T 2 =b2
s20
H1
≷H0
η
where η is an appropriate threshold related to the PFA.
Probability of false alarm and probability of detection
PFA = 2φ(−√η)
PD(b) = 1 + φ
(−s0√η − b
s1
)− φ
(s0√η − b
s1
)
where φ(·) is the cdf of the N (0, 1) Gaussian distribution.
Grenoble 2013 – p. 35/45
J.Y. Tourneret
Unknown parameters a and σ2
Maximum Likelihood estimator of a, b and σ2
θ = (a, b, σ2)
Detection strategy
T 2 =b2
s20
H1
≷H0
η with s20 = CCRLB(b = 0; a, σ2)
Threshold determination
T ∼ N (0, 1)
Grenoble 2013 – p. 36/45
J.Y. Tourneret
Cramér-Rao Lower bound (CRLB)
Fisher information matrix of θ = [aT , b, σ2]T
[JF ]i,j = −E
[∂2 ln f(y|θ)
∂θi∂θj
]i, j = 1, . . . , R + 2
Unconstrained CRLB B
βi,j = [B]i,j =[J−1
F
]i,j
i, j = 1, ..., R + 2
Constrained CRLBa of any unbiased estimator of b under H0
CCRLB(a, σ2) = βR+1,R+1 −(
R∑
i=1
R∑
j=1
βi,j
)(R∑
j=1
βR+1,j
)2
aGorman and Hero, “Lower bounds for parametric estimation with constraints,” IEEE Trans. IT, Nov. 1990.
Grenoble 2013 – p. 37/45
J.Y. Tourneret
Unknown parameters a and σ2
Left: Variance of b under H0 (blue crosses) and CRLB (black line) versus
σ2.Right: Distribution of T under H0 for σ2 = 10−3 (black line) and
standard Gaussian probability density function (red line). R = 3
endmembers, L = 826 spectral bands, a = [0.3, 0.6, 0.1]T .
Grenoble 2013 – p. 38/45
J.Y. Tourneret
Simulation results
Synthetic data (50× 50 pixels)
Four images (4 mixing models)
S1 : LMM
S2 : Bilinear Fan model (γi,j = 1)
y =
R∑
r=1
armr +
R−1∑
i=1
R∑
j=i+1
γi,jaiajmi ⊙mj + n
S3 : Generalized Bilinear model (γi,j ∼ U(0;1))
S4 : PPNMM (b ∼ U(−0.3;0.3))
R = 3 endmembers, L = 826 spectral bands, a uniform in the simplex.
Grenoble 2013 – p. 39/45
J.Y. Tourneret
Synthetic Data
Left: the four sub-images S1 (LMM), S2 (FM), S3 (GBM) and S4 (PPNMM).
Right: Detection maps using PFA = 0.05. Black (resp. white) pixels
correspond to pixels detected as linearly (resp. nonlinearly) mixed.
Grenoble 2013 – p. 40/45
J.Y. Tourneret
Synthetic data
Pixels detected as linear (red crosses) and nonlinear (blue dots) for the
four sub-images S1 (LMM), S2 (FM), S3 (GBM) and S4 (PPNMM).
Grenoble 2013 – p. 41/45
J.Y. Tourneret
Real data: Cuprite Image
AVIRIS image of 190× 250 pixels extracted from Cuprite image observed
in composite natural colors.
R = 14 endmembers, L = 189 spectral bands.
Grenoble 2013 – p. 42/45
J.Y. Tourneret
Real data: Cuprite Image
Left: map of b for the Cuprite image. Detection maps for PFA = 10−3
(middle) and for PFA = 10−4 (right). Black (resp. white) pixels correspond
to pixels detected as linearly (resp. nonlinearly) mixed.
Grenoble 2013 – p. 43/45
J.Y. Tourneret
Conclusions
Post nonlinear models have shown promising results for
spectral unmixing in the presence of nonlinear mixtures
pixel-by-pixel nonlinearity detection to determine which
parts of an hyperspectral image are characterized by
nonlinear mixtures.
Future work within the ANR HYPANEMA
Derive a fully unsupervised algorithm for abundance and
endmember estimation for nonlinear mixtures
Introduction of spatial correlation between adjacent pixels
for parameter estimation and nonlinearity detection
Grenoble 2013 – p. 44/45
J.Y. Tourneret
An Example (Ultrasound Images)
(a) (b) (c)
Actual location of skin lesion (left) and the corresponding estimated labels
(healthy = white, lesion = red) without spatial correlation (middle) and with
spatial correlation (right)a.
aPereyra et al., “Segmentation of skin lesions in 2D and 3D ultrasound images using a spatially coherent
generalized Rayleigh mixture model,” IEEE Trans Med Imaging., Mar. 2012.
Grenoble 2013 – p. 45/45