Post Nonlinear Independent SubspaceAnalysis
Zoltán Szabó, Barnabás Póczos,Gábor Szirtes and András Lorincz
Neural Information Processing Group,Department of Information Systems,
Eötvös Loránd University,Budapest, Hungary
ICANN 2007
Z. Szabó, B. Póczos, G. Szirtes and A. Lorincz PNL ISA
Post Nonlinear Independent Subspace Analysis
Cocktail party problem:
independent groups of people / music bands,
nonlinear (post nonlinear) mixing.
Z. Szabó, B. Póczos, G. Szirtes and A. Lorincz PNL ISA
PNL ISA Equations
The PNL ISA model:
x(t) = f[As(t)]. (1)
Assumptions (s = [s1; . . . ; sM ] ∈ RMd = R
D):source s is d-independent: I(s1, . . . , sM) = 0,s(t) ∈ R
D is i.i.d. in time t ,A ∈ R
D×D invertible and ‘mixing’, that is: A = [Aij ∈ Rd×d ],
∀ i ⇒ ∃(j , k) : Aij and Aik are invertible.f : R
D → RD invertible, acts component-wise.
Goal: s.
Z. Szabó, B. Póczos, G. Szirtes and A. Lorincz PNL ISA
Ambiguites of PNL ISA
PNL mixing structure⇒ mirror demixing [s = Wg(x)].
Question: d-independence of s⇒ true s has been found?
⇓
Yes: PNL ISA separability theorem.
Z. Szabó, B. Póczos, G. Szirtes and A. Lorincz PNL ISA
Separability; PNL-ISA Ambiguities withLocally-Constant Nonzero C2 Densities
Theorem
Supposing that:
A, W: invertible and ’mixing’ matrices,
s: (i) existing covariance matrix, (ii) somewhere locallyconstant, C2 density function,
h = g ◦ f is a component-wise bijection, with analyticalcoordinate functions.
In this case, if e := [e1; . . . ; eM ] = Wh(As) is d-independentwith somewhere locally constant density function, then:
e recovers the hidden source (up to ISA ambiguities +constant translation within subspaces).
Z. Szabó, B. Póczos, G. Szirtes and A. Lorincz PNL ISA
Separability⇒ PNL ISA algorithm
Sketch:1 Estimate g = f−1:
d-dependent Central Limit Theorem
⇓
As is asymptotically Gaussian (D →∞)
⇓
g: ’gaussianization’ transformation
2 Estimate W: linear ISA to g(x).
Z. Szabó, B. Póczos, G. Szirtes and A. Lorincz PNL ISA
Test databases (s)
I.i.d. tests:1 3D-geom (d = 3, M = 6),2 celebrities (d = 2, M = 10),3 letters (d = 2, M ≤ 50).
Non-i.i.d. test: IFS (self-similar structures; d = 2, M = 9).
Z. Szabó, B. Póczos, G. Szirtes and A. Lorincz PNL ISA
Simulations
Performance index: Amari-index (r ∈ [0, 1]) to measure theblock-permutation matrix property of the linearapproximation of
s− E [s] ∈ RD 7→ s := Wg[f(As)]− E [s] ∈ R
D.
Components of the PNL ISA algorithm:gaussianization based on ranks of samples,ISA by joint f-decorrelation (JFD).
Simulation parameters:goodness: average of 50 random (A, s, f) runs,mixing matrix A: random orthogonal,coordinate-wise distortions: fi(z) = ci [aiz + tanh(biz)] + di .
Z. Szabó, B. Póczos, G. Szirtes and A. Lorincz PNL ISA
Illustrations-1: r(T )
Amari-index as a function of
the sample number (T ): 3D-geom, celebrites, IFS.
dimensionality of the problem (↔ M): letters.
1 2 5 10 20 50 10010
−3
10−2
10−1
100
Number of samples (T)
Am
ari−
inde
x (r
)
3D−geom, celebrities, IFS
3D−geomcelebritiesIFS
x1031 2 5 10 20 50 100
10−3
10−2
10−1
100
Number of samples (T)
Am
ari−
inde
x (r
)
letters
M=2M=3M=4M=10M=20M=50
x103
Power-law decline: r(T ) ∝ T−c (c > 0), as Dր.
Z. Szabó, B. Póczos, G. Szirtes and A. Lorincz PNL ISA
Illustration-1: demo (T = 100, 000)
−8 −6 −4 −2 0 2 4 6 8−6
−4
−2
0
2
4
6
8
Functions fi
Z. Szabó, B. Póczos, G. Szirtes and A. Lorincz PNL ISA
Illustrations-2: D (T = 10, 000)
Estimation of D = dim(s) - in the background: Dx = 2D.
Gaussianization→ ordered eigenvalues of cov [g(x)].
Results: average over 50 random runs (A, f).
0 5 10 15 20 25 30 35 400
0.5
1
1.5
2
2.5
3Distribution of Eigenvalues: celebrities (3D−geom, IFS)
Eig
enva
lues
: Mea
n ±
Dev
iatio
n
Number of Eigenvalue2 3 4
0
0.5
1
1.5
2
2.5
3
Number of Components (M)
Eig
enva
lues
: Mea
n ±
Dev
iatio
n
Distribution of Eigenvalues: letters
Z. Szabó, B. Póczos, G. Szirtes and A. Lorincz PNL ISA
Summary
PNL ISA problem
Separability of PNL ISA
⇓ (← d-dependent Central Limit Theorem)
PNL ISA = gaussianization + ISA
Simulations:Estimation error vs. sample number:
power-law decline, as Dր.Possibility to estimate the dimension of the hidden source.
The dimensions of the hidden sources: can also beestimated using the ISA Separation Theorem [Szabó et al.,JMLR 8 (2007), 1063-1095] . . .
Z. Szabó, B. Póczos, G. Szirtes and A. Lorincz PNL ISA
Thank you for the attention!
Z. Szabó, B. Póczos, G. Szirtes and A. Lorincz PNL ISA