Image Analysis & Retrieval
Lec 15
Graph Laplacian Embedding
Zhu Li
Dept of CSEE, UMKC
Office: FH560E, Email: [email protected], Ph: x 2346.
http://l.web.umkc.edu/lizhu
p.1Z. Li, Image Analysis & Retrv, Spring 2017
Outline
Recap:
Eigenface
Fisherface
HW-3: Fisherface and Laplacian face
Graph Embedding
Laplacianface
Graph Fourier Transform
Summary
p.2Z. Li, Image Analysis & Retrv, Spring 2017
Subspace Learning for Face Recognition
Project face images to a subspace with basis A
Matlab: x=faces*A(:,1:kd)
eigf1
eigf2
eigf3
= 10.9* + 0.4* + 4.7*
p.3Z. Li, Image Analysis & Retrv, Spring 2017
PCA & Fisher’s Linear Discriminant
• Between-class scatter
• Within-class scatter
• Where
– c is the number of classes
– i is the mean of class i
– | i | is number of samples of i..
T
ii
c
i
iBS ))((1
c
i x
T
ikikW
ik
xS1
))((
1
2
12
p.4Z. Li, Image Analysis & Retrv, Spring 2017
Eigen vs Fisher Projection
p.5
• PCA (Eigenfaces)
Maximizes projected total scatter
• Fisher’s Linear Discriminant
Maximizes ratio of projected between-class to projected within-class scatter, solved by the generalized Eigen problem:
WSWW T
T
WPCA maxarg
WSW
WSWW
W
T
B
T
Wfld maxarg
12
PCA
Fisher𝑆𝐵𝑊 = 𝜆𝑆𝑊𝑊
Z. Li, Image Analysis & Retrv, Spring 2017
LDA implementation
myLDA.m
p.6
% compute class mean
mx = mean(x);
ids = unique(y); m = length(ids);
Sb = zeros(kd, kd);
for k=1:m
indx = find(y==ids(k)); nk(k) = length(indx);
% class mean
m_cx(k,:) = mean(x(indx, :));
% between class scatter
Sb = Sb + nk(k)*(m_cx(k,:) - mx)'*(m_cx(k,:) - mx);
end
% compute intra-class scatter
Sw = zeros(kd, kd);
for k=1:m
indx = find(y==ids(k)); nk(k) = length(indx);
% remove mean
xk = x(indx, :) - repmat(m_cx(k,:), [nk(k), 1]);
% adding up
Sw = Sw + (xk'*xk);
end
Z. Li, Image Analysis & Retrv, Spring 2017
LDA Implementation
Find projection by generalized Eigen problem solution
p.7
% generalized eigen problem
[A, v]=eigs(Sb, Sw);
if dbg
figure(31);
subplot(2,2,1); imagesc(Sb);
colormap('gray'); title('S_b');
subplot(2,2,2); imagesc(Sw);
colormap('gray'); title('S_w');
z = x*A; dist = pdist2(z, z);
subplot(2,2,3); imagesc(dist);
title('dist(j,k)');
subplot(2,2,4); stem(diag(v), '.'); grid on;
hold on; title('eig v');
end
𝑆𝐵𝑊 = 𝜆𝑆𝑊𝑊
Z. Li, Image Analysis & Retrv, Spring 2017
Fisherface Basis
It is interesting to compare Fisherface with Eigenfacebasis
p.8
FisherfaceEigenface
Z. Li, Image Analysis & Retrv, Spring 2017
Fisherface Performance
Fisher vs Eigenface performance: (fisherface.m)
1200 face images, 144 subjects
Eigen kd=32
p.9
144 subjectsROC
Z. Li, Image Analysis & Retrv, Spring 2017
Outline
Recap:
Eigenface
Fisherface
Graph Embedding
Locality Preserving Projection
Laplacian Face
Graph Fourier Transform
Summary
p.10Z. Li, Image Analysis & Retrv, Spring 2017
Locality Preserving Projection
Recall the dimension reduction formulations: find w, s.t y=wx:
PCA:
LDA:
p.11
max𝑤
wTSw
S = SB + SW
Z. Li, Image Analysis & Retrv, Spring 2017
LPP Formulation –Affinity
To preserve local affinity relationship
Affinity map
Selection of heat kernel size and threshold are important
Hint: affinity matrix should be sparse
p.12
affinity histogram
Z. Li, Image Analysis & Retrv, Spring 2017
LPP Formulation –Affinity Supervised
How to utilize the label info ?
Heat map mapping is good for intra-class affinity modelling, but how about intra-class affinity ?
One direct solution is to set affinity to zero for intra class pairs
p.13
% LPP - compute affinity
f_dist1 = pdist2(x1, x1);
% heat kernel size
mdist = mean(f_dist1(:)); h = -
log(0.15)/mdist;
S1 = exp(-h*f_dist1);
id_dist = pdist2(ids, ids);
subplot(2,2,3); imagesc(id_dist);
title('label distance');
S2=S1; S2(find(id_dist~=0)) = 0;
subplot(2,2,4); imagesc(S1);
colormap('gray'); title('affinity-
supervised');
Z. Li, Image Analysis & Retrv, Spring 2017
LPP- Affinity Preserving Projection
Find a projection that best preserves the affinity matrix
p.14
min𝑊
𝑖
𝑗
𝑆𝑖,𝑗 𝑦𝑖 − 𝑦𝑗2, 𝑠. 𝑡. 𝑦 = 𝑊𝑥
min𝑊
𝑖
𝑗
𝑆𝑖,𝑗 𝑊𝑥𝑖 −𝑊𝑥𝑗2
𝑠. 𝑡, 𝑆𝑖,𝑗 = −exp 𝑥𝑗 − 𝑥𝑖
2
ℎ, 𝑖𝑓 𝑥𝑗 − 𝑥𝑖 ≤ 𝜃
0, 𝑒𝑙𝑠𝑒
Z. Li, Image Analysis & Retrv, Spring 2017
LPP Formulation
L = D-S:
nxn matrix, called graph Laplacian
Normalizing factor: nxn D
Diagonal matrix, entry Dii = sum of col/row affinity
The larger the value, the more important data point is
Constraint on D:
p.15Z. Li, Image Analysis & Retrv, Spring 2017
Generalized Eigen Problem
Now the formulation is,
Lagranian
by KKT (Karush-Khun-Tucker) Condition, it is solved by a generalized Eigen problem
p.16
𝐿 𝑤, 𝜆 = 𝑤𝑇𝑋𝐿𝑋𝑇𝑤 − 𝜆(𝑤𝑇𝑋𝐷𝑋𝑤 − 1)
Z. Li, Image Analysis & Retrv, Spring 2017
X He, S Yan, Y Hu, P Niyogi, HJ Zhang, “Face Recognition Using Laplacianface”, IEEE Trans PAMI, vol. 27 (3), 328-340, 2005.
Matlab Implementation
laplacianface.m
p.17
%LPP
n_face = 1200; n_subj = length(unique(ids(1:n_face)));
% eigenface
kd=32; x1 = faces(1:n_face,:)*A1(:,1:kd); ids=ids(1:n_face);
% LPP - compute affinity
f_dist1 = pdist2(x1, x1);
% heat kernel size
mdist = mean(f_dist1(:)); h = -log(0.15)/mdist;
S1 = exp(-h*f_dist1);
figure(32); subplot(2,2,1); imagesc(f_dist1); colormap('gray'); title('d(x_i, d_j)');
subplot(2,2,2); imagesc(S1); colormap('gray'); title('affinity');
%subplot(2,2,3); grid on; hold on; [h_aff, v_aff]=hist(S(:), 40); plot(v_aff, h_aff,
'.-');
% utilize supervised info
id_dist = pdist2(ids, ids);
subplot(2,2,3); imagesc(id_dist); title('label distance');
S2=S1; S2(find(id_dist~=0)) = 0;
subplot(2,2,4); imagesc(S1); colormap('gray'); title('affinity-supervised');
% laplacian face
lpp_opt.PCARatio = 1;
[A2, eigv2]=LPP(S2, lpp_opt, x1);
Z. Li, Image Analysis & Retrv, Spring 2017
Laplacian Face
Now, we can model face as a LPP projection:
p.18
Eigenface Laplacian Face
Z. Li, Image Analysis & Retrv, Spring 2017
Laplacian vs Eigenface
1200 faces, 144 subjects
p.19Z. Li, Image Analysis & Retrv, Spring 2017
LPP and PCA
Graph Embedding is an unifying theory on dimension reduction
PCA becomes special case of LPP, if we do not enforce local affinity
p.20Z. Li, Image Analysis & Retrv, Spring 2017
LPP and LDA
How about LDA ?
Recall within class scatter:
p.21
This is i-th classData covariance
Li has diagonal entry of 1/ni,Equal affinity among data points
Z. Li, Image Analysis & Retrv, Spring 2017
LPP and LDA
Now consider the between class scatter
C is the data covariance, regardless of label
L is graph Laplacian computed from the affinity rule that,
p.22
𝑆 𝑖, 𝑗 =
1
𝑛𝑘, 𝑖𝑓 𝑥𝑖 , 𝑥𝑗𝑎𝑟𝑒 𝑓𝑟𝑜𝑚 𝑐𝑙𝑎𝑠𝑠 𝑘
0, 𝑒𝑙𝑠𝑒
Z. Li, Image Analysis & Retrv, Spring 2017
LDA as a special case of LPP
The same generalized Eigen problem
p.23Z. Li, Image Analysis & Retrv, Spring 2017
Graph Embedding Interpretation of PCA/LDA/LPP
Affinity graph S, determines the embedding subspace W, via
PCA and LDA are special cases of Graph Embedding
PCA:
LDA
LPP
p.24
𝑆𝑖,𝑗 = −exp 𝑥𝑗 − 𝑥𝑖
2
ℎ, 𝑖𝑓 𝑥𝑗 − 𝑥𝑖 ≤ 𝜃
0, 𝑒𝑙𝑠𝑒
𝑆𝑖,𝑗 =
1
𝑛𝑘, 𝑖𝑓 𝑥𝑖 , 𝑥𝑗 ∈ 𝐶𝑘
0, 𝑒𝑙𝑠𝑒
𝑆𝑖,𝑗 = 1/𝑛
Z. Li, Image Analysis & Retrv, Spring 2017
Applications: facial expression embedding
Facial expressions embedded in a 2-d space via LPP
p.25
frown
sad
happy
neutral
Z. Li, Image Analysis & Retrv, Spring 2017
Application: Compression of SIFT
Compression of SIFT, preserve matching relationship, rather than reconstruction:
Z. Li, Image Analysis & Retrv, Spring 2017 p.26
𝐴 = argmin𝑊
𝑘
𝑗
𝑠𝑗,𝑘 𝑊𝑥𝑗 −𝑊𝑥𝑘2
Homework-3: Subspace Methods
Objective:
Understand the graph embedding connections among popular subspace methods like PCA, LDA and LPP
Practical experiences with serious size data set
Data Set:
https://umkc.app.box.com/s/0qu7tc3jb88at2h53l1dpcuqkt9pn7ww
417 subjects, 6650 image face data set, pre-processed to 20x20 pel images, intensity normalized to [0, 1]
Add your own face images, 10~15, frontal
Tasks:
Compute Eigenface, Fisherface and Laplacianface models
ROC plot on verification performance
mAP for retrieval/identification performance
Z. Li, Image Analysis & Retrv, Spring 2017 p.27
HW-3 test run
Laplacian face is powerful.
Z. Li, Image Analysis & Retrv, Spring 2017 p.28
Graph Fourier Transform
David I. Shuman, Sunil K. Narang, Pascal Frossard, Antonio Ortega, Pierre Vandergheynst:The Emerging Field of Signal Processing on Graphs: Extending High-Dimensional Data Analysis to Networks and Other Irregular Domains. IEEE Signal Process. Mag. 30(3): 83-98 (2013)
Z. Li, Image Analysis & Retrv, Spring 2017 p.29
Signal on Graph
non-uniformly sampled
p.30Z. Li, Image Analysis & Retrv, Spring 2017
Graph Fourier Transform
GFT is different from Laplacian Embedding:
p.31Z. Li, Image Analysis & Retrv, Spring 2017
GFT Example
Graph Laplacian
p.32Z. Li, Image Analysis & Retrv, Spring 2017
Normalized Graph Laplacian
Normalize by edge pair degree
p.33Z. Li, Image Analysis & Retrv, Spring 2017
Graph Frourier Transform
Analogous to FT
p.34Z. Li, Image Analysis & Retrv, Spring 2017
Graph Spectrum
p.35Z. Li, Image Analysis & Retrv, Spring 2017
Graph Signal Smoothness
Quadratic form on L:
p.36Z. Li, Image Analysis & Retrv, Spring 2017
Summary
Graph Laplacian Embedding is an unifying theory for feature space dimension reduction PCA is a special case of graph embedding
o Fully connected affinity map, equal importance
LDA is a special case of graph embeddingo Fully connected intra class
o Zero affinity inter class
LPP: preserves pair wise affinity.
GFT: eigen vectors of graph Laplacian, has Fourier Transform like characteristics.
Many applications in Face recognition
Pose estimation
Facial expression modeling
Compression of Graph signals.
p.37Z. Li, Image Analysis & Retrv, Spring 2017