Face Face IdentificationIdentificationby by FittingFitting a a
3D3D Morphable Model Morphable Modelusing using LinearLinear Shape and Texture Error Shape and Texture Error
FunctionsFunctions
Sami Romdhani Volker Blanz Thomas Vetter
University of Freiburg
Supported by DARPA
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 2/26
The ProblemThe Problem
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 3/26
MenuMenu
Historical Methods
3D Morphable Model
LiST : a Novel Fitting Algorithm
Identification Experiments on more than 5000 Images
Identification Confidence = Fitting Accuracy
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 4/26
Historical Methods : Historical Methods : Active Appearance ModelActive Appearance Model
Use of a generative model:
1. View based (2D), Correspondence basedex: AAM of Cootes and Taylor
Drawbacks:- small pose variation statistically
modeled !
- large pose var. necessitates many models !
- illumination not addressed !
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 5/26
Historical Methods : Illumination ConeHistorical Methods : Illumination Cone
2. Shape from Shading= Recovering 3D shape from Illumination variations
ex: Illumination Cone of Georghiades, Belhumeur & Kriegman
Limited use : up to 24° azimuth variation !
Drawback:Impractical: requires many imagesRestrictive assumptions : constant
albedo, lambertian,no cast shadows
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 6/26
3D Shape
3D Morphable Model - Key Features 13D Morphable Model - Key Features 1
1. Representation = 3D Shape + Texture Map
Texture Map
s
t
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 7/26
3D Morphable Model - Key Features 23D Morphable Model - Key Features 2
2. Accurate & Dense Correspondence
PCA accounts for intrinsic ID parameters only
2 3 4 1s S α
4 3 2 1t T β
...
...
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 8/26
3D Morphable Model - Key Features 33D Morphable Model - Key Features 3
3. Extrinsic parameters modeled using Physical Relations:- Pose : 3x3 Rotation matrix
- Illumination : Phong shading accounts for cast shadows and specular highlights
No Lambertian Assumption.
, ,
1 0 0
0 1 0xk
kk y
tx
yf
t
αR S
( )k
ambient dir speculark k k k
k
r
g
b
A TA aβ
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 9/26
3D Morphable Model - Key Features 43D Morphable Model - Key Features 4
4. Photo-realistic images rendered using Computer Graphicsα
S
, , R
1 0 0,
0 1 0f
,xt yt
, ,x y z
, ,x y z
,x y
β
T
, ,ambient dir specularaA A
, ,r g b
, ,r g b
( , , , , )w x y r g b ( , )mI x y
, ,x y zn n n
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 10/26
Model Fitting : DefinitionModel Fitting : Definition
IterativeModelFitting
,α β,ρ
ModelRenderin
g
( , )mI x y
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 11/26
Model Fitting - History : Standard Optimization Model Fitting - History : Standard Optimization TechniquesTechniquesJones, Poggio 98 : Gradient DescentBlanz, Vetter 99 : Stochastic Gradient DescentPighin, Szeliski, Salesin 99 : Levenberg-Marquardt
-
2
2
2
I
I
I
Model EstimateInput
Difference I
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 12/26
Model Fitting - History : Image Difference Model Fitting - History : Image Difference DecompositionDecomposition
IDD introduced by Gleicher in 97 and used by Sclaroff et al. in 98, and Cootes et al. in 98
-
I
A
Input
Difference I
Model Estimate
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 13/26
LiST : Non-linearity LiST : Non-linearity
1. Non-linear warping
2. Non-linear parametersinteraction
α
S
, , R
1 0 0,
0 1 0f
,xt yt
, ,x y z
, ,x y z
,x y
β
T
, ,ambient dir specularaA A
, ,r g b
, ,r g b
( , , , , )w x y r g b ( , )mI x y
, ,x y zn n n
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 14/26
LiST : Shape & Texture Parameters recoveryLiST : Shape & Texture Parameters recovery
α
S
, , R
1 0 0,
0 1 0f
,xt yt
,,x y z
, ,x y z
,x y
β
T
, ,ambient dir specularaA A
, ,r g b
, ,r g b
( , , , , )w x y r g b ( , )mI x y
, ,x y zn n n
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 15/26
( , , , , )w x y r g b
α
S
, , R
1 0 0,
0 1 0f
,xt yt
, ,x y z
, ,x y z
,x y
β
T
, ,ambient dir specularaA A
, ,r g b
, ,r g b
, ,x y zn n n
LiSTLiST
( , )mI x y
( , )I x y
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 16/26
Optical Flow
α
S
, , R
1 0 0,
0 1 0f
,xt yt
, ,x y z
, ,x y z
,x y
β
T
, ,ambient dir specularaA A
, ,r g b
, ,r g b
, ,x y zn n n
( , )mI x y
( , )I x y
LiST : Optical FlowLiST : Optical Flow
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 17/26
α
S
, , R
1 0 0,
0 1 0f
,xt yt
,,x y z
, ,x y z
,x y
β
T
, ,ambient dir specularaA A
, ,r g b
, ,r g b
, ,x y zn n n
Optical Flow( , )mI x y
( , )I x y
Lev.-Mar.
LiST : Rotation, Translation & Size RecoveryLiST : Rotation, Translation & Size Recovery
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 18/26
α
S
, , R
1 0 0,
0 1 0f
,xt yt
,,x y z
, ,x y z
,x y
β
T
, ,ambient dir specularaA A
, ,r g b
, ,r g b
, ,x y zn n n
Optical Flow( , )mI x y
( , )I x y
Lev.-Mar.Lev.-Mar.
LiST : Illumination RecoveryLiST : Illumination Recovery
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 19/26
LiST : DiscussionLiST : Discussion
• Shape and Texture recoveries are interleavedThe recovery of one helps the recovery of the other
• Takes advantage of the linear parts of the model
• Recovers out-of-the-image-plane rotation & directed illumination
• 5 times faster than Stochastic Gradient Descent
Drawbacks:• Still requires manual initialization• Still not fast enough
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 20/26
Experiments : The CMU-PIE Face DatabaseExperiments : The CMU-PIE Face Database
• Publicly available
• Systematic pose & illumination variations
• 68 Individuals
• 4488 Images with combined Pose & Illumination var.
• 884 Images with Pose var.
-20
-15
-10
-5
0 0
5
10
15
20
-5
0
5 head
flashescamerashead
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 21/26
Experiments : FittingExperiments : Fitting
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 22/26
Experiments : Identification across PoseExperiments : Identification across Pose
Identification Results across Pose
0102030405060708090
34 31 14 11 29 9 27 7 5 37 25 2 22
Gallery Pose
Pe
rce
nta
ge
LiST, average=76%FaceIt, average=21%
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 23/26
Experiments : Identification across Illumination & Experiments : Identification across Illumination & PosePose
Identification on 4488 imagesacross Pose & Illuminationaveraged over Illumination
Front Side Profile
Front 97 91 60
Side 93 96 71
Profile 65 71 86
Gallery
Probe
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 24/26
Identification Confidence : TheoryIdentification Confidence : Theory
Can we be sure to have correctly identified someone ?
Identification Confidence depends mostly on the Fitting
We think:
Classification Support Vector MachineInput:Mahalanobis distance from the average
SSE over 5 regions of the face
Output: Good Fitting Y/N ?
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 25/26
-2-1.25-0.75-0.250.250.751.2520
5
10
15
20
25
30
35
Fitting Score = SVM Output
% o
f Exp
erim
ents
29 % 33 % 12 % 6 % 4 % 7 % 7 % 3 %
Iden
tific
atio
n P
erce
ntag
e
Identification vs. Fitting Score
97.4 %95.1 %
83.7 %
76.5 %
58.9 %
43.2 %
38.2 %
26.8 %
20
30
40
50
60
70
80
90
100
Identification Confidence : ResultIdentification Confidence : Result
The model is goodwe only need to improve the fitting accuracy
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 26/26
ConclusionsConclusions
Novel Fitting Algorithm :• Use of Optical Flow to recover a Shape Error• Recovers most of the parameters linearly• Recovers a few non-linear parameters using
Lev.-Mar.
State of the art identification performances across
Pose & Illumination
Drawbacks:• Still not fast enough• Still requires manual initialisation
