Model-based Image Interpretation with Application to Facial Expression Recognition Matthias Wimmer...

Model-based Image Interpretation with Application to

Facial Expression Recognition

Matthias Wimmer

[email protected]

2007, December 11th

2/25

Technische Universität MünchenMatthias Wimmer

Communication Schemes

Natural human-computer interaction

2007, December 11th

3/25


Example 1: Nissan Pivo2

2007, December 11th

4/25


Example 2: Sony’s Smile Shutter

2007, December 11th

5/25


Outline of this Presentation


Model-based image interpretation

Adaptive skin color extraction

image

facial expression

2007, December 11th

6/25



2007, December 11th

7/25


What are Facial Expressions? Six universal facial expressions (Ekman et al.)

Laughing, surprised, afraid, disgusted, sad, angry

Cohn-Kanade-Facial-Expression database Performed Exaggerated

Determined by Shape Muscle motion

2007, December 11th

8/25


Why are they difficult to estimate?

Faces look differently Hair, beard, skin-color, …

Different facial poses

Only slight muscle activity

2007, December 11th

9/25


Our Approach

motion features and structural features

2007, December 11th

10/25


Model Fitting with Learned Objective Functions

2007, December 11th

11/25


Model-based image interpretation The model

The model contains a parameter vector that represents the model’s configuration.

The objective function Calculates a value that indicates how accurately a parameterized model matches an image.

The fitting algorithm Searches for the model parameters that describe the image best, i.e. it minimizes the objective function.

2007, December 11th

12/25


Local Objective Functions

2007, December 11th

13/25


Ideal Objective FunctionsP1: Correctness property:

Global minimum corresponds to the best fit.P2: Uni-modality property:

The objective function has no local extrema. ¬ P1 P1

¬P2

P2

Don’t exist for real-world images

Only for annotated images: fn( I , x ) = | cn – x |

2007, December 11th

14/25


Learning the Objective Function

x x x xx

xxx x xxx x x x

x x xx x

x xx x x x x

x xxx x

Ideal objective function generates training data Machine Learning technique generates calculation rules

2007, December 11th

15/25


Benefits of the Machine Learning Approach Accurate and robust calculation rules

Locally customized calculation rules Generalization from many images

Simple job for the designer Critical decisions are automated No domain-dependent knowledge required No loops

2007, December 11th

16/25


Evaluation: Fitting Accuracy

2007, December 11th

17/25


Adaptive Skin Color Classification

2007, December 11th

18/25


Basics about Skin Color Classificationgr

een

image 1

image 2

Skin color depends on several image conditions Skin color occupies a large cluster

Skin color varies greatly within a set of images. Skin color varies slightly within one image.

image 2

gree

n

red red

2007, December 11th

19/25


Our Approach Learn image-specific skin color characteristics Parameterize a skin color classifier accordingly

Offline: Learn the skin color mask

Specific for the face detector

Online: Detect the image specific skin color model

Using the face detector Using the skin color mask

Adapt skin color classifier

2007, December 11th

20/25


Results Robustness:

Detection of facial parts:eyes, lips, brows,…

Exact shape outline Ethnic groups

Correctly detected pixels: fixed classifier: 90.4% 74.8% 40.2% adapted classifier: 97.5% 87.5% 97.0% improvement: 1.08 1.17 2.41

adapted

classifier

fixed

classifier

original

image

2007, December 11th

21/25


Additional Work lip classifier eye brow classifier

iris classifier tooth classifier

2007, December 11th

22/25


Conclusion and Outlook

2007, December 11th

23/25


Conclusion

Possible to derive information from face images Model-based image interpretation is beneficial Learn crucial decisions within algorithms

Don’t specify parameters by trial and error Adaptive skin color classifier Learned objective functions

Not yet reached goal for natural HCI Progress is clearly visible.

→ Goal is achievable!

2007, December 11th

24/25


Outlook Learn global objective function

Learn discriminative function (direct parameter update)

Rendered AAM provides training images Many images Exact ground truth (no manual work required)

Learn with further features Higher number of features SIFT, LBP, …

Learn with better classifiers Relevance Vector Machines Boosted regressors

2007, December 11th

25/25


Thank you!

Online-Demonstration: http://www9.cs.tum.edu/people/wimmerm

2007, December 11th

26/25


Publications2008

Tailoring Model-based Techniques for Facial Expression Interpretation. ACHI08 Face Model Fitting with Generic, Group-specific, and Person-specific Objective Functions. VISAPP Low-level Fusion of Audio and Video Feature for Multi-modal Emotion Recognition. VISAPP Facial Expression Recognition for Human-robot Interaction - A Prototype. Robot Vision

2007 Audiovisual Behavior Modeling by Combined Feature Spaces. ICASSP Emotionale Aspekte in Produktevaluationen. Multimediatechnik Application of emotion recognition methods in automotive research. Emotion and Computing Human Capabilities on Video-based Facial Expression Recognition. Emotion and Computing SIPBILD - Mimik- und Gestikerkennung in der Mensch-Maschine-Schnittstelle. INFORMATIK Learning Robust Objective Functions with Application to Face Model Fitting. DAGM Automatically Learning the Objective Function for Model Fitting. MIRU Initial Pose Estimation for 3D Models Using Learned Objective Functions. ACCV Estimating Natural Activity by Fitting 3D Models via Learned Objective Functions. VMV Learning Local Objective Functions for Robust Face Model Fitting. PAMI (journal paper) Enabling Users to Guide the Design of Robust Model Fitting Algorithms. ICV

2006 Learning Robust Objective Functions for Model Fitting in Image Understanding Applications. BMVC A Person and Context Specific Approach for Skin Color Classification. ICPR Adaptive Skin Color Classificator. Journal on Graphics, Vision and Image Processing (journal paper) Bitte recht freundlich. Zukunft im Brennpunkt (journal paper)

2005 Sensor-based Situated, Individualized, and Personalized Interaction in Smart Environments. INFORMATIK Adaptive Skin Color Classificator. GVIP

2004 Experiences with an Emotional Sales Agent. Affective Dialogue Systems

Date post:	14-Dec-2015
Category:	Documents
Upload:	earl-harrow
View:	217 times
Download:	4 times

Model-based Image Interpretation with Application to Facial Expression Recognition Matthias Wimmer...

Documents