Date post: | 12-Jan-2016 |
Category: |
Documents |
Upload: | francis-ryan |
View: | 217 times |
Download: | 0 times |
Graphite 2004Statistical Synthesis of Facial
Expressions for the Portrayal of Emotion
Lisa GralewskiBristol UniversityUnited [email protected]
Aim and Motivation:
Generate new emotion specific video sequences.
Facial expressions + head motion. Potentially infinitely long No repeated footage
Provide an animation tool: User driven. Allows emotion selection. Automatic generation of novel facial expression
sequences.
Approach: Statistical Approach
Divides into 2 stages
Stage 1 Modelling: To form generative emotion models.
• Shape model •point data, potential to drive animations
• Combined appearance space model•texture and shape for the generation of new video frames.
Stage 2 Synthesis: Method to generate new video footage / point data
Emotion video footage is considered as a time series of changing facial expressions.
Time series can be modelled with Auto regressive processes (ARPs) Advantages:
constructed models generate new data similar to training data. Potentially infinitely long generated sequences.
Disadvantages: For successful modelling, input data must have stationary signal.
Use ARP’s to form generative emotion specific models. These allow synthesis of new facial expression emotion sequences.
Stage 1: Modelling Approach
Time
Modelling: Data Collection
Acquire video footage Emotions: Happy, sad
and, angry.
Split video into:
1) Shape representation Using hand labelled
control points.
2) Texture representation Using warped image set
Neutral pose mean shape of control point set.
Modelling: Facial expressions
Apply principle components analysis (PCA) video data to calculate eigenvectors for:
Shape space (point data) Texture space (warped pixel data) Combined space (shape + texture).
Calculate shape and combined space responses.
Data projected from a higher-D space down to lower D-space. Each video frame has a response Together the responses for a sequence form an eigensignature.
Form linear shape and combined space models. Allow reconstruction of original data.
Modelling: Eigensignatures Visualise eigensignature:
Plot of 1st and 2nd principle components (responses).
Each point of the eigensignature represents a single video frame.
AngryHappy
Sad
SHAPE EIGENSIGNATURE
Happy
Sad
Angry
COMBINED EIGENSIGNATURE
Modelling: Construct ARP’s
3 video clips yield: Happy eigensignature Sad eigensignature Angry eigensignature
Consider eigensignature as a multivariate time series.
Separate application of multivariate-ARP’s Results: separate ARP emotion models.
Models used to generate new emotion specific eigensignatures-(stage 2: SYNTHESIS)
Stage 2: Synthesis
Generate eigensignatures Potentially infinitely long Retains look and feel of original.
Original Eigensignature: Emotion Angry
Generated ARPEigensignature: Emotion Angry
Synthesis: Generated Shape ARP sequences:
Original Angry ARP Generated Angry
Using linear shape space PCA models.Project generated shape eigensignature back to shape space.
Synthesis: Generated Shape ARP sequences:
ARP Generated Happy ARP Generated Sad
Synthesis: Generated Combined space ARP sequences:
Original Angry ARP Generated Angry
Using combined appearance PCA models. Project generated eigensignatures back to combined appearance
space.
Synthesis: Generated Combined space ARP sequences:
ARP Generated Happy ARP Generated Sad
Application: User-driven synthesis
Approach: Self organising maps (SOMs) SOM’s project the n-D ARP emotion data into a 2D
image space, defined as ‘expression space’.
Tool development Expression Space visualises emotion ARP models. Navigate space. Automatic generation new emotion specific videos.
Self-Organising Maps
Clustering technique Kohonen Neural Network Visualises n-observations as
g-groups.
SOM splits into 2 elements1) Data – provided by ARP
models Requires sufficient training
data.
2) Weight vectors components. xy position on map After training weight vectors
equivalent to ARP models.
x1,y1
W1, W2 W3 W4
Weight Vectors ARP-coefficients
x2,y2 x3,y3 x4,y4
SOM UNITS
Self-Organising Map Structure
Technique: Acquisition of SOM training data.
COMPLETE EMOTION EIGENSIGNATURE
• Use N emotion ARP’s to train 50x50 unit SOM.
Split complete eigensignature into frame segments with (length-1)
frame overlap.
1 2 N-segments
• Model each segment with an ARP.• Result: obtain N ARP models for SOM training.
Results After training :•Each unit of the SOM is equivalent to a single ARP model.•ARP emotion models are clustered.
Technique: Forming Expression Space
Generate Expression space by colour coding the SOM units. Use an Euclidean distance scheme, based on comparison between the sample ARP
emotion models used for training and SOM map unit ARP’s.
Red units most similar to angry ARP models
Green units most similar
to happy
Blue unit associate to
sad ARP models
Intensity of colour indicates the better match to an emotion group.
Results: SOM Shape Space SOM Combined Space SOM
Emotions cluster into 3 distinct regions. Distinct separation between angry and all other emotions. Large transitional region between sad and happy. Dark regions not closely related to any emotion.
Angry Cluster
Sad Cluster
Happy Cluster
Angry Cluster
Sad Cluster
Happy Cluster
Navigating Expression Space
This tool provides an intuitive interface.
Aimed at non-specialist users
Allows easy traversal of the highly dimensional expression space.
Automatically generates new facial expression sequences for selected emotions.
SOMVisualisation
Generated Expression
Navigating the Combined Appearance Space
SOMVisualisation
GeneratedVideo texture
Complete Eigensignature
Generated Eigensignature
Results: Using different ARP models from the SOM to generate sad crowd.
Mixed Crowd Scene
Conclusions
Demonstrated approach which can reconstruct and generate novel footage.
Successful use of PCA and ARP’s: To construct emotion models that correctly co-
articulate head motion and facial expression.
Eigensignature representation unique for each emotion. Emotion ARP’s cluster distinctly Clustered space easily navigable by non-
specialists.
Finally……
Any questions?