M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Computer vision approaches to identifying Computer vision approaches to identifying people and possible malfeasant behavior. people and possible malfeasant behavior.
Dimitris N. Metaxas Mark G. FrankDimitris N. Metaxas Mark G. Frank Director, Computational Biomedicine School of Communication, Imaging & Modeling Center Information & Library Studies
with special thanks to: Paul Ekman, UCSFUCSF;Sinuk Kang, Amy Marie Keller, Anastacia Kurylo, Maggie Herbasz, Belida Uckun, RutgersRutgers;Jeff Cohn, PittPitt; Takeo Kanade, CMUCMU; Javier Movellan & Marni Bartlett, UCSDUCSD.David Dinges, UPENN
Also thanks to: Office of Naval Research, National Science Foundation (ITR program), AFOSR, DARPA
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Signals relevant to counter-terror.Signals relevant to counter-terror.
• Identification of bad guysIdentification of bad guys• Changes in gait with loads as small as 1 kgChanges in gait with loads as small as 1 kg• Anger prior to imminent attackAnger prior to imminent attack• Fear/distress when lyingFear/distress when lying
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Take a closer look…Take a closer look…
Kim Philby, 1960’sKim Philby, 1960’s (Frank & Ekman, 2003)(Frank & Ekman, 2003)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
How lies are betrayed.How lies are betrayed.
LieLie
Cognitive cluesCognitive clues
- Contradictory statements- Contradictory statements- HesitationsHesitations
-Speech errorsSpeech errors-Reduced illustratorsReduced illustrators
-Contradictory emblemsContradictory emblems-Reduced detailReduced detail
-Etc.Etc.
Emotional cluesEmotional clues
Lying about feelingsLying about feelings Feelings about lyingFeelings about lying
-Look for -Look for reliablereliable
signs of signs of emotionemotion
-Duping delight-Duping delight-GuiltGuilt
-DetectionDetectionapprehensionapprehension
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
The Facial Action Coding System The Facial Action Coding System (FACS)(FACS)
Ekman & Friesen, 1978
Action code: 1, 2, 4, 5, 7, 20, 26
1 Inner brow raise2 Outer brow raise4 Brow lower5 Upper lid raise7 Lid tighten20 Lip stretch26 Jaw drop
46 Action Units
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Challenges facing behavioral science:Challenges facing behavioral science:
• Advantages:Advantages:- reliably identify people & behaviors - non-obtrusive- non-inferential, allows for discovery
• Disadvantage:Disadvantage:- laborious - mistaken identity via cognitive capacity, disguise, etc
• SolutionSolution- automatic computer vision techniques
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Gait recognition
• Identify people from the way they walk
• Important for surveillance and intrusion detection
• What are good features for identifying a person?– i.e., what features are person-specific?
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Background
Sagittal plane - divides body into left and right halves
Limb segment - a vector between two sites on a particular limb
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Elevation Angles
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
The trajectories of the sagittal elevation angles are invariant across different subjects.
As a consequence, person-independent gait recognition will require less training data.
(Borghese et al., 1996)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
The cyclogram
• Elevation angles trace curve in a 4D space
• Curve is called “cyclogram”
• Cyclogram lies in a 2D plane– Well, almost
• Hypothesis: deviation of cyclogram from plane is person-specific
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Cyclogram example
Curve is cyclogram projected into best-fit plane
Green points are real points of cyclogram
Red lines trace the deviation of points from plane (exaggerated scale)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Cyclogram sequence
• Deviation from cyclogram plane can be represented as a sequence
• e.g., CCCGTTTTATATTTTTAAAAGCCGGTAAATTAGGGG
• Compare sequences between people via longest common subsequence (LCS) matching– Well-known dynamic programming algorithm,
used in computational biology
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Examples of People Detection
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Examples of Gait Analysis
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Face: Tracking: Stress Recognition
• Identify which Facial Features (space and time) are important to recognize stress
• Assymetries and Movements around the mouth and eyebrows
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Slope = Asymmetry
A horizontal line would indicate no asymmetry. This facial expression, however, is generally slanted upward and to the left.
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Plots of high and low stress
Expression of high stress in form ofasymmetrical facial expression
time
asym
met
ry
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
In contrast, low stress
time
asym
met
ry
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Some more high stress from different subjects
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Face: Tracking: Stress Recognition
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Face: Tracking: Stress Recognition
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Subtle brow changes important.Subtle brow changes important.
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Technical Challenges.Technical Challenges.
• PosePose• Head motionHead motion• Occlusion from glasses, facial hair, rotation, Occlusion from glasses, facial hair, rotation,
handshands• TalkingTalking• Video qualityVideo quality• Frame rate (blinks)Frame rate (blinks)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Conclusions.Conclusions.
• There are reliable means to identify people as There are reliable means to identify people as well as behaviors associated with deception and well as behaviors associated with deception and hostile intent.hostile intent.
• We can detect these behaviors. We can detect these behaviors.
• We can represent them digitally.We can represent them digitally.
• Can this make us more secure?Can this make us more secure?