Hand Gesture Recognition
By Jonathan Pritchard
Outline • Motivation
• Methods
o Kinematic Models
o Feature Extraction
• Implemented Algorithm
• Results
Motivation • Virtual Reality – Manipulation of virtual objects with
one’s hands.
• Robotics/Telepresence – Precise control of
machinery from remote locations.
• Sign Language – Help the disabled interact with
computers. ASL can be used as test bed for
different algorithms.
Murthy, G. R. S., & Jadon, R. S. (2009). A review of vision based hand gestures recognition. International Journal of Information Technology and Knowledge Management, 2(2), 405-410.
Kinematic Models • Simplifying assumptions about hand motion used to
limit the degrees of freedom in the model
• Many model based approaches use a form of
causal tracking to ease computation.
o Filtering used to estimate state (pose, gesture covariance
matrix) based on previous state(s)
• Wire Frame and Silhouette models
J. M. Rehg and T. Kanade. “Visual tracking of high DOF articulated structures: an application to human hand tracking”. In J.-O. Eklundh, editor, Proc. 3rd European Conf. on Computer Vision, volume II of Lecture Notes in Computer Science 801, pages 35–46. Springer-Verlag, May 1994.
Kinematic Models: Wire Frame
J. M. Rehg and T. Kanade. “Visual tracking of high DOF articulated structures: an application to human hand tracking”. In J.-O. Eklundh, editor, Proc. 3rd European Conf. on Computer Vision, volume II of Lecture Notes in Computer Science 801, pages 35–46. Springer-Verlag, May 1994.
Stereo Vision - Hand Features Identified
Pose Estimation
3D Model
Filtering
Kinematic Models: Silhouette
Stenger, B., Mendonca, P. & Cipolla, R. “Model-Based 3D Tracking of an Articulated Hand”. In IEEE Conference on Computer Vision and Pattern Recognition, (2001) 310–315.
Silhouette matched to gesture outline with error minimizing
Kalman Filtering
Silhouettes From 3D Model
3D Model (Truncated Quadratics)
Feature Extraction • “Getting your man without finding his body parts”
• Low level image features used to extract
information without estimating pose
• Not nearly as robust as model based approaches,
but far simpler and faster to compute.
R. Polana and R. Nelson, “Low level recognition of human motion”, in Proc. of IEEE Workshop on Motion of Non-Rigid and Articulated Objects, Austin, 1994, pp. 77–82.
Feature Extraction: Number of Fingers
New, J. R., Hasanbelliu, E. and Aguilar, M. “Facilitating User Interaction with Complex Systems via Hand Gesture Recognition.” In Proc. of Southeastern ACM Conf., Savannah, (2003).
Threshold applied to saturation space, only largest connected contour kept
Image separated into HSL color spaces
Wrist removed, centroid calculated Circle centered at centroid used to calculate number of fingers
Feature Extraction: Fingertips
J. Raheja, K. Das & A. Chaudhary “Fingertip Detection: A Fast Method with Natural Hand“. International Journal of Embedded Systems and Computer Engineering , Vol. 3, No. 2, July-December 2011, pp 85-88
Orientation found by comparing oriented
histograms Fingertips detected through
algorithm looking at top edge of hand, and it’s
derivative
HSV color space used to obtain binary image
Implemented Algorithm • Fingertip Detection using MATLAB image processing
toolbox.
• Combination of previous feature extraction
algorithms
o HSV color space used to threshold binary image
o Detect orientation of hand, find outline of top
o Use outline values, derivative filter, and knowledge of hand
orientation to locate fingertips.
Preliminary Results
Binary Image
Result
Top Outline
Fingertips
Preliminary Results
Binary Image
Top Outline
Fingertips Result
Preliminary Results
Binary Image
Top Outline
Fingertips Result
Preliminary Results
Binary Image
Top Outline
Fingertips Result
Continued Work • Orientation invariance through wrist detection
• Calculate centroid of binary image
o Reject detected fingertips that are too close to centroid
• Subtract wrist for more accurate centroid
calculation.
Questions?
M. Randall “Questions”, XKCD, no. 1256 Available: http://imgs.xkcd.com/comics/questions.png