1
Supervisors:
Professor Dr. Mohammed Roushdy
Dr. Haythem El-Messiry
T.A. Ahmad Salah
Multi-touch Interactive Surface
2
Team Members
Mennat - Allah Mostafa Mohammad Computer Science
Nada Sherif Abd El Galeel Computer Science
Rana Mohammad Ali Roshdy Computer Science
Sarah Ismail Ibrahim Computer Science
3
Agenda
1. Introduction
2. Physical Environment and Framework
3. Project Modules and Applications
4. Challenges
5. Conclusion and Future work
6. Tools and References
4
Motivation A more natural and direct way of Human Computer
Interaction (HCI).
Current Multi-touch devices are: Expensive Heavy Fragile Consume space
5
Problem Definition
It would be more comfortable, effective and user friendly if the user could interact directly with the display device without any hardware equipments, just using his hands’ gestures.
Our goal is to deliver an interactive surface characterized by low cost, efficiency and ease of use in real life applications.
6
Overview
Optical camera-projector system
Generic Framework for Human Computer Interaction (HCI) using hand gestures.
7
Physical Environment Physical environment consists of:
1. A projector.
2. A webcam placed over the projector’s lens capturing the projected surface.
8
1.25 m
0.8 m
9
Physical Environment
Camera
Projector
Surface
2.15 m
10
Framework
Controller
Configuration Module
Input Module
Hand Tracking
Hand Segmentation
Hand gesture Recognition
Interface
11
Controller Module
12
Controller Module
Detect Corners
SegmentationConstruct the
search windowTrack the
hand
Color Mapping
Search for hand in entry point
Fire EventGesture
Recognition
13
Configuration Module
14
Colors Mapping Maps the colors between the desktop and the captured
image colors. A set of colors are projected and captured for the color
calibration process.
Desktop Colors Projected Colors
15
Corner Detection
The four corners of the image are automatically detected using fast corner detection algorithm.
16
Input Module
17
Calibrate captured image according to the four calibration points.
Geometric Calibration
Captured Image Calibrated image
18
Hand Tracking Module
19
Kalman Filter
The Kalman filter algorithm is essentially a set of recursive equations that implement a predictor-corrector estimator.
Steps: Initialization Prediction Correction
20
Hand Segmentation Module
21
Skin Color Detection
The hand is affected by the projector’s light which results in generating different texture patterns on the hand’s surface which excludes any skin detection algorithm.
Captured image
Skin detection applied
22
Subtraction using Color Calibration
Subtract captured image form the desktop image.
Convert colors of the desktop image to that of the captured image.
23
Color Calibration Get colors’ training set,
each desktop color and its
corresponding projected color.
Divide each pair of images into regions (3x3).
Calculate the transformation matrix A for each region.
b=A * x ; where b is the calibrated color, A is the transformation matrix, x the desktop color.
24
Segmentation Results
Desktop Captured Segmented Global thresholding
Largest BlobExtraction
Blob Analysis A heuristic method to extract the hand from the arm is
applied using morphological operations.
A bounding box (60 * 60) is constructed around the largest blob.
Original Closed Original - Closed
26
Hand Gesture Recognition Module
27
Hand Gesture Recognition
EFDDB
Gesture Type
Contour Tracing
EFD
Contour Re-
sampling
28
Elliptical Fourier Descriptors Elliptical Fourier descriptors are a parametric representation
of closed contours based onharmonically related ellipses.
Any closed contour can be constructed from an infinite set of Elliptical Fourier descriptors.
29
TrainingTraining Set :
60 training images from each gesture
30
Testing Results
Gesture Testing set 1 (68 images) Testing set 2 (119 images)
98 % [1 misclassified] 99% [1 misclassified]
95% [3 misclassified] 97% [3 misclassified]
97% [2 misclassified] 98% [2 misclassified]
98% [1 misclassified] 93% [8 misclassified]
95 %[3 misclassified] 95% [5 misclassified]
31
Interface Module
32
Interface Module
A gesture event is fired whenever the framework recognizes a gesture, the event contain the position and the gesture type.
The interface handles the raised events.
The user can map the gestures to events.
33
Main GesturesEvent Hand Gesture
Pointer
Click
Drag
Zoom
Right Click
34
ApplicationsPuzzle game
Image Viewer
Painter
35
Demo
36
Challenges
Controller Module: Multi hands tracking
Gesture Recognition Module: Similar gestures
Segmentation Module Dark and complex backgrounds Arm Extraction
37
Conclusion
Human Computer Interaction field is still an open field.
Image processing can be very powerful if used in the appropriate environment.
38
Future Work
Using the depth Z- axis besides X and Y axes for determining the hand position.
Multi hands’ and multi users’ interaction.
Interactive Wall can be used with another surface other than the projector, for example a large screen can be used.
Recognize dynamic gestures.
39
ToolsSoftware Microsoft Visual Studio 2008 MatLab OpenCV
Hardware Optical Camera Projector
40
References Edward Rosten, Reid Porter, and Tom Drummond, Faster and better: a machine learning
approach to corner detection, Los Alamos National Laboratory, Los Alamos, New Mexico, USA, 87544, Cambridge University, Cambridge University Engineering Department, Trumpington Street, Cambridge, UK, CB2 1PZ, October 14, 2008.
Yongwon Jeong and Richard J. Radke, Reslicing axially-sampled 3D shapes using elliptic
Fourier descriptors, Department of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute, USA, 2007.
Louis Patrick Nicoli, Automatic Target Recognition of Synthetic Aperture Radar Images
using Elliptical Fourier Descriptors, Florida Institute of Technology, Melbourne, Florida, August, 2007.
G. Amayeh, G. Bebis, A. Erol, and M. Nicolescu, A New Approach to Hand-Based
Authentication, Computer Vision Laboratory, University of Nevada, Reno, 2007.
Asanterabi Malima, Erol Özgür, and Müjdat Çetin, A Fast algorithm for vision-based hand gesture recognition robot control, Faculty of Engineering and Natural Science, Sabancı University, Tuzla, İstanbul, Turkey, 2006.
41
References Greg Welch and Gary Bishop, An Introduction to the Kalman Filter, Department of
Computer Science University of North Carolina at Chapel Hill, NC 27599-3175, Updated: Monday July 24, 2006.
E. Rosten and T. Drummond, Machine learning for high-speed corner detection, European Conference on Computer Vision, May 2006.
Rafael C.Gonzalez, Richard E.Woods, Digital Image Processing ,Second Edition, 2006.
Erik Cuevas, Daniel Zaldivar and Raul Rojas, Kalman filter for vision tracking, Freie Universität Berlin, Institut für Informatik Takustr. 9, D 14195 Berlin, Germany Universidad de Guadalajara Av. Revolucion No. 1500, C.P. 44430, Guadalajara, Jal., Mexico, August 10, 2005.
Jason J. Corso, Techniques for vision based Human computer interaction, A dissertation
submitted to The Johns Hopkins University in conformity with the requirements for the degree of Doctor of Philosophy, Baltimore, Maryland, August 2005.
.
42
References Marcelo Bernardes Vieira, Luiz Velho, Asla S´a, Paulo Cezar Carvalho, A Camera
Projector System for Real-Time 3D Video, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), Instituto de Matem´atica Pura e Aplicada Est. Dona Castorina, 110, Riode Janeiro, Brazil, 2005
Ngon T.Truong, Jae-Gyun Gwag, Yong-Jin Park, and Suk-Ha Lee, Genetic Diversity of Soybean Pod Shape Based on Elliptical Fourier Descriptors, Dep. of Plant Science, Seoul National University, Seoul 151-742, Korea Dep. Of Crop Sciences, Can Tho University, Can Tho, Viet Nam Genetic Resources Div., National Institute of Agricultural Biotechnology, Suwon 441-707, Korea, 2005.
E. Rosten and T. Drummond, Fusing Points and Lines for High Performance Tracking, ICCV, 2005.
Attila Licsár1, Tamás Szirányi, Dynamic Training of Hand Gesture Recognition System,
Proceedings of the 17th International Conference on Pattern Recognition (ICPR’04), University of Veszprém, Department of Image Processing and Neurocomputing, H-8200 Veszprém, Egyetem u. 10. Hungary. Analogical & Neural Computing Laboratory, Computer & Automation Research Institute, Hungarian Academy of Sciences, H 1111 Budapest, Kende u. 13-17, Hungary, 2004.
43
References Attila Licsár1, Tamás Szirányi, lecture notes in computer science, University of Veszprém,
Department of Image Processing and Neurocomputing, H-8200 Veszprém, Egyetem u. 10. Hungary. Analogical & Neural Computing Laboratory, Computer & Automation Research Institute, Hungarian Academy of Sciences, H 1111 Budapest, Kende u. 13-17, Hungary, 2004
Stephen Wolf, Color Correction Matrix for Digital Still and Video Imaging Systems, U.S. DEPARTMENT OF COMMERCE, December 2003.
Qing Chen, Evaluation of OCR Algorithms for Images with Different Spatial Resolutions and Noises, School of Information Technology and Engineering Faculty of Engineering University of Ottawa ©, Ottawa, Canada, 2003.
Vladimir Vezhnevets _ Vassili Sazonov Alla Andreeva, A Survey on Pixel-Based Skin Color Detection Techniques, Graphics and Media Laboratory † Faculty of Computational Mathematics and Cybernetics Moscow State University, Moscow, Russia, 2003.
Yasushi HAMADA, Nobutaka SHIMADA, Yoshiaki SHIRAI, Hand Shape Estimation Using Sequence
of Multi-Ocular Images Based on Transition Network, Department of Computer-Controlled Mechanical System, Osaka University, Japan, 2002.
Dengsheng Zhang and Guojun Lu, A Comparative Study on Shape Retrieval Using Fourier Descriptors with Different Shape Signatures, Gippsland School of Computing and Information Technology Monash University Churchill, Victoria 3842, Australia, 2001.
44
References Douglas Chai and AbdElsalam Bouzerdoum, A Bayesian approach to skin color
classification in YbCr color space, School of engineering and mathematics, Edith Cowan University, Australia, 2000
Kenny Teng, Jeremy Ng, Shirlene Lim, Computer Vision Based Sign Language Recognition for Numbers.
Nguyen Dang Binh, Enokida Shuichi, Toshiaki Ejima, Real-Time Hand Tracking and Gesture Recognation System, GVIP 05 Conference, 19-21 December 2005, CICC, Cairo, Egypt, Intelligence Media Laboratory, Kyushu Institute of Technology 680-4, Kawazu, Iizuka, Fukuoka 820, JAPAN.
G. Amayeh, G. Bebis, A. Erol, and M. Nicolescu, A New Approach to Hand-Based
Authentication, Computer Vision Laboratory, University of Nevada, Reno.
A. M. Hamad, Fawzia shaaban, Mona Gabr, Noha Sayed, Rabab Hussien, Robot Vision, Faculty of computer and Information Sciences Ain Shams University, Cairo, Egypt, 2008.
45