Srinivas Ganapathyraju, Ph.D Sheridan Institute of Technology and Advanced Learning

Brampton, Ontario, Canada. ganapats@sheridaninstitute.ca

Abstract— This paper presents a method for hand gesture recognition to control a 6 axis industrial robot. The image is acquired by means of a web cam system, and undergoes several processing stages before a meaningful recognition can be achieved. Some of these stages include skin detection to effectively capture only the skin region of the hand, noise elimination, application of the convex hull algorithm to get the outline of the hand, and apply convexity hull defects algorithm to determine the finger count. Once the finger count has been determined, the information is transmitted via serial communication to the industrial robot controller. The end effector of the robot moves in four different directions based on the finger count input received from the hand gesture recognition module.

Keywords- Hand Gesture Recognition, Robotics, Control, Skin Detection, Convexity Hull.

I. INTRODUCTION Hand gesturing or gesturing using bodily actions has been

used by man since the dawn of civilization as a means of communicating messages, and also used in parallel with spoken words. The meanings of hand gesturing, varies depending on geographical and cultural backgrounds. Gesturing can also be profession specific, such as hand gesturing used by the military or navy personnel to communicate very specific information. Human computer interaction (HCI) generally requires the use of hardware devices. For robotic devices, HCI is achieved by means of the teach pendant which is used to powering up the robot, entry and editing of programs, and for program execution. With advances in computer vision technology, hand gesture communication is seeing a lot of application in human computer interaction. An application in computer gaming were gesturing is used is the Microsoft Kinetic device [1], which recognizes hand gestures by means of a vision system, to perform gaming actions in a virtual gaming environment. Other applications using a vision system for hand gesture recognition have been developed for multimedia device control [2] and also simulating the actions of a computer mouse [3]. Research has also been done is controlling a wheel chair using head gestures [4]. Methods without using vision systems have also been developed, such as wearable gloves [5], which have inbuilt sensors to recognize hand and finger motion and orientation, which in turn is used to control a

hardware device. Some of the major drawbacks of wearable gloves are the cost and also the need for sophisticated sensors and hardware. Several different methods have been proposed for hand gesture recognition using vision systems. One method proposed involves the use of machine learning based on the K- Nearest Neighbors (KNN) algorithm to recognize hand gestures used for sign language [6]. The advantage of this method is that the system can be trained to recognize a vast number of different hand gestures. The major drawback of such a system would be the vast number of samples required for training the system. Another method proposed for hand gesture recognition uses Artificial Neural Networks (ANN) for the training of hand gestures [7]. This method also requires a large set of training samples. To overcome the need of a large set of training samples, the method proposed in this paper uses logical heuristics, through the application convex hull and convexity hull defects for hand gesture recognition.

II. OPEN CV AND EMGU CV The image processing part of this project was developed

using Microsoft Visual C# and OpenCV [8]. OpenCV is an open source computer vision library developed by Intel Corporation. OpenCV is written entirely in C++, which allows for fast image processing. It has a library of over 2500 different algorithms. OpenCV has interfaces that allow for programming using C++, C, Python and soon JAVA, and can be executed on different platforms such as Windows, Linux, and Android. One major drawback is that GUI interfaces cannot be developed directly using OpenCV. In order to create GUI applications using the .Net platform using programming languages such as Visual C++, C# and Visual Basic, third party wrappers need to be integrated with the OpenCV platform. For our project, the Emgu CV [9], platform was used, since it was entirely developed using C#, and the GUI interfaces for this project were developed in C# it made development and integration of OpenCV easier.

III. PROPOSED SYSTEM ARCHITECTURE The system architecture proposed is divided into two

modules, firstly the Image processing and serial communication module which acquires images of hand gestures using a webcam system and processes the images

Bali, Indonesia, August 28-30, 20132013 3rd International Conference on Instrumentation Control and Automation (ICA)

63

Hand Gesture Recognition Using Convexity Hull Defects to Control an Industrial Robot

978-1-4673-5798-2/13/$31.00 ©2013 Crown

through several stages and communicates the information using serial communication to the second module which is the robot control module. The robot control module gets the finger count from the image processing module and sends the hand gesture commands to the 6 axis robot via the robot controller. The flow diagram of the proposed system is shown in Figure 1. The Graphical User Interface (GUI) is shown in Figure 2.

A. Image Processing and Serial Communication module The image acquisition uses a 5 megapixel webcam system,

and the image processing and serial communication module was developed using OpenCV and the C# programming language using Microsoft Visual Studio 2010, 64 bit version. To integrate the OpenCV with the C# language, the EmguCV wrapper was used. This module is divided into several stages to improve the hand gesture recognition process. Each stage of the process is described in detail in the following sections below.

Figure 1: Proposed System Architecture 1) Image Acquisition

The image acquisition is accomplished by means of a 5 mega pixel web cam, which captures images frame by frame. Images from the web camera are acquired at a rate of 30 frames per second (FPS). OpenCV has functions that allow image acquisition directly from a web cam. The graphical user interface was designed to allow the selection of web cams from a selection list for all web cams that are connected to the computer’s USB port. Each frame is processed through a series of stages as shown in Figure 1, before an actual finger count can be derived.

2) Skin Detection Due to the environment in which the hand gesture

recognition will be performed, it is important to be able to isolate the hand gesture from the rest of the environment in order to accurately apply the contour detection, convexity hull and convexity hull defects algorithm. Different algorithms have been developed for skin detection, the most widely used ones being the RGB color space, HSV (Hue, Saturation, Value) and the YCbCr color space [ ][ ]. For this project the YCbCr algorithm was chosen as it is being widely used for digital video processing and also the higher success rates that have been obtained using this algorithm. In order to achieve robust skin color detection under varying illumination conditions, it is important to separate the luminance from the chrominance of the skin. The YCbCr algorithm does this by separating the luminance in the Y component from the chrominance described through Cb and Cr components. The corresponding skin pixels can be classified with the following values 60 Y 255, 100 Cb 125, 135 Cr 170, Where Y,Cb,Cr =[0,255].

Figure 2: The Graphical User Interface 3) Noise Filtering After applying skin detection to the acquired image, there

will still be some noise that needs to be removed, which is done by deleting noisy pixels from the image, by applying morphological image processing on the image. Firstly image erosion is applied, which trims down the image area where the hand is not present. The second stage is to apply image dilation which effectively enlarges the area image pixels that have not been eroded. The functions in OpenCV for erode and dilation are cvErode and cvDilate respectively.

4) Contour Detection A contour by definition is an outline of an object or a body

usually defined by a line. In computer vision the contour of an object can be used to compute the convex hull of the object from which the convexity defects can be computed. After the skin detection and noise filtering algorithms have been applied, an outline of the hand needs to be generated by applying the contour detection algorithm in OpenCV. This is achieved by applying the cvDrawContours() function in OpenCV.

64

5) Convexity Hull and Convexity Hull Defects In order to determine the shape of objects or a contour that

has already been derived, the objects convex hull and convexity defects [6] can be computed. The Figure 3, shows the outline of the human hand. The convex hull of the image is shown as the dark outline, and the defects which are relative to that outline are labeled A through H. These defects can be used to distinguish the condition of the hand. Three different functions need to be applied in OpenCV to determine these defects. The first function computes the hull of the contour as already described. The second functions in OpenCV is the cvCheckContourConvexity(), which checks to see if the contour is a convex or not. The third function applied in OpenCV is the cvConvexityDefects() function which determines the possible defects present in the image and also their sequence.

Figure 3: Convex Hull and Convexity Defects [8]

6) Finger Counting In order to get the count of the fingers, the convexity defects

described previously are used. Four major points are required from the convexity defects as shown in Figure [4]. They are the start point SX, the Depth Point DY, the Box center point BY, and the length of the defect LD. For each convexity defect CD, the following algorithm [10] is applied to get the total finger count.

Count = {1} If (SY < BY or DY < BY) and (SY < DY) and (LD > box height/ n) Where, LD = (SX – DX)2 + (SY – DY)2

Else Count = {0}. Number of Fingers counted = Σ Count.

7) Serial Communication The next step, once the finger count has been achieved is to

transmit that information to the robot controller. This is done by means of serial communication. The computer is fitted with a USB to Serial cable which is connected to the COM port of the robots controller. The graphical user interface (GUI) on the computer side allows the user to select the baud rate, the COM port, and the number of data bits. The GUI also has text

boxes to display data sent to the robot controller and also data received from the robot controller.

Figure 4: Points used to compute finger count

B. Robot Control Module In this project, an ABB IRB 120 6 axis robot was used. The

robot controller is an IRC5 compact controller. The robot control module is responsible for a number of functions in this experimental work. The finger count information received from the hand gesture recognition is used to move the robot arm in different directions as described below. The USB to serial cable from the computer is connected to the COM 1 port of the IRC5 controller. Programming the controller to read and write through the serial port as well as robot joint manipulation is done using the RAPID programming language. The sections below describe the serial communication and the manipulation of robot arm in more detail.

1) Computer to Controller Serial Communication The first stage of the robot control module is to be able to

communicate with the computer by means of serial communication. The graphical user interface on the computer allows the user to select the COM port on the computer that is connected to the robot controller, the baud rate, and also the data bits for the information that will be communicated back and forth between the computer and the robot controller. Programming the serial communication on the controller side was done using the RAPID programming language.

2) Robot Joint Manipulation The second stage of the RAPID program that was written

for the robot controller gets the finger count integer value via serial communication as described above and manipulates the joint accordingly as shown in Table 1. In order to understand how the robot joint motions are manipulated, one will need to understand the different coordinate systems that are used to describe the motion of the robots manipulator joints, and tool. The main coordinate systems used in describing the work

65

space for the robotic manipulator are shown in the Figure 5. The world coordinate system defines a reference to the floor, which is the starting point for the other coordinate systems. Using this coordinate system, it is possible to relate the robot position to a fixed point in the workspace. The world coordinate system is also very useful when two robots work together or when using a robot carrier. The base coordinate system is attached to the base mounting surface of the robot. The tool coordinate system or the tool center point (TCP) as it is sometimes referred to is a point that can be located anywhere on the tool. The positions recorded during robot motion are the position of the TCP. The user coordinate system or frame specifies the position of a fixture or work piece manipulator. The object coordinate system specifies the coordinates of the object to be manipulated and usually programmed with reference to the user coordinate system. The advantage of doing this is if the location of the fixture which is defined by the user coordinate system is located to a new position, an entirely new program will not be required to be written; rather only the new user coordinate system will need to be defined for the new location of the fixture. The object coordinate system will then reference itself to the new user coordinate system. It is based on the concept that we manipulate the robots end effector in this project. When the finger count is number 2, for example, the object coordinate system will be modified by a (–ve) 100mm in the X coordinate. By using this method it is possible to move the robot joint in the X,Y and Z coordinates as well as also rotate the manipulator joint, though this was not done in this project since only four hand gestures were used.

TABLE I. HAND GESTURE AND EQUIVALENT ROBOT MOVEMENT

Hand Gesture Number Robot Movement

1

Move Forward 100mm

2

Move Back 100mm

3

Move Right 100mm

4

Move Left 100mm

Figure 5: Robot Coordinate System [11]

IV. RESULTS AND DISCUSSION The web camera acquires hand gesture images which are

processed through a number of stages such as skin detection, noise reduction, and application of the convex hull algorithm and finger count heuristic algorithm to determine the finger count. The finger count information is then relayed via serial communication to the robot controller which directs the robotic arm to move in the prescribed direction. During the experimental stage of the system, a number of factors were observed to affect the accuracy of the hand gesture recognition. These factors include background noise from background images that had a similar color contrast to that of the skin color. Poor lighting also affected the gesture recognition. Other factors that affected the recognition were the angle of tilt of the hand and orientation. These draw backs could be rectified by improving the background lighting, ensuring that the background environment hand a color that was different from the skin color. The hand orientation problem could be rectified by practicing the right orientation to ensure more accurate finger count, though there are other factors that might also be affecting the accuracy of the system that needs to be investigated further.

V. APPLICATIONS There are number of areas that hand gesture recognition to

control an articulated robot arm can be used, such as tele-operated robots controlled over the internet or other wireless communication methods. Applications in telemedicine or surgery would also be an area of application. The traditional way of programming industrial robots is by means of using the teach pendant or software loaded on a computer which can be used to generate simulations as well as the path program. By enhancing and improving the hand gesture recognition techniques it would be possible to program the robot paths similar in function to that of the teach pendent.

VI. CONCLUSIONS AND FUTURE WORK The system presented in this paper shows that hand

gestures can be used to control an industrial robot. There are some limitations that need to be addressed, such as the finger count which is five with the current system. A way to implement and recognize more gestures that can be used to enable the robot to move in any direction at any distance and

66

also to be able to control the orientation of the tool would be very useful. Controlling the speed would also be a very useful attribute. The experimental results also show there is a limitation in terms of accuracy of finger count recognition. Alternate methods to improve the recognition of the hand gesture will need to be explored.

ACKNOWLEDGMENT I would like to thank Dean Jocelyn Piercy and Associate

Dean Farzad Rayegani for making the attendance to the conference possible. I would also like to thank Mr. Andrew Orton for all his help with the robot setup and useful tips on Rapid programming. A special thanks to Mr. Patrick Keenan for his useful tips in C# programming.

REFERENCES [1] http://www.xbox.com/en-CA/KinectJ. [2] S.G.Rayo, “Hand Gestures and Hand Movement Recognition for

Multimedia Player Control”, February 2008, unpublished thesis.

[3] V.Veeriah, P.L.Swaminathan, “Robust Hand Gesture Recognition Algorithm for simple Mouse Control,” Int. Journal of Computer and Comunication Eng, vol.2, No 2, pp.219- 221, March 2013.

[4] P.Jia, H.H. Hu, T.Lu, K. Yuan, "Head gesture recognition for hands-free control of an intelligent wheelchair", Industrial Robot: An International Journal, Vol. 34 Iss: 1, pp.60 - 68. 2007.

[5] N.X Tran, H.Phan et al, “Wireless Data Glove for Gesture Based Robotic Control”, Proc. 13th Int.Conf. on Human Computer Interaction part II: Novel Interaction Methods and Techniques, pp 271-280, 2009.

[6] J.P Rakhman et al, “Sign Langauage Translation”. [7] G Ariyanto, P.K Patrick, H.W Kwok, G Yan, “Hand Gesture

Recognition Using Neural Networks For Robotic Arm Control”, Proceedings of the National Conf. On Computer Sci & Information Technology, Jan 29-30, 2007, Faculty of Computer Science, University of Indonesia.

[8] G. Bradski and A. Kaehler, ‘Learning OpenCV Computer Vision with the OpenCV Library”, O’Reilly Publications, 2008.

[9] http://www.emgu.com/wiki/index.php/Main_Page [10] P.Kathuria, A.Yoshitaka, “Hand Gesture Recognition by Using Logical

Heuristics”, Japan Adv.Inst. of Science and Tech. School of Info.Science, Report 2012.

[11] ABB, Rapid programming Manual.

67