GESTURE BASED REAL-TIME WIRELESS CONTROL OF AN …Institute of Technology- Nirma University,...

http://www.iaeme.com/IJARET/index.asp 284 [email protected]

International Journal of Advanced Research in Engineering and Technology (IJARET) Volume 9, Issue 3, May – June 2018, pp. 284–292, Article ID: IJARET_09_03_034

Available online at http://www.iaeme.com/ijaret/issues.asp?JType=IJARET&VType=9&IType=3

ISSN Print: 0976-6480 and ISSN Online: 0976-6499

© IAEME Publication

GESTURE BASED REAL-TIME WIRELESS

CONTROL OF AN ELECTRICAL MACHINE

USING IMAGE PROCESSING

Akshay Raju Krishnani and Aman Choudhary

Student, Department of Electrical Engineering,

Institute of Technology- Nirma University, Ahmedabad, Gujarat, India

Dhara M Mehta

Assistant Professor, Department of Electrical Engineering,

Institute of Technology- Nirma University, Ahmedabad, Gujarat, India

ABSTRACT

With the view of reducing number of mechanical devices and in order to make

processing more efficient, a significant research is being carried out in the domain of

Image Processing. In addition to it one of the fastest growing field in this domain is

Gesture Recognition. Implementing the concept of gesture recognition in machine

makes the communication more natural and intuitive. In this paper, an HMI based

system is developed using Raspberry Pi. The real time processing of hand gesture is

executed and accordingly signals are generated which are further used to control the

electrical machine.

Keywords: Gesture recognition, Raspberry-Pi, Contours, Convexity defects.

Cite this Article: Akshay Raju Krishnani, Aman Choudhary and Dhara M Mehta,

Gesture Based Real-Time Wireless Control of an Electrical Machine Using Image

Processing, International Journal of Advanced Research in Engineering and

Technology, 9(3), 2018, pp 284–292.

http://www.iaeme.com/ijaret/issues.asp?JType=IJARET&VType=9&IType=3

1. INTRODUCTION

Image processing is a concept commonly used in 3D imaging, pattern recognition and

navigation. One of its current researches of study is gesture recognition. Gesture recognition

is a non-verbal form of communication. It often originates from hands or faces. It is a way for

computers to understand and interpret human body languages, hence helps in building a

strong co-relation between machines and humans. In this paper the goal is to interpret human

gesture and generate the commands accordingly. Gestures obtained using camera modules are

pre-processed through Raspberry-Pi. Raspberry-Pi is a low cost computer that helps in

learning computing on different programming platforms. It features integrated micro-chip

with ARM compatible CPU and GPU along with Broadcom system on it. The SD card slot is

also provided, where we can store our memory card containing operating system (Raspbian)

and program memory. In addition to this, it is HDMI supported for video output, with 3.5 mm

Gesture Based Real-Time Wireless Control of an Electrical Machine Using Image Processing


phono jack for audio output [1]. There are 40 GPIO pins in Raspberry-Pi model 3, which are

used to implement the functions according to the generated commands or codes.

The Raspbian is a computer operating system for Raspberry Pi. It has an inbuilt python

IDE command window where computer vision algorithms are implemented to interpret

human gestures. Python is used as the programming language to implement the mathematical

equations which interpret the gestures and process the command accordingly. The

calculations of these equations are carried out using NumPy, a scientific computing package

for python coding. The proposed model works efficiently even in poorly illuminated system.

2. RELATED WORK

Several sensor based techniques were developed in past for recognising hand gestures i.e.

instrumented gloves and optical sensors etc. These techniques need the help of infrared lights

and involve complex configurations. In order to overcome this, the approach of image

processing was introduced which did not require any kind of physical device. One way is to

utilize computer vision, a library specifically developed for performing image processing.

Ron Oommen Thomas [2] controlled a Robotic Arm using Raspberry Pi, moreover

programming was done in Python lDE software to generate signals for forward and backward

motion. Tohari Ahmad et al [3] proposed a comparatively economical monitoring system.

The system is capable of discovering an object, as well as locating it in the form of its

coordinates and taking its picture. The data is received in the form an email

AEl-Sawah, N. Georganas, et.al [4], designed a prototype for 3-D hand tracking and

dynamic gesture recognition. Objective is to be able to continuously track the hand in a

general background and to be able to recognize dynamic gestures in real time.

Vinod P. R, Usha Gopalkrishnan, et.al [3], proposed a method for the automatic

recognition of finger spelling in Indian sign language. The proposed method uses digital

image processing techniques and artificial neural network for recognizing different signs.

3. METHODOLOGY

3.1. Image capturing

Raspberry pi camera module is used to capture gestures in BGR (Blue Green Red) format.

The module used here is in 5 Megapixel fixed focused mode, supporting 1080p, 720p and

VGA90 video modes at 30 frames per second. It is light weight and compact in size making it

an ideal choice for mobile based applications. The Raspberry Pi can capture 90

frames/second (fps) for high-speed photography using its camera module [5]. In the

algorithm, the image lying inside region of interest will only be processed. With the use of

Region of Interest, background noises are eliminated. Hence it increases efficiency of the

model. The only drawback is that the gestures must lie inside region of interest otherwise

they would not be recognized.

3.2. Image Processing

3.2.1. Convert image from BGR to grey format

The gestures are captured in BGR format. These gestures are recognized by identifying the

edges through them. This task is difficult as the image is in BGR format. Hence, the images

are required to be converted into grayscale, which also eliminates external noises. The images

are 3-dimensional images in BGR format; whereas grayscale images are 2-dimensional that

further makes the processing fast and simpler.

Akshay Raju Krishnani, Aman Choudhary and Dhara M Mehta


3.2.2. Filtering the grey scaled image

The process of filtering is used to eradicate unwanted noises and details from an image. The

image is filtered using 2D Gaussian filter. This Gaussian filter architecture implements a

convolution module. This convolution module is widely used in computer vision and image

processing including object recognition and image matching. Multiplication operation is the

main block used in convolution operation. The filter reduces the high frequency components

and thus acts as a low pass filter. Gaussian Filter can be implemented by multiplying

Gaussian mask with the original image which is to be blurred [6]. Following are the equations

applied on each pixel for the filtering purpose. The Gaussian filter is defined as:

G(x) =

√

In two dimensions, it is the multiplication of two such Gaussians, one in each dimension.

G(x) =

√

Where G is the Gaussian mask at the location with coordinates x and y, σ is the parameter

which defines the standard deviation of the Gaussian. As the value of σ increases, the effect

of image smoothing will be more. Smoothing can then be implemented by the convolution of

original image I(x,y) of a certain height h and width w with a Gaussian mask G(x,y) as shown

in equation below. It is actually the result obtained by the product of input image with the

small Gaussian matrix. A 2D convoluted image is of same size as that of the Gaussian masks

[6]. When applied in two dimensions, the results obtained are concentric circles with a

Gaussian distribution from a point which is situated at the centre. Each pixel's new value is

set to a weighted average of that pixel's neighbourhood. The original pixel's value receives

the heaviest weight and neighbouring pixels receive smaller weights as their distance to the

original pixel increases [5].

3.2.3. Thresholding of the image

At this stage the grey scaled filtered image is converted into binary form i.e. 1 and 0 form,

which helps in segmenting the hand from the background, where hand is changed to 0 bit or

white color format and background is set to 1 bit or black color format. Segmentation is done

by first selecting a proper threshold value T, all pixels below the threshold are turned into

zero and all pixels above the threshold are turned into one. There are different thresholding

techniques for thresholding an image. In this algorithm, Otsu’s thresholding method is used.

Otsu’s thresholding is an automatic threshold selection based segmentation method. In

this method, a grey level histogram is computed and a probability of each intensity level is

calculated [7].

P0=∑ ∑

P1=∑ ∑

Two types of corresponding mean vectors µ0, µ1 are:

µ(µ0i)T=(∑ ∑

)T

µ(µ0j)T=(∑ ∑

)T

µ(µ1i)T=(∑ ∑

)T



µ(µ1j)T=∑ ∑

)T

The total mean vector µT or the two-dimensional histogram is:

µ(µTi)T=(∑ ∑

)T

µ(µTj)T=∑ ∑

)T

The following relations can be easily verified:

µ0P0 + µ1P1 = µT

P0 + P1 1

Figure 1 Gesture obtained after filtering and thresholding the image captured

3.2.4. Contour Plotting

Contours are the curves drawn around the boundary of continuous points having similar color

or intensities. It is widely used in shape analysis and object detection. In this algorithm, after

thresholding the image, contour is plotted across the hand. Here initially the hand gestures are

detected by using the command as find Contours(). Following the the two steps for featuring

the contours from the edges. First step is scan the frame from left to right and from top to

bottom to find first contour pixel marked. The next is to scan the frame clockwise until the

next pixel value is equal to 1 [5]. Contour with maximum boundary area is found and all the

identified boundary points are stored.

3.2.5. Convex Hull

Convex hull is a set of continuously connected points in the Euclidean space [8]. It is drawn

around the contour. These contour points lie within the convex hull. In this paper, convex hull

acts as an envelope around the hand. Finger tips are identified with the help of kinks formed

in convex hull. This is done by going around each point in a convex hull and calculating the

angles at those points [9].

Figure 2 Convex hull marked across the gesture (green curve) and generation of stop command



3.2.6. Finding convexity defects

Convex hull maintains convexity by using minimum number of points to form itself (hull).

As there is no (very less) overlapping between hull and contour plotted, the defect points are

generated. Defect points are present wherever contour of object is away from the convex hull.

This defect is/are returned in the form of vector using convexity defects. Convexity defects

are calculated using an inbuilt function “cv2. Convexity Defects ()”, this returns 4 values-

start point, end point, farthest point and approximate distance from farthest point. These

points form a triangle. From the triangle obtained by fingers angle is calculated. Using these

angles it can be determined if a finger is held up [10]. Lengths of sides of triangle are

calculated using:

a = √(start[0] − end[0]) 2 + (start[1] − end[1]) 2

b = √(start[0] − far[0]) 2 + (start[1] − far[1]) 2

c = √(end[0] − far[0]) 2 + (end[1] − far[1]) 2

Using cosine rule, angle between the fingers is calculated and if angle 90; Number of

defect points are ignored, If angle 90; Number of defect points are calculated.

Figure 3 Generation of defect points (red dots) with respect to generated gesture

3.3. Control through Raspberry-Pi

Initially, GPIO pins are set low, then they are set high according to the gesture recognized.

Pin numbers used to achieve the outputs are 35, 36, 37, 38 (according to the GPIO pin

configuration of Raspberry Pi model 3B) and pin number 34 is used as the ground for the

circuit. For generating a particular command, the pin numbers of the driver L298 are set high

depending on the control action required. The pin numbers 35 and 37 are set high to move

forward, the pin numbers 36 and 38 are set high to move backward (reverse), the pin numbers

36 and 37 are set high to turn left and the pin numbers 35 and 38 are set high turn right. The

only drawback of using Raspberry-Pi GPIO pins is that it generates garbage value when the

pins are set low. This creates problem because encoder has active low input data pins, hence

it always reads the GPIO pins as high.

In this paper, NPN transistors are used to overcome the issue of garbage value because

the garbage values generated were 0.1-0.4 volts. The transistor is used as a switch in the

circuit, with base energized using output of GPIO port as it operates at a voltage greater than

0.7 volts. Vcc of +5 voltage is given to the collector and emitter is connected to ground using

a pull down resistor of 3.2 KΩ. Output is taken from emitter of the transistor and given to

input data pin of encoder. With the help of pull down resistors, when GPIO pins are set low,

switch is in off state and data pins are grounded. When GPIO pins are set high, switch is

turned on and data pins are deactivated using a high signal.



3.4. Wireless control of motor-

3.4.1. Encoder-

An encoder converts the 4-bit data which is given to its input data pins into serial output

which is further delivered to RF module for transmitting. Using 5 volt supply, the encoder is

turned on. In order to enable the transmission of data the TE pin of encoder IC HT12E must

be set low. Address pins A0-A7 are used to provide data security. It can either be set high or

low resulting in a unique address generated which must match the address generated by the

decoder. Moreover, 1 MΩ resistor is externally provided for operation of internal oscillation

[11].

3.4.2. RF Module

The serial data obtained from encoder is wirelessly transmitted to decoder using an RF

transmitter receiver module. Using an antenna the data is transmitted in the form of radio

waves to the receiver. Frequency of operation is 433MHz. Both the modules, transmitter and

receiver are operated at 5 volts supply [5].

3.4.3. Decoder

Output of the receiver is given to the decoder, which then converts the data into its initial 4

bit data form. This 4 bit data is then given to the input pins of L298 driver. Decoder is turned

on using a 5 volt supply. It has address pins which generate 8 bit address similar to that of

generated by encoder [5]. In addition, LED connected at pin number 17 gives the indication

whether the data is being transferred or not.

3.4.4. L298 driver

L298 is a high volt, high current driver with the capability to accept TLL logic levels. It

operates at 12-35 volts. It consists of dual H-bridge system, where direction of the motor can

be controlled using H-bridge. As it is a dual bridge system, 2 DC motors can be controlled

simultaneously. For the operation of each DC motor, 3 control pins are provided- 2 for

direction control and 1 as an enable. Enable is used as a control pin for operation of motor.

When enable is set high then only the motor operates otherwise no signals are received by

motor [12]. Figure 4. and Figure 5. show the complete hardware at the transmitting side.

Figure 4 Transmitting circuit of the proposed model



Figure 5 Transmitting circuit excluding the raspberry-pi

4. RESULTS

Images shown below are the outputs obtained using the program executed in Raspberry-Pi

Model 3B. A unique command is generated with each gesture and implemented accordingly.

As shown earlier, Figure 2 indicates that the stop command is generated corresponding to the

captured gesture, Figure 3 indicates that the forward direction command is to be executed

corresponding to the gesture captured. These commands and gestures are correlated to each

other with the help of number of defect points obtained. Similarly, Figure 6 indicates that the

backward direction command is generated when gesture is in the form of three consecutive

fingers. Figure 7 indicates that the left direction command is generated when gesture contains

four consecutive fingers. Figure 8 indicates that the backward direction command is

generated when all the five fingers are displayed in front of the camera. Figure 9 displays the

final complete hardware model at the receiving end, all the crucial components used to design

this model are marked on the figure. The power supply for HT12D has been placed beneath

the car because of which is not visible.

Figure 6 Backward command generated corresponding to the gesture captured

Figure 7 Left command generated corresponding to the gesture captured



Figure 8 Right command generated corresponding to the gesture captured

Figure 9 Complete hardware at the receiving side

5. CONCLUSION

Performance of the system is evaluated based on hand gestures recognition. Gesture

controlled machine when designed carefully proved to be an efficient model for applications

like navigation, motion control and pattern recognition. Here embedding the system with

controller (raspberry-pi), it proves out to be low-cost and interactive technique. The controller

utilized here has high computing speed, hence producing better results.

The proposed model is a real time based system, implemented along with python

programming language, Open CV library and raspberry-pi module. Five types of gestures are

considered here for controlling purpose and based on the detected gesture, signals are

transmitted. Commands obtained through the algorithm is routed to the GPIO pins of

controller, and according to the generated command, machine is driven to one of the four

defined directions. The signals are transmitted wirelessly using radio frequency. The

generated commands are encoded and data is serially sent to the transmitter. The receiver

captures that data and then decoder decodes it. The decoded data is then given to the motor

driver L298 which further controls the motor accordingly.

6. FUTURE SCOPE

The control scheme used in this project i.e. gesture recognition can be further extended to

skin detection method which will eliminate any changes occurring in the background. Even

with a slight improvisation in our program, it can directly focus on the subject i.e our hand

and thereby removing the background in the frame. In order to make the code more efficient

a data set containing 500-1000 images of gestures can be generated, any gesture developed

by the person can be compared to these stored images and commands can be generated

accordingly. Moreover, a strategy to control the speed of a machine can be imparted by

implementing PWM technique. Hence it would work like a conventional car by controlling

the speed of the vehicle along with the direction.



REFERENCES

1. Raspberry Pi 3 Model B datasheet, RS Company, www.rs-components.com/raspberrypi.

2. Thomas, Oommen, Ron and Rajasekaran, K. Remote control of robotic arm using raspberry

pi. International Journal of Emerging Technology in Computer Science &Electronics

(IJETCSE), 8(1), April 2014 pp.186-189.

3. Ahmad, Tohari, Studiawan, Hudan and Ramadhan, T. Developing a Raspberry Pi-based

Monitoring System for Detecting and Securing an Object. International Electronics

Symposium (IES), 2014 pp. 125-129.

4. Allen, John G., Xu, Richard Y. D. and Jin, Jesse S. Object tracking using camshift algorithm

and multiple quantized feature spaces. Proceedings of the Pan-Sydney area workshop on

Visual information processing. Australian Computer Society, Inc., 36, 2004 pp. 3-7.

5. Abed, Ali A. and Rahman, Sarah A. International Journal of Computer Applications (0975-

8887) 173(4), September 2017.

6. Kabbai, Leila, Sghaier, Anissa, Douik, Ali and Machhout, Mohsen. FPGA implementation

of filter image using 2D Gaussian filter. International Journal of Advanced Computer

Science and Applications. 7(7), 2016.

7. Yu, Chen, Dian-ren, Chen, Yang, Li, Lei, Chen. Otsu’s Thresholding Method Based on

Gray Level-Gradient Two-Dimensional Histogram. 2nd International Asia Conference on

Informatics in Control, Automation and Robotics, 2010.

8. Dhawan, Amiraj and Honrao, Vipul. Implementation of hand Detection based Techniques

for Human Computer Interaction. International Journal of Computer Applications (0975-

8887), 72(17), June 2013.

9. Farooq, Javeria and Ali, Muhaddisa Barat. Real Time Hand Gesture Recognition for

Computer Interaction. International Conference on Robotics and Emerging Allied

Technologies in Engineering Islamabad, Pakistan, April 22-24, 2014.

10. Marium, Athiya, Rao, Deepthi, Crasta, Divina Riya, Acharya, Kavya, D’Souza, Rio. Hand

Gesture Recognition using Webcam. American Journal of Intelligence Systems, 7(3), 2017

pp. 90-94.

11. Cardozo, Mrs Samantha, Salkar, Miss Amira, Mendes, Master Shaine and Helawar, Miss

Seema. Wireless Communication using RF Module and Interfacing with 8051

Microcontroller. International Journal of Science Technology and Engineering, 3(7),

January 2017.

12. Walunj, Bhushan, Ingle, Amol and Patil, Suresh. Solar Powered Android Controlled

Inspection Car by using IP Camera. Proceedings of the Modern Era Research in Mechanical

Engineering-2016.

Date post:	10-Mar-2020
Category:	Documents
Upload:	others
View:	2 times
Download:	0 times

GESTURE BASED REAL-TIME WIRELESS CONTROL OF AN …Institute of Technology- Nirma University,...

Documents