Click here to load reader
Date post: | 14-Sep-2014 |
Category: |
Documents |
Upload: | saumya-badachi |
View: | 181 times |
Download: | 2 times |
Click here to load reader
241-401 Computer Engineering Project 1
Term 1/2554
Gesture-Based Control of Software
Mr. Boonyasak Natetawesak, 5110110316
Supervisor
Dr. Andrew Davison
Department of Computer Engineering
Faculty of Engineering
Prince of Songkhla University
19th
July, 2554
i
Gesture-Based Control of Software
ABSTRACT
Using body to control computer is new technology so it still lacks accuracy and speed. But
many scientists and engineers are doing hard work to develop techniques to read human
movements correctly and fast. It makes gesture-based system grow so fast. That why many
industrials look at it, and develop it for industry.
Testing of gesture system in this project will use a little game. That game is not complicate
but can use hand to control instead of mouse.
I will divided project in 2 parts, first is game implementation that can adapt gesture-based
control to play game. The other is image processing field, which use JMF to capture pictures
and JavaCV to process the pictures.
CONTENT
Abstract i
1. Introduction 1
1.1 Introduction 1
1.2 Objectives 1
1.3 Scope 2
2. Literation review 3
2.1 J2SE 3
2.2 JMF 3
2.3 JavaCV 5
2.4 Related Chapter about Gesture 7
2.4.1 Chapter 3 Motion Detection 7
2.4.2 Chapter 5 Blobs Drumming 8
3. Material and methods 9
3.1 Webcam 9
3.2 Netbeans IDE 9
3.3 Notepad++ 10
4. Task 11
4.1 Schedule 11
4.2 Task Complete 12
Reference 14
LIST OF FIGURES
Figure 1 Minority Report screenshot 1
Figure 2 Mgestyk’s application 1
Figure 3 JMF webcam device tested 3
Figure 4 YUV Snapshot test 3
Figure 5 Using JMF libraries to connect with webcam 4
Figure 6 Light test output 6
Figure 7 Koala test output 6
Figure 8 Motion Detector test-rig result 7
Figure 9 Face detect 7
Figure 10 Center of moving 7
Figure 11 HSV Selector 8
Figure 12 Blobs Drumming 8
Figure 13 Ordinary webcam 9
Figure 14 Humanoid shape webcam 9
Figure 15 Netbeans screenshot 10
Figure 16 Notepad++ screenshot 10
Figure 17 GUI concept design 12
Figure 18 Cursor moving 13
Figure 19 User control cursor 13
Figure 20 Mouse clicking 13
Figure 21 User click cursor 13
1
Gesture-Based Control of Software
1. INTRODUCTION
1.1 Introduction
As you know body language is important way to communication with another person. And
gesture-based system is takes those advantages of body language to be controller. But it still
lacks of flexibility or accuracy. Many scientists and engineers are researching and
developing about techniques of read face, body and hands movements correctly and fast.
Even if today gesture-based systems are incomplete system. But many industrials are
interested in gesture-based like MS’s Kinect, Sony’s Candy Eye or Mgestyk are developed
gesture-based control and apply it to their applications.
For example, think about you’re in Sci-fi Holly Woods movies e.g. Minority Report that you
can use your hand to move, select, zoom or close application windows, that is gesture-based
control system but in real, technology still doesn’t reach technology in the movies.
Figure 1 Minority Report screenshot Figure 2 Mgestyk’s application
1.2 Objectives
In this project have 2 main parts, one is about gesture which is heart of project. That I must
implement. And the other is the game to test my gesture working. The game is matching
card game that player can select card by move cursor with their hand and pick it.
2
Gesture-Based Control of Software
1. Design aims
This project divide to two parts; the game and the gesture. The game will use ordinary Java
and I’ll design the GUI first in this project. And the gesture I’ll plan about how to use gesture
in the game.
2. Implementation aims
I’ll design the GUI and coding it without gesture first. And when I’d implemented gesture,
I’ll adapt it to the game. Gesture will use JMF to snap pictures and JavaCV to process them.
3. Learning aims
Doing this project, I think I’ll learn about programing in field of image processing. I’ve to
learn how to use JMF to connect with camera, JavaCV to process image data, and combined
it together to control game.
1.3 Scope
Program will capture using JMF libraries and process by JavaCV libraries. This things will
make users control computer by waving a hand like “Minority Report” movie.
This term I’ve designed the GUI of the game. And study about how image processing work
with example code I’ve got from my advisor.
For the game I’ll implement without gesture first. And adapt gesture in it.
3
Gesture-Based Control of Software
2. LITERATURE REVIEW
2.1 J2SE
Java 2 Standard Edition is an object-oriented language. [3], [4] Developed at Sun
Microsystems. Java platform derived much of its syntax from C/C++ but simpler and fewer
low-level facilities. Compile to bytecode and run on JVM that not depend to any computer
architectures that make java can run anywhere without recoding.
Because this is Java project so I’m using java for coding in this project, the version I’ve used
is JDK 1.6.0_23. Because it’s pure object-oriented programming and less complicate than
C/C++ so it’s easy to code in object style applications, even if their lack of image processing
part in Java but I can use JavaCV for wrapping Java code with OpenCV instead.
2.2 JMF
Java Media Framework (JMF) [5], [6], [12] is an API that enables audio, video and other
time-based media to be add to applications or applets which built on Java. JMF also supports
development of cross-platform multimedia applications. The version will be used is JMF
2.1.1e which is stable version on window.
1. JMF Studio
JMF Studio is application to capture a picture from webcam.
2. JMF Registry
JMF Registry is preference for JMF Studio. It can setting a programs, devices, plugins etc.
Figure 3 JMF webcam device tested Figure 4 YUV Snapshot test
4
Gesture-Based Control of Software
Example code
The full code is in Appendix part.
First ShowJPics create frame and set container to contains picture from webcam, then create
JPicsPanel object and add to container, and check for event come from close button, if it can
receive an event then exit. Fix size and set for visibility. After that create a ShowJPics object
in main method.
pp = new JPicsPanel(); // the sequence of snaps appear here
c.add( pp, "Center");
addWindowListener( new WindowAdapter() {
public void windowClosing(WindowEvent e)
{ pp.closeDown(); // stop snapping pics
System.exit(0);
}
Figure 5 Using JMF libraries to connect with webcam
5
Gesture-Based Control of Software
2.3 JavaCV
JavaCV is wrappers to used libraries in field of computer vision. [7] Public licensed under the
GNU General Public License version 2 (GPLv2). JavaCV work with JNA or JNI to map
Java code with native code but in this project I’ve use JNA instead of JNI. But the bad thing
is it still lacks of documentation so it make JavaCV is a hard part to study.
Features of JavaCV: JavaCV has many options for users to use.
JavaCV can use many libraries of computer vision: OpenCV, ARToolkitPlus,
PGRFlyCapture and FFmpeg.
Come with hardware accelerated full-screen image display.
Easy-to-use methods to execute code in parallel on multi-cores.
Geometric and color calibration of cameras and projectors.
Detection and matching of feature points.
In this project I’ll use it in part of image processing, because its features very support image
processing, it’ll makes thing easier.
6
Gesture-Based Control of Software
Example code
package testjavacv;
import com.googlecode.javacv.*;
import static com.googlecode.javacv.jna.cxcore.*;
import static com.googlecode.javacv.jna.highgui.*;
public class TestJavaCV {
public static void main(String[] args) {
CanvasFrame frame = new CanvasFrame("Cute_Koala");
// CanvasFrame frame = new CanvasFrame("Lighthouse");
frame.setCanvasSize(320, 320);
IplImage img = cvLoadImage("Koala.jpg");
// IplImage img = cvLoadImage("Koala.jpg");
if(img == null){
System.err.println("Could not load image file.");
}
frame.showImage(img);
}
}
Explain code
Start with declare a variable points to CanvasFrame object to create picture frame then set
frame size. After that use IplImage class keep image which load with cvLoadImage. And
show out via showImage method.
Output
Figure 6 Light test output Figure 7 Koala test output
7
Gesture-Based Control of Software
2.4 Related Chapter About Gesture
2.4.1 Chapter 3 Motion Detection
This chapter explains how to detect movement via images that snapped from webcam and
drawn rapidly on panel. And on COG (Center of Gravity) will draw crosshair there. [15] By
using JMF and JavaCV in this chapter.
I got these from Dr. Andrew’s work and test it on my device.
Example
Figure 8 Motion Detector test-rig result
Figure 9 Face detect Figure 10 Center of moving
Motion Detector use new image snapped by webcam to compare with previous image and the
difference will be white while the same view is black via using Threshold.
White point in figure 17 and target in figure 18 using calculation COG (Center of gravity) to
know where is center of movement.
8
Gesture-Based Control of Software
2.4.2 Chapter 5 Blobs Drumming
This chapter use simple technique for detect blobs in an image for looking same color. This
application looks for light blue and red blobs. [15] I’ve got from Dr. Andrew too. And I got
it work on my device too.
Example
Figure 11 HSV Selector
Figure 12 Blobs Drumming
This Blob detectors use HSV instead of RGB because of advantage in brightness ignorance.
HSV Selector is for configure the suitable HSV ranges for each card and save to text file.
Drumming is example on how to use blobs in game. By drawing drums and stick them with
detectors. By using configured HSV values from HSV selector make program knows the
card.
9
Gesture-Based Control of Software
3. MATERIAL AND METHODS
3.1 Webcam
Webcam is a camera that can be accessed by and displayed on website.[1], [2] Have USB,
Ethernet or Wi-Fi connection. Often use for videoconference, security surveillance and
computer vision.
For this project, I didn’t decide for which one will exactly be used, but for now I’m using my
notebook built-in webcam for test the codes.
Figure 13 Ordinary webcam Figure 14 Humanoid shape webcam
3.2 Netbeans IDE
Netbeans is IDE (Integrated Development Environment) [8] was developed by Sun, available
for Windows, MacOS, Linux and Solaris. Netbeans consists of an open-source IDE, many
plugins, and application platform that make developers easier to create web, desktop and
mobile applications.
Platform which Netbeans supports:
Maven
All of Java platform include JavaScripts
HTML
PHP
Ruby on Rails
Groovy on grails
C/C++
I’ll use Netbeans as main editor because it has many tools, plugins and it easy to create
applications.
10
Gesture-Based Control of Software
3.3 Notepad++
Notepad++ is open-source code editor, [9] run on MS Windows environment under GPL
License. It supports several languages.
Features:
Tabbed document interface.
Multiple Clipboards (plugin).
Split screen.
Find and replace.
Supports text encoding formats.
Auto-completion
Syntax highlighting/folding
Macro (recording and execution)
Many plugins
I’ll use Notepad++ as sub editor co with Netbeans, Notepad++ will be used in small part of
code because it less complicate than Netbeans.
Figure 15 Netbeans screenshot Figure 16 Notepad++ screenshot
11
Gesture-Based Control of Software
4. TASK
4.1 Schedule
4.1.2 Project 1 1/2011
No. Topic June July August September
1. Study on example code
about motion detection.
2. Study about blob using.
3. Design GUI of game.
4. Writing draft proposal.
5. Implement game
without gesture.
6. Start implement
gesture.
7. Writing final proposal.
4.1.3 Project 2 2/2011
No. Topic November December January February
1. Continue gesture
implementation.
2. Adapt gesture in to
game
3. Improve program.
4. Writing draft proposal.
5. Integrate all part and
get its result.
6. For maintenances.
7. Writing final proposal.
Finished
12
Gesture-Based Control of Software
4.2 Task complete
4.2.1 Read papers to get ideas.
Gesture Recognition for a Webcam-Controlled First Person Shooter.
Vision-Based Virtual Control Mechanism via Hand Gesture Recognition.
Recognition-Based Gesture Spotting in Video Games.
Visual Touchpad: A Two-Handed Gestural Input Device.
4.2.3 Design the GUI of the game.
Concept of this project is the game that can play without use mouse or keyboard but gestures.
The game is Matching Cards. There will be even number of cards and player needs to choose
the same pair of cards to win the game. If player lose all life, game over.
1. GUI Design
Figure 17 GUI concept design
2. Details
The blocks is layout in Java which use BoxLayout, BorderLayout and GridLayout.
Name: Show player name.
Life: The chances which player still left.
Time: Time counting.
Card: First time cards will show the front.
Play / Pause Button: Play is starts the game card flip back, time counter start. Pause will
pause the time and can’t select any cards.
13
Gesture-Based Control of Software
Game play: When chose a card, the card will flip. If a pair of cards is the same, they will stay
open. If wrong cards flip back and a life disappear. Either last pair of correct cards or lose
all life game will stop.
Adaptation of gesture to the control:
Moving cursor:
Figure 18 Cursor moving Figure 19 User control cursor
Moving cursor by waving hand in front of camera. [11] [13] [14] [15]
Clicking:
Figure 20 Mouse clicking Figure 21 User click cursor
Clicking by clenching a hand. [12] [14]
4.2.2 Test example codes.
Snaps code is about snaps a picture via webcam using JMF.
Motion Detector has 2 codes, one is Motion Detector Test-rig that show movement
that webcam can detect. The other one is Motion Detector, that target user face even
user move it’ll change position by combination of JMF and JavaCV.
Blob Drumming is using blob in drumming game using cards instead of drum sticks.
Using JMF and JavaCV + Blob in this code.
4.2.3 Writing draft proposal.
14
Gesture-Based Control of Software
REFERENCE
[1] Wikipedia, http://en.wikipedia.org/wiki/Webcam, access date: 01/28/11.
[2] Wise geek, http://www.wisegeek.com/what-is-a-webcam.htm, access date: 01/28/11.
[3] Oracle, http://www.oracle.com/us/technologies/java/index.html, access date: 01/28/11.
[4] Wise geek, http://www.wisegeek.com/what-is-j2se.htm, access date: 01/28/11.
[5] Wikipedia, http://en.wikipedia.org/wiki/Java_Media_Framework, access date: 01/28/11.
[6] Oracle, http://www.oracle.com/technetwork/java/javase/tech/index-jsp-140239.html,
access date: 01/28/11.
[7] JavaCV, http://code.google.com/p/javacv/, access date: 01/30/11.
[8] Oracle, http://netbeans.org/community/releases/69/, access date: 01/30/11.
[9] Notepadd++ team, http://notepad-plus-plus.org/, access date: 01/30/11.
[10] Fivedots, http://fivedots.coe.psu.ac.th/~ad/jg2/ch09/, access date: 01/30/11.
[11] R. W. Wilson, A Salgian. 2008. “Gesture Recognition for a Webcam-Controlled First
Person Shooter”, Lecture Notes in Computer Science. Vol. 5359 : 889-896.
[12] C. H. Yeh, S. J. F. Jian, J. C. Bai, J. S. Liou, R. N. Yeh, S. C. Wang, P. Y. Sung. 2010.
“Vision-Based Virtual Control Mechanism via Hand Gesture Recognition”, Journal of
Computers. Vol. 21, No. 2.
[13] H. Kang, C. W. Lee, D. Jung. 2004. “Recognition-Based Gesture Spotting in Video
Games”, Pattern Recognition Letters. Vol. 25, No. 15.
[14] S. Malik, J. Laszlo. 2004. “Visual Touchpad: A Two-Handed Gestural Input
Device”,ICMI’04.
[15] Davison, A. 2007. Pro Java 6 3D Game Development, Apress.