SIXTH SENSE TECHNOLOGY

Seminar report submitted in partial fulfillment of the

requirements for the award of the degree of

Bachelor of Technology (B. Tech)

In

Information Technology

Of

University of Calicut

By

JISMI K JACOB

Under the Guidance of

Ms. JISNA V A

Asst.Professor, IT Department

April 2014

Department of Information Technology

JYOTHI ENGINEERING COLLEGE,

Cheruthuruthy,Thrissur – 679 531.

DEPARTMENT OF INFORMATION TECHNOLOGY

JYOTHI ENGINEERING COLLEGE

CHERUTHURUTHY, THRISSUR- 679 531

April 2014

CERTIFICATE

This is to certify that the seminar report entitled “SIXTH SENSE

TECHNOLOGY” being submitted by Ms. JISMI K JACOB in partial

fulfillment of the requirements for the award of degree of Bachelor of

Technology of University of Calicut is bonafide record of work carried out at

Department of Information Technology, JECC by her during the period

December 2013 – April 2014 under our supervision.

Seminar Guide Seminar Coordinator Ms. DIVYA M MENON

Ms. JISNA V A Ms. SABNA A B H.O.D in charge,

Asst.Professor Asst.Professor IT Department

ABSTRACT

Sixth Sense technology is a technology with which a system could be trained to

recognize and percept real world objects and react as desired. Sixth Sense technology bridges

the gap between the physical world and the digital world, bringing intangible, digital

information out into the tangible world, and allowing us to interact with this information via

natural hand gestures. Sixth Sense Technology is implemented in 'Sixth Sense/WUW (wear

your world) using gesture recognition, augmented reality, computer vision and radio

frequency identification. It’s just born concept which allows user to connect with the internet

seamlessly. Without use of keyboard, mouse we can see videos access, change, move data

simply .But this concept bottle necks lead to modification of the same by using commands

instead of gestures. Sixth Sense technology could be integrated with voice recognition.

Bluetooth device and laser projectors could be used.

INDEX

Contents Page no:

1. INTRODUCTION 1

2. OVERVIEW 3

3. ORIGIN OF IDEA 4

3.1 What is Sixth Sense Technology? 5

3.2 Earlier Sixth Sense Prototype 5

3.3 Recent Prototype 6

4. DETAILED DESCRIPTION 8

4.1 Components 8

4.2 Working of Sixth Sense Technology 11

4.3 Related Technologies 13

4.4 Applications 19

4.5 Advantages 23

4.6 Future Enhancements 23

5. CONCLUSION 24

6. BIBLIOGRAPHY 25

List of Figures

3.1 Six Senses 5

3.2 Earlier Prototype 6

3.3 Recent Prototypes 7

4.1.1 Camera 8

4.1.2 Projector 9

4.1.3 Mirror 9

4.1.4 Smartphone 10

4.1.5 Color Marker 11

4.1.6 Microphone 11

4.2 Working of the Sixth Sense 12

4.3.1 Augmented reality 15

4.3.2 Computer vision 15

4.3.3 Hand gestures 17

4.3.4 Components of RFID 18

CHAPTER 1

INTRODUCTION

We use our five natural senses to perceive any information; that information helps us

make decisions and choose the right actions to take. But arguably the most useful information

that can help us make the right decision is not naturally perceivable with our five senses,

namely the data, information and knowledge that mankind has accumulated about everything

and which is increasingly all available online. Although the miniaturization of computing

devices allows us to carry computers in our pockets, keeping us continually connected to the

digital world, there is no link between our digital devices and our interactions with the

physical world. Information is confined traditionally on paper or digitally on a screen. Sixth

Sense Technology bridges this gap, bringing intangible, digital information out into the

tangible world, and allowing us to interact with this information via natural hand gestures.

‘Sixth Sense’ frees information from its confines by seamlessly integrating it with reality, and

thus making the entire world your computer. WUW was developed by Pranav Mistry, a Ph. D

student at Fluid Interfaces Group at the MIT Media Lab. The Sixth Sense prototype

implements several applications that demonstrate the usefulness, viability and flexibility of

the system acts as the computer and your connection to the Cloud, all the information stored

on the web.

Sixth Sense recognizes the objects around you, displaying information automatically

and letting you access it in any way you want, in the simplest way possible. The device

brings us closer to reality and assists us in making right decisions by providing the relevant

information, thereby, making the entire world a computer. The technology is mainly based on

hand gesture recognition, image capturing, processing, and manipulation, etc. The software of

the technology uses the video stream, which is captured by the camera, and also tracks the

location of the tips of the fingers to recognize the gestures. This process is done using some

techniques of computer vision. He invented ‘ Sixth Sense / WUW ( Wear UR World)’ which

is a wearable gestural , user friendly interface which links the physical world around us with

digital information and uses hand gestures to interact with them. This technology is a

revolutionary way to interface the physical world with digital information. Modern

technologies include the touch screen techniques which is used widely and it makes ease of

operation and saves utilization time.

This paper deals with the latest technology called the sixth sense. It’s a wearable

interface that augments the physical world around us with the digital information. It’s just

born concept which allows user to connect with the internet seamlessly. Without use of

keyboard, mouse we can see videos access, change, move data simply .But this concept bottle

necks lead to modification of the same by using commands instead of gestures. Speech IC is

used as a database for commands which will be initially trained for storage. It performs the

corresponding commands accessing the operation from the mobile device connected to it and

action is projected using a projector over any surface.

CHAPTER 2

OVERVIEW

Previously many technologies evolved such as augmented reality which is to add

information and meaning to real object or place. Unlike virtual reality, augmented reality

does not create a simulation of reality instead it takes a real object or space as the foundation

and incorporates technologies that add contextual data to deepen a person understanding of

the subject. It’s a term for live direct or indirect view of a physical real world environment

whose elements are augmented by virtual computer generated imagery. Gesture recognition is

a term with a goal of interpreting human gestures through mathematical gestures and

mathematical algorithms. Computer vision is the science and technology of machines that is

concerned with the theory behind artificial systems that extract information from the images.

As a technological discipline, computer vision seeks to apply its theories and models

to the construction of computer vision systems. The examples include the controlling

processes, detecting events, organising information, modelling objects or environments and

interaction. Recently speech integrated circuits evolved which is used widely in car

automation and home appliances. It eases the operation and saves the utilization time of the

manual operations performed by the human’s every day. The speech recognition process is

performed by a software component known as speech recognition engine. The primary

function of this is to process the spoken input and translate it into text which the application

understands. The application then can do one of the two things, 1.The application can

interpret the result of the recognition as a command, in this case application is a command

and control application. If the application handles the recognized text as simply text, then it’s

considered as dictation application. When the user says something, it is known as utterance.

An utterance is a stream of speech between two periods of silence. The speech IC can be used

for all sorts of data, statistical models, and algorithms to convert spoken input into text.

CHAPTER 3

ORIGIN OF IDEA

This technology is a revolutionary way to interface the physical world with digital

information. Modern technologies include the touch screen techniques which is used widely

and it makes ease of operation and saves utilization time. Sixth sense is a wearable gestural

interface that augments the physical world around us with digital information and lets us use

natural hand gestures to interact with that information. But the bottle necks of this method

such as position of camera, for capturing gestures interprets the accuracy in the projected

output, lead to use of commands instead of hand gestures. The position of camera is a major

constraint in the image capturing and projected output efficiency and accuracy. Therefore the

actions which we regularly perform in our daily life, are converted to commands and are

trained to a speech IC .They are stored as a database in the integrated circuit and

corresponding actions are performed when the speech is recognized from the user. It’s a hi-

tech device seamlessly integrate Analog information with our every day physical world. The

voice is directly performed into operation within fractions of seconds, and the action is

projected on the surface. It’s a portable device and eases the operation which we regularly

perform. Basically the sixth sense technology concept involves the use of hand gestures .the

finger tip will contain colored markers and hence gestures performed will be captured by the

camera. Then it’s given to the mobile device for the corresponding action to be performed.

The action is projected on the surface through the projector. Software algorithms and

computer vision technologies will be used to enable the action from the mobile device for the

corresponding gesture captured in the camera. This gesture based technology is used for

variety of applications like performing basic actions, locating points in the map, watching

video in news paper, dialing number in hand etc. The slight modification of this method leads

to the use of commands that is analog information into real world. The analog data is

converted into digital and performed as action, as all times the hand gestures cannot be used.

This was how the wearable device is fit to the human body. Here color markers are used in

the finger tips.

In our technology we use commands for performing the same operations. Many high

technology speech integrated circuits evolved which makes our operation enhanced with

more advanced features. To ensure accurate gesture recognition and an intuitive interface a

number of constraints are applied. A region in the front of the projection screen is defined as

the active zone and the gestures are ignored, if the gestures are performed out of this area.

Gestures are also defined by a set start posture, end posture and dynamic motion between the

start and end postures. Perhaps the use of gestures is most powerful when combined with

other input modalities, especially voice. Allowing combined voice and gestural input has

several tangible advantages. The first is purely practical-ease of expression .Ease corresponds

to the efficiency with which commands can be remembered and expressiveness, size of

command vocabulary.

3.1 WHAT IS SIXTH SENSE?

Sixth sense in scientific or non scientific terms defined as Extra Sensory Perception. It

involves the reception of information not gained through any of the five senses. Namely the

data, information and knowledge that mankind has accumulated about everything available

online.

Figure 3.1 six senses

3.2 EARLIER SIXTH SENSE PROTOTYPE

Steve Mann is the father of Sixth Sense Technology who made of wearable

computer in 1990. He implemented the neck worn projector with a camera system. Maes’

MIT group, which includes seven graduate students, were thinking about how a person could

be more integrated into the world around them and access information without having to do

something like take out a phone. They initially produced a wristband that would read an

Radio Frequency Identification tag to know, for example, which book a user is holding in a

store. They also had a ring that used infrared to communicate by beacon to supermarket smart

shelves to give you information about products. As we grab a package of macaroni, the ring

would glow red or green to tell us if the product was organic or free of peanut traces —

whatever criteria we program into the system. They wanted to make information more useful

to people in real time with minimal effort in a way that doesn’t require any behavior changes.

The wristband was getting close, but we still had to take out our cell phone to look at the

information. That’s when they struck on the idea of accessing information from the internet

and projecting it. So someone wearing the wristband could pick up a paperback in the

bookstore and immediately call up reviews about the book, projecting them onto a surface in

the store or doing a keyword search through the book by accessing digitized pages on

Amazon or Google books. They started with a larger projector that was mounted on a helmet.

But that proved cumbersome if someone was projecting data onto a wall then turned to speak

to friend — the data would project on the friend’s face.

Figure 3.2: Earlier prototype

3.3 RECENT PROTOTYPE

WUW was developed by Pranav Mistry, a Ph. D student at Fluid Interfaces Group at

the MIT Media Lab. The Sixth Sense prototype implements several applications that

demonstrate the usefulness, viability and flexibility of the system acts as the computer and

your connection to the Cloud, all the information stored on the web. The key here is that

Sixth Sense recognizes the objects around you, displaying information automatically and

letting you access it in any way you want, in the simplest way possible. The device brings us

closer to reality and assists us in making right decisions by providing the relevant

information, thereby, making the entire world a computer. The technology is mainly based on

hand gesture recognition, image capturing, processing, and manipulation, etc. The software of

the technology uses the video stream, which is captured by the camera, and also tracks the

location of the tips of the fingers to recognize the gestures. This process is done using some

techniques of computer vision. He invented ‘ Sixth Sense / WUW ( Wear UR World)’ which

is a wearable gestural , user friendly interface which links the physical world around us with

digital information and uses hand gestures to interact with them.

Figure 3.3 Recent prototypes

CHAPTER 4

DETAILED DESCRIPTION

4.1 COMPONENTS

The hardware components are coupled in a pendant like mobile wearable device. The

components are:

1. Camera

2. Projector

3. Mirror

4. Mobile component

5. Color markers

6. Microphone

4.1.1 Camera

Camera captures an object in view and tracks the user’s hand gestures. It sends the

data to smart phone. Camera recognizes and tracks user's hand gestures and physical objects

using computer-vision based techniques. Sixth Sense system implements a gestural camera

that takes photos of the scene the user is looking at by detecting the ‘framing’ gesture. It acts

as a digital eye, connecting you to the world of digital information.

Figure 4.1.1: Camera

4.1.2 Projector

A tiny LED projector displays data sent from the smart phone on any surface in view-object,

wall, or person. The projector projects visual information enabling surfaces, walls and

physical objects around us to be used as interfaces. The

projector projects visual information enabling surfaces,

walls and physical objects around us to be used as

interfaces. We want this thing to merge with the physical world in

a real physical sense. You are touching that object and

projecting info onto that object. The information will look like it

is part of the object. The projector itself contains a battery inside, with 3 hours of battery life.

Figure 4.1.2: Projector

4.1.3 Mirror

The mirror reflects the projection coming out from the projector and thus helps in

projecting onto the desired locations on walls or surfaces. The user manually can change the

tilt of the mirror to change the location of the projection. For example in application where

the user wants the projection to go on the ground instead of the surface in front, he can

change the tilt of the mirror to change the projection. Thus, the mirror in the Sixth Sense

helps in overcoming the limitation of the limited projection space of the projector.

Figure 4.1.3 Mirror

4.1.4 Mobile component

The Sixth Sense system uses a mobile computing device in user’s pocket as the

processing device. The software program enabling all the features of the system runs on this

computing device. This device can be a mobile phone or a small laptop computer. The

camera, the projector and the microphone are connected to this device using wired or wireless

connection. The detail of the software program that runs on

this device is provided in next section. The mobile computing device is also connected to the

Internet via 3G network or wireless connection.

A Web-enabled smart phone in the user's pocket processes the video data, using

vision algorithms to identify the object. Other software searches the Web and interprets the

hand gestures.

Figure 4.1.4: Smartphone

4.1.5 Color marker

Color marker is at the tip of the user’s fingers. Marking the user’s fingers with red, yellow,

green, and blue tape helps the webcam recognize gestures. The camera tracks the movements

of the color markers. The movements and arrangements of these makers are interpreted into

gestures that act as interaction instructions for the projected application interfaces.

The software program processes the video stream data captured by the camera and

tracks the locations of the colored markers (visual tracking fiducials) at the tip of the user’s

fingers using simple computer-vision techniques. The movements and arrangements of these

fiducials are interpreted into gestures that act as interaction instructions for the projected

application interfaces. The maximum number of tracked fingers is only constrained by the

number of unique fiducials, thus Sixth Sense also supports multi-touch and multi-user

interaction.

Figure 4.1.5: Color marker

4.1.6 Microphone

The microphone is an optional component of the Sixth Sense. It is required when

using a paper as a computing interface. When the user wants to use a sheet of paper as an

interactive surface, he or she clips the microphone to the paper. The microphone attached this

way captures the sound signals of user’s touching the paper. This data is passed to computing

device for processing. Later, combined with the tracking information about user’s finger, the

system is able to identify precise touch events on the paper. Here, the sound signal captured

by the microphone provides time information whereas the camera performs tracking. The

applications enabled by this technique are explained earlier.

Figure 4.1.6 Microphone

4.2 WORKING OF SIXTH SENSE TECHNOLOGY

The hardware that makes Sixth Sense work is a pendant like mobile wearable

interface. It has a camera, a mirror and a projector and is connected wirelessly to a Bluetooth

smart phone that can slip comfortably into one’s pocket. The camera recognizes individuals,

images, pictures, gestures one makes with their hands. Information is sent to the Smartphone

for processing. The downward-facing projector projects the output image on to the mirror.

Mirror reflects image on to the desired surface. Thus, digital information is freed from its

confines and placed in the physical world.

Figure 4.2: working of the sixth sense

The entire hardware apparatus is encompassed in a pendant-shaped mobile wearable

device. Basically the camera recognizes individuals, images, pictures, gestures one makes

with their hands and the projector assists in projecting any information on whatever type of

surface is present in front of the person. The usage of the mirror is significant as the projector

dangles pointing downwards from the neck. To bring out variations on a much higher plane,

in the demo video which was broadcasted to showcase the prototype to the world, Mistry uses

colored caps on his fingers so that it becomes simpler for the software to differentiate

between the fingers, demanding various applications.

The software program analyses the video data caught by the camera and also tracks

down the locations of the colored markers by utilizing single computer vision techniques.

One can have any number of hand gestures and movements as long as they are all reasonably

identified and differentiated for the system to interpret it, preferably through unique and

varied fiducials. This is possible only because the ‘Sixth Sense’ device supports multi-touch

and multi-user interaction.

The technology is mainly based on hand gesture recognition, image capturing,

processing, and manipulation, etc. The map application lets the user navigate a map displayed

on a nearby surface using hand gestures, similar to gestures supported by multi-touch based

systems, letting the user zoom in, zoom out or pan using intuitive hand movements. The

drawing application lets the user draw on any surface by tracking the fingertip movements of

the user’s index finger.

4.3 RELATED TECHNOLOGIES

Previously many technologies evolved such as augmented reality which is to add

information and meaning to real object or place. Unlike virtual reality, augmented reality

does not create a simulation of reality instead it takes a real object or space as the foundation

and incorporates technologies that add contextual data to deepen a person understanding of

the subject. It’s a term for live direct or indirect view of a physical real world environment

whose elements are augmented by virtual computer generated imagery. Sixth Sense

technology takes a different approach to computing and tries to make the digital aspect of our

lives more intuitive, interactive and, above all, more natural. We shouldn’t have to think

about it separately. It’s a lot of complex technology squeezed into a simple portable device.

When we bring in connectivity, we can get instant, relevant visual information projected on

any object we pick up or interact with The technology is mainly based on hand augmented

reality, gesture recognition, computer vision based algorithm etc.

4.3.1 AUGMENTED REALITY

Augmented reality is a term for a live direct or indirect view of a physical real world

environment whose elements are augmented by virtual computer generated imagery.

Augmented reality, blurs the line between what's real and what's computer-generated by

enhancing what we see, hear, feel and smell. Augmented reality is one of the newest

innovations in the electronics industry. It superimposes graphics, audio and other sense

enhancements from computer screens onto real time environments. Augmented reality goes

far beyond the static graphics technology of television where the graphics imposed do not

change with the perspective. Augmented reality systems superimpose graphics for every

perspective and adjust to every movement of the user's head and eyes. The basic idea of

augmented reality is to superimpose graphics, audio and other sensory enhancements over a

real world environment in real time. However, augmented reality is more advanced than any

technology you've seen in television broadcasts, although some new TV effects come close,

such as RACEf/x and the super-imposed first down line on televised U.S. football games,

both created by Sport vision. But these systems display graphics for only one point of view.

You can even point the phone at a building, and Layar will tell you if any companies in that

building are hiring, or it might be able to find photos of the building on Flickr. Augmented

reality adds graphics, sounds, haptic feedback and smell to the natural world as it exists.

Everyone from tourists, to soldiers, to someone looking for the closest subway stop can now

benefit from the ability to place computer-generated graphics in their field of vision.

The main hardware components for augmented reality are: display, tracking, input

devices, and computer. Combination of powerful CPU, camera, accelerometers, GPS and

solid state compass are often present in modern Smartphone, which make them prospective

platforms. There are three major display techniques for Augmented Reality:

1. Head Mounted Displays

2. Handheld Displays

3. Spatial Displays

1. Head Mounted Displays

A Head Mounted Display (HMD) places images of both the physical world and

registered virtual graphical objects over the user's view of the world. The HMD's are either

optical see through or video see-through in nature.

2. Handheld Displays

Handheld Augment Reality employs a small computing device with a display that fits in a

user's hand. All handheld AR solutions to date have employed video see through techniques

to overlay the graphical information to the physical world. Initially handheld AR employed

sensors such as digital compasses and GPS units for its six degree of freedom tracking

sensors.

3. Spatial Displays

Instead of the user wearing or carrying the display such as with head mounted displays or

handheld devices; Spatial Augmented Reality (SAR) makes use of digital projectors to

display graphical information onto physical objects.

Figure 4.3.1 Augmented reality

4.3.2 COMPUTER VISION

Computer vision is the science and technology of machines that see. It is concerned

with the theory behind artificial systems that extract information from images. An image is a

huge array of gray level (brightness) values of individual pixels. Taken individually, these

numbers are almost meaningless, because they contain very little information about the scene

. A robot needs information like "object ahead", "table to the left", or "person approaching" to

perform its tasks. The conversion of this huge amount of low level information into usable

high level information is the subject of computer vision. Earlier algorithms were too

computationally expensive to run in real-time, but also required any type of memory and

modeling. We concentrate on two types of images frequently used in computer vision:

Intensity images Photograph like images encoding light intensities, Range images encoding

shape and distance (sonar and laser).

Figure 4.3.2: Computer vision

4.3.3 GESTURE RECOGNTION

Gesture recognition is a topic in computer science and language technology with the

goal of interpreting human gestures via Mathematical algorithms. Gestures can originate from

any bodily motion or state but commonly originate from the face or hand. Current focuses in

the field include emotion recognition from the face and hand gesture recognition. The

keyboard and mouse are currently the main interfaces between man and computer. Humans

communicate mainly by vision and sound, therefore, a man-machine interface would be more

intuitive if it made greater use of vision and audio recognition. Another advantage is that the

user not only can communicate from a distance, but need have no physical contact with the

computer. However, unlike audio commands, a visual system would be preferable in noisy

environments or in situations where sound would cause a disturbance. Many approaches have

been made using cameras and computer vision algorithms to interpret sign language.

However, the identification and recognition of posture, gait, proxemics, and human behaviors

is also the subject of gesture recognition techniques. Gesture recognition can be seen as a way

for computers to begin to understand human body language, thus building a richer bridge

between machines and humans than primitive text user interfaces or even GUIs (graphical

user interfaces), which still limit the majority of input to keyboard and mouse. Gesture

recognition enables humans to interface with the machine (HMI) and interact naturally

without any mechanical devices. Using the concept of gesture recognition, it is possible to

point a finger at the computer screen so that the cursor will move accordingly. Gesture

recognition is useful for processing information from humans which is not conveyed through

speech or type. As well, there are various types of gestures which can be identified by

computers.

Figure 4.3.3: Hand gestures

4.3.4 RADIO FREQUENCY IDENTIFICATION

Radio Frequency Identification is basically an electronic tagging technology that

allows the detection, tracking of tags and consequently the objects that they are affixed to

Radio frequency identification (RFID) is a technology that uses communication via radio

waves to exchange data between a reader and an electronic tag attached to an object, for the

purpose of identification and tracking. Some tags can be read from several meters away and

beyond the line of sight of the reader. The application of bulk reading enables an almost

parallel reading of tags. Radio frequency identification (RFID) is a generic term that is used

to describe a system that transmits the identity (in the form of a unique serial number) of an

object or person wirelessly, using radio waves. It's grouped under the broad category of

automatic identification technologies. Radio-frequency identification involves interrogators

(also known as readers), and tags (also known as labels). Most RFID tags contain at least two

parts. One is an integrated circuit for storing and processing information, modulating and

demodulating a radio-frequency (RF) signal, and other specialized functions. The other is an

antenna for receiving and transmitting the signal. RFID is in use all around us. If you have

ever chipped your pet with an ID tag, used EZ Pass through a toll booth, or paid for gas using

Speed Pass, you've used RFID. In addition, RFID is increasingly used with biometric

technologies for security.

The idea of Sixth Sense is to use Radio Frequency Identification technology in

conjunction with a bunch of other enterprise systems such as the calendar system or online

presence that can track user activity. Here, we consider an enterprise setting of the future

where people (or rather their employee badges) and their personal objects such as books,

laptops, and mobile phones are tagged with cheap, passive RFID tags, and there is good

coverage of RFID readers in the workplace. Sixth Sense incorporates algorithms that start

with a mass of undifferentiated tags and automatically infer a range of information based on

an accumulation of observations. The technology is able to automatically differentiate

between people tags and object tags, learn the identities of people, infer the ownership of

objects by people, learn the nature of different zones in a workspace (e.g., private office

versus conference room), and perform other such inferences. By combining information from

these diverse sources, Sixth Sense records all tag-level events in a raw database. The

inference algorithms consume these raw events to infer events at the level of people, objects,

and workspace zones, which are then recorded in a separate processed database. Applications

can either poll these databases by running SQL queries or set up triggers to be notified of

specific events of interest. Sixth Sense infers when a user has interacted with an object, for

example, when you pick up your mobile phone. It is a platform in that its programming

model makes the inferences made automatically available to applications via a rich set of

APIs. To demonstrate the capabilities of the platform, the researchers have prototyped a few

applications using these APIs, including a misplaced object alert service, an enhanced

calendar service, and rich annotation of video with physical events.

Figure 4.3.4: Components of RFID

4.4 APPLICATIONS

The basic operations such as enabling clock, inbox, browsing, searching gallery,

calendar, seeing contact list etc are performed regularly in the mobile every time. These

operations can be stored as commands in the IC and then can be accessed on the screen or

over any surface using our technology within fractions of seconds.

4.4.1 MAKE A CALL

The sixth sense can project the keypad onto one’s hand which can be used as a virtual

screen to make a call. You can use the Sixth Sense to project a keypad onto your hand, then

use that virtual keypad to make a call. Calling a number also will not be a great task with the

introduction of Sixth Sense Technology. No mobile device will be required, just type in the

number with your palm acting as the virtual keypad. The keys will come up on the fingers.

The fingers of the other hand will then be used to key in the number and call.

Figure 4.4.1: Making a call

4.4.2 CHECK THE TIME

When you draw a circle on your wrist, a virtual watch appears that gives you the correct

time. Sixth Sense all we have to do is draw a circle on our wrist with our index finger to get a

virtual watch that gives us the correct time. The computer tracks the red marker cap or piece

of tape, recognizes the gesture, and instructs the projector to flash the image of a watch onto

his wrist.

Figure 4.4.2: Checking time

4.4.3 ZOOMING FEATURES

The user can zoom in or zoom out by using their intuitive hand movements.

Figure 4.4.3: zooming

4.4.4 GET PRODUCT INFORMATION

The sixth sense uses image recognition or a marker technology to recognize the products

we pick up and gives us the information on those products. For example, if you're trying to

shop "green" and are looking for paper towels with the least amount of bleach in them, the

system will scan the product you pick up off the shelf and give you guidance on whether this

product is a good choice for you.

Figure 4.4.4: Product information

4.4.5 GET FLIGHT UPDATES

By using our boarding pass, the system will recognize it and would let you know whether

your flight is on time and if the gate has changed.

Figure 4.4.5: Flight updates

4.4.6 TAKE A PICTURE

If you fashion your index finger and thumbs into a square (“framing” gesture), the

system snaps a photo at that time. After taking the desired number of photos, we can project

them onto a surface to view them.

Figure 4.4.6: Take pictures

4.4.7 CALL UP A MAP

It is helpful to call up the map of our choice and then use our thumbs and index

fingers to navigate the map. The sixth sense also implements map which lets the user display

the map on any physical surface and find his destination and he can use his thumbs and index

fingers to navigate the map, for example, to zoom in and out and do other controls.

Figure 4.4.7: Virtual map

4.4.8 CHECK THE EMAIL

Draw an @ symbol on any surface helps to check the email using the web service.

4.4.9 CREATE MULTIMEDIA READING EXPERIENCES

The Sixth Sense system also augments physical objects the user is interacting with by

projecting more information about these objects projected on them. For example, a

newspaper can show live video news or dynamic information can be provided on a regular

piece of paper. Thus a piece of paper turns into a video display.

Figure 4.4.9: Multimedia reading

4.4.10 FEED INFORMATION ON PEOPLE

It helps to display the relevant information about a person we are looking at. Sixth

Sense also is capable of "a more controversial use”. When you go out and meet someone,

projecting relevant information such as what they do, where they work, and also it could

display tags about the person floating on their shirt. It could be handy if it displayed their

facebook relationship status so that you knew not to waste your time.

Figure 4.4.10: info about person

4.5 ADVANTAGES

Portable, supports multi-touch and multi-user interaction, connectedness between

world and information, cost effective, data access directly from machine in real time, mind

map the idea anywhere, assists us in making right decisions, supports multi touch and multi

user interaction, the device serves the purpose of a computer plus saves time spent on

searching information. Sixth Sense also recognizes user’s freehand gestures (postures) and

saves electricity.

4.6 FUTURE ENHANCEMENTS

Imagine the world where Sixth Sense Technology is applied everywhere. In

educational field, the number of hardware components could be reduced. Usage of papers and

electricity could decrease. Students could use any wall or any surface wherever they are to

carry out activities that are done in a PC. Security will be assured for everyone. It could be

helpful in rendering defense services. In medical field, it could be implied to check the

genuinity of drugs. It could be implemented to monitor the agricultural lands. Blind people

could be able to read books and recognize objects. It could be used for the betterment of

handicapped people. Sixth sense could make the world magical.

CHAPTER 5

CONCLUSION

Sixth Sense technology recognizes the objects around us, displaying information

automatically and letting us to access it in any way we need. The Sixth Sense prototype

implements several applications that demonstrate the usefulness, viability and flexibility of

the system. Allowing us to interact with this information through natural hand gestures. The

potential of becoming the ultimate "transparent" user interface for accessing information

about everything around us. Currently the prototype of the device costs around $350 to build.

It could change the way we interact with the real world and truly give everyone complete

awareness of the environment around us. The Sixth Sense prototype implements several

applications that demonstrate the usefulness, viability and flexibility of the system. It will

definitely revolutionize the world.

The Sixth Sense software will be open source. As far as this seems to be a little set of

items, there will not be user interfaces or much advanced programs for the users. There will

be much harder and secured coding inside the device to make sure the security of the

software. It will be interesting to know the new language for coding for a sixth sense device.

BIBLIOGRAPHY

[1] Kirishima, T. Sato, K. Chihara, K.Dept. of Electr. Eng., Nara Nat. Coll. of Technol.,

Japan Robotics, “Gesture Spotting and Recognition for Human–Robot Interaction”, IEEE

Transactions on Volume: 23, Issue:2 pp256 – 270., April 2007.

[2] Alon,J.Athitsos, V.Quan, YuanSclarof,” A Unified Framework for Gesture Recognition

and Spatiotemporal Gesture Segmentation”, IEEE transactions on Pattern Analysis and

Machine Intelligence, Volume: 31, Issue:9 pp 1685 - 1699 ., Sept. 2009.

[3] Gomez, A.M. Peinado, A.M. Sanchez, V. Rubio, A.J.dept. eoria de la Senal,” Recognition

of coded speech transmitted over wireless channels Wireless Communications”, IEEE

Transactions on Volume: 5, Issue: 9, pp-2555 – 2562., Sept. 2006.

[4] Evans, J.R. Tjoland, W.A. Allred, L.G.Ogden Air Logistics Center, Hill AFB,” Achieving

a hands-free computer interface using voice recognition and speech synthesis”, IEEE

Volume: 15, Issue:1, pp 14 16., Jan 2000.

[5] Pelaez-Moreno, C. Gallardo-Antolin, A. Diaz-de-Maria,” Recognizing voice over IP: a

robust front-end for speech recognition on the world wide webMultimedia”, IEEE

Transactions on Volume: 3, Issue:2, pp-209 – 218., Jun 2001.