Haptic control and feedback in mobile gaming

Tommi Urtti

University of Tampere Department of Computer Sciences

“Haptics in mobile contexts”-course thesis December 2008

Table of Contents

Haptic control and feedback in mobile gaming .................................................................... 1

Introduction ................................................................................................................................................... 3

Pervasive elements ..................................................................................................................................... 4

The accelerometer ...................................................................................................................................... 5

Multi-touch screens in gaming ............................................................................................................... 8

Tactile feedback ........................................................................................................................................... 9

Conclusion ................................................................................................................................................... 10

References ................................................................................................................................................... 11

Keywords: Pervasive gaming, haptics, gaming, accelerometer, multi-touch

Introduction

Gaming is all about immersion: a good game is one that takes you into the game and

doesn’t let you go, the type of game that you play ”just one round more”. Many

successfully designed features of the game are needed to achieve the best possible

immersion. These include graphics, sound, plot, characters and challenge.[5,6,7]

One element that has not been fully integrated until late has been the use haptic

feedback and use of haptics as input. Including haptics has been shown by Ramsamy

et al. [14] to increase immersion considerably. There are other cases in which haptics

has been integrated as only an inconsistent and irrelevant vibration of the game

controller. In these cases, most of them console or PC games, many players find the

addition annoying instead of positively contributing to the experience.

In the past few years many interesting innovations have electrified the usage of

haptics in games. Nintendo Wii was marketed before announcement as a “revolution”

of gaming instead of an “evolution” like its competitors the Xbox360 and the PS3.

Wii’s graphics are not on par with the other consoles, but its motion-sensing haptic

game controller has made it a success story worldwide, outselling [15] both

Microsoft’s and Sony’s consoles.

Apple’s iPhone was introduced in June 2007. The much-hyped smartphone has many

technical capabilities that profile it as a sound mobile gaming platform. These include

a 3,5 inch multi-touch display, 3D-graphics processing unit from PowerVR and an

accelerometer. Game developers have welcomed the accelerometer and multi-touch

display. We introduce these features through example games.

Sony Playstation’s DualShock controller sparked interest in vibrating controllers in

the 1990’s. Afterwards there has been little progress in designing and giving players

haptic feedback in attempts to achieve a more immersive experience. We’ll take a

look at few innovations in this area, particularly in the field of pervasive gaming,

where haptics is expected to play a bigger role than with mainstream gaming.

Pervasive elements

Pervasive elements are designed to bridge real life objects into virtual game

environments. We decided to include this part in the survey, since they introduce

interesting ideas to gaming in general and mobile gaming in particular.

Ullmer and Ishii [17] have in their work presented two levels of haptic feedback. In

tangible interfaces the first one is the representation level. Immediately by picking up

a tangible object like a baseball bat or a real life PDA you get haptic sensations from

the device, what is its form and functions. The other level is the interaction level, this

the haptic feedback given to the player when the physical objects and digital

information interact in pervasive games. For instance when you hit an image of a

tennis ball with an actual tennis racket (or a controller) the racket vibrates.

Martin Faust has accomplished some interesting work with pervasive elements in

games. Doom 3, a first-person shooter game, has the typical method of guiding the

player with in-game emails and a heads-up-display. Faust modified the game [8] in a

way that instead of reading the emails on the on-screen virtual PDA, they were

pushed to an actual PDA device. Haptic feedback of the game was improved on the

representation level, since the player could hold their real PDA in their hand and read

the emails on it, instead of watching an on-screen image of a PDA.

The new version of Airkanoid [8] is another work by Faust. Airkanoid is an old game

from 1985 where the player controls a paddle with which he reflects a ball around the

screen to break a number of bricks. Once the bricks are broken the player reaches the

next level. A new version of Airkanoid in 2006 took the game concept as it was, but

made a few changes. The paddle movement would not be vertically limited as it had

been before (paddle on the bottom of the screen, horizontal movement for the width

of the screen).

The big invention however was to replace the virtual paddle with actual physical

objects called Airbats. The Airbats in the game are optically detected, so they can

essentially be anything, even a rolled up paper. The game is projected on to a screen.

The image has been altered so that the game elements are added on top of the video

stream of the players. When the players position the Airbat so that the ball hits it, the

ball is reflected in the game (see figure 1).

Figure 1: Haptic Airkanoid.

In “Haptic Airkanoid” Faust and Yoo [10] further improved the concept of Airbats.

By adding two vibrating elements into the Airbats, a stronger bond could be created

between the virtual and physical world. When the image of the physical Airbat hits

the image of the ball, the Airbat vibrates like the collision had actually happened. The

participants described the sensation as not very realistic, but all of them agreed that

the game was more fun with the haptic feedback than without.

The accelerometer

The accelerometer (or motion sensor) is not a new invention, but lately it has

appeared more commonly in consumer devices. Hard drives have integrated the

motion sensor to prevent damage from sudden movement. This is especially useful in

laptops. Mobile phones, including the Nokia N95 and Apple iPhone, are equipped

with accelerometers. Other mobile phones equipped with this feature include models

from Samsung, Motorola and Nokia. In terms of software distribution the iPhone has

an advantage in comparison to competitors. Games and other software can be

purchased exclusively from the App Store. App Store has proven a success; Vodafone

and other operators are now developing similar services for other phone models.

iPhone is the main platform for which we surveyed accelerometer games.

iPhone’s accelerometer composes of a silicon mass, a set of silicon springs, and an

electrical current. Motion is sensed by registering fluctuations in the electrical current

when the silicon mass moves. The silicon springs are in place to lead the current

through the mass. Fluctuations are registered and software developers may use the

data as events. [2]

Figure 2: Accelerometer demonstration [Source: Apple.com].

The accelerometer in the iPhone is used by many of the games available on App

Store; one of these games is Labyrinth. The game’s idea is rather simple, to guide a

small ball through a maze with holes for the

balancing the board was done via

type of game that would be practically impossible to implement with good immersion

using the typical keypad for control. Labyr

to measure the angle of the phone. When the phone is slanted the ball

realistic physics in the correct direction w

through the mass. Fluctuations are registered and software developers may use the

Accelerometer demonstration [Source: Apple.com].

The accelerometer in the iPhone is used by many of the games available on App

ne of these games is Labyrinth. The game’s idea is rather simple, to guide a

small ball through a maze with holes for the ball to fall through. In the original game

was done via 2 wheels that rotate the axis of the planes

type of game that would be practically impossible to implement with good immersion

using the typical keypad for control. Labyrinth on the iPhone uses the accelerometer

to measure the angle of the phone. When the phone is slanted the ball rolls

in the correct direction with speed relative to the angle

through the mass. Fluctuations are registered and software developers may use the

Accelerometer demonstration [Source: Apple.com].

The accelerometer in the iPhone is used by many of the games available on App

ne of these games is Labyrinth. The game’s idea is rather simple, to guide a

n the original game

that rotate the axis of the planes. This is

type of game that would be practically impossible to implement with good immersion

inth on the iPhone uses the accelerometer

rolls with fairly

ith speed relative to the angle.

Figure

Other available games that use the accelerometer include racing, snowboarding

fishing and flying (a bi-plane)

as they would a steering wheel

immersion of accelerometer games believing it to be something that wears off quicker

than a captivating plot or a well

Asier Arranz developed software called NiiMe

2008. The software is an innovative design tha

other Nokia models equipped with the accelerometer to be used as a control

peripheral of a PC. Software installed in the

(x,y,z-coordinates) via Bluetooth to a nearby PC that has been

The phone can then be used as a mouse or a game controller. Game controller usage

has been demoed with games that use the phone as a drumstick, steering wheel,

airplane flight control and

and jumping to jump. The last one

player’s pocket and registering a slight horizontal coordinate change for running and

clearer vertical change for jumping.

Figure 3: iPhone game “Labyrinth”.

available games that use the accelerometer include racing, snowboarding

plane). Generally the player moves the phone around using it

wheel or a fishing rod. Skeptics have questioned the actual

accelerometer games believing it to be something that wears off quicker

than a captivating plot or a well-designed learning curve.

Asier Arranz developed software called NiiMe [3] (Nokia2MovingExperience) in

2008. The software is an innovative design that enables the use of Nokia N95 and

models equipped with the accelerometer to be used as a control

peripheral of a PC. Software installed in the Symbian S60 phone sends location data

coordinates) via Bluetooth to a nearby PC that has been paired with the phone.

The phone can then be used as a mouse or a game controller. Game controller usage

has been demoed with games that use the phone as a drumstick, steering wheel,

airplane flight control and playing Super Mario by actually moving your

and jumping to jump. The last one I believe is done by wearing the phone in the

player’s pocket and registering a slight horizontal coordinate change for running and

vertical change for jumping. In my opinion this represent a clever u

available games that use the accelerometer include racing, snowboarding,

the phone around using it

a fishing rod. Skeptics have questioned the actual

accelerometer games believing it to be something that wears off quicker

(Nokia2MovingExperience) in

t enables the use of Nokia N95 and

models equipped with the accelerometer to be used as a control

S60 phone sends location data

paired with the phone.

The phone can then be used as a mouse or a game controller. Game controller usage

has been demoed with games that use the phone as a drumstick, steering wheel,

playing Super Mario by actually moving your feet to run

is done by wearing the phone in the

player’s pocket and registering a slight horizontal coordinate change for running and a

In my opinion this represent a clever usage of

technology together, taking the accelerometer of the Nokia phone and using to control

a gaming platform that can run more advanced games.

It is early to say if the accelerometer will become a common feature of smartphones

aiming the gaming market. Nevertheless it gives developers the possibility of

designing a completely different game experience compared to the traditional keypad.

Multi-touch screens in gaming

Multi-touch has been one of the most discussed features of display techniques since

Jefferson Han’s (NYU's Courant Institute of Mathematical Sciences) impressive

demonstrations in the TED (Technology, Entertainment, Design) conference of 2006.

In the following years a similar concept was implemented with different techniques

into the Apple iPhone and Microsoft Surface. Multi-touch will also be supported in

the upcoming Windows 7 operating system, scheduled for launch late 2009 or early

2010.

Multi-touch makes gestures a possible method for input in a user interface. For

example the iPhone has the flick, tap, pinch and stretch gestures [1]. The gestures are

used in an attempt to create more intuitive interfaces with less buttons, scales and

menus. By reducing the number of physical buttons in mobile devices the frame can

fit a larger display. A larger display providing visual feedback has been found to

increase game immersion. Previous generations of touch screens had error situations

when the touch screen was used too quickly. Multi-touch has been designed for

simultaneous touches and thus will register touches with better accuracy being less

prone to errors. Disadvantages of touch screens quickly become obvious when

multiplayer games are played on a single screen. Fingers block the view of the

display.

AirHockey is one of the games that use multi-touch on the iPhone. Two players can

play the game either on one phone or over the network. AirHockey is a game where

you control a circular tool with which you hit the puck trying to score a goal in the

other player’s end. The game is fast-paced and requires constant attention. As a result

typically you will have both players controlling their sticks all the time. This game

could not have been implemented on a touch screen without the multi-touch

capabilities of the display. A normal touch screen would only register the one player’s

movements.

A different type of example usage of multi-touch is Simcity, a classic city

construction simulation game on the PC. Simcity on the iPhone uses the pinch and

stretch gestures to zoom in and out. Previously it has been published for the Nintendo

DS and the Windows Mobile platform. Once this game is published it could be well

be used for comparing gaming with gestures or a stylus on mobile platforms.

Tactile feedback

Typically haptics are used in games for feedback instead of an input. Most mobile

phones have the vibration function that is used in games for simple feedback. Problem

with the typical vibration function of a mobile phone or a game controller is that it is

too ambiguous: it doesn’t give specific information to the player of what has

happened. In attempts to significantly improve the usefulness of tactile feedback

innovative engineering is needed. Well designed haptic feedback was proven by Jiang

et al to reduce error rates and thus improve performance [11].

TN Games was founded by a medical surgeon who invented a new method of using

old aircompression techniques. The TN Games 3rd Space Vest has 8 pockets that

inflate rapidly creating the sensation of a tap. Depending on a game this will be

getting shot or hit. Giving the player a hint where the attacks are coming from enables

a quick response and a more immersive experience [16].

Figure 4: TN Games 3rd

Space Vest.

There are many different ways of giving haptic feedback to the player. Techniques

include usage of ultrasound or deep bass sounds. If a company wants to concentrate

on making a product as mobile as possible sound and air inflation devices are heavy-

set. Also in order to achieve the most mobile experience, cables must get rid of. 3rd

Space Vest has a USB cable running to the PC and an air tube going to the external

compressor, which has an AC cord. With this setup moving is going to be limited and

the user has to weigh the value of the haptic feedback. Scott Lowe, of IGN Gear, was

one of many that was impressed by the TN Games device, but would not wear it

regularly. [13]

Conclusion

In this paper we have given an overview of some of the recent developments in the

field of enhancing gaming through the use of haptics as either an input or output. As

future work it would be interesting to do user studies comparing using haptics as input

against keypads or joystics.

References

1. Apple Inc., iPhone multitouch specification sheets, http://www.apple.com/iphone/features/multitouch.html, (1.12.2008)

2. Apple Inc., iPhone accelerometer specification sheets, http://www.apple.com/iphone/features/accelerometer.html, (1.12.2008)

3. ARRANZ, A. http://www.niime.com/ (13.12.2008)

4. BROWN, E. and CAIRNS, P., 2004. A grounded investigation of game immersion, CHI’04: CHI '04 extended abstracts on Human factors in

computing systems, 2004, ACM pp. 1297-1300.

5. CHANG, D. "Haptics: Gaming's New Sensation," Computer, vol. 35, no. 8, pp. 84-86, Aug., 2002

6. CHENG, K. and CAIRNS, P.A., 2005. Behaviour, realism and immersion in games, CHI'05: CHI '05 extended abstracts on Human factors in computing

systems, 2005, ACM pp. 1272-1275.

7. ERMI, L. and MÄYRÄ, F., 2005. Fundamental components of the gameplay experience: Analysing immersion. Proceedings of DiGRA 2005 Conference: Changing Views - Worlds in Play 2005.

8. FAUST M., Airkanoid, http://www.e56.de/Airkanoid.html, 2006 (1.12.2008)

9. FAUST M.: “Doom 3 PDA Mod”, http://www.e56.de/PDA.php 2005. (1.12.2008)

10. FAUST M., YOO Y.-H.: Haptic Feedback in Pervasive Games. In Proceedings of the Third International Workshop on Pervasive Gaming Applications - PerGames 2006

11. JIANG, L. et al.: Reducing error rates with lowcost haptic feedback in virtual reality-based training applications. In: Proceedings world HAPTICS, Pisa, Italy (2005) 420–425603-609.

12. KIM, S.-Y.: ”Interactive Racing Game with Graphic and Haptic Feedback”, p.69-77, LNCS Volume 4813/2007

13. LOWE, S. 3rd Space Gaming Vest Review, http://gear.ign.com/articles/931/931587p1.html (14.12.2008)

14. NEXGENWARS, http://nexgenwars.com/ (1.12.2008)

15. RAMSAMY, P. et al: Using Haptics to Improve Immersion in Virtual Environments. In Computational Science – ICCS 2006, Springer 2006, pp.

16. TN GAMES www.tngames.com/ (13.12.2008)

17. ULLMER, B., ISHII, H.: Emerging frameworks for tangible user interfaces. In Caroll, J.M., ed.: Human-Computer Interaction in the New Millenium. Addison-Wesley (2001) 579–601

18. WANG, Q. et al: “A Haptic Memory Game using the STRESS2

Tactile Display” CHI 2006, April 22–27, 2006