An Adaptable Rear-Projection Screen Using Digital Pens And HandGestures

Peter Brandl1, Michael Haller1, Michael Hurnaus1, Verena Lugmayr1, Juergen Oberngruber1,Claudia Oster1, Christian Schafleitner1, Mark Billinghurst2

1Upper Austria University of Applied Sciences, Hagenberg, Austria2HITLabNZ, University of Canterbury, Christchurch, New Zealand

firstname.lastname@fh-hagenberg.at

Abstract

INTOI is a rear-projection setup which combines ac-curate pen tracking with hand gesture recognition. Thehardware consists of an Anoto pattern printed on a spe-cial rear-projection foil and an infrared tracking sys-tem. INTOI is a low-cost system that is scalable andprovides highly accurate input (to less than 1mm). Fi-nally, our setup supports a novel multi-user interactionthat combines simultaneous interaction of both handand pen gesture input.

1 Introduction

Whiteboards, flip charts, walls, and tables are stillthe primary tools used for explaining, developing, andcommunicating ideas during the early phases of design[4]. While traditional flipcharts and whiteboards areused in many settings, users still face a range of re-strictions: for example, to draw with different colorsone needs different colored pens, it is cumbersome toflip between pages and add additional sheets, and thereis a limited size of the available drawing area. In ad-dition, although people often use PowerPoint presen-tations in combination with paper flipcharts, there isstill no simple way to combine both in a powerful pre-sentation tool.

Over the last decade, large, interactive vertical dis-plays have become increasingly popular. To better un-derstand the design requirements for interactive dis-plays in a business setting, we carried out an explo-rative field study at Voestalpine, an Austrian steel com-pany. An overview of the most important features isdepicted in table 1.

Reasonable latency Hardware robustnessInexpensive to manu- Drawing area of in-facture finite sizeMulti-point interaction Direct interactionFew additional devices Physical objects should not(e.g. just one controller) interfere (e.g. body)

Table 1. Design Requirements for an interac-tive, large, vertical display

In this paper, we present a low-cost interactive ver-tical display that fulfills these requirements. Our dis-play combines digital pen technology from Anoto1 andinfrared optical tracking in a new way. We describean appropriate presentation application, INTOI, thatdemonstrates the benefits of the system. In contrastto most related work, we use digital pens as styli foran intuitive and easy-to-use interaction with large sur-faces. The research described in this paper presentsa new way for tracking on large surfaces and presentsdesign guidelines for large interactive vertical displays.

2 Related Work

In the late 1988, Xerox PARC developed the Live-Board [1], the first blackboard-sized touch-sensitivescreen capable of displaying an image. SMART Tech-nologies Inc.2 introduced its first interactive white-board in 1991. The tracking is based on the DViT(Digital Vision Touch) technology and uses small cam-eras mounted in each of the four corners of the panel

1http://www.anoto.com2http://www.smarttech.com

1

to track the user input [7]. The system is mainly de-signed to be used with pens, but it can also track fingertouches. However, not more than two inputs can bedetected simultaneously. A similar technology is thetouch frame provided by NextWindow3. Again, em-bedded cameras can track up to two points at the sametime. The MIMIO4 and eBeam5 ultrasonic tracking de-vices, where participants use special styli, are a goodand cheap alternative tracking surface. However, theyare limited in their range, and line-of-sight restrictionsreduce the tracking performance.

Matsushita and Rekimoto built the HoloWall, a ver-tical surface allowing tactile interaction [6]. The au-thors achieve good tracking results using a special dif-fuse rear-projected screen, infrared (IR) LEDs and acamera with an IR pass filter. The system tracks anyobject which is near enough to the surface detectedby the camera. Wilson’s TouchLight [10], is a simi-lar imaging touch screen technology, which uses simpleimage processing techniques to combine the output oftwo video cameras placed behind a semi-transparentHoloscreen in front of the user. Wilson’s research re-sults also strongly influenced the success of Microsoft’sSurface which uses four IR cameras for tracking users’input. Starner et al. [9] used an infrared tracking en-vironment for the Perceptive Workbench tabletop sys-tem. The application features the recognition of ges-tures on the surface that enhance selection, manipu-lation, and navigation tasks. The tracking is basedon shadows created by infrared illuminants that aremounted above the table. Finally, Han demonstratedin [3] an impressive scalable multi-touch interactionsurface that takes advantage of frustrated total internalreflection (FTIR). This technology introduces a newway to create scalable multi-touch displays at a man-ageable price.

Unfortunately, in most of this related work, userscan either interact with fingers or hand gestures or withan input device. A combination of both is only possiblewith the DViT and the NextWindow system. Theseinterfaces, however, do not allow a multi-user/fingerinteraction. An alternative to using computer vision orother technology for hand tracking is capturing inputthrough digital pens. Many researchers are workingwith digital pens from Anoto [5]. Although the Anototracking technology has been available for more thanfive years, in the last year it became possible to usea real-time Bluetooth connection to retrieve live pendata. In contrast to most related work, we use theAnoto digital pen as a stylus that allows the tracking on

3http://www.nextwindow.com4http://www.mimio.com/5http://www.e-beam.com/

large augmented surfaces. The setup itself is scalable,easy-to-manufacture, and accurate - even on very largesurfaces.

3 Our Approach

The Shared Design Space [2] was our first demon-stration, where we combined digital pens with surfacetracking for a large tabletop setup. Figure 1 shows ourcurrent prototype and illustrates the hardware setup ofan interactive table and a digital whiteboard in combi-nation with a rear-projection screen.

Figure 1. The rear-projection screen has tinydots printed on a special foil.

Figure 2 depicts the different layers of our setup.The tracking is realized by using a large Anoto patternprinted on a special rear-projection foil (d) and digitalpens (a). Anoto-based pens are ballpoint-pens with anembedded infrared (IR) camera (f) that tracks the penmovements simultaneously. The pen has to be usedon a specially printed 600dpi paper with a pattern ofsmall dots with a nominal spacing of 0.3mm.

The pattern was printed by using black ink. In oursetup we used the HP Desginjet Z2100 plotter in com-bination with the HP Colorlucent Backlit UV foil togenerate the pattern. The Backlit foil is mainly de-signed for back lighted signs so it generates a diffuselight. Thus, no spotlights from the projectors are visi-ble at the front of the screen. Moreover, the renderingand the brightness of the projected image is still ofhigh quality. In our setup, we used one A0 sized pat-tern sheet (118.0cm×84.1cm). The pattern is clampedin-between two acrylic panels (cf. figure 2 (b),(c)). Thepanel in the back has a width of 6mm and guaranteesa stable and robust surface while the panel in the front

(b)

(a)

(c)

(d)

(e)(f)

Figure 2. Users can interact with the pro-jected image using digital pens from Anoto.

has a width of only 0.8mm to protect the pattern fromscratches. We noticed that the acrylic cover in thefront does not diffract the Anoto pattern at all. How-ever, using thicker front panels (e.g. ≥4mm), producesbad tracking results.

Once the user touches the board with the pen, thecamera tracks the underlying Anoto pattern. It canthen derive its absolute coordinates on the pattern andsend them to a computer over Bluetooth at a rate of50Hz. Both the surrounding light and the lights comingfrom the projectors (placed in the back of the panel)do not interfere with the pen tracking, because thecamera tracks the pattern with its IR camera. Cur-rently, Anoto pens with Bluetooth are available fromNokia (SU-1B), Logitech (io-2), and Maxell (PenIT).All of these pens are pressure sensitive which allowsfor additional functionalities (i.e., better control in asketching/drawing application). In our setup we usedthe pen from Maxell. From the pen, we receive thepen ID, the ID of the pattern sheet, and the positionof the pen tip on the pattern. Theoretically, there isno limit to how many people interact simultaneously.However, only 8 devices can be connected to a singleBluetooth dongle. We have tested our setup with eightparticipants interacting simultaneously without havingserious performance penalties.

In addition to the pen tracking, our system alsosupports hand tracking. Behind the display surface,we mounted a common PAL camera (WATEC WAT-502B), and we track the user’s hands on the screen byusing brightness differences. The camera behind thescreen captures a grey surface and objects coming nearthe surface will appear as blurred shadow. Thus, onlyobjects directly touching the surface are recognized assharp outlined shapes (cf. embedded image of figure3).

Figure 3 illustrates our general hardware setup thatenables a stable tracking in an IR light scenario. Thelight illuminating the scene from the back casts a high

Figure 3. Infrared lights at the back of theuser and on top of the screen guarantee anoptimal lighting condition for the hand track-ing. The camera with the IR-pass filter ismounted on the back of the screen.

contrast shadow of the user’s body on the screen.Without additional light from the front, the resultingcontrast image seen by the camera would include thefull body shape. Notice that the IR lights at the frontof the panel do not interfere with the tracking resultsof the digital pens.

3.1 Multi-User Interaction

Our method has many advantages over related so-lutions. One of the most important features is themulti-user interaction and identification. Using a sin-gle Bluetooth dongle, we can support a maximum of8 users, who can interact simultaneously with the sys-tem. Since the pen data is sent over UDP to the ren-dering software, there is no problem in using additionalBluetooth dongles attached to other machines to in-crease the number of users. Thus, the application isvery scalable. All of the Anoto pens have unique IDsin order to keep track of each users’ drawings. More-over, pen settings such as color, stroke width and thecurrently selected tool are stored for each pen sepa-rately.

3.2 Simultaneous interaction of hand andpen

Another important feature of our display is the sup-port for simultaneous hand and pen input. The infi-nite sized drawing area of the demo application can bepanned by sliding one hand over the projection surface.Using two hands together enables the user to zoom inand out of the current page by moving the hands eitherapart or together (cf. figure 4).

Figure 4. While zooming the content with atwo-hand gesture, users can perform accu-rate interaction with the pen device.

In combination with the highly accurate digital penwe were able to achieve very intuitive interaction. TheIR light from the spots in the front is reflected by theuser directly onto the semi-transparent surface. When-ever a user touches the panel, the hand produces ashadow that can be captured by the camera behindthe projection surface. If a digital pen is used at thesame time, the size of the stylus tip is too small to berecognised by the camera and therefore does not affectthe visual tracking. This fact enables the simultaneoususe of stylus and hand gestures without interferenceproblems.

3.3 Accuracy and Performance

We achieved a pen tracking resolution of less than1mm. For the hand tracking we use a PAL-resolutioncamera that is calibrated to the actual area of the IN-TOI surface. Even though the accuracy of the shadowboundaries tracked by the camera is too low to rec-ognize individual finger tips, it is accurate enough toenable robust hand tracking for navigating throughthe page. While our application is idle, it rendersaround 350 frames per second on an Acer Travel-mate 8200, Mobile DualCore Intel Pentium M, 2.0GHz,

2GB RAM, ATI Mobility Radeon X1600 with 512MB.During writing/drawing with the pen we still achieve130fps. As all the stroke data is stored on the graphicshardware, rendering of processed strokes does reducethe performance significantly. The hand tracking is ac-complished by a separate PC that streams the datavia network to the INTOI client. Using common com-puter vision techniques, we achieve a framerate of 30fpson the tracking server, regardless of how many handstracked.

3.4 Using the pattern anywhere

The maximum size of the pattern that can be suc-cessfully printed is A0. However, stitching multipleA0-sized patterns together can result in larger track-ing surfaces without any tracking penalties. Palettesallow efficient and fast interaction with the system onlarge tracking surfaces. Figure 5, for example, depictsa scenario where application properties can be chosenby a simple shortcut palette. The tangible palette al-lows a fast and efficient interaction with the systemand does not require any hardware. Again, we use anAnoto pattern, which is printed on a paper. The pentracks the feature and triggers the according shortcut.

Figure 5. Palette-interaction with the system.

4 Interaction Techniques

Using a pen is a very common and natural way towrite or draw on a sheet of paper. In a pilot study,we found that most subjects moved the page by hand,while writing with the pen (see figure 6). In order tofollow this natural interaction model, the applicationuses pen devices for sketching while the infinite sizeddrawing area is panned and zoomed by hand gestures.

Figure 6. The system offers a pie menu forleft- and right-handed people. The pie menufor right-handed users has placed all itemson the left side of the menu.

While related work also supports hand and pen in-teraction (since most hand tracking setups can auto-matically handle pen tracking), our system can pro-vide a distinction between pen and hand input. Thecombination of both interaction techniques, using thehand-based gestures and the pen-interaction guaran-tees an optimal interaction with the interface . Preciseand accurate interaction is provided by using the penand large movements such as scaling or moving thepanel were realized by intuitive hand gestures.

In contrast to classical WIMP-interfaces, we used alot of pen- and gesture-based metaphors such as theCurve Dial technique [8]. Using a Curve Dial, userscan easily browse through an image gallery, by rotatingthe pen in the middle of the pie menu. This methodtracks the curvature of the path created by the penand changes the values accordingly (cf. figure 7). Thespeed of how fast the user can browse through the con-tent is controlled by the circle size made by the user:the larger the radius the slower the increase in the pa-rameter and thus the slower the speed of the browsingof content. In the same way, users can change otherparameters, cf. stroke width.

In addition to the Curve Dial, a pie menu appears bytapping the pen on the display surface. Consequently,users can change properties or functions. Each usergets his/her own menu and can interact individuallywith the system (e.g. change the colors or the strokewidth). More complex functions (e.g. change to anoverview of the slides) locks the system and avoids un-

Figure 7. The Curve Dial technique allows anintuitive interaction on a large surface.

wanted interactions. Loading content, such as images,presentation slides and stored INTOI-presentations canbe done by simply browsing through the media libraryusing the Curve Dial technique [8]. Finally, INTOIalso supports simple pen-gestures (e.g. for removingobjects, users just have to cross out the object). Whilewe have not conducted formal user studies yet, over thepast several weeks a number of groups of people havetried our prototype interface. In general they foundthat the simultaneous use of both hand and high accu-rate pen input are one of the most promising featuresof this system.

5 Conclusions and Future Work

In this paper we have presented a rear-projectionscreen based on the Anoto technology in combinationwith digital pens and a simple IR-tracking setup. Thescalable system allows multiple users to interact simul-taneously. The hardware for tracking the digital pens isrobust and allows a really precise interaction with highaccuracy. Finally, the setup is easy-to-manufacture andcost effective.

We are currently optimizing the hand-gesture track-ing. Our current tracking setup is depending on thelighting conditions of a room and requires a light roomenvironment. Therefore, we are focusing on improvingthe hand feature tracking by using Han’s FTIR ap-proach [3] or a similar method. Finally, in the near fu-ture we will conduct detailed user studies to evaluatethe usability of our interface and observe the impactthe technology has on meeting dynamics.

6 Acknowledgement

This project is sponsored by the Austrian Sci-ence Fund FFG (FHplus, contract no. 811407) andvoestalpine Informationstechnologie GmbH. The au-thors would like to express their gratitude to the anony-mous reviewers and Jakob Leitner for figure 2.

References

[1] Scott Elrod, Richard Bruce, Rich Gold, DavidGoldberg, Frank Halasz, William Janssen, DavidLee, Kim McCall, Elin Pedersen, Ken Pier, JohnTang, and Brent Welch. Liveboard: a large in-teractive display supporting group meetings, pre-sentations, and remote collaboration. In CHI ’92,pages 599–607, New York, NY, USA, 1992. ACMPress.

[2] Michael Haller, Daniel Leithinger, Jakob Leit-ner, Thomas Seifried, Peter Brandl, Jurgen Za-uner, and Mark Billinghurst. The shared designspace. In SIGGRAPH ’06: ACM SIGGRAPH2006 Emerging technologies, page 29, New York,NY, USA, 2006. ACM Press.

[3] Jefferson Y. Han. Low-cost multi-touch sensingthrough frustrated total internal reflection. InUIST ’05, pages 115–118, New York, NY, USA,2005. ACM Press.

[4] Scott R. Klemmer, Mark W. Newman, Ryan Far-rell, Mark Bilezikjian, and James A. Landay. Thedesigners’ outpost: a tangible interface for collab-orative web site. In UIST ’01, pages 1–10, NewYork, NY, USA, 2001. ACM Press.

[5] Chunyuan Liao, Francois Guimbretiere, and KenHinckley. Papiercraft: a command system forinteractive paper. In UIST ’05: Proceedings ofthe 18th annual ACM symposium on User inter-face software and technology, pages 241–244, NewYork, NY, USA, 2005. ACM Press.

[6] Nobuyuki Matsushita and Jun Rekimoto.Holowall: designing a finger, hand, body, andobject sensitive wall. In UIST ’97, pages 209–210,New York, NY, USA, 1997. ACM Press.

[7] Gerald D. Morrison. A camera-based input de-vice for large interactive displays. IEEE Comput.Graph. Appl., 25(4):52–57, 2005.

[8] Graham Smith, M.C. Schraefel, and Patrick Baud-isch. Curve dial: eyes-free parameter entry for

guis. In CHI ’05: CHI ’05 extended abstracts onHuman factors in computing systems, pages 1146–1147, New York, NY, USA, 2005. ACM Press.

[9] T. Starner, B. Leibe, D. Minnen, T. Westyn,A. Hurst, and J. Weeks. The perceptive work-bench: Computervision -based gesture tracking,2003.

[10] Andrew D. Wilson. Touchlight: an imaging touchscreen and display for gesture-based interaction.In ICMI ’04: Proceedings of the 6th internationalconference on Multimodal interfaces, pages 69–76,New York, NY, USA, 2004. ACM Press.