Rapid prototyping of mobile applications foraugumented reality interactions

Michele Di CapuaUnlimited Software

Centro Direzionale, Isola F/1180143 Napoli, Italy

Email: m.dicapua@u-s.it

Gennaro Costagliola, Mattia De Rosa, Vittorio FuccellaUniversity of Salerno

Via Ponte Don Melillo84084 Fisciano (SA), Italy

Email: {gencos, matderosa, vfuccella}@unisa.it

Abstract—The progress achieved in the field of computer visionand the great improvement and diffusion of mobile technologiesenable the exploration of new models of human-computer in-teraction, especially with respect to Augmented Reality (AR)scenarios. On the other hand, the lack of rapid prototypingenvironments for the development and testing of new systems ofAR and Mixed Reality, may slow down the process of adoptionof these technologies. This document, based on the state of theart of AR technology and on particular of AR development tools,describes the work in progress for the definition of a tool for therapid prototyping of applications using mobile devices. The scopeof the tool is twofold: to simplify the business development andsimulation process of specific scenarios and to provide a tool tosupport the analysis of the quality of human interactions in ARenvironments.

I. INTRODUCTION

Augmented reality (AR) is a term for the live direct orindirect view of a physical, real-world environment whoseelements are augmented by computer-generated sensory in-put [1]. The applications and related technologies for ARare attracting increasing attention from both the scientificcommunity and companies originally involved in differentresearch areas. In particular, the progress achieved in thefields of computer vision and mobile computing are mainlyshifting the focus towards the development of systems forAR for mobile devices [2]. AR is thus creating newer andnewer opportunities for exploring the mechanisms of interac-tion between humans, and virtual and physical environments.However, even if in strong expansion, the current state ofthe art of AR technologies and applications is still belowmarket expectations, especially when considering the qualityof the interaction offered. While some aspects, that are closelylinked to the AR technology (i.e. marker-tracking, rendering,etc.), are gradually evolving, on the other end, there are stillseveral aspects, both technical and social, requiring furtherinvestigation. One of these aspects is the creation and analysisof appropriate interaction techniques for AR applications,which allow the user to interact with virtual content in anintuitive manner [3]. It is possible to explore the developmentof new interaction techniques in different directions including:ubiquitous computing [4], tangible computing [5], and socialcomputing [6]. In ubiquitous computing, we can analyze theinteractions with the user and its activities, within a dynamicenvironment. In tangible computing the user interacts withinterfaces, modeled as physical objects belonging to the everyday life and associated with digital information. The area ofsocial computing tends, in AR environments, to analyze the

psychological aspects underlying user interactions.The state of the art in AR technologies and applications is

analyzed in section II; section III describes the architecture andthe objectives of the proposed tool; lastly, some final remarksand an outline of future work conclude the paper.

II. STATE OF THE ART

At present, the development of AR applications is still anexpensive and not intuitive task. Typically, the applicationstend to privilege a specific context in order to maximize theactual capacity of the particular system used, not allowingthe user to experience different scenarios that would allow theanalysis of complex interactions. In the literature, especially inthe last decade, we can find several tools that tend to simplifythe AR process of application development, trying to reducethe programming effort, and that are often tied to particulardomains and to particular technologies.

It is possible to divide the recent production of developmenttools within the AR systems according to 3 distinct levels [3].At the lower level there are the libraries that provide a basicintegration between the artificial “vision” and the computer. Atthe intermediate level we can find programming environmentswhich provide an initial infrastructure for building applica-tions. On the top level there are the GUI-based applicationenvironments that are mainly oriented towards end users withfew programming skills.

Below we now analyze briefly the main tools developedso far. ARToolkit [7], is a library used for the developmentof AR applications. The library consists of a user friendlyframework, which provides its basic functionalities includingseveral features such as: tracking of markers, OpenGL render-ing and VRLM 3D support. However, the use of the libraryrequires code development in C++ or Java only. Porting ofthis toolkit on some mobile platforms such as Android is alsopossible. Studierstube [8] is a stable framework, developed atthe University of Graz, which includes rendering capabilities,tracking and content management. The environment has beendesigned with a particular inclination to the collaborativeaspects of AR systems. Studierstube is a high-level tool thatrequires the knowledge of C++ and the availability of the OpenInventor graphics toolkit. DWARF (Distributed Wearable ARFramework) [9], is developed at the University of Monaco andis a CORBA-based framework that enables rapid prototypingof distributed applications of AR. The development philosophyof the tool is strongly focused on a communication and dis-

2011 IEEE Symposium on Visual Languages and Human-Centric Computing: Posters and Demos

978-1-4577-1247-0/11/$26.00 ©2011 IEEE 249

tributed service-oriention. However, good programming skillsare still required for the use of this framework.

III. ARCHITECTURE AND OBJECTIVES OF THE AR TOOL

Starting from the idea that a rapid prototyping system[10], [11] may be helpful for the process design of ARapplications, the idea proposed here has, as its main goal, thecreation of a tool for rapid prototyping of mobile applicationsin augmented environments [12]. Several tools were alreadyproduced in this area, as discussed above, but all of themdeal with specific technical issues within the landscape ofAR (i.e. tracking, 3D rendering, etc.). In addition, very fewof these tools work directly within the mobile, and none ofthese tools automatically produces mobile applications able tointeract in AR systems. The tool is made of the following maincomponents (see figure 1):

1) A Visual Environment;2) A Business Logic Server;3) An RDBMS for content data storage.The visual environment is used to map fiducial markers (or

QRcode) to simple actions defined by a user, such as, ”opena web page” or ”play a video”. The user can also chooseto interact with some kind of sensors (i.e. temperature), forexample, getting the data provided from a sensor visualizedon a fiducial marker. The configuration of these actions, linkedto fiducial markers, is stored in an XML file and then passed tothe Business Logic Server, which provides dynamic integrationbetween actions and data (i.e. video or 3d model) stored inthe RDBMS. The tool finally generates a mobile application(i.e. for the Google Android platform), which is configuredto interact with some markers, in the way the user visuallyprogrammed them.

Besides the usual systems for tracking fiducial markersand their 3D representation, the tool integrates other tech-nologies that are recently emerging, and that can improveuser interactions within an environment, such as RFID tags.The development philosophy of the tool provides an abstractapproach to the task of modeling each single element ofthe AR environment (such as markers, QR code, etc..). Theprogramming environment of the tool for the development ofmobile AR applications and their interactions will be mainlycomposed of visual elements. Some of them will be enhancedwith scripting capabilities, still preserving the basic philosophyof not requiring advanced programming skills. Low level codedevelopment will only be required for the possible integrationof external components, such as particular sensors [13] orfor the development of advanced interactive services (i.e.actuators).

As regards the implementation details of the basic featuresof AR systems (i.e. tracking markers), low-level libraries willbe analyzed and integrated (e.g. NyARToolkit). The integrationof these libraries in the tool will allow the continuous updateof the developed mobile applications, without making themdependent on the libraries themselves.

IV. CONCLUSION AND FUTURE WORK

In this paper we have shown the main features of thearchitecture of a tool for the rapid prototyping of AR appli-cations for mobile devices. The advantages that this new toolis expected to provide are:

Fig. 1: Tool working schema

• Rapid prototyping in an augmented environment.• Development of new interfaces (metaphors) of interac-

tions for particular contexts.• Possible development of new formal techniques to “pre-

dict” and model user interactions.There are several possible scenarios in which the proposed

tool can be used. Among these we can assume prototypingand simulation of user interactions with different environmentssuch as mobile shopping (viewing and purchase of goods ina shop “augmented” with interactive elements), and mobilemuseum, in an interactive augmented museum tour.

REFERENCES

[1] D. Wagner, T. Pintaric, F. Ledermann, and D. Schmalstieg, “Towardsmassively multi-user augmented reality on handheld devices,” in In ThirdInternational Conference on Pervasive Computing, 2005, pp. 208–219.

[2] I. M. Zendjebil, F. Ababsa, J.-Y. Didier, E. Lalague, F. Decle, R. Del-mont, L. Frauciel, and J. Vairon, “Realite augmentee en exterieur. etat del’art,” Technique et Science Informatiques, vol. 28, no. 6-7, pp. 857–890,2009.

[3] Feng Zhou, H. B.-L. Duh, and M. Billinghurst, “Trends in augmentedreality tracking, interaction and display: A review of ten years of ismar,”in Proceedings of the 7th IEEE/ACM ISMAR ’08, 2008, pp. 193–202.

[4] M. Weiser, “The computer for the 21st century,” SIGMOBILE Mob.Comput. Commun. Rev., vol. 3, pp. 3–11, July 1999.

[5] H. Ishii and B. Ullmer, “Tangible bits: towards seamless interfaces be-tween people, bits and atoms,” in Proceedings of the SIGCHI conferenceon Human factors in computing systems, ser. CHI ’97. New York, NY,USA: ACM, 1997, pp. 234–241.

[6] L. A. Suchman, Plans and situated actions: the problem of human-machine communication. New York, NY, USA: Cambridge UniversityPress, 1987.

[7] ARToolkit, http://www.hitl.washington.edu/artoolkit/, 2011.[8] Studierstube, http://studierstube.icg.tugraz.at/, 2011.[9] DWARF, http://ar.in.tum.de/Chair/ProjectDwarf/, 2011.

[10] J. Verlinden and I. Horvath, “Analyzing opportunities for using interac-tive augmented prototyping in design practice,” Artif. Intell. Eng. Des.Anal. Manuf., vol. 23, pp. 289–303, August 2009.

[11] Y.-K. Lim, E. Stolterman, and J. Tenenberg, “The anatomy of prototypes:Prototypes as filters, prototypes as manifestations of design ideas,” ACMTrans. Comput.-Hum. Interact., vol. 15, pp. 7:1–7:27, July 2008.

[12] M. Bauer, B. Bruegge, G. Klinker, A. Macwilliams, T. Reicher, S. Riß,C. Sandor, S. Christian, and M. Wagner, “Design of a component-basedaugmented reality framework,” in In Proc. ISAR 2001, 2001, pp. 45–54.

[13] B. Goncalves, J. G. P. Filho, and G. Guizzardi, “A service architecturefor sensor data provisioning for context-aware mobile applications,” inProceedings of the 2008 ACM symposium on Applied computing, ser.SAC ’08. New York, NY, USA: ACM, 2008, pp. 1946–1952.

250