DEPARTMENT OF INFORMATION TECHNOLOGY
Augmented reality Display of possibilities of augmented reality using Java and C++
Marcus Agelii & Gösta Agerberg Supervisor: Maya Neytcheva & Sergio Courtade Project in computer sience: Report January 2013
Jan 2013
PR
OJE
CT
RE
PO
RT
Abstract
The purpose of this project is to explore augmented reality and investigate its possibilities.
Another target was to have software which ran on a mobile device. In this report we
summarize the conducted study on the current progress in the field and describe two
applications, for Android, developed under the project work. The applications
demonstrate what augmented reality can be used for. The applications are written in Java
and C++.
3
Contents
1. Introduction
What is augmented reality ............................................................................................... 4
Markers ............................................................................................................................ 4
Augmented reality - early history and current development ........................................... 5
Augmented reality for mobile devices ............................................................................ 6
Android applications ....................................................................................................... 6
2. Project description and aims
Problem description ......................................................................................................... 7
Limitations ....................................................................................................................... 7
Goals ................................................................................................................................ 7
3. Workflow and implementation ..................................................................................... 8
4. Results and conclusions
Logotype app ................................................................................................................. 12
Virtual buttons app ......................................................................................................... 13
Conclusions .................................................................................................................... 13
5. Discussion
Difficulties ..................................................................................................................... 14
Possible developments of our apps ................................................................................ 14
Possibilities of Augmented Reality ................................................................................ 15
6. Acknowledgements ...................................................................................................... 17
7. References ..................................................................................................................... 18
4
1. Introduction
What is augmented reality
Augmented Reality (AR) is the process of detecting objects in the reality and adding
information to that reality. The best way to explain this further is by an example: Imagine
that you are looking at a screen; the screen shows a camera feed of today’s newspaper.
When the camera comes to a certain picture in the newspaper, instead of just showing that
newspaper with the picture, it instead shows a video taking the picture’s place in the
newspaper.
The effect is achieved by enabling the program that detects the picture to search through
its database looking for what actions to take when it finds that exact image. This specific
program has videos as actions, and uses it by overlaying the image by the video.
Markers
To detect something in reality an AR application needs certain input; this is usually from
a camera but could also be from other sources such as GPS or accelerometers. If the AR
application uses a camera feed as an input and recognizes an object using image
recognition software, this object is called a marker. Often specific images are used as
markers but it could also be real objects such as a car or a building. There is also an AR
technology called markerless based AR, instead of trying to recognize specific markers
from a camera feed, this technology tries to do a mapping of the surrounding environment
and uses this map to augment reality.
Figure 1: Picture in a newspaper being replaced with a video using augmented reality
5
Augmented reality - early history and current development
AR as a concept and idea has been around for many years but it’s only in the recent years
that technology has been made small and fast enough to make AR available for everyday
use. The first AR system was developed in the 60’s but the term AR was not coined until
1990 by a Boeing researcher called Tom Caudell.1 One of the first uses of simple AR
technology was by pilots in fighter jets, where the pilot had a HUD (Heads Up Display)
showing altitude, velocity and other useful flight data in their main field of view. This can
however be discussed as being real AR but when this technology evolved into also
showing waypoints and cues over targets (Target designators) this becomes more of what
people see AR as today.
In recent years as mobile phones and handheld devices have been made cheaper and more
powerful, the potential of AR applications have increased significantly. Today there are
many apps available to download from both Google play and Apples App store. The
application field of these apps ranges from navigation to entertainment, home furnituring
etc. There is however still a lot of developing needed before these apps surpasses the
usefulness and the reliability of other methods and apps not using AR.
Figure 2: A HUD display in a fighter jet.
1 Kangdon, Lee; http://link.springer.com/article/10.1007%2Fs11528-012-0559-3
6
Augmented reality for mobile devices
AR right now is optimal for mobile devices and especially for smartphones. The
smartphones have the ingredients needed for the AR to work, a camera, a CPU and a
screen, all in the same small package. Another feature that makes smartphones very
suitable for implementing AR apps is their accessibility both to consumers and to
developers. It is easy to use a smartphone and it is easy to develop software for a
smartphone. The smartphone is also very mobile and as such very easy to move around an
augmented object. Since the smartphones also have other detectors such as
accelerometers and GPS, these can also be used when handling AR.
Android applications
Android is the name of an operating system primarily used for mobile and tablet devices
but also in other devices such as cameras and watches.2 Android is based on Linux but
have been further developed by Google to better suit smartphones and tablets (such as
energy management). Android is open source and primarily developed by Google, but due
to its open source nature many carriers and smartphone manufacturers have adapted
Android to fit their products. For Android there is a wide variety of third-party
applications that can be installed on devices running Android. These applications (apps)
are either distributed by Google’s own store called Google play or by other available
stores such as Amazon store, or by third party websites since it’s possible to
download .apk files to an android device and install them manually.
To develop applications for Android, Google have released an Android Software
Development Kit (SDK) and the applications are developed using Java. This software kit
includes all that is needed to start developing applications for Android including a
debugger, software libraries and tutorials. There is also a native development kit available
for applications or extensions in C or C++. The official Integrated Development
Environment (IDE) supported is Eclipse3 using the Android Development Tools plugin.
2 http://www.android.com
3 http://www.eclipse.org
7
2. Project description and aims
Problem description
The project description for our project was rather general and did not contain any specific
details and limiting conditions. Our supervisor gave us the “AR for mobile devices” topic
and ideas in this topic to investigate and let us set the goals and limitations by ourselves.
The ideas to focus on were basically three; the image analysis involved in reading the
images, the creation and reading of information from the database, or the projection of
information. Therefore our first task was to actually come up with a more specific project
description. An advice we got from our supervisor was to “not invent the wheel again”
and by having this in mind we decided to focus on already built up SDK:s and investigate
how these worked and what could be done with them.
Since we didn’t want this to become merely a literary study, we decided that if possible,
we wanted to create an application that would display the possibilities of AR in an easy
way. In the short description we got, it was preferred that we would work on mobile
devices so we decided that our applications target would be a smartphone of some kind.
Limitations
We had to set up our own limitations, but since neither we nor our supervisor had any
experience in the subject we had no idea how to set them up initially. We did some
research to find a suitable library to work in, and then some more research to see what
this library could do for us, and what we could do with it.
This resulted in that we limited ourselves to the third alternative for specialization – the
displaying of information. We also narrowed our project down to using a library called
NyARToolkit4 and working in Windows. The last two limitations proved to be
unnecessary, since we early in the project found out that the limitations of NyARToolkit
were too strict to fit what we were interested in doing.
More research paved way for a library called Vuforia; this library had more functions
built in, was up to date and offered the possibility to more easily write an application for a
mobile device. This meant that our new limitations sent us towards an Android
application, and also the new experience in programming for a mobile device.
Goals
To merge our project description together with the limitations we posed, we decided to
focus on the following three goals. First, do a literary research of AR together with
Vuforia5, to better know what is possible to do, and also how to do this. Second, to keep
our research from becoming merely a literary study we wanted to create an application for
Android that displayed some of the AR features. Third we decided to provide ideas for
further development, as an inspiration for other projects within the same area of research.
4 http://nyatla.jp/nyartoolkit/wp
5 https://www.vuforia.com
8
3. Workflow and implementation
We started by using a library, called NyARToolkit, and managed to successfully
implement this on a PC. Though easy to implement, NyARToolkit didn’t contain all the
functionalities we needed, neither was it especially suitable for a mobile platform, and we
switched to Vuforia.
The library we have been using, Vuforia, is a library developed by Qualcomm, for AR
and smartphones. This library contains a lot of resources, along with a couple of tutorials
which makes for quick learning. After a period of researching Vuforia, setting up the hard
– and software, we started by successfully implementing all of the tutorials and making
sure we understood the code.
Next step was to create our own application. Since we decided to create an application for
Android, we used Vuforia together with the Android SDK and Eclipse.
We wanted to do something that involved both Uppsala University and Schlumberger, so
we decided to try to do something with their logotypes. To do this we had to first make
the application be able to recognize the logotypes. This is currently being done using a
web service that Qualcomm is providing. It works by uploading an image of the marker
you want to find, let their classifier find characteristics in the image, and finally it
presents the marker together with a grade of how easy the marker will be to spot. For
example our markers for Uppsala University and Schlumberger were not that good. This
depends on the markers not using the entire image space; there are a lot of gaps of plain
white which doesn’t help the image recognition. More so, the logotype of Uppsala
University is almost perfectly rotational symmetric, this is making it difficult for the
algorithm to find out what is up and what is down. As a last problem, both images don’t
have sharp edges, or differ greatly in contrast, which is also something that the image
analysis algorithm is looking for. When done you can download the markers as two files
and attach them to you application. You get an .xml file containing the names and sizes of
the markers and a .dat file which is containing the data.
9
Figure 3: The Eclipse development environment.
As soon as the application can recognize the marker it’s time to decide what you want it
to do when it finds the marker. We wanted to produce something that was showing the
possibilities of AR, and that also was linked to the project and the companies. Therefore
we decided to project text strings and one rotating logotype, all done in 3D.
The creation of text is a bit tricky; this is because OpenGL, which is used in Android,
doesn’t support easy dynamical text creation and projection. It is possible to display text
on the screen relatively easy, but since we wanted the text to be produced on the marker,
we had to do find a way to do this.
We ended up having to create the text in Blender6, an open source 3D content creation
suite. This meant that the text ended up being in 3D, but at the cost of it taking much
more time than just writing a string like one would do for a normal JAVA or C++ program.
The logotypes were also created using Blender, together with Inkscape7, a program that
can automatically trace an image and turn the image into scalable vector graphics in
the .svg format.
6 http://www.blender.org
7 http://inkscape.org
10
Figure 4: Rendering the Uppsala University logotype in Blender.
Finally to import the objects into our application, we had to use a Perl8 script that turned
the .obj file into OpenGL graphics.
The objects don’t contain any textures, simply for the reason that they were already large,
and were content because we could apply generic textures as long as we accepted that our
objects could only have one color.
To display the objects as we wanted them to be displayed, we had to rewrite and write a
lot of new code. There are a lot of smart functions inside the Vuforia library though, and
this has helped us a lot. We decided to let the logotypes spin, and be displayed above the
marker, while the text should be static and displayed in the marker’s plane – all of which
would help to show the possibility of walking around the object since it is locked to the
marker and not to the screen.
To make the object spin we created a function that uses the difference in time in between
two calls for the system time, and rotating the object before projecting it. Doing this often
makes it look like the object is spinning.
8 http://www.perl.org
11
We created one more app as well, to show the possibilities of virtual buttons. Virtual
buttons are a function that creates buttons on the marker that when pressed sends a signal
to the application to react. In our example, we created a spaceship that can rotate and
move forward and backward. This also makes for a nice AR effect, where if you look at
the spaceship right on, and push the button to make it fly away, it goes out of screen
quickly. But, if you move the screen so that you see the space ship from behind, you can
see it fly all the way out to periphery and disappear.
12
4. Results and conclusions
Two different applications for Android are implemented. One shows text and an
augmented 3D-logotype of Uppsala University or Schlumberger when the app detects
their respective logotype.
Logotype app
This application has five different markers. Three of the markers are different versions of
the Uppsala University logotype and two markers are different versions of the
Schlumberger logotype. Depending on whether the Uppsala University logotype or the
Schlumberger logotype has been found it does different but similar things. If the Uppsala
University logotype (in any of the three different versions) has been detected, a red 3D
augmented logotype is shown spinning above the marker. The position of this augmented
logotype will be in the middle of the found marker but a small bit above the plane of the
marker. In addition two pieces of text that lies in the plane of the marker will also be
visible. These text pieces are 3D-objects but since it is placed in the plane of the marker it
will give the appearance of being 2D. A third feature is also that when the Uppsala
University logotype has been detected an options-menu button is changed from linking to
google.com to instead linking to uu.se.
When the Schlumberger logotype has been detected, the red augmented Uppsala
University logotype is changed to a spinning Schlumberger logotype and the link is
instead to Schlumbergers homepage slb.com.
Figure 5: Our logotype app displayed on an Android device.
13
Virtual buttons app
This app uses a generic image as a marker. This image consists of a wood cabinet
combined with four different buttons. The main feature of this image is the four buttons
with which the user can interact. The Image itself is taken from the Qualcomm examples
and is used because of its good properties as a marker.
When the marker has been found by the app a spaceship will be drawn on the screen
above the plane of the marker. The buttons are best described as virtual buttons and can’t
be interacted with on the screen of the smartphone, but must instead be interacted with as
if they existed for real on the image. To press and interact with the buttons the user must
cover the button that he wants to interact with from the view of the smartphone camera.
The buttons have different uses and you can accomplish four different things by pressing
the different buttons. The leftmost button makes the spaceship roll right in its flight path.
The second button makes the spaceship roll left and the third and fourth button makes the
spaceship travel either forward or backward. The buttons works as “press and hold-
buttons” meaning that when the first button is pressed, the spaceship will start rolling
right with a constant speed. When the button is released the spaceship will stop rolling
and maintain the angle at which it stopped. The same approach is used for all of the other
buttons as well.
Figure 6:A screenshot from an Android device using the virtual buttons app.
Conclusions
We think that we have succeeded in reaching our goals, by researching AR and creating
two applications. On the way we managed to present a couple of possible uses for AR,
that in our meaning, points in the direction that AR is a medium that is worth developing
and can be of good use.
14
5. Discussion
Difficulties
The main difficulties within our project were first and foremost to decide what we were
going to focus on. AR is such a big area, with a lot of possibilities. The project course
plan practically only stated that we should do something within AR.
Further on, we had to do some research before we found Vuforia, and we also tried with
another library before Vuforia. We weren’t really content with the functionalities and the
demands of NyARToolkit and therefore we came to the conclusion that we might have to
do some more research to find a better suited library.
We also had some problems due to the fact that almost all new computers nowadays runs
on 64-bit systems, and the first library that we used (NyARToolkit) used some very old
components that demanded a 32-bit system. This was however quickly solved when we
found Vuforia, since it is updated and supports almost all platforms.
The biggest challenge perhaps was that we had to learn how to create objects in 3D, and
to convert these into OpenGL standard. This is due to the lack of easy support for writing
text within Vuforia and OpenGL. There are a few workarounds but due to the limitations
in time for this project we decided to do this the same way it was presented in one of the
tutorials we found. To do this we had to learn how to create 3D objects, and we chose a
program called Blender. This was completely new to us and therefore took some time to
get a hold of.
Possible developments of our apps
As a part of this project, we also decided that we would present a couple of possible
improvements, so that further developing could be done by new groups. Since the project
course only is 15 credits, this means that no group will be able to do all of the proposed
tasks, but our propositions should instead be considered as inspiration to new project
works.
First of all we would like to see improvement being made in the image analysis. This is a
part where we haven’t done anything and as a result we ended up relying solely on the
built-in image analysis in Vuforia. Though it works fine, it has some restrictions that keep
us from using whichever picture we would like, and this is something we see as a
potentially big improvement.
The creation of markers is the second area we have thought of. Right now, markers are
being made using the Vuforia homepage, by uploading the desired image, and then
downloading one .dat and one .xml file. We would like to see the creation of markers
being so simple that the user should be able to take a photo with a camera, and then
choose to use it as a marker. This together with the improved image analysis could make
it possible for any marker at all to be used.
The third proposition is to work on simplifying the creation and displaying of text. This
should be one of the simpler tasks, and make a big difference. In the end a lot of
information is written, thus you can easily imagine that text will be the most projected
object. If it would be possible to create the information dynamically, one could also have
15
updated information being projected. Together with internet access it would be possible to
use updated information from homepages, process these and display the info. One can
also use the example above and add, to the newly taken photo of a marker, a couple of
sentences describing the marker – or maybe even a link to the homepage where the
information should be gathered.
The probably biggest and most important improvement however might be to be able to
use some kind of cloud-based storage system. If all the markers and information can be
kept online, it would dramatically decrease the size of the application, and in the
meantime making it possible to store almost unlimited amount of data. Again using the
previous example, one can think of a database that isn’t local anymore, but expanded by
users worldwide. This would also make for a database that expands very quickly and at
almost no cost at all. The application has to be connected to the internet at all times, but
together with some smart search algorithm, should be quick in finding if the current
picture is a marker, and if so – what information is connected to this marker. The search
algorithm also has to keep a check so that two users don’t upload two versions of the
same image and by doing so making it hard for the image analysis to detect which of the
pictures is most similar to the camera feed.
Possibilities for Augmented Reality
We can see a lot of areas where AR could be used, some of them now and some in the
future.
Video games is a market that is constantly developing and pushing its frontiers forwards
toward more realistic gameplay and new and spectacular gaming experience. With that in
mind, we think that AR could be a huge success in certain video games. With the use of
an arena and gaming glasses, one could create football, war or adventure games. You
would need to keep track of the players, their movements and actions, but since this is
already accomplished using for example the Kinect9 tool for Xbox 360; this shouldn’t be
a main concern. To be able to offer a Battlefield or Call of Duty10
experience where
you’re playing using your body to move, having a 360 degree of sight and the feeling of
being inside the game, would dramatically improve the experience.
One can also think of more practical possibilities, such as furniture fitting. Here you can
imagine persons that are thinking of buying a shelf; they want to know whether or not it
fits in their home – by size, color and style. Produce a simple marker and put it where you
want the shelf, and by using your mobile device you can see the actual size and style of
the object, and of course change the color – all by using AR.
AR could also be used as installation walkthrough, whether we’re talking furniture
assembling with IKEA products or for example electronic devices. Recently an American
show produced an example of how to use AR as a tool to install a router, with the mobile
device recognizing the type of router and also which cables should be inserted in which
9 http://www.xbox.com/sv-SE/Kinect/GetStarted
10 Battlefield and Call of Duty are two popular video games, in the FPS (First Person Shooter) genre. They
exist in numerous editions but all of them are in a war setting.
16
order and position.11
Another idea is to use AR as a presentation tool. This can for example be used for
companies who want to demonstrate their latest product, and to make it possible for
potential buyers to see the actual product in 3D and be able to move around it using a
more physical approach than just watch the product on a computer screen. It can also be
thought of as accompanying other information, and extending its possibilities. Since May
2012 there is an app for iPhone that together with the Metro newspaper lets the user see
video clips of news that the paper otherwise would be restricted to describe in text and
maybe publish a picture of.12
Since most papers already feature a web version where they
have video contents, this is a simple and cheap way to offer this functionality to
customers who prefer to read a physical newspaper but still are interested in additional
material.
The best example of how highly rated AR is today is the looming introduction of Google
glasses. That such a big company has invested a lot of resources into an AR project
indicates that they really have confidence in the technique. Google glasses are believed to
have a huge impact on the way and speed at which we will retrieve information. Without
the obligation of using your hands, they can let you find out which bus that will take you
home, who built the building across the street or what groceries is on sale in the local
shop and which recipes that goes hand in hand with these.
11
Mills, Matt;
http://www.ted.com/talks/matt_mills_image_recognition_that_triggers_augmented_reality.html 12
http://www.metro.se/nyheter/i-dag-far-din-metro-tidning-liv/EVHleB!n64olTSHiS1VQ
17
6. Acknowledgements
We would like to thank Maya Neytcheva for her help and support with our project. She
gave us valuable information and set up our office were we worked on this project.
We would also like to thank Sergio Courtade, and Schlumberger, who came up with the
idea to the project. It’s been a really interesting and challenging project, and we have
really enjoyed working together with Schlumberger.
At last we also would like to thank Data-Mats for setting us up with computers, other
hardware and the software that we needed.
Thank you all!
18
7. References
Kangdon, Lee. “Augmented Reality in Education and Training.” Tech Trends 56, no. 2 (March
1, 2012): 13–21. doi:10.1007/s11528-012-0559-3.
Mills, Matt. “Image Recognition That Triggers Augmented Reality | Video on TED.com,” July
2012.
http://www.ted.com/talks/matt_mills_image_recognition_that_triggers_augmented_realit
y.html.
“Android.” Accessed January 8, 2013. http://www.android.com/.
“Blender.org - Home.” Accessed January 8, 2013. http://www.blender.org/.
“Eclipse - The Eclipse Foundation Open Source Community Website.” Accessed January 8,
2013. http://www.eclipse.org/.
“Get Started with Kinect - Xbox.com.” Accessed January 8, 2013. http://www.xbox.com/en-
US/Kinect/GetStarted.
“I Dag Får Din Metro-tidning Liv – Metro,” May 29, 2012. http://www.metro.se/nyheter/i-
dag-far-din-metro-tidning-liv/EVHleB!n64olTSHiS1VQ/.
“Inkscape. Draw Freely.” Accessed January 8, 2013. http://inkscape.org/.
“NyARToolkit Project | The NyARToolkit Project Website.” Accessed January 8, 2013.
http://nyatla.jp/nyartoolkit/wp/.
“The Perl Programming Language - Www.perl.org.” Accessed January 8, 2013.
http://www.perl.org/.
“Vuforia | Augmented Reality Unleashed.” Accessed January 8, 2013.
https://www.vuforia.com/.