Augmenting immersion:

The implementation of the real world in virtual rea lity

Dragoş GHEORGHIU1 | Livia ȘTEFAN2

1 National University of Arts, Bucharest, Romania | 2 Institute for Computers, Bucharest, Romania

Abstract: The last few decades have seen the rise of a new archaeology characterized by a propensity for

high realism in the representation of the past under the form of virtual reconstructions, which allow the viewer

to have an immersive experience. Our project proposes the insertion of experiments performed in real re-

constructed contexts inside complex 3D virtual reconstructions, thus augmenting the degree of “reality” and of

immersion. The result is a creative mix of virtual reality and reality (mixed reality) under the form of “windows

of immersion”, which can stimulate the archaeological imagination.

Keywords: high realism, virtual archaeology, mixed reality, immersion.

Introduction

Following the modern tradition of 'visual science‘(ARNOLD 2000, 71), the archaeological imagination

(SELLEN 2012) used and uses ‘intermediaries’ (BONDE and HOUSTON 2013, 1) such as photographic (see

SHANKS 2012, 102;1997) images (see SMILES and MOSER 2005) due to their ‘absolutely analogical’

(MITCHELL 1986, 61) character.

Continuing this visual trend of mimetic representation (see BERGER 1984), supported by the new digital

technologies, and due to a sort of ‘technophilia’ (SHANKS and WEBMOOR 2013, 88) or ‘technological

fetishism’ (HUGGETT 2004), the last decades have seen the rise of a trend in archaeology characterized by a

propensity for high realism in the representations of the Past (SHANKS and WEBMOOR 2013, 89), under the

form of virtual reconstructions (BARCELÓ 2001; ABBOTT 2012). Bucking the trend there are however voices

that warn us of the impossibility of a perfect replication (BRYSON 1983, 6; LEIBHAMMER 2000, 131).

The high realism promoted by ‘Virtual Archaeology’ (FORTE and SILIOTTI 1997; BARCELÓ et al. 2000;

NICCOLUCCI 2002) transformed it into an emergent popular subject, because of its capacity to create

photorealistic products, i.e. images with a high content of similitude with the real.

This tendency in archaeology can also be discerned in contemporary culture which also gravitates towards

photorealism (see HYATT 1946; MILLS 1998; EARL 2009), or to a physically-based rendering (BRÖCKER

2010, 3) due, in great measure, to the notable progress of digital imaging techniques. But this tendency to

reproduce nature in all its minutest details has a long tradition in European culture. As early as the

Renaissance, when the difference between artists and scientists was not so clear cut, the new ‘machines for

seeing’ (ARNOLD 2000, 71), i.e. the microscopes, telescopes, and camera obscura (MILLS 1998) permitted

an accurate perception of detail and generated a tendency to a sort of photographic realism, a phenomenon

observed in the scientific illustration and in the paintings of different art schools.

International Conference on Cultural Heritage and New Technologies | Vienna | 2015

2

In archaeology as well one can observe this gradually increasing tendency towards photorealism (in scientific

illustrations), as the last two centuries have seen the transition from bi-dimensional images of objects and

architecture (see BRUSIUS 2012) to the three dimensional reconstructions in Virtual Reality (VR).

Today, when applied to the reconstruction of ancient objects and architecture, the hyperrealism (see

PAPADOPOULOS and EARL 2009) created with the help of digital technologies has proven remarkably

efficient since it induces in the viewer a strong sense of ‘authenticity’ (CHALMERS et al. 2006) and a ‘high

fidelity’ (MCNAMARA et al. 1998; ROUSSOS and CHALMERS 2003) in a ’realistic’ reconstruction of the past

(DEVLIN and CHALMERS 2001; HAPPA et. al. 2009a; HAPPA et al. 2009b).

In the present paper the authors address the “reality” issue of the virtual reconstructions from the point of view

of the visitors’ immersion in the virtual reconstructed medium (KATEROS et al. 2014) presenting the Time

Maps Project. A website (TIMEMAPS 2016a) was designed to support the research and the proposed

solutions for the immersion employing other methods along with the hyper-realistic design and rendering.

Current research

From this perspective of “improvement” of the reality, VAN BOXTEL (2013) asserts that cultural and historical

conventions affect the users’ understanding of virtual reconstructions and pleads for developing a visual

literacy on the example of the “Rome Reborn” project. The visual literacy is defined by PAUWELS (2008) as

“learning to look more consciously at visual manifestations of reality, being able to place images and visual

representations in a broader context of production and consumption, and becoming aware of the personal and

cultural coloring in visual reflection and action”.

To enhance the sense of physically being present according to the definitions of SANCHEZ-VIVES, &

SLATER (2005), KATEROS et al. (2014) employed a gamification paradigm and the newly available VR

equipment Oculus Rift HMD in two different 3D simulation frameworks, Unity3D and an own-developed

gaming framework. In their research the authors concluded that a low-realism but coherent 3D world, “can

lead to higher presence []. If you’re targeting a visually realistic environment, it is more likely to generate

breaks in presence [] because the human brain will expect many things that we are not yet able to achieve

technically: perfect physics, sound, force feedback“ (KATEROS et al. 2014:10).

Unity3D as a research tool

Unity3D is a free gaming engine with 5 million registered developers and 600 million gamers (UNITY3D

2016b). Even though it is a proprietary framework Unity3D is very popular among game developers and

researchers due to its ease of use, high-quality rendering and rapid deployment to web, standalone, mobile

target platforms and recently to WebGL supported browsers. It is also well documented, with a wealth of

examples as well as with an asset store. The programming model employs modern software paradigms and is

flexible, being supported mainly by C# language and JavaScript for Unity (UNITY3D, 2016d).

At its basic level Unity3D can be employed by less experienced programmers, i.e. for creating a 3D scene with

different 3D assets, light and camera settings. For more experienced users, Unity3D offers physical

simulations, real-time lighting, graphical editing, animations, Artificial Intelligence (UNITY3D 2016a). Moreover,

Unity3D is an augmented/virtual reality (AR/VR)-ready platform, by allowing the integration with AR libraries,

Gheorghiu | Ştefan – Augmenting immersion

3

e.g. Vuforia (VUFORIA 2016), and with Oculus Rift or Samsung Gear VR devices (UNITY3D 2016c). Unity3D

web applications require a web plugin, downloadable as an ActiveX component.

A competitive authoring tool is the Unreal gaming engine (UNREAL 2016), which also supports the

development of VR applications with Oculus Rift HMDs. Unreal requires C++ programming which is not so

popular but which provide better web performance and compatibility with the web standards.

Time Maps’ technical solution

The Time Maps project intends to recover the ancient technologies and to study the spatial experience of the

contexts in which these have been utilized (GHEORGHIU and ȘTEFAN 2012). A case study of the project was

the experimental reproduction of the technologies of a Roman villa rustica. One corner of the villa has been

reconstructed in reality (Fig. 1) and several archaeological experiments with glass, ceramics and textiles have

been filmed. Afterwards, the whole villa was modelled in 3dMax and a Unity3D-based VR application was

implemented and integrated in the TimeMaps website. The rendering is performed by the Unity3D web plugin.

The architectural form and the context were rendered with textures and colors taken from the archaeological

record. The illumination (GHEORGHIU in press) took account of a specific moment of the day, thus trying to

provide a more accurate (realistic) image of the site. In other experiments, e.g. Mangalia and Albești, the

forms and textures of the ancient objects were scanned for a hyper-realistic representation.

Another solution for improving the “reality” in the 3D reconstructions was the scanning of objects and of

historical characters, (i.e. actors dressed in epoch costumes), and their transfer in the virtual reconstructions.

In the “Mangalia” (TIMEMAPS 2016b) and “Albeşti” (TIMEMAPS 2016c) pages of the TimeMaps website, the

scanned objects and ”live persons” were integrated in the VR applications, beside the video hotspots with the

same characters performing in re-enactments, their highly definition details augmenting the degree of

immersion of the visitor in the 3D simulation (Figs. 2, 3)

In the VR application several video hotspots linking to the video films already mentioned have been inserted in

the exact place where the technologies have been utilized. The insertion have been made possible by means

of sensitive images that when clicked open a video player (Fig. 4).

International Conference on Cultural Heritage and New Technologies | Vienna | 2015

4

Fig. 1 – Experimental reproduction of a Roman villa rustica (Gheorghiu and Ștefan 2012)

Fig. 2 – A Hellenistic villa from Kallatis-Mangalia

Gheorghiu | Ştefan – Augmenting immersion

5

Fig. 3 – A Hellenistic villa in Albeşti village

Fig. 4 – The insertion of video hotspot in the 3D reconstruction

International Conference on Cultural Heritage and New Technologies | Vienna | 2015

6

Results

In the case of the virtual reconstruction of the Roman villa rustica the viewer has the possibility to perceive the

whole reconstructed space (rendered in a highly realistic manner) and to have a spatial experience of the

original context. Several hotspots (Fig. 5) lead to videos (Fig. 6) depicting technologies existent at that time

(metal, glass, ceramics, textiles) and augmented with historical and contextual information

By accessing the hotspots, the user navigating through the VR application exits the virtual medium through a

“window” and enters the “real” world. Thus, the “reality” of the hotspot offers more credibility (reality) to the

highly-rendered virtual reconstruction, and improves the immersion.

The educational value of the project due to its high immersive potential was proved in various schools in urban

and rural areas in Romania (ȘTEFAN and GHEORGHIU 2014) and Portugal. Students have used the virtual

reconstructions as a game, and explored both the 3D space and the immersive “windows”.

Fig. 5 – Hotspots leading to videos depicting technologies existent in Roman villa rustica

Fig. 6 – A video film showing the use of a kiln.

Gheorghiu | Ştefan – Augmenting immersion

7

Lessons learned

Experiments

The experience of mixing the virtual with the real underlined the importance of the presentation of the

experiment in a high-realistic context. Even if the context was virtual, this has situated the human action in a

larger frame of forms, materials and illuminations, and permitted a spatial experience of the environment,

difficult to implement in reality.

Experientiality

A second lesson is that of the experientiality (i.e. subjective experience) (see CUNNINGHAM et al. 2008, VI)

which is generated by the recuperation of the “reality” of the past accomplished by combining the scanned

objects and humans with the video films. The experientiality of the virtual space (the first phase of the

immersion) is continued by the experientiality from the real hotspot along with the scanned objects.

The double experientiality, which implies the body agency, also permits a greater perception of “presence”

(LOMBARD and DITTON 1997; HERRERA et al. 2006; JI and WAKEFIELD 2016:17).

Altogether the methods enhanced the reality of the reconstruction and provided an improved educational

value.

Imagination

Last but not least, the advanced state of immersion has stimulated the archaeological (SHANKS 2012) and

creative imagination (see MCCAULEY 2009, 62ff) both of the youngsters and of the specialist users who

participated in the project.

Future work

In spite of the positive results among history teachers, primary school students and scientists, our Mixed

Reality (MR)-based solutions can be improved for a tighter mixture of the virtual and real, to produce a

continuum virtual-real-virtual. In the current solution, the users are exiting the virtual context to be able to

immerse in the video re-enactments. This limitation of our design is caused by the Unity3D web plugin, which

acts as a “weak link”: the requirements to install this plugin provide delays in the loading of the web VR

applications and incompatibilities with mobile access of the VR applications. Furthermore, the future browsers,

such as WebGL, will no longer support the plugins (CHROME 2016), including the Unity3D web plugin, as a

method to promote the use of standard web technologies and provide a more natural web experience.

Running the present TimeMaps MR application on the web will not be possible, provided that older Internet

Explorer or Mozilla browsers will be employed.

As a future work we plan to create a continuum comprising the scanned objects and the video films. For these

purposes we shall experiment the new Unity3D 5.0 deployment for WebGL (WEBGL 2016), a JavaScript

library for rendering interactive 3D and 2D graphics within any compatible web browser and without the use of

plugins. We also will attempt a solution based on X3DOM, a cross-platform JavaScript open-source

framework. X3DOM (X3DOM 2016) supports native implementations (iOS8, Chrome and Firefox for Android),

International Conference on Cultural Heritage and New Technologies | Vienna | 2015

8

so this solution could solve the compatibility with the mobile devices. X3DOM supports more immersive MR

applications (X3DOM 2016), by a natural integration of the videos within the 3D content, which could better

serve the objectives of our project.

Acknowledgements

The authors thank Mr. Wolfgang H. Börner for his kind invitation to publish the paper presented at the 20th

International Conference on Cultural Heritage and New Technologies, November 2015, Vienna, and Dr.

Willem Beex, Prof.Dr. Giorgio Verdiani and Prof. Dr. Peter Ferschin for their useful remarks during their

session “Virtual, augmented reality and other techniques in Cultural Heritage for the general public”. Many

thanks to M. Bogdan Căpruciu for his help in improving the translation of the present text.

The reconstructions from the Vadastra site were coordinated by Professors Andreea Hasnaş and Dragoş

Gheorghiu and were integrated in the Unity3D-based VR application by Golem Company. The reconstruction

from the Albeşti site was coordinated by Dr. Robert Constantinescu, and the scanning and the animation from

Mangalia and Albeşti were performed by Marius Hodea and Liviu Ungureanu.

TimeMaps is funded by an exploratory research, grant PN II IDEI (Time Maps. Real communities, Virtual

worlds, Experimented Pasts).

References ABBOTT, M. (2012). Visualising Stonehenge: A virtual archaeology, In: D. Arnold, J. Kaminski, F. Niccolucci, and A. Stork, eds., The 13th

International Symposium on Virtual Reality, Archaeology and Cultural Heritage VAST (2012). pp.1-4.

ARNOLD, K. (2000). Between explanation and inspiration: Images in science. In: E. Sian, ed., Strange and charmed. Science in the

contemporary visual arts. London: Calouste Gulbenkian Foundation. pp.68-83.

BARCELÓ, J.A. (2001). Virtual reality for archaeological explanation beyond ‘picturesque’ reconstruction. Archeologia e calcolatori,

pp.221-244.

BARCELÓ, J. A., M. FORTE and D. H. SANDERS (eds.) (2000). Virtual reality in archaeology. British Archaeological Reports

International Series 843. Oxford: Archaeopress.

BERGER, J. (1984). Ways of seeing. London: BBC.

BONDE, S. and S: HOUSTON (2013). Re-presenting the Past. Archaeology through text and image. Oxford and Oakville: Oxbow.

BOXTEL, S. van (2013). Reviving the Past. Conventions of realism in the virtual reconstruction of Rome Reborn, Retrieved from

https://siemvanboxtel.files.wordpress.com/2012/12/reviving-the-past.pdf, Accessed January 2016.

BRÖCKER, M. (2010). Photorealistic rendering in the context of spatial Augmented Reality. Techniques and implementation.

Saarbrücken: VDM Verlag Dr.Müller.

BRUSIUS, M. (2012). Misfit object: Layard’s excavations in Ancient Mesopotamia and the Biblical imagination in mid-nineteenth century

Britain. Journal of Literature and Science 5 (1), pp.38-52.

BRYSON, N. (1983). Vision and painting: the logic of the gaze. New Haven and London: Yale University Press.

CHALMERS, A., I. ROUSSOS and P. LEDDA (2006). Authentic Illumination of Archaeological Site Reconstructions. CGIV, pp.431-434.

CUNNINGHAM, P., J. HEEB and R. PAARDEKOOPER (eds.) ( 2008). Experiencing archaeology by experiment. Oxford: Oxbow Books.

DEVLIN, K., and A. CHALMERS (2001). Realistic visualisation of the Pompeii frescoes, Proceedings of the 1st international conference

on Computer graphics, virtual reality and visualization, ACM, pp. 43-48.

Gheorghiu | Ştefan – Augmenting immersion

9

EARL, G. (2009). Physical and photo-realism: the Herculaneum Amazon. Arqueologica, 16 - 19 Jun 2009, 2.0, Seville, Spain.

EVANS, A., M. ROMEO, A. BAHREHMAND, J. AGENJO and J: BLAT (2014). 3D graphics on the web: A survey, Accessed January

2016 www.Academia.edu.

FORTE, M. and A. SILIOTTI (eds.) (1997). Virtual Archaeology: Great Discoveries Brought to Life through Virtual Reality. London:

Thames and Hudson.

GHEORGHIU, D. (in press). Lighting in reconstructed contexts: Archaeological experiments and experientiality with pyrotechnologies, In:

Papadopoulos, C. (ed.), Archaeology and the Language of Light: Interdisciplinary Studies in Experience and Perception, Oxford, Oxford

University Press.

GHEORGHIU, D. and L. ȘTEFAN (2015). Preserving Monuments In The Memory Of Local Communities Using Immersive MAR

Applications As Educational Tools, The 10th International Conference on Virtual Learning, Universitatea de Vest, Timisoara.

GHEORGHIU, D. and L. ȘTEFAN (2014). Memory and Immersive Applications. The use of MAR to preserve Local Tangible and

Intangible Heritage (poster), E-iED 2014: 4th European Immersive Education Summit, University of Applied Science BFI, 24-26 November

2014, Vienna, Austria.

GHEORGHIU, D. and L. ȘTEFAN (2012). Mobile Technologies and the Use of Augmented Reality for Saving the Immaterial Heritage,

The 13th International Symposium on Virtual Reality, Archaeology and Cultural Heritage VAST2012, Brighton, pp.21-24.

HAPPA‚ J., A. ARTUSI‚ P. DUBLA‚ T. BASHFORD−ROGERS‚ K. DEBATTISTA‚ V. HULUSIĆ and A: CHALMERS ( 2009a). The Virtual

Reconstruction and Daylight Illumination of the Panagia Angeloktisti. Proceedings of VAST: International Symposium on Virtual Reality‚

Archaeology and Cultural Heritage 2009. pp.49-56.

HAPPA, J., M. A. WILLIAMS, G. A. TURLEY, G: EARL, P.DUBLA, G. BEALE, G. GIBBONS, K. DEBATTISTA and A. Chalmers (2009b).

Virtual relighting of a Roman statue head from Herculaneum: a case study. Afrigraph 2009. pp.5-12.

Herrera, G., R. JORDAN, and L. VERA (2006). Agency and presence: A common dependence on subjectivity?, Presence 15 (5), pp. 539-

552.

HUGGETT, J. (2004). Archaeology and the new technology fetishism, Archeologia e Calcolatori.

HYATT, M. A. (1946). The Photographic Eye. Bulletin of the Metropolitan Museum of Art, New Series, Vol.5, No.1. pp.15-26.

JI, H., and G. WAKEFIELD (2016). Endogenous biologically inspired art of complex systems, IEE Computer graphics and Applications 36

(1), pp.16-21.

KATEROS, S., S. GEORGIOU, G. PAPAGIANNAKIS, and M. TSIOUMAS (2014). A comparison of gamified, immersive VR curation

methods for enhanced presence, Eurographics 2014, Strasbourg.

LEIBHAMMER, N. (2000). Rendering realities. In: I. Hodder, ed. Towards Reflexive Method in Archaeology: The Example at Çatalhöyük,

BIAA Monograph 28, Cambridge: The McDonald Institute of Archaeological Research. pp.129-142.

LOMBARD, M. and T. DITTON (1997). At the hearth of it all: The concept of presence, Journal of Computed-Mediated Communication 3

(2).

MCCAULEY, D. (2009). Night and shadows: The space and Place of Darkness. Environment, Space, Place, vol. 1. Bucharest: Zeta

Books, pp.51-76.

MCNAMARA, A., A. CHALMERS, T. TROSCIANKO and E. REINHARD (1998). Fidelity of Graphics Reconstructions: A Psychophysical

Investigation. In Rendering Techniques, pp.237-246.

MILLS, A. A. (1998). Vermeer and the Camera Obscura: Some Practical Considerations. Leonardo, Vol.31, No.3, pp.213-218.

MITCHELL, W. J. T. (1986). Iconology: Image, text, ideology. Chicago: University of Chicago Press.

NICCOLUCCI, F., (ed.) (2002). Virtual Archaeology, Proceedings of the VAST Conference, Arezzo, Italy. B.A.R. International Series 1075.

Oxford: Archaeopress.

International Conference on Cultural Heritage and New Technologies | Vienna | 2015

10

PAPADOPOULOS, C. and G. EARL (2009). Structural and Lighting Models for the Minoan Cemetery at Phourni, Crete, In: Debattista, K.,

Perlingieri, C., Pitzalis, D. and Spina, S. eds. Proceedings of the 10th VAST International Symposium on Virtual Reality, Archaeology and

Cultural Heritage, VAST2009, Malta.

PAUWELS, L. (2008). Visual Literacy and Visual Culture: Reflections on Developing More Varied and Explicit Visual Competencies. The

Open Communication Journal, 2, 79-85.

ROUSSOS, I., and A. CHALMERS (2003). High Fidelity Lighting of Knossos. Proceedings of the 4th VAST International Symposium on

Virtual Reality, Archaeology and Cultural Heritage, VAST 2003, Brighton, pp.195-202.

SELLEN, A. T. (2012). Nineteenth-Century photographs of archaeological collection from Mexico. In: Pillsbury, J. ed., Past Presented.

Archaeological illustration and the Ancient Americas, Dumbarton Oaks Pre-Columbian symposia and colloquia, Washington D.C. pp.207-

229.

SHANKS, M. (2012). The Archaeoological imagination. Walnut Creek: Left Coast Press.

SHANKS, M., and T. WEBMOOR (2013). A political economy of visual media in archaeology, In: Bonde, S. and Houston, S., Re-

presenting the Past. Archaeology through text and image. Oxford and Oakville: Oxbow. pp.85-108.

SMILES, S. and S. MOSER, S. (eds.) (2005). Envisioning the Past: Archaeology and the Image. Oxford: Blackwell.

SANCHEZ-VIVES, M., and M. SLATER (2005). From presence to consciousness through virtual reality. Nature Reviews Neuroscience 6,

4, pp. 332–339.

ȘTEFAN, L., and D. GHEORGHIU (2014). 3D cyber-communities of learning. An immersive educational strategy for rural areas,

International Conference Smart 2014 Social Media in Academia: Research and Teaching, 18-21 September 2014, Timisoara, Romania.

CHROME (2016). https://www.chromium.org/developers/npapi-deprecation

X3DOM. (2016). http://www.x3dom.org/documentation/tutorials/generic-3d-data-conversion

WEBGL (2016). http://www.khronos.org/webgl

TIMEMAPS. (2016a). http://timemaps.net

TIMEMAPS. (2016b). http://timemaps.net/timemap/mangalia/?page_id=3401

TIMEMAPS. (2016c). http://timemaps.net/timemap/albesti/?page_id=3288

UNITY3D. (2016a). http://www.unity3d.com

UNITY3D. (2016b). http://unity3d.com/public-relations

UNITY3D. (2016c). http://docs.unity3d.com/Manual/VRDevices-Oculus.html

Unity3D. (2016d). http://docs.unity3d.com/Manual/CreatingAndUsingScripts.html

UNREAL ENGINE. (2016). http://www.unrealengine.com

VUFORIA. (2016). https://developer.vuforia.com/library/articles/Solution/Installing-the-Unity-Extension

Imprint:

Proceedings of the 20th International Conference on Cultural Heritage and New Technologies 2015 (CHNT 20, 2015)

Vienna 2016

http://www.chnt.at/proceedings-chnt-20/

ISBN 978-3-200-04698-6

Editor/Publisher: Museen der Stadt Wien – Stadtarchäologie

Editorial Team: Wolfgang Börner, Susanne Uhlirz

The editor’s office is not responsible for the linguistic correctness of the manuscripts.

Authors are responsible for the contents and copyrights of the illustrations/photographs.