Date post: | 04-Apr-2018 |
Category: |
Documents |
Upload: | ictexperimedia |
View: | 277 times |
Download: | 0 times |
of 56
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
1/56
This deliverable is an intermediate progress report for EXPERIMEDIAs embedded
experiment focusing on shared, real-time, immersive and interactive cultural and educational
and executed by the Foundation of the Hellenic World at its premises at Hellenic Cosmos in
Athens. Starting from the more abstract scenario description provided in the earlier D2.1.2,
exploiting the architectural blueprint described in the D2.1.3, taking into consideration the
methodological guidelines described in D2.1.1 as well as the ethical oversight principles
described in D5.1.1, and of course based on the preliminary work described in D4.3.1, the
document provides an overview of the work that has been done to this day, the data
gathered, the conclusions drawn and the plans for the future of the experiment.
D4.3.2
Experiment progress report including
intermediate results
2012-10-31
Manolis Wallace (FHW), Pavlos Mavridis (FHW), Anthousis Andreadis (FHW)
www.experimedia.eu
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
2/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 1
Project acronym EXPERIMEDIA
Full title Experiments in live social and networked media experiences
Grant agreement number 287966
Funding scheme Large-scale Integrating Project (IP)
Work programme topic Objective ICT-2011.1.6 Future Internet Research andExperimentation (FIRE)
Project start date 2011-10-01
Project duration 36 months
Activity 4 Experimentation
Workpackage 4.3 EX3: shared, real-time, immersive and interactive cultural andeducational experiences
Deliverable lead organisation FHW
Authors Manolis Wallace (FHW), Pavlos Mavridis (FHW), AnthousisAndreadis (FHW)
Reviewers Stephen C. Phillips (ITInnov)
Version 1.0
Status Final
Dissemination level PU: Public
Due date PM14 (2012-11-30)
Delivery date 2012-10-31
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
3/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 2
Table of Contents
1. Executive Summary ............................................................................................................................ 4
2. Introduction ........................................................................................................................................ 6
3. Background .......................................................................................................................................... 7
3.1. Goals and success evaluation criteria ..................................................................................... 7
3.2. Experiences offered before EXPERIMEDIA ..................................................................... 8
3.3. The experience examined in the embedded experiment ..................................................... 9
3.3.1. First stage of the experiment ............................................................................................... 9
3.3.2. Second stage of the experiment ........................................................................................ 10
3.4. Constraints ............................................................................................................................... 10
4. Experimental facility architecture ................................................................................................... 11
4.1. Definition of agents ................................................................................................................ 11
4.2. Live streaming experiment during the show ....................................................................... 11
4.3. Augmented reality experiment after the show .................................................................... 13
5. Experimental facility implementation ............................................................................................ 16
5.1. Streaming experimental facility ............................................................................................. 16
5.2. Augmented reality experimental facility ............................................................................... 16
5.3. Social networks experimental facility ................................................................................... 23
5.4. Monitoring experimental facility ........................................................................................... 25
6. Experiment execution ...................................................................................................................... 26
6.1. Before the beginning of the experiment .............................................................................. 26
6.1.1. Deploy the facility ............................................................................................................... 26
6.1.2. Familiarize ourselves........................................................................................................... 29
6.1.3. Inform staff and schedule accordingly ............................................................................ 29
6.1.4. Prepare the informed consent forms ............................................................................... 296.1.5. Prepare the questionnaire. ................................................................................................. 30
6.2. Experiment procedure ............................................................................................................ 30
6.2.1. Preparatory phase................................................................................................................ 30
6.2.2. Execution phase .................................................................................................................. 30
6.2.3. Data acquisition phase ....................................................................................................... 30
7. Data analysis ...................................................................................................................................... 32
7.1. Statistical views of the data .................................................................................................... 32
7.2. Interesting observations ......................................................................................................... 33
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
4/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 3
7.3. Evaluation ................................................................................................................................ 34
7.3.1. Internal evaluation .............................................................................................................. 34
7.3.2. External evaluation ............................................................................................................. 35
8. Current status and future plans ...................................................................................................... 36
8.1. Experimental facility ............................................................................................................... 36
8.2. Experimental methodology ................................................................................................... 36
9. Ethics, privacy, PIA .......................................................................................................................... 38
9.1. Minimum ethical principles ................................................................................................... 38
9.1.1. Doing good .......................................................................................................................... 38
9.1.2. Doing no harm .................................................................................................................... 39
9.1.3. Risk management ................................................................................................................ 39
9.1.4. Consent................................................................................................................................. 39
9.1.5. Confidentiality ..................................................................................................................... 40
9.1.6. Data protection ................................................................................................................... 40
9.2. Ethical oversight principles ................................................................................................... 40
9.2.1. Informed consent ............................................................................................................... 41
9.2.2. Deception ............................................................................................................................. 41
9.2.3. Data collection..................................................................................................................... 41
9.2.4. Withdrawal from the investigation ................................................................................... 41
9.2.5. Observational research ....................................................................................................... 42
9.2.6. Data protection regulation ................................................................................................. 42
9.2.7. Consortium partner responsibility .................................................................................... 42
9.3. PIA ............................................................................................................................................ 42
10. Risks .................................................................................................................................................... 44
10.1. Evolution and handling of risks ............................................................................................ 44
10.2. Current risk registers ............................................................................................................... 4510.2.1. Risks for the participants ................................................................................................... 45
10.2.2. Risks for the experiment .................................................................................................... 46
11. Conclusion ......................................................................................................................................... 48
Appendix A. Informed consent ........................................................................................................ 50
Appendix B. Device lease .................................................................................................................. 51
Appendix C. Structured questionnaire ............................................................................................. 52
Appendix D. Gathered data ............................................................................................................... 54
Appendix E. Independent experiment evaluation .......................................................................... 55
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
5/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 4
1.Executive SummaryThis document presents the working report of the 3rd EXERIMEDIA embedded experiment.
The document covers all aspects of the experiment, ranging from the purely technical to the
purely theoretical ones. More specifically:
Section 2 provides a brief introduction to the document, including a summary of the reasoning
that led us to split the experiment in two stages.
Section 3 highlights relevant elements of the experiment's background in order to improve the
readability and completeness of this document, whilst the reader is advised to refer to previous
deliverables of the project for further information.
Section 4 makes the architecture descriptions of D4.3.1 more detailed and adopts them to the
specifics of the first stage of the experiment. The architecture is presented for the two distinctparts of the experimental facility and the information flow between the different components is
also shown. Section 5 continues on the same path and explains how these architectures where
actually implemented.
Moving on to the actual implementation of the experiment, Section 5 describes how the
experimental facility was implemented. It is worth noting that due to the early timing of our
experiment additional work had to be carried out from our part in order to address issues not yet
addressed by our technical partners; occasionally not even successfully.
Section 6 focuses on the actual execution of the experiment and gathering of the data. Using theimplemented experimental infrastructure and following a reduced version of the scenario already
outlined in D4.3.1 we explain how we invited a number of participants to experience the
EXPERIMEDIA extensions to the venue and provide us with their feedback on them.
Unfortunately we were only able to do this for the augmented reality component at this stage,
but we are confident we will also cover the other components for the second stage of the
experiment.
Section 7 presents the gathered data and based on a brief statistical processing reaches some
initial conclusions. Despite the small sample they are based on, these conclusions will be very
useful in guiding the preparations for the second stage of the experiment. Evaluations of ourwork, both internal and independent, are also presented.
Having presented the work performed to this day, in Section 8we also outline the work planned
for the second stage of the experiment, so that all of its goals may be reached.
Section 9 discusses ethics and privacy. The precautions taken with respect to ethics and privacy
were already analysed quite rigorously in D4.3.1. We do revisit the subject here for completeness
but the reader should be made aware that much of the text of this section is in fact also found in
section 6 of D4.3.1, but here it is adapted to the specifics of the first stage of the experiment, as
they finally formed.
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
6/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 5
Risks have been monitored and handled throughout the design, implementation and execution
of the experiment, following the guidelines specified in D1.1.2 and using the registers defined in
D4.3.1. A complete review of the evolution of the risk registers is provided in Section 10.
Appendices include the form used to get informed consent, the form used to track devices, the
questionnaire used during the experiment, and the complete listing of the data gathered. We also
include a copy of the external evaluators report.
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
7/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 6
2. IntroductionThis deliverable is an intermediate progress report for EXPERIMEDIAs embedded experiment
focusing on shared, real-time, immersive and interactive cultural and educational and executed by
the Foundation of the Hellenic World at its premises at Hellenic Cosmos in Athens. Startingfrom the more abstract scenario description provided in the earlier D2.1.2, exploiting the
architectural blueprint described in the D2.1.3, taking into consideration the methodological
guidelines described in D2.1.1 as well as the ethical oversight principles described in D5.1.1, and
of course based on the preliminary work described in D4.3.1, the document provides an
overview of the work that has been done to this day, the data gathered, the conclusions drawn as
the plans for the future of the experiment.
Very early in our work towards the execution of this experiment we came to realize some
difficulties with the scheduling of different tasks and events in the timeline of the project that
could prove problematic. Most notable among them the fact that the first working versions ofthe technical partners contributions were expected late in the project and in fact after the
finalization of the plans for the experiment as well as the fact that there would not be a chance
for us to have feedback from the project reviewers at a mature stage of the work in order to be
sure that we are headed in the right direction.
Both constitute very serious dangers for the experiment and therefore appropriate action was
required. Therefore, in order to navigate away from the aforementioned dangers we decided to
split the experiment in two stages with very distinct character and goals.
The first stage is aimed to run quickly, provide an early confirmation of the projectsmethodology and technical approach, put technical components to a practical test and generate
feedback for the technical partners who are working on them, gather know-how that will help
the new partners who are just joining the project to seek even higher goals, allow the consortium
to have a demonstrable output from early on in the project and give us an opportunity to present
our approach to experimenting to the project reviewers during the first year review.
The second stage is aimed to build on the experiences and know-how of the first stage in order
to perform the complete experimental work that was envisaged for the 3rd embedded experiment
in the most suitable manner possible.
This document reports on the first stage of the experiment, which has just been completed. The
document covers all aspects of the experiment, ranging from the purely technical to the purely
theoretical ones, and also hints at the work planned towards the successful implementation and
execution of the second stage of the experiment as well.
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
8/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 7
3. BackgroundThe experiment's background has already been discussed in D4.3.1 as well as in D31.1, D31.2
and D4.33. We briefly review here some main elements of it in order to enhance the readability
and completeness of the current text and advise the reader to refer to the abovementioneddocuments for further details.
3.1. Goals and success evaluation criteriaThe embedded experiment involves the development of an experimental facility and its testing
with the participation of real users in real settings. Clearly something will be implemented, some
data will be gathered and some analysis will happen. Still, a core question remains unanswered
and critically subjective: when has the experimenter done enough? In order to remove the
subjectivity and provide a clear measure of success, we identified in D4.3.1 a set of goals for the
experiment and defined the corresponding objective criteria.
Given the fact that four different levels of success are defined, the objective is to achieve a
considerable level of success in the first stage of the experiment reported herein and then build
on that in the second and final stage of the experiment in pursuit of even higher success.
We review the experiment's goals here briefly, together with their corresponding success criteria,
as defined in D4.3.1:
Goal 1: Be an EXPERIMEDIA test bed (Baseline success)
The experiment can be executed. This entails having implemented the experimentarchitecture, having made all of the included components operational and having been
successful in their integration.
Know-how has been gathered. This refers to the gathering of know-how related to thefurther implementation of the embedded experiment.
Goal 2: Explore suitability of FIRE technologies for the field under examination (Moderate
success)
Identify differentiation between using and not using the FIRE technologies. In otherwords, we need to establish that there is a substantial difference for the visitors betweenthe conventional experience currently offered and the one that will be offered in the
scope of the experiment.
Classify the impact of each component as positive, negative or neutral. This is a morespecific version of the previous criterion, as here it is not enough to establish that there is
a difference. What is also required is a clear indication regarding whether this difference
has an impact that QoE of the visitors and if so whether this impact is positive or
negative.
Goal 3: Measure impact of FIRE technologies (Success)
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
9/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 8
Quantify and measure QoE. In other words we need to have designed a measure thatquantifies QoE and we also need to have applied this to data gathered from the
experiment.
Correlate measured QoE to utilized FIRE technologies. Moving a step further, to meetthis criterion we should be able to identify the contribution of each component in theQoE, so that strategic decisions can be made regarding the directions that warrant
further examination.
Goal 4: Identify parameters that affect impact (Exceptional success)
Measure QoE for different parameters. This criterion is met if data gathered whenrunning different instances of the experiment and different QoE values are computed.
The compared instances need to be such that a direct comparison related the differences
in QoE to differences in QoS of some kind (e.g. bitrate), differences in the design and
execution of experiment (e.g. duration of show), differences in demographics etc. Gathered insight for the design of future experiments. This refers to the gathering of
insight related to the implementation of future EXPERIMEDIA installations at Hellenic
Cosmos, for example in order to run future experiments.
3.2. Experiences offered before EXPERIMEDIAThe embedded experiment is built around the VR immersion experience offered by the Tholos.
When EXPERIMEDIA extensions are not considered, this is offered mainly as a standalone
experience that is not combined with any of the other exhibits or services of Hellenic Cosmos.
The typical operation of the Tholos and of the service it offers to its visitors may be graphicallymodelled as in Figure 1. It is easy to see that this is a mainly one-way communication system, as
the museum educator controls the system, thus specifying what the Tholos system will render
and project to the visitors, while at the same time commenting on it. As a sole exception to this,
visitors are able to participate in electronic polls which determine the path that the Educator will
follow altering in this way the flow of the presentation in real time.
The main reason for this extremely structured and predefined approach is that the museum
educator is working with predefined scenarios, i.e. descriptive texts prepared by the FHW
experts. These texts provide information on the 3D worlds in a specific order and therefore the
tour in the 3D world has to follow the same order, otherwise the museum educator would beunable to provide synchronized information.
Before the start of the show there is a brief pre-show which informs the visitors about the
characteristics of the VR immersion system. This contains information such as whether they will
feel dizzy, what to do in that case and so on. The pre-show is not related to the content of the
show and does not add to the value that is offered to the visitors.
After the end of the show the visitors exit the Tholos and this completes the offered experience.
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
10/56
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
11/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 10
3.3.2.Second stage of the experimentFor the second stage of the experiment we shall implement and test the whole experimental
facility as planned. The only note is that based on the evolution of the risks listed in section 10
there may arise the need for slight deviations from the plans presented in deliverable D4.3.1.
As the plans have now, to move to the second stage of the experiment we will also have to
integrate the experiment monitoring component as well as the social component in the current
experimental facility and we will have to run the experiments with more people and analyse the
data.
3.4. ConstraintsThe main attendants of FHW shows are children and adolescents. In the EXPERIMEDIA
experiments only adults will be considered, which creates a question regarding the validity and
generality of the results.
For the same reasons, and given the fact that the navigation in a virtual world is a group
experience, it is not possible to apply some monitoring techniques (for example video recording),
unless if the experiment is executed only when pure groups of participants are present, i.e.
when there is no one present who is either not eligible or has not agreed to participate in the
experiment.
Regarding the experiment timeline, since some of the real exhibits connected to tags in the
virtual content are in open areas of the Hellenic Cosmos venue, the weather may have an impact
on the execution of the experiment.
The execution of the experiment is also constrained by the timeline of the development of
EXPERIMEDIA components by the technological partners of the project. This has already
become more than evident during the definition and implementation of the first stage of the
experiment.
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
12/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 11
4. Experimental facility architectureAs has already been made clear in D4.3.1, the embedded experiment had and still has a very tight
schedule. As a result, and for reasons that will be discussed in further detail in section 10, minor
adjustments to the original plans had to be made during implementation and execution, as tomake sure that the experiment remained both on track and on schedule.
Therefore, before moving on with the detailed description of the implementation work that has
been carried out since the submission of D4.3.1, we review in the following the design for the
implementation of the experimental facility and provide its most current instance. Specifically,
we discuss the parts of the experimental facility and their architecture.
Our experiment consists of two parts. The first part takes place during the show, inside the
dome installation of the Foundation of the Hellenic World, and aims to improve the overall
experience of the audience through their interaction with scientists and experts. The second partof the experiment takes place after the show, and uses augmented reality technologies to provide
additional information to the audience. Below we provide an architectural overview of each part.
4.1. Definition of agentsThe agents participating in this experiment are
- Visitors: The audience that participate in the interactive shows of the "Tholos" dome theater.
We will refer to them as Visitors.
- Museum educator or simply "educator" is a person that interacts with the audience andcontrols the navigation through the virtual 3D environment.
- Experts: One or more scientists, located remotely, that provide additional commentary on the
content that is shown during the show and answer questions from the visitors. We will refer to
these scientists as "Experts".
4.2. Live streaming experiment during the showSince the Experts are on a remote location, the main motivation behind this experiment is to
allowreal-time interaction between the Experts and the Visitors. To this end, the experts and
the audience must see and hear the same part of the interactive show what is shown in the dome.Additionally, the experts must be able to hear any questions from the audience, and the audience
must hear the experts. Figure 2 shows a high level overview of the locations and the desired flow
of information between the agents that participate in this experiment.
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
13/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 12
Figure 2. High-level overview of the locations and the desired information flow between the agents of theexperiment. In this particular example two experts participate in the experiment from remote locations.
The actual connectivity and communication between the components is shown in Figure 3. In
this figure we can see that the museum educator holds the navigation control, which specifies the
content that should be displayed to the visitors. Based on this input, the cluster in the Tholos
dome processes the loaded 3D world in order to render the according location and viewpoint
and display it to the visitors in the dome. This is the part that was already supported before
EXPERIMEDIA and it is in fact the typical scenario for the utilization of the Tholos.
With the EXPERIMEDIA extensions, the Tholos system, in addition to the local projections,
also forwards the rendered stream (actually a downsized and 2D version of it) to the video
stream server, which in turn makes it available to the experts' application. In this way the experts
will be aware of the presented content in real-time. The video stream from Tholos is captured
from another PC through a video capture card (AVERMEDIA Game Broadcaster HD). At this
PC the video is transcoded along with the audio feed from the educators microphone and are
transmitted using Adobes Flash Live Encoding to ATOS Server. The Experts will be using
ATOS Flash Player to visualize the video stream.
Additionally, the museum educator is able to see video-feeds from the experts, as shown in
Figure 11. This video and audio feed from the experts is also passing through the ATOS Server,
where it is transcoded, and it is made accessible to the educator through a regular website. In
order to see this website, the adobe flash player is required, a plugin that is available in most
modern operating systems. A minor limitation of the system is that since adobe flash is required,
the web-based application cannot run on mobile or embedded devices, which usually lack
support for this plugin. Nevertheless, the usage of web-technologies like flash in our system
makes the video and audio streams easily accessible from multiple computers.
Remote Location 1
Expert 1
Remote Location 2
Expert 2
Audio + Video
from "Tholos"Audio + Video
from "Tholos"
"Tholos" Theater
Educator
Audio + Video
from Experts
Visitors
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
14/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 13
4.3. Augmented reality experiment after the showThe second part of the experiment takes part after the interactive show in the Tholos Theatre.
During this experiment, the visitors can learn additional information about the content of the
show, by visiting specific locations on the premises of the Hellenic World facilities. In these
locations we have installed markers and using augmented reality techniques, these markers canbe recognized by an application that runs on commodity mobile devices. The application can the
super-impose virtual objects on top of the real ones, by tracking the position and orientation of
the markers, or can simply open a website with additional information, depending on the
location of the marker. It is worth noting that this experiment does not require the participation
of the experts, the visitors will take the additional information directly from their mobile device.
This is an significant advantage, because it minimizes the operating expenses.
The architecture and information flow of this part of the experiment is shown in Figure 4. With
our application, the input video feed from the real world is augmented with virtual 3D objects.
Alternatively, when a specific marker is detected in the input video feed, the user is redirectedautomatically to a website that is related to his particular location. The association of the markers
with the virtual objects or the websites is performed using a configuration file.
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
15/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 14
Figure 3. Flow of information and component diagram for the experiment.
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
16/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 15
Figure 4. Flow of information and component diagram for the experiment. Our application configurationcan support up to 50 markers and associate them with different 3D objects or links to websites.
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
17/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 16
5. Experimental facility implementationBellow we outline the implementation efforts that were required in order to support this
experiment.
5.1. Streaming experimental facilityThe implementation of the first part of the experiment requires streaming the content shown at
the dome to multiple remote locations. The interactive content for the dome is produced by a
cluster of PCs, as shown in Figure 2. Each PC renders of this cluster renders only one part of the
dome. We have augmented this cluster with one additional PC that renders one extra view of
the dome. This required a change in the configuration files of our cluster. The video output of
this PC is directed to another PC that encodes the content in MPEG4 H.264 format and streams
it to the ATOS central server. The h.264 format was chosen among many others because it is
one of the most advanced video codecs and it is known to provide state-of-the-art encoding
quality.
The quality of the streaming experience is highly sensitive to the available bandwidth. Therefore
we have experimented with various encoding settings for the video and audio stream, in order to
maximize the image quality and the responsiveness of the streaming content. One technical
limitation that was imposed from the ATOS media server was that the stream should be at
60fps. A 30fps stream would be perfectly adequate for our purposes and would require less
bandwidth but at the time of this experiment we could not use this option. After some
experiments with our line, we have settled on a 1440x1080 video resolution for the stream. This
is highly depended to the bandwidth of the internet connection in the FHW facilities.
It should be noted that the ATOS media server is located in Spain. Therefore some latency is to
be expected in the communication. Aside from the network latency, which is to be expected,
some amount of latency was also introduced by the h.264 encoder in our streaming PC.
Nevertheless, this latency was only a few seconds, and did not pose a serious problem in our
experiment. In the future it might be worth investigating methods to reduce this latency even
further.
5.2. Augmented reality experimental facilityThe second part of our experiment, as noted before, involves a mobile application that usesaugmented reality technologies. The augmented reality framework that was provided to us for
testing was the Metaio Mobile SDK. Using this SDK, we have developed our own mobile
application. During the implementation of this application we had to make a several design
decisions and tests. Bellow we outline these implementation efforts.
First we had to decide which mobile platform to support. The provided SDK supports both the
Android OS from Google and the Apple iOS. After consideration, we chose to implement our
application in the Android OS, because we have found that the development process is more
open. Developing and testing on Apple devices required a registration with Apple, something
that could take several weeks for corporate entities and could potentially delay our efforts, thuscompromise the project. Additionally, the social networking API that is required for another part
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
18/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 17
of the experiment supports only the Android OS. Therefore the application was made for
Android, using the JAVA language and the Eclipse SDK and the Android Development Toolkit.
Figure 5 shows a screenshot of the development environment. The exact versions of the
software used are Eclipse SDK 3.7.2, ADK 18.0.0.v201203301601-306762. The target Android
OS version was 2.3.7 or higher.
Figure 5. The development environment for the mobile application.
After the decision about the Operating System, we had to choose the particular device that was
going to be used in our experiment. We performed our first experiments in a "Sony Ericson
Live" mobile phone, shown in Figure 6.
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
19/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 18
Figure 6. Initial tests were performed in this mobile phone (Sony Ericsson Live). The small screen made itdifficult to use our application.
Our application was successfully running in this phone, but we have observed that the screen
size was rather small, 3.2inches. The small size of the screen was making it hard for the visitors
to see the provided information in the form of 3D Objects or Websites. The details in the 3D
objects were not apparent and the websites were difficult to navigate and read. Therefore we
have used a device with larger screen, the Sony XPERIA P, shown in Figure 7. This phone has a
4.0Inch screen, which makes using the application more comfortable.
Figure 7. The actual device that was used in the experiment (Sony XPERIA P). The 4Inch screen is a goodtrade-off between usability and portability.
Ideally, we could also use some Tablet PCs running the Android OS, but the larger size makes
these devices less portable. We believe a screen around 4-5 inches provides the right balance
between usability and portability. Portability is required for our experiment, because the
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
20/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 19
augmented reality markers are scattered at various locations in the FHW facilities, and the
visitors are expected to carry the device with them, while walking around.
Figure 8. Our application running on the Android Emulator on a PC. Although the application can run inthe emulated device, the actual experiments and testing cannot be performed in the emulator, becauseaugmented reality applications inherently require the video feed from a mobile device. Thus, all of ourtesting and development for the augment reality experiment was performed on actual mobile devices.
Next, we had to adapt our virtual reality content to the needs of a mobile device. Our 3D models
are mostly designed to be displayed by powerful workstations. These workstations are several
times faster than a mobile device, so our content had to be simplifies in order to be usable in a
mobile phone. After considerable experimentation, we have found that our particular mobiledevice can show 3D models with up to 20000 triangles. Another limitation was that the model
should have only one texture, instead of multiple layers of textures that we use in our
workstations. To overcome this limitation, our artists merged the information from all the layers
in a single texture. This texture also has the information about the lighting. This process is called
"baking" in technical terms. Therefore, we have concluded that after reducing the polygon count
to around 20000 triangles and baking all the information to a single texture, the 3D models were
usable in our mobile devices.
Next, we have performed various experiments with the type of markers that we were going to
use in our experiments. The Metaio SDK supports both "marker-less tracking", where the
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
21/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 20
application can track the location and the orientation of an arbitrary image, and tracking with
markers, where the application tracks specific markers, as the ones shown in Figure 9.
Figure 9. Example of markers used in the augmented reality application.In both cases, the 3D model that is superimposed in the video feed should follow the position
and the orientation of the image or the marker respectively. Nevertheless, we have found that
the tracking with markers instead of images is more robust. In particular, we have found that
when using the markers, the tracking is more stable, meaning that the 3D object is actually
closely following (tracks) the marker in the video feed, and does not appear to be moving
independently, which was the case when using markerless tracking. In particular, the
performance of markerless tracking depends on the contents of the actual image that is used for
the tracking, something that we have found unacceptable for our application. Furthermore,
tracking with markers was less sensitive to lighting conditions, and was working reliably even on
environments with relatively low illumination, thus we have used tracking with markers.
The markers are integrated on an information sheets that are placed on various locations at the
facilities of Hellenic Cosmos. Figure 10 shows an example of such a sheet, along with the
application running on the mobile device.
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
22/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 21
Figure 10. Left: The augmented reality information sheet. Right: The visitor places the phone over thesheet and observer the 3D object.
Figure 11. The PC screen of the museum educator, communicating with the remotely located Expert.
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
23/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 22
Figure 12. Two museum educators guide the audience through a virtual representation of the ancient Cityof Miletus inside the Dome Installation of FHW. Low light conditions were required for the projectors.
One educator controls the navigation with the joystick, while the other one narrates some historicalinformation. On the right side we can see the PC with the video feed from the experts.
Figure 13. Another view of the educators during the experiment.
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
24/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 23
5.3. Social networks experimental facilityOur intention was to use social networks to allow direct communication of the Visitors with the
Experts. To this end, we have used the Social Networking API, which is provided by our
partners, in order to build a mobile application that connects the users to the Twitter and
Facebook mobile networks. During our testing, we have found that the Social Networking API
is not robust and ready for deployment, and further development is needed in order to fix
various issues and bugs.
Bellow we have a comprehensive description of our tests, the errors occurred and how to
reproduce them:
1. The provided source at the Experimedia SVN has a small error and fails compilation. Inparticular at the link path there are two versions of the twitter4j library. To make it
compile and perform the tests that follow we had to remove the "twitter4j-core-2.1.1-
SNAPSHOT.jar" library and keep only the "twitter4j-core-android-2.2.5.jar".
2. After the above correction, I have launched the example app and tried to login to twitter.When pressing the button I insert my twitter account credentials and I authorize the app.
After this step, which is performed in a browser that pops-out, the control redirected
back to the android app, but it crashes immediately and shows the typical crash message.
See the screenshot in Figure 14.
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
25/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 24
Figure 14. An error during the testing of the Social Networking API in the Android Emulator. The sameerror also occurs on the devices.
3. After this step, if you relaunch the app, it appears that the twitter account is logged-in.Perhaps it kept the credentials from the previous step, before the crash. But if I sent a
message, the app crashes again.
4. I have also tried to login only using Facebook. In the Facebook case the login happensagain in a browser window that pops-out, but there is not any crash, like twitter. After
the login, when trying to send a test message, the app crashes again. Interestingly enough,
judging from the logs, the app crashes on the twitter components (TwitterBaseImpl.java).
I would expect if the user provided only a Facebook login, and not a Twitter one, theAPI should not try to use Twitter (and crash)
5. The last test was to try to send a message using both Twitter and FB (it could be an APIlimitation that it needs both). This use-case also leads to a crash. It seems that the error
has again to do with the twitter integration/components of the API.
This has been tested on a Sony Experia P with android 2.3.7, on a virtual device with android
2.3.3 and on a virtual device me 4.0.3. All the above configurations exhibited the same
behaviour. All the errors appear deterministic and reproducible in our setup (i.e. if we follow the
exact same steps, we always get the errors described above)
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
26/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 25
More recently we have received an updated version of the Social Networking API from our
partners, and we are in the process of integrating it in our experimental facility so that we can
include it in our tests, as originally planned.
5.4. Monitoring experimental facilityTo this day we have not worked with any other EXPERIMEDIA components. Clearly, this will
be altered for the second stage of the experiment. If nothing else, we shall at least have to
integrate the experiment monitoring component that is being prepared by our partners, since the
projects methodology requires the experiments to be built and executed around it.
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
27/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 26
6. Experiment executionIn this section we present the actions taken and procedures followed in order to conduct the
experiment.
6.1. Before the beginning of the experimentA great deal of effort was required before even approaching the first candidate participant.
Specifically, we needed to:
deploy the experimental facility, familiarize ourselves with the experimental facility, so that we could provide assistance as
required,
inform staff about the upcoming experiment and schedule accordingly,
prepare the informed consent forms and, prepare the questionnaire.
6.1.1.Deploy the facilityThis refers to the actual selection and development of the interest points for which AR services
will be offered. With the help of FHW historians and archaeologists 5 points were identified that
are related to Miletus (which is the topic of the Tholos projection that will be used in the context
of EXPERIMEDIA) and for which additional content is available. For 4 points a pure AR
service is provided as a 3D object will be superimposed on their camera input, whilst for the 5th
point there will be a redirect to an online link. Specifically:
Point 1 is a CD on Miletus offered in the Hellenic Cosmos store and is coupled with a 3D
reconstruction of a bed that could be found in the city.
Figure 15. Point 1
Point 2 is a Book on Priene offered in the Hellenic Cosmos store and is coupled with a 3D
reconstruction of a building that could be found in the city.
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
28/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 27
Figure 16. Point 2
Point 3 is a presentation of Thales in the Mathematics exhibition. It is coupled with a link to the
online Encyclopedia of the Hellenic World (www.ehw.gr) and specifically to the lemma on
Thales. As we found out during testing the wireless signal at that location is not stable enough
and at times it is not possible to access the page. In face the wireless connection failed us duringall the tests and therefore we did not manage to have feedback on this point of interest.
Figure 17. Point 3
Point 4 is a physical reconstruction of an ancient ship. It is coupled with a digital reconstruction
of a different ancient ship and the descriptive text outlines the differences. Depending on the
time of day and most importantly on whether the weather was cloudy or not, difficulties were
observed with the clarity of the monitor.
http://www.ehw.gr/http://www.ehw.gr/http://www.ehw.gr/http://www.ehw.gr/7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
29/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 28
Figure 18. Point 4
Point 5 is a sample excavation site that we use to show pupils how an archaeologist actually
works. It is coupled with the reconstruction of an amphora.
Figure 19. Point 5
One core concern is the relative location of these points. As can be seen in Figure 20 the points
are located at quite distant locations from each other. In fact the walking distance between points
3 and 4 is approximately 900 meters. Considering that a participant would have to be approached
and informed about the project and the experiment at the main entrance of the venue (close to
points 1 and 2) but should walk to points 4 and 5 and then back again in order to have
experienced all 5 elements that we have developed, it is clear that the experiment will take long
to run.
In fact, we have had to accept that only one round of experiments will run in a day and that in
the participant selection criteria we would also add an evaluation of the physical state of the
individual, so that they would not have a problem covering this distance.
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
30/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 29
Figure 20. Venue outline
6.1.2.Familiarize ourselvesThis was the easy part, at lease for the first stage of the experiment. For the reasons that are
presented below we were forced to only run the part of the experiment with the augmented
reality component. The related application is quite easy to use (once it is turned on all a user has
to do is point to the card) so not much training was needed for Hellenic Cosmos personnel.
For the second stage of the experiment we expect to have a considerably harder task inproviding both the museum educators and the experts with effective operational guidelines.
6.1.3.Inform staff and schedule accordinglyWhen it comes to the augmented reality component no scheduling was necessary. For the
Tholos, on the other hand, we needed to find free time in the Tholos schedule, book it for
EXPERIMEDIA, arrange with ATOS so that streaming would be available, arrange with the
museum educators and make sure the experts were also available a the same time.
In trying to arrange for this we came to realize that the expert has to be a person that knows the
content much better than the museum educators (who are also historians and archaeologists) forthis to make sense. There are only two such individuals in FHW (they lead our archaeological
research) and as it should be expected their schedule is very full and their time very limited due
to the fact that they hold critical roles in multiple projects. Unfortunately we did not manage to
find a time slot that could accommodate all the above constraints and had to leave this part of
the experiment for the second stage, so that we have more time to either re-examine the scenario
or book ahead.
6.1.4.Prepare the informed consent formsThe informed consent form was prepared based on a sample that was provided by the EAB. It
can be found (in Greek) in Appendix A.
5.Amphorea
4. Ship
2. Building
1. Bed
3. Link aboutThales
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
31/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 30
6.1.5.Prepare the questionnaire.The questionnaire was prepared based on the ethical and privacy guidelines that were agreed
upon. Therefore only multiple choice questions are included. The questions examine the
participants views regarding the clarity of the image, the relevance of the content to the point of
activation, the participants interest in the presented content, the added educational value and thefun they had using the application. It also asks whether the participants would accept to pay a fee
of 1 euro to have access to this application, or to similar application with more points of interest.
All the above are asked both generally and also separately for each point of interest.
The questionnaire can be found (in Greek) in Appendix C
6.2. Experiment procedure6.2.1.Preparatory phaseIn this stage we selected participants. This was generally done early in the day so that enoughtime would be available for the participants to stroll to all points considered in the experiment.
In fact candidate participants were asked beforehand whether they would be willing to take a
stroll to the other end of the venue and then return. As agreed, no minors were approached.
The experiments goals, the overall procedure, their role in the experiment, the nature of the
gathered data, the handling of the data etc were explained in detail to every candidate aurally, and
then the written consent form was presented. We were available to answer any additional
questions posed, but there were no such incidents.
The experiment run over a duration of 4 days, and since only 3 devices are available this meansthat we worked with a total of 12 participants.
6.2.2.Execution phaseThis is the phase in which the visitors that participate in the experiment visit the Tholos as well
as the physical exhibitions of the Hellenic Cosmos and use the augmented reality application.
The participants were required to sign for the smart devices. This documentation was not
combined with any of the experiments data and was only maintained until the end of the tour,
when the device was returned.
The form signed for the lease of the smart device is presented in Appendix B.
The brochure described in D4.3.1 was not created, as we chose to physically place the markers at
the interest points. This meant of course that we had to provide participants with directions
regarding the location of the interest points.
6.2.3.Data acquisition phaseSince only the AR part of the experiment was executed, Visitors were the only participants. We
used questionnaires and not focus groups to poll them for information.
6.2.3.1. QuestionnairesThe questionnaire used has already been presented.
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
32/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 31
6.2.3.2. Focus groupsIn the first stage of the experiment it proved impossible to hold focus groups. The reason is
purely technical: focus groups work well when a large number of participants has first been
polled through structured questionnaires and then based on the analysis of these questionnaires
the researchers determine some points on which they would like to have additional informationand organize focus groups. Each focus group has a smaller but still considerable number of
members and through discussion the point of focus is examined more to get a more concrete
idea of the participants view.
In the first stage of the experiment it was first of all impossible to first examine the data and then
have the focus groups, simply because the data was gathered over different days and when it was
analysed the experiment execution was already over and no participants were available to hold a
focus group. But even if we had manually selected some focus questions and decided to hold
focus groups we still could not do it because:
The three participants that we have as a maximum number for any given moment is stilla very small number
The participants are scattered over a very large area and randomly return to thereception, hand in the equipment, fill in the questionnaires and leave. There is no way to
synchronize the departure of the three participants and therefore it is not possible to
hold focus groups since it is not possible to have the participants at one place together at
the end of the experiment.
For the second stage of the experiment we plan to hold focus groups. The execution of the partof the experiment that refers to Tholos will facilitate this, as considerably larger numbers of
individuals can take part at the same time.
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
33/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 32
7. Data analysisGiven the limited data that has been gathered, data analysis is much less rigorous that one might
expect from such a complex and ambitious experiment. We do envisage more powerful data
analysis being needed for the second stage of the experiment.
The small size of the gathered data allows us to include the complete list in this document. The
list can be found in Appendix D. Despite the small number of samples, the number of questions
allows a wide range of statistical values, correlations and graphical charts to be produced. We
present below some statistics that we find most informative and interesting.
Questions are translated roughly and abbreviated. A standard 1-5 Likert scale is used for most
questions. The exception is Yes/No questions which are treated as 1/0 values.
7.1. Statistical views of the dataTable 1. Average values of picture clarity.Question Value
Clarity of picture for the bed 4,58
Clarity of picture for the building 4,58
Clarity of picture for the ship 1,33
Clarity of picture for the amphorae 1,75
Table 2. Average values of relevance to the point of interest.
Question Value
Relevance for the bed 2,25
Relevance for the building 2,33
Relevance for the ship 3,83
Relevance for the amphorae 4,08
Table 3. Average values of interest.
Question Value
Interest for the bed 2,17
Interest for the building 2,25
Interest for the ship 3,4
Interest for the amphorae 2,75
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
34/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 33
Table 4. Would pay 1 euro for educational value
Question Value
Educational value for the bed 1,67
Educational value for the building 1,41
Educational value for the ship 3,25
Educational value for the amphorae 2,75
Table 5. Average values of fun.
Question Value
Fun for the bed 2,08
Fun for the building 2,2
Fun for the ship 4,42
Fun for the amphorae 3,41
Table 6. Would pay 1 euro for more content like this
Question Value
Content and points of interest similar to the bed 8%
Content and points of interest similar to the building 8%
Content and points of interest similar to the ship 92%
Content and points of interest similar to the amphorae 92%
7.2. Interesting observationsSince the focus of EXPERIMEDIA is on users we pay particular attention to Table 5 and Table
6, which probably provide the best measure of QoE. The fun that the participants have is a
direct measure of the value they have given to their experience, whilst the question aboutwhether they would be willing to pay for similar services is an indirect measure of the same
thing.
The first important observation is that people would be willing to pay money for such an
application. This is probably the safest way to conclude that the augmented reality component
did enhance their experience considerably.
Quite interestingly there are huge differences in the average values for the different points. It is
clear that points such as the ship and the amphorae have made a bigger impression than for
example the bed. Although one can make guesses, it is the goal of EXPERIMEDIA to focus onusers. Therefore, instead of attempting to guess which elements of these points make them more
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
35/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 34
fun and valuable for the users, we note the observation and hope to have the opportunity to
explore the reasons behind it in a focus group during the second stage of the experiment.
EXPERIMEDIA aims to correlate QoE measures with QoS measures in order to assess which
technical parameters influence the value that the user receives and in which way. Not having
integrated the experiment monitoring component in our facility yet, the closest we have to a QoS
measure is the participants feedback regarding the clarity of picture.Table 7 allows us to review
correlations more easily.
Table 7. Summative data
Point of interest Bed Building Ship Amphorae
Clarity of picture 4,58 4,58 1,33 1,75
Relevance 2,25 2,33 3,83 4,08
Interest 2,17 2,25 3,4 2,75
Educational value 1,67 1,41 3,25 2,75
Fun 2,08 2,2 4,42 3,41
Would pay 8% 8% 92% 92%
It is quite interesting, and perhaps shocking, that the QoS and QoE appear to have a negative
correlation. Most probably a better explanation would be to conclude that they do not have a
strong correlation in this experiment. Even so, it is quite interesting to see that the lower quality
of picture that is available at the outdoors location is not enough to make the experience lessenjoyable and valuable. Clearly there are other parameters related to these points that have a
stronger correlation with the QoE, and it is our goal to explore which are these parameters
during the second stage of the experiment.
7.3. Evaluation7.3.1.Internal evaluationFirst of all we are thrilled to have managed to execute a fully cycle of the experiment, starting
from the specification of an abstract scenario and reaching to the point of discussing correlationsbetween QoE and QoS. Clearly, we have implemented a scaled down version and still have a
large part of the work ahead of us, but we now have for the first time a practical overview of the
process.
We are of course to some extent disappointed not to have managed to execute the part of the
experiment that runs in the Tholos, particularly when this was the first to be implemented and
has already been demonstrated repeatedly to the consortium and to the broader public. But we
did manage to gather valuable data and we will use this in the organization and execution of the
second stage of the experiment.
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
36/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 35
Overall our internal evaluation is that this has been a very successful first stage, and that the
accumulated experience will facilitate even further the implementation of the rest of the work.
7.3.2.External evaluationAs promised in D4.3.1, we have sought the independent evaluation of our results from an
individual who is not directly linked to EXPERIMEDIA or FHW. Ms Evi Togia fits this
description. She holds a masters degree in social sciences and engineering from the University of
Manchester. Her independent evaluation is copied in Appendix D as is, without any editing
from our side.
This independent evaluation is only intended to help us have a more objective examination of
our current achievements and outlook and not to affect the project reviewers opinions.
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
37/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 36
8. Current status and future plansAlthough we have clearly and repeatedly touched upon the issue of the first and second stages of
the experiment, we find it fitting to devote at least a few lines to outlining the plans for the
remaining of our work on this experiment.
8.1. Experimental facilityWe are currently happy with the technical work done on the augmented reality part of the
experimental facility, so we do not expect to be doing and core development work in this
direction. We will of course have to examine if and how the augmented reality software that we
are using can be linked to the EXPERIMEDIA experiment monitoring component.
As far as the data itself is concerned, we do hope to make additions. Specifically, our limited
initial testing has indicated that the augmented reality points are too sparse and only some of
them are truly interesting and relevant to the items they have been coupled with. We havealready asked our content experts (historians and archaeologists) to examine whether more
points can be created, given the existing physical exhibitions available in Hellenic Cosmos.
We are also quite happy with the technical work done on the video streaming end, and grateful
to ATOS for their huge support. We also hope to be able to join this with the experiment
monitoring component.
We did not manage to integrate the social component with our facility the first time round, but
we are now convinced that the updated version that ICCS has circulated will operate as planned.
We therefore plan to do work on integrating this component with our venue and with theexperiment monitoring component.
As has already become available, the integration of the experiment monitoring component with
our venue and experiment is a core goal for the upcoming period.
As far as the involved hardware is concerned, we hope to be able to have more smart devices for
the second stage of the experiment, so that we can run the augmented reality part of the
experiment with more participants.
8.2. Experimental methodologyAlthough we have only managed to run few experiments with the augmented reality component
and none with the video streaming component, we have already had the chance to identify some
weaknesses in our methodology and examined scenario, which may call for the corresponding
adjustments.
Specifically:
It is hard to perform extensive tests with the augmented reality component becausehaving only 3 devices we can only have 3 participants at a time. Considering the size of
the venue and the dispersion of the augmented reality points, this practically means that
only 3 participants may be considered per day, which does not allow for much flexibility.
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
38/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 37
Moreover, a single damaged or stolen device can have a major impact on our ability to
run experiments.
We have already painfully realized that the planned scenario asks for some of our mostvaluable (and therefore busy) staff members to stop their work for the whole duration of
the experiments shows (about 45 minutes each time) in order to be readily available to
help. This is clearly not realistic for a large scale implementation. We may need to
consider an alternative scenario where an expert is not standing by doing nothing but
either has a more meaningful role or is only alerted when there is actual need.
We have not designed a new version of the experimental scenario at this time, nor are we sure
that we will, but this is a direction that we will have to examine seriously over the next period if
we are to maximize the benefits from the execution of the experiment.
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
39/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 38
9. Ethics, privacy, PIAEXPERIMEDIA will conduct research with human participants and is, in particular, interested
in human behaviour and experience with Future Internet technologies to understand how to
provide meaningful collective experiences to individuals and society. Given that participants insocial and networked media research should have confidence in the experimenters, good
research will only be possible if there is mutual respect and confidence between experimenters
and participants. As some areas of human experience and behaviour may be beyond the reach of
experiments, observation or other form of investigation and may raise ethical considerations,
EXPERIMEDIA will provide an ethics management process that incorporates ethical and data
protection review of experiments.
Appropriate management of ethical issues will be guaranteed by the project management
through a mixture of measures to ensure the right technical, physical and administrative
environment. The project identified an ethical issues coordinator, as well as a data protectioncoordinator, which will be incorporated within the overall project management structure.
Additionally, an Ethics Advisory Board (EAB) and Data Protection Board (DPB) have been
created. The EAB will advise the EXPERIMEDIA consortium on ethical, privacy and data
protection issues. The DPB is responsible for ensuring that EXPERIMEDIA is compliant with
data protection requirements and that the technical partners develop a system that considers
privacy.
These are described in more detail in D5.1.1, which also includes the results of extensive work
towards the establishment of guidelines that should be followed in the experiments, in order to
ensure compliance with ethical requirements and respect for the privacy of those involved in the
experiments.
In addition to that, D5.1.2 identifies points that specifically the FHW embedded experiment
should consider and D2.1.1 details the Privacy Impact Assessment (PIA) methodology that
should be followed by all EXPERIMEDIA experiments.
In the following we elaborate on how these have been considered and incorporated specifically
in the design of this embedded experiment. Given the overlapping nature of the topics and the
documents, there may be some repetition of concepts. We keep it in the text by choice, in order
to also maintain the point by point reference to the above documents.
9.1. Minimum ethical principlesIn D5.1.1 a set of ethical principles has been identified for the embedded experiments. They
have all been considered in the design of this embedded experiment, as explained in the
following.
9.1.1.Doing goodThe experiment assesses the added value provided to end users by the extension of the Tholos
infrastructure and of the Hellenic Cosmos venue in general via exploitation of the
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
40/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 39
EXPERIMEDIA technology. Therefore it does good both for those participating in the
experiment as well as in general.
As far as the participants of the experiment are concerned, they will have the opportunity to
benefit from enhanced services that were previously unavailable. In the more general sense, this
experiment will be a first step towards making these technologies a part of the normal operation
of the Tholos (remote experts participating) and of the venue (augmented reality points scattered
in the venue), so that more people can benefit from them in the future.
9.1.2.Doing no harmAlthough people working in EXPERIMEDIA will be monitoring closely the execution of the
experiment in all stages in order to analyse every relevant piece of information that becomes
available, the actual navigation in the virtual worlds is still performed by the properly trained
personnel who do that task in the conventional Tholos shows. Their training and expertise
guarantees the quality of the experience that will be provided to all participating user groups.
For general public the Tholos is meant to provide a feeling of what it was like to live in another
era, which will be achieved for the groups of the experiment as well. For a special case of
visitors, though, there is a different goal: the students and pupils that visit the Tholos are mainly
meant to receive assistance in their history courses. No harm will be done in this direction either,
as these user groups are not considered as eligible participants for the experiment.
When it comes to the augmented reality component, this merely adds a paper sign to the venue.
It makes no difference to those not carrying a specialized device and only offers additional and
very relevant information to those who do. Moreover the devices are owned by FHW and novisitor devices are ever tampered with. Therefore no harm is done.
9.1.3.Risk managementAs the experiment design was formulated, and now as the two stages of the experiment are
executed, risks are constantly analysed, evaluated and treated, in the same sense as in D1.1.2. In
the experiments risk register, two types of risks are identified: risks for the participants and risks
for the experiment itself. In the context of ethical oversight of the experiment of course, it is the
former that is of core interest.
The current instance of the risk register for the participants is displayed in Table 8.
As can be seen all identified risks have been treated with the AVOID option. In fact this is a
strategic choice for the experiment: risks for the participants will be avoided, even if that moves
the risk to the experiment itself. In this manner we can be assured that the participants of the
experimenters will not be facing any risks.
9.1.4.ConsentThe preparatory phase of the experiment involves the explicit communication of any relevant
information to the eligible participants (i.e. what the experiment is about, what it entails, which is
their role, etc). Only those eligible participants that have agreed and have signed a note ofinformed consent are considered in the experiment. This consent has a predetermined duration
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
41/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 40
of two months. Differently to what was stated in D4.3.1, this consent is not revocable after the
submission of the information. The reason is that all information is fully anonymised from its
very creation and therefore it is technically impossible to locate and remove the information
provided by a specific person.
9.1.5.ConfidentialityDuring the experiment only the required data is gathered, this data will only be made available to
the individuals that are needed to process that data and no part of this data will be disclosed to
any third parties. Gathered data is fully anonymised. All data will be purged after the analysis has
been completed and at the latest two months after its gathering.
Figure 21. One of the completed questionnaires. No personal data is listed and all options are multiplechoice so that no handwriting is required either
9.1.6.Data protectionFollowing the review of the experiments plan by the EAB we have reached the conclusion that
we do not need to follow a rigorous data protection plan.
9.2. Ethical oversight principlesD5.1.1 has also produced a more detailed set of ethical principles, more customized to the
specifics of EXPERIMEDIA and the embedded experiments. These have also been considered
and adopted in the design of the experiment, as seen in the following.
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
42/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 41
9.2.1.Informed consentAll participants are informed of the research objectives and all aspects of the research that might
reasonably be expected to influence willingness to participate. We explain all other aspects of the
research about which the participants may enquire. This is done in the preparatory phase of the
experiment, before participants are asked to join the experiment, so their decision constitutesinformed consent.
We do make sure that participants do not feel pressured to take part in the experiment. In fact,
in order to make sure that we do not put pressure on people to take part in the experiment, we
have decided not to set daily goals for the number of participants, so that the people in charge of
recruiting do not become extra pushy when the numbers do not add up.
There is not and will not be any payments to participants. We have stated in D4.3.1 that
incentive mechanisms used may include the offer of free passes for Hellenic Cosmos exhibitions,
but we have not implemented this during the first stage of the experiment. Even if applied, thiswill not put visitors that accept to participate in any higher risk than that of visitors who do not
participate and pay for their passes for the exhibitions.
9.2.2.DeceptionWe will never intentionally deceive, mislead or withhold information from participants over the
purpose and general nature of the investigation.
9.2.3.Data collectionFor the first stage of the experiment we did not collect any personal data about the participants.
We mentioned in D4.3.1 that we will collect some personal data about participants during the
experiment (for example demographics) and this option does remain open for the second stage
of the experiment. In that case we will provide participants with any information to complete
their understanding of the nature of the research. We will discuss with the participants their
experience in order to monitor any unforeseen negative effects or misconceptions, in the scope
of the focus groups.
We will adopt a principle of data minimisation: only the necessary information will be collected
and processed and the information will be stored only for as long as is necessary. It may not
possible not to record specific individuals during the second stage experiment, as we may need
to keep track of the individuals that have the EXPERIMEDIA devices in their possession, but
we will anonymise the data when the devices are returned at the end of each session of the
experiment.
User profiles will only be stored when the consent of the users is acquired and only for the
purpose and lifetime of corresponding experiments session. There will be no commercial
exploitation of user profiles.
9.2.4.Withdrawal from the investigationWe will make it plain to participants that they have the right to withdraw from the research atany time, irrespective of whether some incentive has been offered and accepted. The participants
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
43/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 42
will NOT have the right to withdraw retrospectively their consent given and to require that their
own data be destroyed, simply because with the fully anonymised data it is not possible to do
that. Our EAB assures us that this is okay.
9.2.5.Observational researchThe experiment will be in part run in Tholos, which is used by user groups. In order to avoid
observing individuals that have not given their consent for this, we will try to run the
experiments only with pure groups, i.e. with groups that contain only people that have agreed
to participate in the project.
In anycase observing in the context of this experiment does not include any kind of recording
and is limited to observing group dynamics during the show. This is something that is done in
the Tholos irrespective of the experiment, as the museum educator that is coordinating the show
is always observing the groups response and adapts the navigation accordingly.
If the technological choice is made to use audio in order to interact with the experts (the current
plan is to post to social networks instead), then only the rooms general sounds will be recorded
and not each individual separately. In any case, this will only be allowed for the pure
EXPERIMEDIA groups.
9.2.6.Data protection regulationAs expected, we do state that all personal data will be captured and processed according to the
applicable data protection provisions, such as Directive 95/46/EC on the protection of
individuals with regard to the processing of personal data and on the free movement of such
data, including Article 29 Working Party 8/2010 opinion, and Directive 2002/58 on Privacy andElectronic Communications) and the Greek data protection legislation that may be applicable.
Having said that, we clarify that this is in fact a void statement for the first stage of the
experiment due to the fact that no personal data whatsoever was captured. It remains to be
determined whether any personal data will be considered for the second stage of the experiment,
as the related questionnaires have not yet been prepared.
9.2.7.Consortium partner responsibilityBe sharing the early version of the experiment description with the consortium we invited our
partners to participate in the ethical review of our plans. As the experiment design progressed weupdated our partners about our plans.
Our partners are of course also invited to monitor the experiment either for the sake of acquiring
a better of the operation of the technical components they have provided or simply for their
information. Any concern that they may have at that time will be considered and treated
accordingly.
9.3. PIAAs was shown in D4.3.1, no further PIA is required. Nevertheless, written consent of a two
duration of two months is acquired. Dr. Manolis Wallace will act as the data controller for the
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
44/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 43
experiment. Since no other personal data is recorded, the data controllers duty is limited to
keeping safe the forms of informed consent (which list the participants names) and deleting
them at the end of the predefined period.
7/31/2019 D4.3.2 FHW Experiment Progress Report v1.0
45/56
EXPERIMEDIA Dissemination level: PU
Copyright FHW and other members of the EXPERIMEDIA consortium 2012 44
10.RisksAt the beginning of the work on the experiment, two risk registers were formulated: risks for the
participants and risks for the experiment itself. These were initially reported in D4.3.1, but were
also constantly monitored as a live document, and updated as the project and its environmentevolved. In the following we start by presenting how the evolution of the risk registers has
affected the work on the experiment to this day, as well as the current instance of the registers
with respect the upcoming work.
10.1. Evolution and handling of risksIn the time that has elapsed since the preparation and finalization of D4.3.1, and as work on the
experiment progressed, the risks associated with it also evolved accordingly.