REMOTE – Desk-top VR for future command and control?R.S.Aylett1, C.Delgado1, J.H.Serna1, R.Stockdale2, H.Clarke2, M.Estebanez2,
P.Goillau2, T.Lynam2.
Centre for Virtual Environments, University of Salford, Business House, University
Road, Salford, M5 4WT, Manchester, UK 1
QinetiQ, St. Andrews Road, Malvern, Worcestershire, WR2 3PS. 2
Email: [email protected], [email protected]
Abstract
The paper discusses a study in supporting collaborative military planning in which groupware, video-conferencing and a desktop Collaborative Virtual Environment (CVE) were used. It discusses the designand implementation of the CVE and the set-up and execution of the study using questionnaires andobservation. The results of the study questionnaires showed that the CVE was not seen by users as the bestof the ways offered to support collaborative planning; these results are discussed and their implication forthe design of such a CVE are assessed.
Keywords
Virtual environment, collaboration, groupware, avatar, net-VE.
1. IntroductionThis paper reports work carried out by QinetiQ and the University of Salford, both in the
UK, to compare the use of a desktop collaborative virtual environment with two other
modalities as support for a distributed military command team. The system produced,
REMOTE (Reach-back Enabling Multiple Objective Team-working Environment), was
included in an initial study evaluating potential novel command technologies for
distributed team working, and more detail can be found in the resulting QinetiQ technical
report [Clarke et al., 2003]. The customer for this work was the UK Ministry of Defence
(MoD), who are, like other national defence departments (for example the United States
Department of Defense), looking at ‘command posts of the future’. Command posts are
responsible for decision-making, the co-ordination of manoeuvres, and issuing orders to
troops, and are made up of commanders with an often substantial support staff. Such
large command posts are difficult to deploy, especially in proximity to a battlefield - not
the safest place to be - and yet there are many advantages in doing so for rapid and
efficient decision-making. As a result the idea of using technology to combine a small
mobile group with all the distributed resources they need – especially experienced people
who should not be risked by putting them close to combat - has become an attractive one.
This produces a geographically distributed team.
Much research has been carried out into distributed team working in the much broader
context of the globalisation of organisations. The issues and problems are comparatively
well known as a result, and mostly arise from the absence of spatial co-location. Co-
location has many advantages, especially for tightly coupled synchronous interaction
[Olson et al92, Covi et al 98]. A shared local context, including shared time frame and
many outside events, supports easy socialising, building trust and mutual understanding
[Bradner and Mark 02] and supporting informal interaction before and after more formal
episodes. Personal identities are known and can be taken into account – including cultural
and language differences - and peripheral interactions can be seen in passing and
interpreted. Mutual awareness is known to be central to collaborative working [Steed and
Tromp 97].
Within interactions themselves, many channels other than the overt semantic content are
available – for example gesture, gaze, posture, facial expression and prosody. It has been
estimated that more than 65% of the information exchanged during a face-to-face
interaction is expressed through these largely non-verbal mechanisms [Argyle 88], which
are mainly analogue, continuous, and support subtle distinctions and nuances. Flowing
conversation depends on many such channels for ‘back-channel’ regulation and
successful turn taking [Cassell et al 01]. Moreover, they play important roles in
establishing a common frame of reference both for concepts and objects, for example
through deictic gestures such as pointing. This in turn relies on the spatiality of such
references – people and objects are located in a common space. In multi-person meetings,
such channels can also be used to express or judge the reaction of other participants to the
current speaker. Finally, co-location minimises communication delays.
This analysis has produced a demand for ‘virtual co-location’ (VCL) – a loosely defined
concept which can be taken to mean capturing some or all of the characteristics of co-
location for participants not actually co-located, using a variety of information and
communication technologies: audio, video and computer-based. The basis for the study
reported here was the hypothesis that some of the characteristics of co-location that are
difficult to support through mechanisms such as email, electronic chat, video-
conferencing and groupware, could be supported through a collaborative virtual
environment (CVE) [Churchill and Snowden 98]. Here, a multi-user 3D graphically-
realised shared virtual space would play some of the role of shared physical space
through the sense of physical presence – the feeling of being physically located in the
virtual space – which with multiple users might then produce a feeling of physical co-
location.
2. Study methodologyThere are known methodological problems in evaluating CVEs [Steed and Tromp 97]
related to the prototypical nature of most applications and the need to deal with complex
social behaviour both inside and outside the CVE. The geographic distribution and
number of users makes conducting proper controlled experiments very difficult, while the
prototypical nature of applications usually makes it infeasibly expensive in time and other
resources to create different experimental conditions. These constraints are particularly
acute where the task involved is unpredictably structured and has an extended duration,
as in military operations (note that Steed and Tromp considered video-conferencing and
booking holidays as their applications, both much less problematic than military
operations).
In addition, studies involving military teams are inevitably hard to set up. Not only is it
inconceivable that wholly novel approaches would be tried in a live situation, even their
use in official exercises would require a degree of maturity in the technology and
acceptance by the prospective users still a long way from being achieved.
Given that this work involved an initial study, or ‘proof of concept’, participant numbers
were bound to be too small to adopt a rigorous quantitative approach, forcing a
qualitative and exploratory stance. Nevertheless, qualitative approaches are well
established in HCI work where they are seen as fundamental to user-centred design,
drawing on cognitive and social psychology and ethnography [Hughes et al 95].
Qualitative methods are also widely used in the broader field of Information Systems
[Myers and Avison 02].
The qualitative technique adopted was that of the exploratory case study. This refers to
the collection and presentation of detailed information about a particular participant or
small group, frequently including the accounts of subjects themselves [Sechrest et al 96].
The case study technique gives special attention to completeness in observation,
reconstruction, and analysis of the cases under study and deals with a situation in which
there are many more variables of interest than there are data points. This contrasts with
controlled experiments in which there are typically several orders of magnitude more data
points than variables. A case study is more concerned with establishing variables and
questions for future research than with final, generalisable conclusions, but it can also be
seen as an individual iteration in a larger process of user-centred design, with particular
relevance to use-in-context and interface and interaction issues [Shneiderman 92]. This is
particularly true where a case study can legitimately be categorized as one involving
‘typical’ users carrying out a ‘typical’ task.
Two teams of four participants made up of ex-service personnel were made available.
These were experienced personnel who had carried out such operations in their military
careers and could therefore be regarded as typical of the user group being targeted.
However these participants were only available for the week-long study in March 2003
itself, and not for prior development of the user requirements. Here, experience of past
studies in other military domains and some comment from a military link officer were the
only input. The case study approach is familiar to these users and is widely-used within
the military environment since it is a basic training method and the framework for
military exercises. One can therefore have some confidence in the typicality of the
scenarios used and their relevance to the evaluation of the technology.
As we have commented, it has often been found infeasible in evaluating CVEs to create
different experimental conditions. However it was possible to do this for the study being
discussed given that the participants were available for a week, rather than the shorter
times of some other studies [Steed and Tromp 97]. An initial view was taken that since
groupware offers the most mature technology for distributed teams, this should form the
base-line condition for the study. The package GROOVE Workspace [Groove-Networks
02] was the groupware used. Two further experimental conditions were then added, as
enhancements to, not replacements for, groupware: the addition of video-conferencing,
using CUSeeMe [CUSeeMee-Networks 01], and the addition of a CVE. It was hoped in
this way to capture differential benefits of the second and third experimental conditions.
Thus it was a fundamental requirement for the CVE that it interface with the chosen
groupware platforms and support the invocation of groupware components from within it.
Finally, it is important to remember that the target users are very unlikely to be
sophisticated users of IT and are used to coordinating activity at a distance by voice only,
using radio. They are however also used to working in a disciplined style controlled by
Standard Operating Procedures (SOPs). SOPs are detailed written instructions to be
routinely followed so as to achieve uniformity in the performance of a specific function.
They are used in many large organizations, especially where the consequences of
procedural errors can be very serious, as for example in military operations, and
experienced personnel, such as those in the current study, are familiar with the relevant
SOPs and will apply them appropriately under case-study conditions.
3. The construction of REMOTEThe experimental constraints were the starting point for the requirements of the
REMOTE software. Objects spatially located within the CVE were to be used as a means
of invoking external applications such as email, web browser, and groupware
components. A degree of fidelity to the physical environment was required – the virtual
world should look something like a command post and include objects such as a bird
table (i.e. large table for the display of maps etc) that would be the central feature in a
real command post. It was decided that a central room with other rooms leading off
would be included, with each room containing a particular piece of functionality. There
were also requirements on the user’s representation, or avatar, that will be discussed
below.
3.1 Choice of development frameworkAn initial stage of the work was to evaluate available CVE platforms. It was considered
desirable to use commercial software since this has generally higher levels of support and
fewer bugs than research platforms and is consistent with the policy of the end-user to use
COTS (Commercial Off-The-Shelf software) whenever possible. For this reason as well
as issues of availability and learning curve, the two large research platforms DIVE
[Frécon 98] and MASSIVE-3 [Greenhalgh et. al 00] were not adopted.
Three commercial packages were evaluated. These were: Adobe Atmosphere [Adobe 02],
a multi-user web virtual world package, then available in beta-release; the game engine
distributed with the game Unreal Tournament [Epic-Games 99]; and the WildTangent
software environment [Wildtangent 02], originally a games engine but now marketed as a
builder of interactive web media environments and a rapid prototyping toolkit. All of
these were capable of supporting a client-server architecture, of handling a 3D graphical
world, and of importing 3D models from widely available modelling packages such as
3ds max [DISCREET]. Atmosphere was however felt to have a rather low frame-rate and
a very limited API for adding new behaviours, only offering JavaScript. It was not clear
that it would be feasible to call other applications from within it, such as GROOVE, as
required.
The standard API for the Unreal Tournament engine was limited to its proprietary Unreal
Script, and while the Gamebots [Kaminka et al. 02] extensions make it possible to run
external code on a socket interface this seemed cumbersome. However WildTangent has
a very open API supporting Visual C, Visual Basic or Java (though it requires the
Microsoft JVM, not updated past Java 1.1.4), and is easy to integrate into web browsers,
unlike Unreal. While Unreal uses OpenGL as its underlying library, and is thus multi-
platform, WildTangent sits on DirectX and is Windows-specific, but this was not seen as
an issue for a proof-of-concept system that was intended to run under Windows. In any
case, this close relationship with DirectX makes WildTangent fast, and as it handles all of
the graphics, there is little speed penalty in using Java as a framework around it with the
accompanying benefits of a modern web-oriented language. Thus WildTangent was
chosen as the development platform and Java was used to access its facilities.
3.2 Embodiment and identityIn considering what user representation to support within REMOTE, certain basic
considerations were important. User representations – avatars1 – are added to a CVE to
meet six generic requirements, as proposed byPandzic et al 97: (i) Perception allows a
user to register the presence of others, and (ii) localization to see where in the
environment they are. (iii)Identification is then the ability to know who they are. These
three capabilities are basic but can be met by fairly crude graphical representations.
A further three capabilities require more sophisticated representations. (iv)Visualization
of interest focus usually requires the representation of gaze direction, most obviously
through a user representation with eyes, whether realistic, as in a humanoid figure for
1 We use the term avatar here in its original VR sense [Stephenson 93] to mean any representation of the user in thegraphical world, not in the more recent sense of any humanoid virtual agent, user-driven or not
example or iconic, such as a personal emblem or object. This is also achievable using a
sensor beam as in virtual robots. (v) Visualization of actions can only occur if the avatar
has the ability to carry out actions, using some form of end-effector, in robot terminology.
This could be a robot gripper, or arms and legs, but again, could also be represented very
simply by extending a line to a manipulated object, or through a 3D cursor or virtual hand
grasping the object. (vi) Avatar Communication poses the heaviest requirement - in real
life, as already argued above, it is supported by gesture, posture, facial expression and
vocal characteristics. Creating an expressive avatar requires a relatively sophisticated
representation, which may then create a problem for the user who has to manipulate this
expressive repertoire while also carrying out whatever collaborative task is at issue.
Perception, localisation and identification of course include self-knowledge too, and this
depends on whether a first-person or third-person view is adopted. In the former case, the
user does not see their own avatar since their position is as if they were looking through
the avatar’s eyes; in the latter case, the user is positioned a little above and behind the
avatar. Atmosphere and Unreal Tournament both take a first-person view (with the
weapon being carried visible in Unreal) but a third-person view was selected in
REMOTE. In the real world we always have some awareness of what we look like and
where our body is in relation to objects around us, and this effect of the third-person view
was felt to outweigh the inconvenience of having the avatar body partially block the
user’s field of vision.
A humanoid form for an avatar is not always essential, but in terms of expressiveness it
has many advantages [Benford et al 97]. It seemed especially relevant to REMOTE’s
target users given their lack of familiarity with graphical environments. It also provided
an intuitive method for displaying the service, rank and role of a particular user,
significant identifiers in military organizations and especially in control rooms where
commanders of all services may collaborate along with civilian or government
specialists.
A natural way of identifying individuals would be to texture their face onto the face of
their avatar, technically quite straightforward. The conditions of the study made this
impossible, as the participants were unknown until shortly before it took place; instead
the name, role and rank of the user were attached above the head of the avatar, using a
billboard so that it always faced towards a viewing user, the data being gathered at logon.
However this was removed from the user’s own avatar (they know who they are) as it
blocked too much of the field of view. Uniform was used to indicate service membership
– camouflage for army, white shirt and dark blue trousers for navy, and light blue shirt
and blue trousers for air force. Both male and female avatars were provided, the gender
user-selected at logon, and figures of somewhat heroic physical proportions were
produced (see Figure 1). Had timescales allowed it, different character designs would
have been developed and evaluated with end-users: as it was, only feedback from the
military liaison officer was available and his response to this design was positive.
Figure 1 Avatars in REMOTE
As already suggested, there is a trade-off between the communicative advantage of giving
avatars more expressive behaviour and the disadvantage of producing a cognitive
overload for users in trying to invoke such behaviour appropriately. A small set of
animations was therefore provided as seen in Figure 2. It was seen as particularly
important to indicate when the avatar was inactive because the user driving it was absent
– the ‘chill-out’ animation.
The rotate and walk animations were invoked when the user navigated their avatar within
the environment. However it was also considered important to a feeling of physical
presence and realism that avatars could not walk through walls or furniture, so that
obstacle avoidance was required. The method of separating axes was used to determine
whether or not two bounding boxes intersected. Using the data for the centre, axes and
extents of the avatar’s box and for the bounding box of the object to be tested, classical
algorithms to test if two boxes have an intersection as described in Eberly 01 can be used.
S t i l l
C h i l l - o u t ( n o t P r e s e n t)
W a l k
R o t a t e
R u n
S a l u t e
l e a n
s t a r t
If they do collide then the velocity of the avatar is reduced to 0 and the position is
updated accordingly.
Latency is always an issue in a
CVE, and if the position of an
avatar is updated only on
coordinates from the server,
latency effects can produce a
very jerky animation. Dead-
reckoning can be used to
improve this situation by
making a local prediction in the client about where an avatar is likely to move, moving
towards that point and updating with the incoming coordinates from the server. Thus
dead reckoning consists of two parts prediction and convergence [Singhal and Zyda 99].
Prediction is the way the entity's current state is computed based on previously received
update packets while convergence defines how the object’s position and trajectory is
corrected as in the following equation:
Pred.pos after t secs = pos + velocity x t
The difference between the real position and the predicted position is next computed and
a new path is calculated, which is used in the convergence algorithm. Finally a point in
the future is selected on the new predicted path, and the navigation is updated
accordingly. Several techniques have been proposed [Singhal and Zydah 1999], although
a linear convergence (using velocity but not acceleration) was selected for REMOTE,
Figure 2 State Diagram of animations.
because latency was not thought to be a big problem in the setup of this study given that
it ran on a LAN in two neighbouring rooms.
3.3 Interacting with other applications
Object Room Action
Board Main room Launch Newsgroup
Table with map Bird table room Groove sketchpad tool
Cabinets Files room Groove files tool
Board Pictures Room Groove Pictures tool
Presentation dais with stairs Meeting room Nothing
Flying doc Documents room Groove notepad tool
Other Avatars All Netmeeting call to Avatar's IP (was
not used in actual test)
Table 1 Objects and locations in REMOTEAs discussed above, the ability of the CVE to interact with other applications was a
fundamental requirement. Applications were associated with particular objects in the
environment, mostly located in specific rooms, each labelled with the function. Table 1
Figure 3:REMOTE CVEand launchedapplications.
shows the objects, associated applications, and the room in which they were located.
A bounding box was implemented around the launch objects and the associated
application for a user launched when their avatar intersected the bounding box. To make
this visible to the user, the bounding box was rendered as partially opaque on intersection
– as in the top left component of Figure 3, in which the Board object in the Pictures room
has been intersected, bringing up the GROOVE Picture seen bottom right in Figure 3.
When the avatar leaves the bounding box the application is closed, and a delay factor was
incorporated to prevent rapid zigzags between opening and closing. Interactions with
another user were initiated by approaching that user’s avatar, and such interactions were
limited to one-to-one. The main purpose here was to invoke Netalk to allow voice
interaction between two users. The bottom-left component of Figure 3 contains the
logging file listing specified events in the VLE, though the text is not readable in this
diagram, and the top-right component contains the login facility.
3.4 Overall architectureFigure 4 shows the client-server architecture of the overall system. Client-side servers
handle the invocation of GROOVE and other applications, a login application records
significant actions of the avatar and the login functionality handles the identification of
users and their attachment to an avatar.
The architecture is divided into two logical components: the client and the server side.
The server side handles the data received from the clients, that is positions and actions of
the avatar and it replicates the data to the clients so that the all clients have the data
regarding the other users' avatars. To handle communication between the server and the
clients, the Observer-Subject Pattern was used, updating information when a message is
received by the listener (as seen in figure 4). This runs in a separate thread.
On the client side, (Figure 4) several subcomponents were developed to handle the
required tasks. Object Orientation and Design Patterns [Gamma et al, 1996] are heavily
used in the architecture offering support for modularity and the use of existing abstract
solutions to common problems like the use of controllers (the singleton pattern was used
to have just one instance of a class of controllers in the code). These controllers are used
to handle the animation of the avatars, the network communications and the data relating
to all the objects displayed in the virtual environment.
Also important was the use of the command pattern to link an object in the virtual
environment to an action, for example to open a GROOVE subcomponent. GROOVE is
itself a set of subcomponents, and as Table 1 shows, four of these – sketchpad, files,
notepad and picture – were invoked by an avatar approach to the relevant object. This
was handled by the GROOVE Server: the object has an associated message, which is sent
to the server, creating a link between object and GROOVE subcomponent. A similar
mechanism was used to open other Windows applications using the application server,
Figure 4 Overall architecture diagram.
and an editable XML file was used to specify the repertoire of connections making the
system easily extensible.
Another external program is a Logger Server, used to store copies of messages so that a
history of the interactions between avatars and objects can be maintained. This was also
configurable via XML. The login is used to enter the profile of the user and allows a user
to choose their sex (male, female), their service, their role and their rank. The selected
profile is used to generate an avatar in the appropriate uniform and the label above the
avatar's head.
4. Study and resultsAs already mentioned, the study involving the use of REMOTE ran in March 2003 and
involved two teams of four ex-military personnel, each forming a HQ. A team contained
a Chief of Staff (COS), an Intelligence coordinator (J2), Operations controller (J3) and
Logistics controller (J4). These are standard and well-understood HQ roles. Each team
was located in a separate room but shared the same LAN. No attempt was made to
simulate latency, bandwidth constraints or other WAN conditions.
4.1 Use of scenariosThe scenarios were all set within the context of a UN peace-keeping operation in the
fictitious country of New Albion, an island archipelago in the North Atlantic
approximately 600Km to the west of UK. New Albion comprises three ethnically divided
provinces: Wessex (Angle), Siluria (Celtic) and Devonia (Celtic). After 10 years of bitter
civil war the UN is invited to intervene to monitor a lasting ceasefire, however UN
intervention fails through lack of a clear mandate and other factors. A new ceasefire is
brokered and NATO agrees to intervene with New Albion consent and a revised UN
mandate. The US, UK and France each agree to send Divisional sized forces (NAFOR).
Despite the ‘ceasefire’ there is an operationally urgent need to get new forces on the
ground as quickly as possible. The UK contributes a framework Multi-National
Divisional HQ, some Divisional troops, a Mechanised Brigade and an embarked
Commando Brigade.
The background information remained the same throughout all three conditions,
minimising the additional read-in time each day and allowing more time for planning.
Each of the three experimental conditions described above in section 2 was introduced by
a new task set within the overall scenario. For condition 1, this was to organise the relief
of the commando brigade by the incoming mechanical brigade; for condition 2 to deploy
forces to secure a town in which ‘ethnic cleansing’ was threatening; for condition 3 to
evacuate threatened settlers from the town back to their ethnic home area and protect
those staying on. The output from each scenario was a plan briefing in which a remotely
situated representative Commander was presented with potential courses of action
(COA). Scenarios were such that they deliberately forced collaboration between HQs -
for example, sources of information were placed at different levels within each HQ. A
control cell ran the simulation, generating military and political events.
The maintenance of Situational Awareness (SA) is usually ranked among the most
important tasks for all command post team members [Taylor 1990] and experienced
personnel such as those used in the study perceive it as a key task. A widely (though not
universally) accepted definition is that of [Endsley 95] in which SA is the perception of
the elements in the environment within a volume of time and space, the comprehension of
their meaning, and the projection of their status in the near future: informally ‘what is
going on’. Military planning in command posts is naturally hierarchical (for example
strategic, tactical or operational), and in the scenario used, planning invariably
commenced with one HQ having better higher-level SA while the other would have better
lower-level SA. This was achieved by having one HQ ‘on the ground’ operating at the
tactical level and the other working at the operational level out of the area.
The study used a within groups (related, same-subject) design [Clarke et al 03] so that all
subjects performed all conditions: GROOVE alone, GROOVE + CUSeeMe and the
GROOVE + REMOTE. The constraints on the availability of participants, and the
resources available, made this the most feasible approach. There are known difficulties
with this design: for example, there can be a learning effect from one condition to the
next and a potential order effect where performance either improves as a result of practice
and familiarity or degrades because of fatigue or boredom. Changing the scenario and the
role of the team members between the conditions can counterbalance some of these
effects. However, this needs to be taken into account when analysing the results and is
discussed in section 4.4 below.
In the five-day programme, each 4-person team spent the first day separately carrying out
non-technology-based team-building exercises, to avoid an increasing team familiarity
and cohesion among the co-located participants impacting the study. During the second
day, familiarisation with GROOVE was carried out. Then one condition was run on each
day, with the last part of days 3 and 4 spent on training in CUSeeMe and REMOTE
respectively. The last part of day 5 was used to complete the questionnaires discussed
below and for a debrief.
4.2 Data collection and analysisData was collected by observations made during each experimental condition and through
questionnaires completed after each experimental condition, see Appendix 1. Two
observers were present in each team room and used an observational ‘crib sheet’ as a
guide – see Appendix 3. Observations covered: the human factors issues associated with
working in a distributed command environment; how the tools supported the VCL
concept though team and task work; the effects of the different tools on local and
distributed team interactions; the effect of the tools on the military task; difficulties in
using the tools. A set of categories was developed to allow observations to be grouped
and this can be seen in Appendix 4. The debrief was carried out using the prompt sheet in
Appendix 2.
Three questionnaires were used, containing a 7 point Likert rating scale and completed
electronically. The three questionnaires covered: aspects of distributed and collaborative
team working effectiveness; the degree to which the collaborative technologies meet
VCL (Virtual Co-Location) requirements; and aspects of the collaborative technologies
including general comments / strengths / weaknesses / missing features, and tool design
aspects and the effect of the collaborative technologies on co-located team working.
Subjects were invited to add any comment they wished after any question. The debrief
followed the questionnaires and was structured around five ‘open’ questions and
associated prompts as seen in Appendix 2.
4.3 Results summaryThe averaged responses to the VCL requirements questionnaire, which was composed of
41 statements each with a 7-point Disagree-Agree response scale, are shown as graphical
profiles in Figure 5. There was considerable variation in responses across individual
questions, of which all but two – questions 2 and 9 - were posed positively (i.e. higher
score = better). Overall, the GROOVE + CUSeeMe condition achieved higher
‘agreement’ ratings than did the GROOVE alone condition, which in turn was rated
higher than the GROOVE + REMOTE condition. Overall means across all conditions
and participants were 4.87 (GROOVE+CUSeeMe), 4.12 (GROOVE alone) and 3.38
(GROOVE+REMOTE). Separating these profiles by team showed that Team 1 rated
GROOVE alone and GROOVE+CUSeeMe similarly to team 2. However,
GROOVE+REMOTE achieved higher agreement ratings from Team 2 (overall mean
3.86) than Team 1 (overall mean 2.9).
This difference appeared to relate to the very different ways of working of the two teams:
observation logs showed that Team 2 spent far more time in own-team meetings to
discuss plan options, details of how they were going to go about the task, the information
that they had available to them, initial ideas, and the tasks each of the team members
would carry out. They would most often turn away from their workstations to hold such
meetings and as a result message alerts and calls from team 1 would be missed. Team 2
members also stood up to leave their workstations and temporarily gather around other
members’ workstations to discuss information far more than team 1 members. A
consequence of this was that Team 1 members were frequently unable to contact Team 2
members and expressed a great deal of frustration. REMOTE had no facility to signal
when such own-team meetings were taking place and thus contributed to rather than
relieving Team 1’s frustration.
REMOTE’s best scores related to Q11 (‘The software enabled me to rapidly exchange
key information with another individual on a one-to-one basis.”) and to Q21 (“The
software enabled all members of the team to undertake their roles and responsibilities”)
strongly suggesting that the overall critical response was not due to functional
incapability or fragile implementation, but rather to the basic design decisions made. Its
worst scores relate to Q18 (“The software effectively supported social interaction (e.g.
chatting or interacting separate to the work task) with another individual on a one-to-one
basis.”) and Q19 (“The software effectively supported social interaction with several
other team members at the same time.”), reflecting the constraint of one-to-one
communication only.
In terms of meeting distributed team members as easily as co-located members,
REMOTE was overall considered better at supporting this than GROOVE, but was
considered worse than CUSeeMe. This was due to an exceptionally high rating by Team
2, which judged REMOTE superior to both GROOVE and CUSeeMe in this regard.
Team 2 rated REMOTE on a par with CUSeeMe, however, in its ability to support a
sense of shared meeting space.
The ranking between the software of the three conditions shown in these subjective scales
is reflected in the participants’ written questionnaire comments, and observers’
observations. The two other questionnaires alternated positive and negative questions, so
that the result profiles are less easy to visualise than the one shown, but they also gave a
compatible ranking of the study conditions.
We return here to the issue raised in 4.1 above: whether there was a learning effect from
one condition to the next and a potential order effect either improving performance as a
result of practise and familiarity or degrading it because of fatigue or boredom which
might itself explain the ranking of the three conditions. First we would restate that the
first day included a scenario based team task without any of the software under
examination in order to minimise the effect of improving social cohesion among subjects
who were new to each other. Secondly, GROOVE was in use for all three days of the
experimental condition as well as in training on the second day. A positive learning effect
for it as between condition 1 and condition 2 was noted in observation. However had this
effect been dominant, then one would not expect the second experimental condition to be
ranked higher than the third one, as it was. If a boredom factor for GROOVE was
involved, one would have expected the observation process to detect it, and it did not.
The area in which boredom and frustration may have impacted the study lay in the
difference between Team 1 and Team 2’s work styles, and the frustration suffered by
Team 1 as a result discussed above. This may have been a further factor in the more
critical assessment of the third experimental condition by team 1. However both teams
ranked the three experimental conditions the same, suggesting that the order effect, if
present, was not decisive. In general, the observation regime acted as a control on both
factors since it produced a detailed picture of action and interaction.
4.4 User -identified problemsThe overall questionnaire results above show that users were critical of REMOTE
compared to GROOVE and CUSeeMe. The underlying problems prompting these results
are brought out in user comments which were collected by the observers, made during the
debrief session, and included in the questionnaires. Table 2 lists the most significant
problems, illustrated with specific comments, which are naturally there taken out of the
context in which they occur. Nevertheless, the typicality of the users and of the scenarios
asserted in section 2, gives us some confidence that our identification of the generic
problems is correct. Discussion follows in the next section.
Positive comment related to creating a feeling of presence and good awareness of other
team members and what they are doing, for example: “The main strength is that it creates
a feeling of the team being present especially the distributed team.”; “The avatars provide
good awareness of other team members and what they are doing”; “ CVE provides a good
sense of virtual presence”.
FIGURE 5: VCL requirements results
0
1
2
3
4
5
6
7
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41
VCL requirements
Dis
ag
ree
- A
gre
e 1
-7
T2 Groove T2 CUseeMe T2 REMOTE
Problem Comment
Navigation slow and
laborious
J2: "Where do I find the notepad?", COS: "In the documents room", J3: "see you there in 10
minutes!"
"I just can't see the logic of it…it's just an arcade game!". Moving people around the
environment is laborious. I can't see the point of moving mannequins around a set of
corridors".
COS: "OK, Let's all go to the bird table", J3: "Could take a while".
Difficulty in finding
colleague’s avatars
"Where is he? He was in the files room, he was there a minute ago". (At the central table)
"OK, he's landed".
Only one app at a time
can be used
"Can I put a map in there/ I want to be able to drag and drop. Need to import the map, but
can't import the map via sketchpad. It's hopeless. If I can't load a map into sketchpad, can
only start from scratch."
"If I want to work on a file, I have to come out of notepad".
”Difficult to know whether individuals in different teams are looking at the same map. Quite
difficult to confirm, as you can't look at the map and chat at the same time”
“The REMOTE environment forces you to only take in one piece of information at a time.
Splits your attention.”
Avatar jams by app
launchers – see Fig 6
"Queuing up at filing cabinets with 4 big guys in the way".
Wrong information
access metaphor
"To look at a file, we need to leave the meeting room, bump into walls and go to the files
room…. I think that's ridiculous".
"You're constantly thinking where is everything… and how do I get there?"
"You shouldn't have to go into another room to get something"
"I just think the analogy is wrong with rooms etc. You put together your folder and carry it
around with you".
Only 1-1 voice
interaction
"This is even worse, we're just one-to-one chatting now". "I'm going to leave this now,
because there's no point"
Table 2. User-identified problems
5. Discussion and Further WorkThe results of the study show that the REMOTE CVE was not seen as the best way of
supporting collaborative team working by its users. However one must take into account
here the inherent difficulties in supporting distributed teams with technology. The
following comment is instructive, arising from a study into the use of non-CVE
collaborative tools to support distributed design teams in manufacturing engineering:
“Findings from this study suggest that collaborative tools must be clearly superior to
existing practices to merit the effort of deployment, adoption and subsequent use, since
the burden of learning and mastering a new tool in a demanding and fast-paced
engineering design environment may not outweigh the perceived benefits.” [Wierba et al
01 p1].
In a similar collaborative design environment, albeit with physically co-located teams, the
technology that was most clearly valued was a printing wallboard that saved participants
the time previously spent copying informal diagrams to take away [Olson and Olson 00].
Here, ‘just one thing’ was added to existing and familiar technologies. Work in the field
suggests that involvement of users in the design of collaborative environments – difficult
to set up in military domains – plays a very important role. For example, a successful
‘collaboratory’ of space physicists [Olson and Olson 00] has involved 10 major redesigns
over 7 years, all involving the users. In this collaboratory, scientists organise themselves
into virtual rooms of objects and remote partners to run experiments using real-time data
from geographically distributed instruments, suggesting that the basic metaphor of the
virtual room may play a positive role.
Figure 6 Avatars queuing to access the File resource.
The key question is whether a CVE is inherently incapable of delivering clearly superior
results both to existing practice and other technological solutions such as the two first
experimental conditions, or whether a different set of design decisions, together with
some maturing of underlying technologies, might produce a more successful CVE. This
question is related to an ongoing ‘space-versus-place’ debate within the CSCW
community as summarised in [Benford et al 2001].
The arguments for the ‘space’ metaphor already outlined suggest that it is independent
movement within a shared coordinate system, combined with the representation of others'
positions through avatars that allows a CVE to support social interaction. Those
supporting the ‘place’ argument suggest that other important aspects of an environment
beyond the provision of a shared coordinate system are responsible for supporting social
interaction. We are more convinced by the potential benefits of a shared space, and here
put forward lessons learned from the study on the design of such a space for distributed
military command teams.
REMOTE applied a relatively naturalistic design to its virtual world, while interfacing it
to applications such as groupware with a desktop metaphor. These two metaphors did not
appear to marry together well for the users, and the conflict between them was sharpened
by locating each application launcher in a different virtual room. Unlike the collaboratory
cited above, in which an experiment is a specific activity and thus fits conceptually into a
separate room, military planning requires a strong integration of different information
sources from different locations.
Thus the bird table plays a very central role in real command posts and this was not
successfully replicated in REMOTE. One user commented: “The key to digitisation is a
version of the electronic bird table. Showing the same picture to all team members, and
can plan concurrently”. REMOTE applied a naturalistic metaphor for physical space, this
was not true of its information metaphor: “Compared to the analogue system, you have a
map and notepad which you carry with you anywhere. So you shouldn't have to go to
other rooms - not realistic". Allowing avatars to carry information sources would also
make it natural to have many open simultaneously and support transfer between them –
the restriction of having one tool only open at a time was felt acutely, especially .for
maps, often used with other documents, and a natural focus for voice interaction. It would
also have avoided queues around central resources such as in the Files Room (Figure 6).
The consequence of the decision to distribute information sources was a high requirement
for navigation. The effort this produced for users underlines a basic difference between
real and virtual spaces. In the real world, most spatial navigation in office environments
is carried out at a sub-cognitive level – we walk and think, or talk, at the same time. This
is not true of navigating an avatar in a desktop CVE and it is hard to conceive of a
navigational mechanism that would be as low in effort. But this does not mean that a 3D
space is in itself a bad thing, merely that one should not try to ‘walk’ avatars about a large
space with separate rooms as in REMOTE.
A further consequence of the design was that users found it hard to locate the avatars of
users with whom they wanted to interact. A ‘plan view’ was suggested as a result, and
this together with some colour-coding would support easier location. A single-room
design would fundamentally reduce the problem, but might then meet over-crowding
problems in a command post with many more participants. Centering interaction on
avatars rather than information sources might produce a single-room design with ‘team
areas’ or stations for particular roles to interact – an intelligence or a logistics desk for
example.
The key issue for these users is visualising the state of the task; one user commented “the
environment doesn't change, it's the task that changes”. This is not entirely true of course
– the changing element in the environment is however the avatars themselves, so that
their identity and activity is what the environment really ought to communicate. However
supporting the visualisation of maps, the key information source for military planning,
would be a better use of 3D for these users: “Really good (virtual) 3D mapping would be
excellent. Especially if you are able to work with and annotate the maps.”
The identification mechanism adopted of labels over avatars’ heads was not completely
successful. Such legends may be hard to read – as can be seen in Figures 1, 3 and 6 - and
at a sensible size take up too much space. Texturing avatar faces with those of the users
themselves ought to be tried –not possible in this study because the identity of the
individuals participating was known only a short time before the study took place.
However individualised faces along with further clothing cues might have been
attempted. As with navigation, it is hard to think of a currently viable mechanism to allow
a user to invoke complex expressive features in an avatar without imposing a high degree
of effort. The positive response of the users to web cam data suggests that compositing a
video head-and shoulders onto the avatar’s face is worth considering in spite of the
technical difficulties, at least when voice interaction between users takes place.
User responses underlined the importance of voice interaction in collaboration, and the
importance of multi-voice conferencing. In spite of the use of a LAN there were still
problems with audio quality in the experiment, and this has been confirmed by a number
of earlier studies [Tang et al 94; Olson and Teasley 96; Finholt and Olson 97; Covi et al
98]. The voice facilities of GROOVE were criticized, and though NetMeeting was seen
as better in quality, participants would have preferred to use the phone. Voice-over-IP is
still less reliable and of a much lower quality than telephony. Attempts to integrate
NetMeeting into REMOTE along with the other applications were not successful, so that
it had in fact to be run in parallel outside the environment instead of being triggered when
two avatars met. Even then, the restriction to one-to-one conversation was far too
constricting.
6. ConclusionsRunning studies such as this one and learning from the results seems fundamental to the
development of CVEs into a successful and useful technology. Collaborative activity is a
particularly difficult area to support with technology, and this is all the more so for
tightly-coupled tasks with a high degree of inter-dependence such as in military planning
[Olson and Olson 00]. A deep understanding of the user requirements and how these can
be supported with the technology as it now is cannot be achieved without such
experimental work. In this case, a desktop VE was poorly received, but as we have tried
to indicate in this final section, this seems more a case of iterating the design concepts
than abandoning the approach. There is still a need to assess more explicitly the impact of
these technologies on task performance and conduct to determine whether they provide
the ability to enable the task to be done more quickly and more efficiently.
ReferencesAdobe (2002). Adobe Atmosphere, http://www.adobe.com/products/atmosphere/main.html.
Argyle, M. (1988) Bodily Communication , New York: Methuen and Co., 1988.
Benford, S. D., Bowers, J. M., Fahlen, L. E., Greenhalgh, C. M., and Snowdon, D. N. (1997) Embodiments,
Avatars, Clones and Agents for Multi-user, Multi-sensory Virtual Worlds, Multimedia Systems, 5, 2, 1997,
93-104.
Benford, S., Greenhalgh, C., Rodden, T. and Pycock, J. (2001) Collaborative Virtual Environments,
Communications of the ACM, Volume 44 , Issue 7 pp79 – 85, July 2001
Bradner, E. and Mark, G. (2002). Why Distance Matters: Effects on Cooperation, Persuasion and
Deception. In proceedings of Computer Supported Cooperative Work (CSCW 2002). November 16-20,
New Orleans, Louisiana
Cassell, J., Nakano, Y., Bickmore, T., Sidner, C., and Rich, C. (2001). “Non-Verbal Cues for Discourse
Structure.” Proceedings of the 41st Annual Meeting of the Association of Computational Linguistics, pp.
106-115. July 17-19, Toulouse, France
Churchill, E and Snowdon, D (1998) Collaborative Virtual Environments: an introductory review of issues
and systems, in Virtual Reality: Research, Development and Application, 1998
Clarke, H; Goillau, P and Stockdale, R. (2003) Distributed Teamworking Experiment, QinetiQ May 2003,
Malvern UK
Covi, Lisa M., Olson, Judith S., Rocco, Elena, Miller, William J., and Allie, Paul. (1998) A Room of Your
Own: What do we learn about support of teamwork from assessing teams in dedicated project rooms. In
Cooperative Buildings: Integrating Information, Organization and Architecture: First International
Workshop, Co'Build '98. Darmstadt, Germany.
CUSeeMe-Networks (2001). CUSeeMe, http://ic2.cuseeme.com/.
DISCREET : http://www.discreet.com/support/max/
Eberly, D. H. (2001). 3D Game Engine Design, San Francisco: Morgan Kauffman Publishers.
Endsley. M.R. (1995) Toward a theory of situation awareness in dynamic systems. Human Factors ,37 (1),
32-64.
Epic-Games (1999). Unreal Tournament, Infogrames.
Frécon, E. and Stenius, M. (1998). DIVE: A scaleable network architecture for distributed virtual
environments, Distributed Systems Engineering Journal (DSEJ), 5, pp 91-100, Special Issue on Distributed
Virtual Environments.
Finholt, T. A., and Olson, G. M. (1997) From laboratories to collaboratories: A new organizational form
for scientific collaboration. Psychological Science. 8(1), 28-35
Gamma, E; Helm, R., Johnson, R and Vlissides, J. (1996) Design Patterns, Addison-Wesley Longman,
Reading Massachusetts.
Greenhalgh, C., Pubrick, J., and Snowdon, D. (2000). Inside MASSIVE-3: Flexible support for data
consistency and world structuring. In Proceedings of the 3rd International Conference on Collaborative
Virtual Environments, (CVE ‘2000), San Francisco, California, USA, ACM Press.
Groove-Networks (2002). Groove Workspace, http://www.groove.net.
Hughes,J; King,V; Rodden, T and Anderson,H. (1995) The Role of Ethnography in Interactive Systems
Design, ACM interactions , v.2 n.2, p.56-65, April 1995
Kaminka, G. A.; Veloso, M.; Schaffer, S.; Sollitto, C.; Adobbati, R.; Marshal, Andrew N.; Scholer,
Andrew, S.; and Tejada, S. (2002). GameBots: the ever-challenging multi-agent research test-bed, In
Communications of the ACM, January issue.
Myers, M.D. and Avison, D.E. (eds.). (2002) Qualitative Research in Information Systems: A Reader, Sage
Publications, London, 2002.
Olson, J. S., and Teasley, S. (1996) Groupware in the wild: Lessons learned from a year of virtual
collocation. Proceedings of the Conference on Computer Supported Cooperative Work. New York: ACM.
pp-419-427
Olson, G.M., and Olson, J.S. (2000). Distance matters. Human Computer Interaction, 15, 139-179.
Pandzic, I; Lee, E; Magnenat Thalmann, N; Capin, T and Thalmann, D. (1997) A flexible architecture for
Virtual Humans in Networked Collaborative Virtual Environments Computer Graphics Forum Vol 16,
Issue 3 Conference Issue (September 1997)
Sechrest, L., Stewart, M., Stickle, T., and Sidani, S. (1996). Effective and Persuasive Case Studies.
Cambridge, MA: Human Services Research Institute.
Shneiderman, B. (1992). Designing the user interface: Strategies for effective human-computer interaction,
2nd edn, Reading, MA: Addison-Wesley
Singhal, S., and Zyda, M. (1999) Networked Virtual Environments, New York:ACM Press, Siggraph
Series.
Steed, A and Tromp, J. (1997) "Experiences with the Evaluation of CVE Applications", in Proceedings of
Collaborative Virtual Environments 1998, 17-19th June 1998, Manchester, UK
Stephenson, N.(1993) Snowcrash. Spectra Books , 1993.
Tang, J. C., Isaacs, E. A., and Rua, M. (1994) Supporting distributed groups with a Montage of lightweight
interactions. Proceeding of the ACM Conference on Computer Supported Cooperative Work (CSCW94),
Pp. 23-34
Taylor, R. M. (1990) Situational awareness rating technique (SART): The development of a tool for
aircrew systems design. In Proceedings of the Situational Awareness in Aerospace Operations, AGARD
CP-478, Paper 3.
Wierba, E.E., Finholt, T.A., and Steves, M. (2001). "What have you done for me lately?" A case study of
barriers to collaborative tool adoption in a manufacturing engineering setting (PDF file). At:
http://www.crew.umich.edu/technical_reports.htm
Wildtangent (2002). Wildtangent Web Driver, http://www.wildtangent.com
Appendix 1 – The Questionnaires
A1.1 the team-working questionnaireQUESTIONNUMBER Questions
1 In the co-located team, it was possible for team members to have a similarunderstanding of the situation.
2 In the distributed team, it was possible for team members to have a similarunderstanding of the situation.
3 When the team was co-located, technology hindered the team members’understanding of the situation.
4 When the team was physically located together, it was possible to have anunderstanding of the overarching team goal.
5 In the distributed team, it was difficult to maintain an understanding of wherethe team fitted into the bigger picture.
6 It was easier to have an awareness of other team member’s roles andresponsibilities in the co-located, rather than the distributed, team
7 In the co-located team, it was easy to monitor and assess the progress of fellowteam members.
8 In the co-located team, it was easy to ask for guidance from the Chief of Staff.9 In the co-located team, it was difficult to notice and prevent team problems.10 It was difficult to notice and prevent team problems, when working in a
distributed team.11 It was more difficult to resolve conflicts when team members were distributed.12 It was difficult to know when to provide constructive feedback to team
members who were not in the same room.13 It was easy to identify when co-located team members required assistance or
support.14 It was easy to ask for support when the team was co-located.15 Feedback was more discrete and less formal when given face to face.16 In the distributed team, it was difficult to know when other team members were
having task-focused problems.17 It was easier to develop trust between members in the same room than team
members that were distributed.18 When separated from the rest of the team, it was possible to experience feelings
of isolation.
of isolation.19 It was easy to develop and maintain interpersonal relationships between co-
located members of the team.20 The technology hindered the development and maintenance of interpersonal
relationships.21 Unfamiliarity within the distributed team, prevented team members from
asking for support.22 It was easy to build and maintain a good team atmosphere in the co-located
team.23 In the co-located team there was a high level of satisfaction.24 The technology hindered the overall level of satisfaction in the co-located team.25 In the distributed team there was a high level of satisfaction.26 The technology hindered the overall level of satisfaction in the distributed
team.27 In the co-located team there was a high level of motivation.28 The technology hindered the overall level of motivation in the co-located team.29 In the distributed team there was a high level of motivation.30 The technology hindered the overall level of motivation in the distributed team.31 In the co-located team, there was high level of cohesion.32 The technology hindered the development and maintenance of cohesion in the
co-located team.33 In the distributed team, there was a high level of cohesion.
34 The technology hindered the development and maintenance of cohesion in thedistributed team.
35 It was difficult to interact and communicate with co-located team members.36 When team members were not co-located, additional co-ordination and
communication demands arose.37 It was difficult to interact simultaneously with co-located and distributed team
members.38 It was difficult to conduct brainstorming sessions with co-located team
members whilst maintaining awareness of the activities of distributed teammembers.
39 It was difficult to maintain eye contact with co-located team members.40 In the co-located team, it was difficult to manage team members’ tasks and co-
ordinate a unified effort.41 When a message was not communicated verbally, face-to-face, it was difficult
to understand whether a team member had understood the message.42 Requesting information was easier when face-to-face.43 In the distributed team, it was difficult to know how and when to update others
with information.44 Lack of face to face communication increased the number of
misunderstandings that occurred.45 Communicating over electronic media was not as effective as face to face
communication.46 The physical layout of the work environment inhibited non-verbal
communication between co-located team members.
A1.2 The Requirements Questionnaire
QUESTIONNUMBER Questions
1 The software did not constrain team working between members of the co-located team.
2 Co-located team members felt forced to stay at their workstations to undertake theirtasks.
3 The software enabled me to communicate with other team members whenever I neededto.
4 It was useful having a choice of ways to contact other team members (e.g. by usingtext or voice).
5 The software made it easy to determine whether team members were currentlyavailable for communication.
6 The software enabled me to communicate effectively with others whilst maintainingawareness of current information.
7 The software made it easy to ensure that all team members had a similar understandingof the situation.
8 The software enabled communications to be brief and concise.9 A high number of misunderstandings and communication breakdowns occurred when
using the software.10 The application enabled me to ‘Reachback’ (i.e. to locate staff functions outside of the
traditional area of operations) to the distributed team location.11 The software enabled me to rapidly exchange key information with another individual
on a one-to-one basis.12 The software enabled me to effectively exchange information with several other team
members at the same time.13 The software enabled information to be produced and circulated, using common
formats with visual or graphic representations where appropriate.14 The software made it easy to check that information was received and understood as
intended.15 The software enabled all team members to interact with one another as required.16 The means of electronically meeting other team members and interacting with them
was appropriate, and natural17 Ad-hoc, spontaneous interactions with other team members could take place
electronically.18 The software effectively supported social interaction (e.g. chatting or interacting
separate to the work task) with another individual on a one-to-one basis.19 The software effectively supported social interaction with several other team members
at the same time.20 The software was better at helping team members to exchange information and facts
relevant to the task than in helping them get to know each other as colleagues.21 The software enabled all members of the team to undertake their roles and
responsibilities.22 The software helped the team to synchronise their activities.23 The software made it easy to keep track of each other’s activities.24 It was easy to switch between individual tasks and team activities using the software.25 The software made it easy to manage the team’s workload effectively.26 It was easy to maintain awareness of all distributed team members through the use of
the software.27 The software made it easy to monitor distributed team members actions.
28 The software made it easy to know when to provide feedback to team members whowere not in the same room.
29 The software made it easy to identify when distributed members of the team requiredassistance and support.
30 The software made it easy to identify and deal with problems within the distributedteam.
31 The application allowed me to ‘Virtually Co-locate’ with team members in the otherroom i.e. to interact with distributed team members as if physically co-located withthem.
32 I could as easily electronically ‘meet’ the team members located in the other room as Icould meet my co-located colleagues in the same room.
33 Overall, I experienced a sense of ‘shared meeting space’ with other team memberswhen using the software.
34 Overall, I felt that the software supported me in feeling present in the distributed team.35 The software enabled me to feel a member of the distributed team.36 The software enabled me to feel familiar with my distributed team members.
37 The software aided the creation of a continuing sense of common team purpose.38 The software made it easy to build and maintain a good team atmosphere.39 The software enabled the distributed team to create and maintain an atmosphere of
trust.40 The software is a useful ‘proof of concept’.41 The software should be developed further for military use.
A1.3 The Technology Questionnaire
QUESTIONNUMBER Questions
1 The training I received on Groove was adequate to conduct the experiment.(Condition 1 only)
2 Overall, I found the software easy to use.3 I liked the software and enjoyed using it.4 The software helped me to do teamworking in an efficient and timely manner.
5 It was straightforward to learn and become familiar with the software.6 I had to frequently ask the advice of technical staff on how to use the system.7 The system behaved in a consistent way enabling me to understand how it worked.8 The system enabled easy access to common documents.9 The system enabled remote briefings to be conducted.10 System functions were well integrated.11 Few actions were required to perform functions.12 The interface enabled me to efficiently access information.13 Contacting individual users was quick and simple.14 I used the voice chat facility frequently15 The voice chat facility was easy to use.16 I felt that the voice chat facility was effective.17 I used the text chat facility frequently.18 The text chat facility was easy to use.19 I felt that the text chat facility was effective.20 I used the discussion tool frequently.21 The discussion tool was easy to use.22 I felt that the discussion tool was effective.23 I used the sketchpad tool frequently.24 The sketchpad tool was easy to use.25 I felt that the sketchpad tool was effective.26 I used the instant messaging tool frequently.
27 The instant messaging tool was easy to use.28 I felt that the instant messaging tool was effective.29 I used the electronic filing system frequently.30 The electronic filing system was easy to use.31 I felt that the electronic filing system was effective.32 I used the contacts tool frequently.33 The contacts tool was easy to use.34 I felt that the contacts tool was effective.35 I used the notepad tool frequently.36 The notepad tool was easy to use.37 I felt that the notepad tool was effective.38 The training I received on CUSeeMe was adequate to conduct the experiment.
(Condition 2 only).39 I used the video conferencing facility frequently (Condition 2 only).40 The video conferencing facility was easy to use (Condition 2 only).41 I felt that the video conferencing facility was effective Condition 2 only).42 The training I received on REMOTE was adequate to conduct the experiment.
(Condition 3 only).43 The virtual environment was natural and intuitive. (Condition 3 only).44 The avatars were effective for interacting with distributed members of the team.
Condition 3 only).45 The avatars were effective for interacting with objects in the virtual environment (e.g.
filing cabinets and post box). (Condition 3 only).
Appendix 2: The debrief prompt sheet
EXP3 - VCL debrief sheet (PJG 21_2_03)
Debrief sessions will occur after electronically completing the questionnaires for eachexperimental condition. This sheet is intended as an ‘aide memoire’ to structure the main themesto be covered in the debriefs.
Five ‘open’ style interview questions are given. Questions with brief ‘Yes / No’ answers are to beavoided. These are not intended to be prescriptive. Priority should be given to asking about theissues and problems that will have surfaced during the experimental condition.
Finally, a number of non-directive questioning and ‘prompt’ techniques have been included assuggested ways to keep the debrief discussions flowing.1. Debrief Topics for explorationALL CONDITIONS (1, 2 and 3)Q1. Were there any particular issues, problems or successes that emergedduring the last condition? (e.g. concerning the scenario, SOP’s, EXCON,software / technologies, teamworking issues – local co-located team vs.distributed team?)(Can you say more? What did you particularly like / dislike / feel was missingduring the condition?)
Q2. Did the software help you in carrying out your task activities?(In what way ? / Can you say more? / Can you think of specific examples?)
Q3. Did the software help you in interacting with and getting to know other teammembers?(In what way? / Can you say more? / Can you think of specific examples?)
Suggestion - probe specific software applications and features:• Groove (voice chat, text chat, document tool, sketch pad tool, instant
messaging facility, electronic filing system, contacts tool, notepad tool)CONDITION 2 ONLY (CU-SEE ME VIDEO CONFERENCING)Q4a. Did the CU-SeeMe video conferencing enable you to accomplish objectivesthat you couldn’t achieve using Groove alone?(What were they? / Can you think of specific examples?)
Suggestion - probe specific software applications and features:• CUseeMe desktop video conferencing (1:1, 1: many)CONDITION 3 ONLY (REMOTE VIRTUAL ENVIRONMENT)Q4b. Did the REMOTE virtual environment enable you to accomplish objectivesthat you couldn’t achieve using Groove alone?(What were they? / Can you think of specific examples?)
Q5. How did REMOTE with its virtual environment and avatars compare with theprevious CU-SeeMe video conferencing condition?(What were the relative strengths and weaknesses / advantages anddisadvantages?)
Suggestion -probe specific software applications and features:• REMOTE (overall virtual environment, avatars – presence / location / state,
avatar-avatar trigger of chat, avatar-object trigger of Groove tools)ALL CONDITIONS (1, 2 and 3)Q6. Is there anything else we forgot to ask, that you’d like to mention?(Catch all – last chance to elicit any remaining issues).2. Example ‘open’ style guidance questions for ‘probing’ topics1. Concerning X, what were your thoughts / overall impressions?2. What issues became evident with X?3. What did you think of the specific feature / aspect of X?4. What were the strengths / what worked well / what did you like?5. What were the weaknesses / what didn’t work well / what did you dislike?6. What was missing / what additions were needed / what would you like to see
added in the future?7. Other specifics (what / when / who / where / how? )3. Prompts – to keep the discussion goingI see. That is interesting. Can you say more? Did that always occur? Can yougive me an example, a ‘for instance’, when that occurred? In what way was that a
problem for you? Broadening to everyone in the team, did you all think that wasan issue for the team?Etc etc.
Appendix 3 – The Human factors ‘crib sheet’ using during observation
Appendix 4: Observer data analysis coding table
Abbrev Definitionusr_str User identified strengthsusr_weak User identified weaknessesusr_ftr User identified areas for future developmentusr_cmnt User commentsapp/funct Functionality of the underlying applicationsco-ord Team Co-ordination and Interaction.Tm_clim Team Climatead_be Adaptive Behavioursmon/fed Performance Monitoring and Feedbackunder Understandingcomm Communicationproc/mow Planning process issues or methods of workinglead Leadershippos Productivity of Staffinfo_shr Information Sharing / Managementhci Human Computer Interface Design Issuespres Sense of Shared Presence and Awareness of Others.train Training Issuesergo/phys Ergonomics or Physical Issuestech Other Techical Issuesease_use Ease of Usescenario Scenario Issuescrash System Crashesz_question Observers’ Questions.