Evaluating the usability of an evolving collaborative product - changes in user type, tasks and evaluation

methods over time.

Chris Nodder, Gayna Williams, Deborah Dubrow Microsoft Corporation

One Microsoft Way Redmond, WA 98052

cnodder/gaynaw/debdu@microsoft.com

ABSTRACT The first users of a new technology are often engineers and enthusiasts. The functionality and interface that they find acceptable may be very different than the requirements of a more mainstream audience. This poses challenges for usability engineers in both defining user groups and then evaluating a product against usability goals, when both users and goals are changing as the technology matures. Usability evaluation methods for collaborative applications must evolve and iterate at least as fast as the products themselves. This paper describes the changes in approach taken by usability engineers between Version 1 and Version 3 of the Microsoft NetMeeting product.

Keywords Usability Evaluation, Development Process, NetMeeting, Application Sharing, Collaborative Tools, Design considerations

INTRODUCTION There’s an old joke that says, “By version three, the product should be worth using,” especially when applied to commercial software. The goal of usability involvement in the product cycle at Microsoft is to make the product compelling for its target users, and for carrying out their target tasks, from the first release onwards.

The benefits of usability involvement in the design and development cycle are well documented [S, 81. However, the textbook approach to usability does not always work. In practice, user types, user tasks and usability evaluation methods will change simultaneously as the product matures from novel technology to desktop tool.

Permission to m&e digital or hard copies ol’all or part oftbis pork for

personal or classroom use is granted without (‘CL’ provitlcd that copies arc not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. ‘1’0 copy otherwise, to republish, to post on servers or to redistribute to lists. requits prior specific permission anii!or a kc.

GROUP 99 Phoenix Arizona USA Copyright ACM 1999 l-581 13-065-l/99/1 1...$5.00

In this paper, rather than trying to mould our experiences to the textbook descriptions, we document what actually happens when real world constraints are introduced. In reality, product usability has to adapt to product teams’ needs, and usability engineers must build a relationship with the team they support before they can expect the co- operation required to introduce user-centered concepts.

Evaluations of communication media have occurred since the early 70’s such as those by Chapanis et al [1] who compared the benefits of high quality video with audio. Much rigorous research has continued in this direction by determining which medium provides the best communication for given situations, for example several studies [9,11] have found marginal differences in the benefit of video over audio communication in laboratory settings. Many evaluations looking at social issues were conducted using customized conferencing systems designed specifically for research projects (e.g; [2,6,12]). Recently in the literature there has been more research based on observing users of communication technology in settings outside the research world as the use of online communication software has become more pervasive [lo].

Until recently using real time collaboration software required all users in the group to be using the same software and be connected to a specific type of network. This type of investment and installation required technical skills and a strong belief that the investment would be worth the communication benefit. In the last few years we’ve seen the speed of connection to the Internet increase, the introduction of several standards-based Internet phone and video products and also recently video cameras as part of computer package deals. The cost of experimenting with real-time communication products has been greatly reduced in terms of both real dollar amounts and support costs. With these cost and technology changes, there has been an increase in the use of real-time communication software outside the research and academic community.

As real time communication products become more common, we can start to see an increase in informal and unplanned communication. The success of America

150

Online’s Buddy List feature is an example of this. Community and collaboration software for the Internet is an important reality and good software must support each user group with an easy to use interface to complete their desired tasks.

Microsoft NetMeeting is a real time conferencing product that has released three versions in 3 years. This is not a long time in comparison to the amount of time conferencing products have been studied in the research community. However, it has been a highly active time in terms of Internet technological advancements and adjustments in the audience for conferencing products. This paper addresses how usability contributed to the development of the NetMeeting conferencing product over this period and how the goals of the product adjusted during its development due to changes in the potential audience and changes in the Internet environment. In addition it highlights some challenges we faced in working with the first synchronous communication product to be usability tested at Microsoft.

Gentner and Grudin [3] describe a human interface design model which shows that early adopters are frequently technically advanced users, and often a good task based UI only develops as the product seeks an increased share in the market place. We describe how both this factor and the significant changes in the infrastructure and technology to support the product encountered during the development of NetMeeting contributed to the design process. There are always costs associated with being an early adopter of technology, especially with collaborative products, as critical mass and network infrastructure have to be in place before a real working benefit is realized. We describe how we dealt with the issue of gradually changing focus from early adopters to main stream office and then home users during the process and what that meant to usability goals and techniques. We offer this case study to creators of real time commercial software to supplement the usual recommendations of how to involve usability into product development [8] by providing insights into what actually happened during the development of a product being delivered on Internet time.

Development Teams and Usability at Microsoft There were about 12 people on the NetMeeting team when product development started in August 1995. The team was located in the Personal Systems Division, with products such as Windows and Internet Explorer. One Usability Engineer within the PSD Usability Group was dedicated to working with the NetMeeting team throughout the development cycle of the product. The NetMeeting team now contains about 40 developers, program managers (PMs), testers and support staff. NetMeeting now commands about 80% of one engineer’s time.

Typically usability engineers work with the user interface program managers on each product, as these are the people who specify the user interaction and interface design. The results of usability activities are communicated throughout

the team. Testing takes many forms, from exploratory site visits to workplaces and homes, where the focus is to learn how users work with the product in their daily lives through to controlled laboratory evaluations targeting specific functionality questions.

Lab studies can use paper prototypes, system mock-ups or the latest working version of the product code. At Microsoft, the working development code is compiled every night, so that the most recent changes are available the next morning for testing. These daily “builds!’ [4] are used for testing, and usability feedback can quickly be incorporated into the builds in time for the next test sessions.

NetMeeting product today Today NetMeeting is at version three. It is a free product for the Windows environment that facilitates application sharing between many people (multi-point), provides chat and a whiteboard, and also provides audio and video communication between any two people in the conference (point to point). These components have been added incrementally as technology, competition and user needs have dictated. There are now over 200 publicly available User/Internet Location Servers (ULSfiLS) serving as directories for callers, as well as many more private ILS within corporations.

Table 1 shows the time line of the product’s evolution and the usability activities associated with the schedule. ,,, PF@a~~~&~?~

W95 Product inception 1 Audio, application sharing, 12 studies, including: i chat, whiteboard, . Wizard of Oz /

i Internet location server . 2,3, 4 participant test 1 f/96 NetMeeting I.0 j31 released. _ ’

6 studies, including first competitor study

V96 NetMeeting 1.0 rekashi, I,~ ‘Ius friends list, jstudies, including: 4uto add to friends, video l Small scale field study

. Trials testing with video IO/96 NetMeeting 2.0 PI &le&d ‘:’ ’ :. Plus Video switching, 12 studies;including: User categories, tab UI . Remote testing using

NetMeeting . Integrated testing with ema . Focus groups

8/97 NetMeeting 2.0 rele&eil : : I Bar UI, 11 studies, including: Integration with other products . Longitudinal home field

study .

W98 NetMeeting 2.1 released , ASDL focus groups

Performance increases 4 studies, some formative for V3 5/99 NetMeeting 3.Ofpl relehsed ; : Refocus interface, 5 studies, including: Add desktop sharing, l Paper prototyping Remote access, security . Wizard of oz features, more ways of locating . Multiple user lab studies users.

Table 1: Product timeline

151

PRELIMINARY STUDIES Getting the technology to work Initially the product consisted of two separate development teams, one working on a point-to-point audio conferencing product and the other designing a multi-point application sharing tool. This functional split, translated directly into the user interface. A user could launch the audio user interface (UI) independently of the application sharing UI. Along with the primitive UI design, in the early stages the developers were focussing on what was technically possible rather than on optimizing performance. This meant that the application was often slow, and interface elements often looked rough. When usability involvement began, the team had already specified that the product would include application sharing, chat, a white board and an audio component.

Puffing the interface together A significant UI change occurred when the audio component was combined with the application sharing interface. Initially when the UIs were combined they were literally placed above one another (Figure 1). Although it wasn’t the intention to have the UI look this way when shipping, for a couple of usability test iterations it appeared this way. Gradually the disparate areas were integrated as appropriate opportunities occurred for change.

Figure 1: Audio UI on top of app sharing UI, as used in some early studies

Another issue related to testing daily “builds” of a software product that was still undergoing major coding effort. It was unusual to find all the parts of the product in a near shipping state in any one build. For example audio was a significant feature in the product, however after a couple of studies it was communicated to the team that users were not happy with the quality of the audio and wanted it to be more like telephone quality. The team had already guessed this fact and rather than continue to test audio and receive these requests every week from participants, we made some compromises. In order to get data on other product areas that wasn’t biased by the poor voice quality, testing of the initial audio setup screens continued, however users were switched from using NetMeeting audio to using speaker telephones as soon as they moved onto other tasks. The phone scenario still had some validity, as it was believed that if a user had two phone lines, or one phone line and a network connection available to them in a work situation they would probably share applications with NetMeeting but use the phone for audio communication. We have since seen this happening many times in site visits.

Users An initial question before launching into usability work is “Who are the intended users?” Although the answer could eventually be “Everyone,” it was anticipated that the early adopters of this technology would be fairly experienced Windows users who were interested in new technologies and had hardware capable of running the product. The first users of a new technology are often engineers and enthusiasts. The ideal or acceptable interface for them can be very different than that for mainstream users [6]. For instance, initial adopters may be happy to work with an interface that exposes the inner workings of the PC, whereas mainstream users often prefer a veneer. Therefore intermediate and advanced Windows 9.5 users were selected to participate in studies. As this was a new product and conferencing products were less familiar in the popular Internet market place at this time (August 1995), there was no need to specify that participants should be unfamiliar with computer conferencing products.

Initial usability Studies Some early studies for NetMeeting were conducted using a wizard of oz technique. One participant would be in the labs, working with the UI, and the communication interactions were faked by having the usabihty engineer play the part of the other person they were to communicate with. Also at this stage the audio technology was not functioning, so although the audio UI was being evaluated, the quality and abilities of the audio technology were not (a speaker telephone was used for communication). Some of the features in the UI were also evaluated using a paper prototype technique, because at this early stage of development it was not necessary to use two participants or a fully working product to evaluate basic task flow through the system.

152

Once the audio and collaboration elements were functioning and stable enough to evaluate, more single participant studies were conducted. Now an additional observer (the ‘confederate’) participated in the studies as the person the participant communicated with in the study. This allowed a high level of control over the way the tasks progressed. There was little need to go straight to testing with two participants because we were still making significant UI changes and it made more sense to identify the glaring problems rather than becoming overwhelmed with confounding factors such as issues with the UI interfering with problems with the interaction model.

Soon the product was stable enough to bring in pairs of participants. We initially did not specify that the participants had to know each other, because the product was not intended to prevent strangers from conversing. However, we discovered that due to the artificial nature of the usability environment and the tasks, we couldn’t guarantee that two unacquainted participants in the lab would feel comfortable and confident enough to complete the tasks in a real manner. Eventually it was decided that a criterion for the studies should be that participants knew each other. We also found that participants who know each other more reliably show up at the same time for tests. So at this point we traded off one of the product scenarios to get a better quality of usability information.

Modifying the testing procedures This was the first conferencing product to be usability tested at Microsoft and it took us a couple of iterations to create the best situation to observe the participants. Initially the usability labs were set up to have two observers and two participants, such that each observer would watch one participant (Figure 2).

Participants with computers communicating with each other using NetMeeting

Two usability engineers with one monitor each, observing one participant computer each, creating two time-stamped records of the interaction.

Figure 2: Original evaluation setup

This created problems when trying to understand the interactions between users. Team members would observe the studies and switch from one room to the other to see what was happening. This meant they sometimes missed the important interaction problems. Eventually the labs were

arranged such that one usability engineer could conduct the study and observe both participants at once (Figure 3).

In usability studies we normally test between six and nine individual participants per study. We decided it was adequate to conduct just three to four sessions when testing pairs, yielding observation of six to eight participants. We were testing NetMeeting every two weeks, and when possible the same task list was used, enabling us to record general interaction issues across studies, thus increasing the number of observations.

Participants with computers communicating with each other using NetMeeting

/ \

aId GIlill

One usability engineer with one monitor showing two video sources. observing both participants, creating one record of the interaction. Figure 3: Modified evaluation setup

NETMEETING 1 .O USABILITY WORK AFTER BETA 1 There are occasions prior to a product release, whether a beta or final release, when the interface is considered to be at visual freeze, and no more UI changes will be accepted. There are continuous tradeoffs in the development of products. The usability engineer must ensure that usability issues are communicated and factored into the tradeoffs. Getting a cutting edge technology to the marketplace, with adoption expected to be mainly among intermediate and advanced users willing to make an effort to use the product, often involves compromises. Considerable development effort was spent on providing sufficient performance, as this was seen as a critical part of user adoption. Thus some known UI issues were left unresolved.

NetMeeting Beta 1 was released April 1996. 12 usability studies had been completed by this stage (see Table I).

Finding out who our users were Frequently with web beta releases the first adopters of the technology are more technical savvy users. There was a large group of early adopters of NetMeeting who were chat room users. To some extent this had come about because to use NetMeeting required two users to have hardware meeting the NetMeeting requirement (and a microphone and speakers). Users who were more familiar with real time communication with strangers had a higher probability of finding someone to communicate with and saw it as an extension of the chat rooms.

153

As the beta versions of the product were released requests for features came to the product team from early adopters. By now this group had expanded beyond the chat users and technicians. The cost/benefit trade-off to learn and persevere with the technology had encouraged others to adopt the product. Two of the more mainstream groups using NetMeeting at this point were friends who wanted to stay in touch internationally using chat and the internet- based (inexpensive) audio, and small businesses who made use of the application sharing capacities.

When beta 1 shipped the team were encouraged to spend time on the servers to contact ‘real end users’. Although this wasn’t the traditional way usability engineers promoted contact between team members and users, it certainly provided high contact time with users, and reinforced some of the issues that had been observed in the labs.

Continuing to test Although we were at visual freeze, we maintained the testing schedule. Having less pressure on evaluating UI change provided an opportunity previously to evaluate different scenarios. There is always a testing tradeoff between ensuring that the ideal user situation works and evaluating the less frequent and more complicated situations. For example there had not been an opportunity to compare application sharing using the telephone, audio, or the chat program to communicate. Another study was conducted to evaluate scrolling behavior when two users’ screens are different sizes and resolutions.

For these evaluation sessions, re-using task lists from previous tests made it easier to understand which issues related to the manipulated UI elements, and which were issues we had seen before.

Studies such as these provide the usability engineer with information so when the team is speculating on features or scenarios for future releases, it might be possible to provide knowledgeable feedback in the initial discussions.

Around the same time as we released our beta, a competitor product was also released to the Internet. The product had a similar set of features, so a quick study was conducted to evaluate how they addressed similar issues. It is often hard to do direct comparison between two products when one is richer in features. The task list has to work for the product with fewer features, which makes it possible for the product with more features to appear more confusing given the scenario. Also, results from such a study need to be carefully communicated to a team as there is a tendency to push aside the findings with the response, “but we support more (or fewer) scenarios”. In this instance, results showed no significant benefits or challenges to users for a basic set of tasks except for one feature in the whiteboard program. The competitor product used a pixel based whiteboard and provided an eraser feature (e.g., very similar to Microsoft Paint), whereas the NetMeeting whiteboard provided an object based whiteboard and so used a delete model which

was harder to discover. There were many higher priority fixes to both the user interface and underlying code required before this feature could be addressed, but today there is an eraser in NetMeeting’s whiteboard program.

Testing larger groups - methodological concerns At this point issues relating to the use of NetMeeting with two users were well understood, even if they hadn’t all been addressed. Now it was time to test the product supporting larger interactions. Pairs of participants who knew each other were scheduled to complete some tasks and get familiar with NetMeeting one week, then they were invited back the following week to work in groups of four. This was a challenge for our usability technicians; we arranged it so one usability engineer could monitor four participants and speak with two, and another engineer could speak to the other two participants. It was as much a test of our method and lab technology as of the product. Eventually we only managed to successfully conduct one session this way, as during one session there was a power failure in the usability wing of the building (an incredibly infrequent event). At this time we didn’t proceed with testing more participants in a four-person scenario because it wasn’t a high priority scenario for the first release of the product. Since this study, the labs have been refitted in a way that makes it easy to observe and interact with multiple participants from one observer station, and several tests with multiple participants have been run.

NetMeeting 2.0 ptanning

Figure 4: Version 1 .O Gold (final release) user interface

NetMeeting 1.0 shipped in August 1996. This product (Figure 4) was available through downloading from the web and also became a shipping component of Windows 95. Version 2 was planned to integrate more closely with Internet Explorer (version 3), and so several interface changes such as toolbar behavior and look were planned to

154

ensure consistency in look and feel between the two products.

Microsoft creates USENET newsgroups for new products, both as a support mechanism and so that we can get user feedback. Through this and other sources, we found that users of the version 1 product were asking for multi-point audio and easier access to the user location directory servers (ULS). Every time they made a NetMeeting call, users who had contacts on more than one ULS directory were changing options that had initially been designed as a one-time configuration setting.

Technology limitations cause interface design issues Multi-point audio was an obvious request from our users, but was not technically feasible within bandwidth and performance constraints. Instead, it was only possible to offer the ability to switch voice communications between two users in a multi-point meeting, even though applications were shared between many users. This easier technical solution led to a harder user interface design solution. Having audio available to only two users meant that a switching mechanism was required. It was not easy to represent half a feature set as point-to-point and half as multi-point, but again it was the early adopters who were pushing for this switching feature.

Video was also about to be introduced into the product and had to be incorporated into the main UI. Again, video would only be point-to-point The point-to-point switching for the audio and video components were tied to each other, so several usability tests focussed on how this should work.

The user location server issue had come about in part through the popularity of the product. The ULS were reaching capacity, forcing users to choose a different server in order to connect. Although work was underway to improve ULS performance and capacity, it became apparent that as users would still be switching directories quite frequently, the task should be easier for them.

Changes in user type By this time, there was a greater awareness of video conferencing and more powerful computers capable of supporting the audio-visual requirements of NetMeeting were becoming generally available. This meant that the prospective user population had shifted from early adoption experts, through those making the cost/benefit tradeoff, to the point where end-users in an organization and intermediate level home users should be able to work comfortably with the technology as part of their daily jobs [7]. The kind of participants for tests changed accordingly. Now pairs of users who worked together in an office environment, with a similar level of expertise and a similar seniority were chosen. We found that differences in expertise or seniority were great for making the less senior or experienced user vocalize their concerns, but did not lead to co-discovery or collaborative work. Also for some

studies, it was necessary to screen out participants who had been exposed to NetMeeting 1 .O as usage increased.

Evaluations To start to get a feel for the usability issues that might occur now that video was to be incorporated, a study was conducted using the latest version of NetMeeting and CU- SeeMe. In this study we didn’t evaluate CU-SeeMe as a product, but used its video window alongside NetMeeting to start to understand video issues from a testing and UI perspective for future work. This usability session helped us understand how to create task lists that would be appropriate for future studies where video was a part of the study. We found, for instance, that it was very important to create tasks where users didn’t have to read information from the task list constantly. Topics such as planning a Halloween party with an unlimited budget gave users the ability to draw on their own ideas and knowledge, rather than having detailed tasks with many instructions and pieces of data, which leads to a lot of time head down reading paper. While the video UI was being planned it was possible to return to conducting paper prototype tests with single participants as it wasn’t necessary to consider user interactions with some of the basic controls.

For a short time, the video UI was on a different tab from the current call information (Figure 5a). From testing it was obvious that this design was not working, as users would focus on the call tab and completely miss the fact that video was available. Iterating the design based on feedback from more test sessions, we ended up with the UI shown in Figure 5b. This design shipped as version 2.0.

Evaluating the new audio-video switching interface introduced new testing problems. Three-person meetings were required to use this feature. Rather than bringing in a third person to participate in this study, the usability engineer became the third person. So when the time came for a participant in the study to contact a third person they contacted the usability engineer (Figure 6).

Figure 5a: Prototype for video switching (note extra ‘video’ tab on left)

155

Figure 5b: final video switching UI

Evaluation Participants communicating with each other using NetMeeting

Usability engineer uses another PC to become the third participant.

Figure 6: Three-way meetings using engineer

Site visits were conducted after version 2.0 was released. We were interested in observing users downloading the product from the Internet, installation, and initial use. We slightly amended the strategy after the first site visit. Our participant had installed the product and was then faced with a list of NetMeeting users he could call but didn’t know, some of whom looked rather dubious! From then on we made sure that when we were out on a site visit someone at Microsoft would be listed who we could call using NetMeeting if necessary. Although we had been aware of the social implications of the product before this time, this was the first real end-user demonstration using this type of product outside the labs.

Social issues appear again The product was becoming popular with diverse segments of the user population and this led to problems managing the User Locator Servers (directories listing other users online at the same time as you) and making sure users were

comfortable using the servers, A high percentage of entries in the ULS were of X-rated content. The NetMeeting team did not want to moderate the lists but wanted to offer a means of using the ULS without being exposed to such content and so introduced user categories (Figure 7).

We had to get feedback from current users on the introduction of user filtering. An additional page was being added to the introduction wizard, along with changes to the directory page. At this time there were an increasing number of NetMeeting users worldwide but not many in the Seattle area and thus available to come in for a usability lab study. So we did the first worldwide NetMeeting usability study. To accomplish this, we used the product itself, calling people using the ULS directory servers. in one evening 14 NetMeeting users were successfully contacted over the Internet using the product, and by using application sharing to show them PowerPoint screen images, they completed a form of on-line paper prototyping. Lessons learned from this study were communicated to other usability engineers at Microsoft to encourage the use of NetMeeting as a remote usability testing tool.

fliicbJan&o &ad

SydmylNSW Au&da

SmAMra.... uritats

Figure 7: User Location Service categories

More competitor evaluations A competitor Internet phone product came on the market during this time, and the team was suddenly concerned about this product being easy to use and more friendly - it had an animated character and looked more appealing as a home Internet phone product. A usability study showed however that having an animated character didn’t mean the product was any easier to use. This event did however force the team to decide who their target users were at this point - did they want to continue supporting just early technical adopters, or were they interested in moving towards an interface that would be accessible to the home market as well.

Cross-product compatibility and integration As new features were added to NetMeeting, some feature areas were becoming similar to feature areas in other of Microsoft’s products. Although the NetMeeting team and the Outlook Express team (a Microsoft mail and news client

156

program) had no plans to integrate, the target users (office workers) may well work with both products. As time allowed, pairs of users were tested in sessions that encompassed office-like tasks where synchronous and asynchronous support would be useful. As we were interested in observing the use of the technology in these scenarios rather than initial learnability, participants in these studies were chosen to be familiar with email and were given some training on NetMeeting, including how to contact each other and share applications. The results of this study were not intended to impact the product at that point in the product development cycle, so this was not a high priority scenario at that time. Later, however, the understanding gained from this study contributed to solving integration issues for the Windows Address Book, Outlook 98 and Office 2000 before specifications had even been created.

NETMEETING 2.1 Many of the changes between version 2 and 2.1 were in the underlying code, increasing performance without requiring changes to the interface.

Interface changes at this point focused less on incorporating new functionality and more on presenting a consistent, simpler and task based experience (Figure 8).

Figure 8: The change from Version 2’s Tab UI to Version 2. l’s Outlook bar

Changing user base Along with Internet Explorer integration, work was also undertaken at this time to integrate NetMeeting more closely into the Office range of products. This happened with the realization that NetMeeting had become a viable collaborative tool for daily office work. For example, the ability to schedule a NetMeeting is a desirable corporate

feature, so usability studies now included evaluation of how to schedule NetMeetings from within Outlook. The previous integration testing was valuable as a starting point here. Several tests were undertaken to ensure that this integration was achieved without detriment to users’ ability to work with either product. This effort required collaboration between the Office and NetMeeting teams to develop integrated features but also required the usability engineers to work together to ensure that appropriate scenarios were tested and that issues were dealt with by the relevant team members.

NETMEETING 3.0 beta 1

Changes in the technology benefit the user interface During the development of the version three product, many technical hurdles were overcome. These technical changes also enabled new, more intuitive ways of working to be developed, again allowing the user interface to be redefined. Now in version three, the application’s “footprint” on the desktop is a lot smaller, and it has become more of a meeting control panel than the collection of allied collaboration tools that it was previously (Figure

9).

Figure 9: Version 3 Beta ldefault view. A compact view, showing just the picture or just the sharing controls is also

available.

157

It had never been the intention to devote so much of the interface to audio controls, but technical constraints had forced their inclusion. Now, many of the interface elements that were concerned with “tweaking” a meeting to get the best performance from audio and video components were no longer required, making the user interaction a lot simpler. Whole new ways of working with the product, such as remote desktop sharing, where a user accesses a machine in one location using NetMeeting from another location, have been introduced. The application sharing process has been re-engineered to interfere less with operation of other programs on the machine at the same time. Items that took up screen real estate have been changed to better meet users’ tasks. For instance it was found that users were only interested in seeing the video image of themselves to check their posture at the beginning of the call. Now this window appears as a picture-in-picture image superimposed upon the image of the sender’s video. This is big enough to reassure the user, but small enough to avoid distracting them. The product’s name has changed to “Windows NetMeeting”’ to better reflect its position as part of the core operating system.

Users become more mainstream Almost a year after the initial release of version 1.0, appealing to a more diverse group of users was becoming important. The user groups were becoming more concrete, giving us a better understanding of their tasks than when the product was in its infancy. Audio/video and Internet technology became more widely available and affordable. This, and the sale of more computers with multimedia hardware, increased the target user base for the product into the home market. Also there was an increased interest in the use of NetMeeting as a tool for corporations.

When version 2.1 reached visual freeze, Planning work started on NetMeeting 3. Work undertaken by usability engineers with the NetMeeting team led to definition of the key user types that they wished to support, and the tasks that these users would want to undertake. This work may not have been possible during earlier product cycles, as more emphasis was placed on the technology than satisfying user requirements at that time. Now, however, with the technology at a more stable and understood point, users became the focus.

Evaluation techniques change again Several field studies were conducted to collect data on these newly defined user types. A group of interviews was conducted in Western Canada on NetMeeting use amongst a self-contained community who were about to be connected with high bandwidth communications (ADSL) into every home and business. Understanding the requirements of this community helped the team to consider features for a future set of more “wired” home and small business users

A local field study was designed to observe the use of NetMeeting in several target market segments, from homes

to large organizations. By this point, a critical mass of people had the requisite hardware and bandwidth, so there was a large enough installed user base from which to draw local examples in all the target areas.

Lab based testing of this next release started again with single users working through paper prototypes and participative or co-discovery evaluations using the daily builds. These were followed by multi-participant evaluations with a specific focus on the user types that the product was now primarily aimed at - users of Office products working on shared applications.

Improved desktop sharing means that it is now possible to control a participant’s desktop remotely. This has been useful in several evaluations, to signal areas of the UI or to get participants back on track. This is another example of the product being used recursively to test itself.

The development team was able to make more radical changes to the underlying code than had originally been expected in the version 3 beta 1 timeframe. This meant that it was feasible for NetMeeting to become a mainstream home user product as well as an offtce productivity product. This gave the team an opportunity to design the product UI to appeal to a broader user base than they had first targeted for this release. Having data from the site visits and initial user-centered planning work enabled UI decisions to be made with minimal extra testing.

NetMeeting Version 3, Beta 1 was released to the Web in April of 1999, and marks the first release of a major user interface redesign.

Conclusions In a perfect world, this paper would have been presented as the description of an iterative usability process from start to finish that showed the orderly development of a product from conception, through its first release to its current version, clearly showing the co-evolution of technology, users and user interface. However that is not how it happened. The technology wasn’t ready, the user groups weren’t ready, and our usability procedures weren’t ready.

Working on a product that consists of cutting edge technology has numerous challenges. This case illustrates where development was initially led by the available technology rather than a specific product goal, which in turn drew in users who were attracted by certain technology-centric scenarios, either just for the novelty, or because they could put up with a sub-optimal interface for the benefits it offered. Just over a year later, many technological constraints had been resolved, and the user scenarios were defining how the technology was integrated.

The usability methods applied to the evaluation of this product had to adapt to match the target user groups as they were redefined, and to match the technological changes in the product. Some usability evaluation methods involve team buy-in, and early in the product cycle this trust did not exist. This meant early compromise in order to ensure

158

continued involvement. As the team began to recognize usability’s contribution, the engineer could become more flexible and thus could suggest better methods for evaluating the product. This led to development of some novel evaluation techniques.

Usability engineers had to keep in touch with the potential future scenarios at the same time as evaluating currently available software, in order to be prepared when the climate changed to the point where scenarios would lead the technologies. Figure 10 shows the interaction of user type and technological evolution. Technically adept users with leading-edge machines were the only people capable of running NetMeeting V 1. With greater processing power, lower cost components and technological advances, the product became accessible to a wider group of users. This led to development being based more on user scenarios.

I 1996 1997 1998 1999

Figure 10: Change in focus from technology to user scenarios as NetMeeting matured

For a joke to be funny, it has to have some basis in reality. With the “By version three, the product should be worth using,” joke, the truth is that this was the version where development was driven to a large degree by considerations of users and their tasks. That is not to say that the previous versions of the product were poorly implemented, but more that they were essential precursors to making what can be called a “usable” product. The initial releases of NetMeeting provided market adoption and learning points, and led to improvements in later versions that would not otherwise have been possible.

As NetMeeting grows, and is used by more people in more situations, the user interface will no doubt change to accommodate new work practices, new technology, and probably new ways of being sociable online.

In dealing with the technical, logistical and social challenges presented by evaluating a new collaborative technology, one element of usability work is critical: the ability to communicate with the product team and work with their schedule. This case illustrates how important it is to stay focussed on end-user scenarios, regardless of the current technical limitations, so that you get ahead of the curve and are in a position to contribute the necessary user- centered information at the beginning of the next design cycle.

ACKNOWLEDGMENTS We thank the NetMeeting team for writing the application, the 260+ participants we studied who helped shape the product, and the reviewers who commented on this paper.

REFERENCES 1.

2.

3.

4.

5.

6.

7.

8.

9.

Chapanis, A., Ochsman, R., Parrish, R., & Weeks, G. (1972). Studies in interactive communication: I. The effects of four communication modes on the behavior of teams during cooperative problem solving. Human Factors, 14, pp. 487-509.

Dourish, P., & Bly, S. (1992). Portholes: Supporting awareness in a distributed work group. Proc. CHI’92, pp. 541-547.

Gentner, D.R. and Grudin, J., (1996). Human interface design models: Lessons for computer human interfaces. IEEE Computer, 29, 6,28-35.

Grinter, R.E., (1998). Recomposition: Putting It All Back Together Again. Proc. CSCw’98 pp. 393-402

Grudin, J (1991). Interactive Systems: Bridging the gaps between developers and users. IEEE Computer, 24, 4, 59-69

Ishii, H., Kobayashi, M., & Arita, K. (1994). Iterative design of seamless collaboration media: From TeamWorkStation to ClearBoard. Communications of the ACM, 37,8, pp. 83-97.

Mark, G., Grudin, J., and Poltrock, SE., (1999). Meeting at the desktop: An empirical study of virtually collocated teams. Proc. ECSCW’99.

Mayhew, D.J., .(1999) The Usability Engineering Lifecycle. Morgan Kaufmann

Olson, J.S., Olson, G. M., & Meader, D. K. (1995). What mix of audio and video is useful for doing remote real-time design work. Proc. CHI’95, pp- 362-368.

lO.Palen, L. (1999). Convergent Perspectives on Groupware Evaluation & Design. CHI 99 Proceedings, May 15-20, 1999, Pittsburgh, PA USA

11. Sellen, A. (1992). Speech patterns in video-mediated communication. Proc. CHI’92, pp. 49-59.

l’L.Tang, J.C., & Rua, M. (1994). Montage: Providing teleproximity for distributed groups. Proc. CHI’94, pp. 37-43.

159