1
PROGRESSREPORT
FIRST YEAR
GRANT CCR-9624846
Improving the Design of Interactive Software
David F. Redmiles
Department of Information and Computer ScienceUniversity of California, Irvine
Irvine, CA 92697-3425
email: [email protected]: (714) 824-3823fax: (714) 824-1715
web http://www.ics.uci.edu/~redmiles/
July 1996 - July 1997
2
Table of Contents
Overall Summary of Progress for Year 1 4
Collaborations 5
Theoretical Progress 5
Human-Centered Software Development Lifecycle 5
Software Design Environments 6
Usage Data Collection 9
Information Review and Project Awareness 10
System Implementation Progress 11
System Integration — A Scenario of Design, Usage, and Review 11
The Argo Software Design Environment 16
EDEM Software for Usage Data Collection 17
The Knowledge Depot for Information Review and Project Awareness 17
Immediate, On-going Work Plan 19
Personnel Supported on the Grant 19
Research Support of Senior Personnel 20
List of Publications—Year 1 20
List of Presentations—Year 1 21
Web Site Guide 22
References 22
3
List of Figures
Figure 1: Human-Centered Software Development Lifeycle 6
Figure 2: The Argo Software Design Environment 7
Figure 3: Mapping between Cognitive Theory and Design Environment Features 8
Figure 4: Differing Expectations between Developers and Users 9
Figure 5: Travel Voucher Application 12
Figure 6: Agents Editor 13
Figure 7: Monitoring Events 14
Figure 8: Organizing Information for Review in Knowledge Depot 14
Figure 9: Architecture of the Travel Voucher Application in Argo 15
Figure 10: Process Model for Constructing the Architecture of an Application 15
List of Tables
Table 1: Characterizing Individual and Group Awareness 10
4
Improving the Design of Interactive Software
Overall Summary of Progress for Year 1
The goals of our research are to combine elements of research in human-computer interactionwith research in software engineering. More specifically, we have made progress with the follow-ing approaches. First, we have focused on the communication that takes place between softwaredevelopers and the intended end users of interactive systems. To facilitate this communication wehave developed an approach we callexpectation agents which automatically collects usabilitydata and reports it back to developers. Second, we have focus on the collection and organizationof usability data and other information which software developers, end users, and other stakehold-ers exchange during the course of a project. Namely, we have explored an approach based ongroup memory andorganizational awareness. Third, we have re-examined the highest level ofactivity in software development, namelysoftware architecture design. We have developed thisapproach in light of current cognitive analyses of human problem solving and design. We contin-ually seek to integrate our efforts and refer to them under the collective term ofhuman-centeredsoftware development.
Three papers are included with this progress report to provide details of these approaches (Rob-bins, Hilbert, Redmiles 1996; Hilbert, Robbins, Redmiles 1997; Kantor, Zimmermann, Redmiles1997). In the body of this progress report, we are including a scenario of how we envision thethree specific approaches to work together in the context of a software development effort. Thisproject is strengthened by our close collaboration with NYNEX/Bell Atlantic in White Plains.This collaboration grounds our theories in software practice and may in the future provide onevenue for evaluation. We also corroborate our approach when possible with other researchprojects of similar interest. Thus, herein we also report on our specific efforts in collaborations.
In summary, this report describes our progress in the following areas:• Collaborations: developments in our on-going collaboration with NYNEX/Bell Atlantic
and collaborations with other research groups;• Theoretical Progress: an integrated theoretical framework for human-centered software
development; and• System Implementation Progress: implementation of our theory through three systems for
supporting, respectively, expectation agents, organizational awareness, and software archi-tecture design.
In addition, this report provides information on the following:• Immediate, On-going Work Plan: specific current goals (consistent with the original pro-
posal);• Personnel Supported on the Grant: people contributing to the effort with or without direct
funding;• Research Support of Senior Personnel: efforts to expand our research through new propos-
als;
5
• List of Publications—Year 1;• List of Presentations—Year 1; and• Web Site Guide.
Collaborations
A key aspect of our research program is our collaboration with outside industry and researchgroups. These collaborations provide valuable feedback on our research ideas. The feedbackcomes both from other researchers and actual practitioners. Foremost, this feedback is a criticalelement of any research. However, it also is extremely valuable to graduate students in that it pro-vides them the opportunity to think more realistically about their work. Finally, it provides oppor-tunities for internships and potential future employment for them.
With respect to this grant, our primary collaboration is with NYNEX/Bell Atlantic based in WhitePlains. Our principle contacts there are Michael Atwood and Bea Zimmermann. Through this col-laboration we gain access to end users in actual work place settings and have the opportunity tojointly develop prototype systems. Some milestones in this collaboration include the following:
• internship at NYNEX/Bell Atlantic for one student during Summer 1996 and for Summer1997;
• licensing agreement, completed in June 1997, for joint development of the KnowledgeDepot software tool described in this report;
• visits by technical staff, UCI to NYNEX/Bell Atlantic in June 1996, NYNEX/Bell Atlanticto UCI in April 1997;
• joint authorship of a technical report on group awareness (Kantor, Zimmermann, Red-miles 1997).
In the past six months, we have cultivated a working relationship with a Los Angeles based com-pany, Computing Services Support Solutions (Cs3). They are working on an event-based languagerelevant to our work on expectation agents. K. “Swamy” Narayanaswamy and Martin Feather areour principle contacts there. To date, we have
• hosted a research visit by Cs3 staff at UCI;• explored plans to jointly evaluate our software.
Theoretical Progress
Our research has three objectives: 1) to develop a model of software development as a process ofon-going communication; 2) to support this model through active mechanisms in software tools;and 3) improve the accessibility and acceptance of usability methods by software practitioners. Ingeneral, the objectives reflect a theory ofhuman-centered software development.
Human-Centered Software Development Lifecycle
Figure 1 shows one way to illustrate a human-centered software development lifecycle. Namely,prototype systems are designed, evaluated in a usage situation, and the results of the evaluationare reviewed and incorporated back into a revised design. Of course, by this diagram, we do notmean to imply that these activities are segregated phases of a lifecycle; rather, the activities are
6
separated out for ease of discussion and to emphasize areas when software tool support is possi-ble.
Figure 1: Human-Centered Software Development Lifeycle
This lifecycle is human-centered in two ways. First, the emphasis on prototypes and evaluation ofusage maintains a focus on the requirements of end users and other stakeholders in the project.Second, there is an underlying assumption that the software developers need cognitive support formanaging the complexity of usability information and incorporating that data and other informa-tion into design. Thus, the applications being developed are engineered to human cognitive needsof the end users and the tools used to developed those applications are engineered to the humancognitive needs of the software developers.
Software Design Environments
The approach we have taken emphasizes the use of software protoypes. This is well motivated bythe research on participatory design (Greenbaum and Kyung 1991). Prototypes provide a focusaround which software developers, end users, and other stakeholders can communicate andexpose requirements. While we acknowledge the necessity of non-executable prototypes (such asmock-ups or informal drawings) early in the development lifecycle, we seek to enable softwaredevelopers to create executable prototypes as soon as possible. The approach we have taken is tofocus on high level design through software architecture and support that design throughdomain-oriented design environments (Fischer 1994). In particular, we have developed a design environ-
Design
UsageReview
ARGOEDEM
EDEMKNOWLEDGEDEPOT
incorporatereviews into
deploy prototype withexpectation agents
deposit observations
prototype and expectation agents
prototype with expectation agentsexpectation agents’ reports andother information
perform direct observations,interviews, other manualusability methods
perform other design reviews,including email discussions
design
7
ment, called Argo, for the domain of software architecture design (Robbins, Hilbert, Redmiles1996). In this domain, it is possible for a designer to generate a software prototype from a highlevel architectural specification. Figure 2 provides a view of Argo (center) being used to design avideo game (left). A “to do” checklist window appears on the right in this figure.
Figure 2: The Argo Software Design Environment
In general, a domain-oriented design environment is a tool that augments a human designer’s abil-ity to design complex artifacts. Design environments must address systems-oriented issues suchas design representation, transformations on those representations (e.g., code generation), andapplication of analysis algorithms. Furthermore, design environments exceed the capabilities ofmost CASE (computer-aided software engineering) tools by addressing the cognitive needs ofdesigners. Specifically, we have examined three cognitive theories of design and problem solving:reflection-in-action, opportunistic design, andcomprehension and problem solving. Figure 3 sum-marizes our work by showing a mapping from these cognitive theories to specific design environ-ment features that address them.
The cognitive theory of reflection-in-action observes that designers of complex systems do notconceive a design fully-formed (Schoen, 1983, 1992). Instead, they must construct a partialdesign, evaluate, reflect on, and revise it, until they are ready to extend it further. Design environ-ments support reflection-in-action with critics that give design feedback. Critics are agents thatwatch for specific conditions in the partial design as it is being constructed and present thedesigner with relevant feedback. Critics can be used to deliver knowledge to designers about theimplications of, or alternatives to, a design decision. Examples of domain knowledge that can be
8
delivered by critics include well-formedness of the design, hard constraints on the design, rules ofthumb about what makes a good design, industry standards, organizational guidelines, and theopinions of fellow project stakeholders and domain experts. In Argo, critics are displayed in the“to do” checklist window on the right in Figure 2.
Figure 3: Mapping between Cognitive Theory and Design Environment Features
The cognitive theory ofopportunistic design explains that although designers plan and describetheir work in an ordered, hierarchical fashion, in actuality, they choose successive tasks based onthe criteria of cognitive cost (Hayes-Roth, Hayes-Roth, 1979; Guindon, Krasner, Curtis 1987;Visser, 1990). Simply stated, designers do not follow even their own plans in order, but choosesteps that are cognitively the least cost among alternatives. The cognitive cost of a task depends onthe background knowledge of designers, accessibility of pertinent information, and complexity ofthe task. Thus, although it is customary to think of solutions to design problems in terms of a hier-archical plan since hierarchical decomposition is a common strategy to cope with complex designsituations, in practice, designers have been observed to perform tasks in an opportunistic order.Design environments can allow the benefits of both an opportunistic and a prescribed design pro-cess. They should allow, and where possible augment, human designers’ abilities to choose thenext design task to be performed. They can also provide information to lower the cost of follow-ing the prescribed process and avoid context switches that deviate from it.
The theory ofcomprehension and problem solving observes that designers must bridge a mentalgap between their model of the problem or situation and the formal model of a solution or system(Kintsch, Greeno, 1985; Fischer, 1987; Pennington 1987). Problem solving or design proceedsthrough successive refinements of the mapping between elements in the design situation and ele-ments in the formal description. Multiple, overlapping design perspectives in design environmentsallow for improved comprehension and problem solving through the decomposition of complex-ity, the leveraging of the familiar to comprehend the unfamiliar, and the capture of multiple stake-holders’ interests. It is our contention that no fixed set of perspectives is appropriate for every
Multiple perspectives
Critics
Produce "to do" items with links
"To do" list
Reminders to oneself Allows choice
Process model
Design Perspectives Multiple, overlapping perspectives Customizable presentation graphics Presentation critics
Theories of Cognitve Needs Support in Argo
Comrehension andproblem solving
Opportunitsic design
Reflection-in-action
Provide missing knowledgeEvaluation during design
Diversity of knowledge
RemindingProcess flexibilityProcess guidanceProcess visibility
Divide complexity
Presents feedback
that match mental models
Timeliness
Kept relevant and timely by control mechs Continuous and Pessimistic Low barriers to authorship
Process editing Process critics Process perspectives
9
possible design; instead perspective views should emphasize what is currently important in theproject. When new issues arise in the design, it may be appropriate to use a new perspective on thedesign to address them.
Usage Data Collection
Usability breakdowns occur when developers’ expectations about system usage do not matchusers’ expectations. A usage expectation determines whether a given interaction (e.g., a sequenceof mouse clicks) is expected or unexpected. For example, a developer may hold the usage expec-tation that users will fill in forms from top to bottom with minor variation, while a user may holdthe expectation that independent sections may be filled out in any order.
Figure 4: Differing Expectations between Developers and Users
Figure 4 shows a conceptual picture of expectations held by two groups of stakeholders (develop-ers and users) and expectations encoded in the system being used. Each lowercase “e” representsa tacit expectation held in the mind of a person or in the code of a program. Developers get theirexpectations from their knowledge of the requirements, past experience in developing systems,domain knowledge, and past experience in using applications themselves. Users get their expecta-tions from domain knowledge, past experience using applications, and interactions with the sys-tem at hand. The software system embodies implicit assumptions about usage that are encoded inscreen layout, key assignments, program structure, and user interface libraries. Each upper case“E” represents an explicit expectation. Several usability methods seek to make implicit expecta-tions of developers and users explicit, e.g., cognitive walkthroughs (Wharton et al. 1994), partici-patory design (Greenbaum, Kyung 1991), and use cases (Rumbaugh, Booch, Jacobson 1997).
Once a mismatch between users’ and developers’ expectations is detected it can be corrected inone of two ways. Developers can change their expectations about usage to better match users’expectations, thus refining the system requirements and eventually making a more usable system.For example, features that were expected to be used rarely, but are used often in practice should bemade easier to access. Alternatively, users can adjust their expectations to better match those ofdevelopers, thus learning how to use the existing system more effectively.
10
Detecting a breakdown or difference in expectations is useful, but aligning expectations requiresknowledge of the other expectations and the specific differences. For developers to learn aboutusers’ expectations they need specific details of actual usage, including context, history, timing,and intent. For users to learn of developers’ expectations they need clear documentation of theintended system operation and rationale to be presented to them at the time of breakdown. Ineither case, dialog between users and developers can help clarify and expose expectations.
Information Review and Project Awareness
In the CSCW literature, systems labeled as group awareness tools are generally designed to pro-vide relatively instantaneous awareness of a group of individuals. This research explores a type ofawareness tool that does not fit within this implicit classification of awareness. The approachinstead focuses on providing individuals with awareness of multiple groups (rather than multipleindividuals), over days or weeks (rather than fractions of seconds to minutes). The types of infor-mational benefits derived from such a system vary greatly from the benefits of a typical groupawareness tool. Table 1 illustrates the types of questions answered by different classes of aware-ness systems. The two dimensions are the frequency with which users receive information and theunit of observation.
A tool whose unit of observation is an individual, helps users maintain awareness of a set of indi-viduals; example tools are Portholes (Dourish & Bly, 1992; Mantei, Baecker & Sellen, 1991; Lee,Girgensohn, Schlueter 1997) and Piazza (Isaacs, Tang & Morris, 1996). A tool whose unit ofobservation is a group helps users maintain awareness of a set of groups; example tools are VideoWindows (Abel, 1990) and wOrlds (Boggy & Kaplan, 1995). Information that users need to beaware of over days rather than over minutes tends to be much more conceptual and task oriented
Table 1: Characterizing Individual and Group Awareness
Unit of Observation
Frequency Individual Group
Seconds toMinutes
What is a person’s location andcurrent activity? (exampletools are Portholes, Piazza)
Is there a group meeting?Where? What types of tasks isthe group working on? Who isin the group? (example toolsare Video Windows, wOrlds).
Days toMonths
What is a person trying toaccomplish this week? Whatare a person’s plans for thisweek? What problems is a per-son working on solving?(example tools are various cal-endar applications and distri-bution lists).
What is a group working onthis week? What kinds of prob-lems is the group encounter-ing? What changes has thegroup made in the task thegroup is working on? Whenwill the task be complete? (weseek to fill this block with ourtool, Knowledge Depot).
11
than physical. Instead of providing awareness through graphical or audio means of some physicalfact (the user is in room X, sitting, talking on the phone and browsing the web), we instead have toprovide information that is much more abstract (a person plans to work on the following tasks,and the tasks have this set of priorities, and the following problems are delaying them) which typeof information is best represented textually, as is done in calendars. Whereas the unit of informa-tion in a typical awareness system is an individual, the unit of awareness in a system aimed atgroups over days is the task or set of tasks of the group. Awareness of the task then is what makesthis type of awareness so important.
We have focused on an approach being developed to fill the Group/Days-to-Months quadrant ofthe diagram. We refer to tools that fit the this quadrant asproject awareness tools. We have devel-oped one such tool, called Knowledge Depot, which we illustrate below.
Features that are needed by all awareness systems for them to function are information capture,information distribution and information presentation. The effectiveness of an awareness tool willlargely depend upon how well it provides these three features. It must capture useful informa-tion, distribute it quickly to the appropriate people, and display the information in a meaningfulmanner. From this, a bulletin board system could be argued to be an awareness system, as it cap-tures information, distributes it by making it available to bulletin board readers, and presents it asa list of messages. However, the information capture requires users with information to deliber-ately start the news reader, navigate to a topic and submit the information, and distributionrequires users to go to appropriate software and search for the information that they want to beaware of.
System Implementation Progress
The human-centered software development lifecycle presented above is being explored throughthree systems. The Argo Software Design Environment implements the design theory componentillustrated in Figure 1. The EDEM (Expectation-Driven Event Monitoring) system substrateimplements the observation of usage component. The Knowledge Depot implements the reviewcomponent. The preceding sections introduced our theoretical understanding of a human-centeredsoftware development software lifecycle and each of the component activities. The following sec-tions illustrate how the components could work together through a hypothesized scenario and thenexplain more about each system’s details. As noted earlier, additional details can be found in thetechnical reports and paper accompanying this annual report (Robbins, Hilbert, Redmiles 1996;Hilbert, Robbins, Redmiles 1997; Kantor, Zimmermann, Redmiles 1997).
System Integration — A Scenario of Design, Usage, and Review
We are working on integrating our tools for design, observation, and review (project aware-ness). Our objective is to provide integrated support of the human-centered software develop-ment life lifecycle described earlier and shown in Figure 1. We present our integration effort bymeans of a scenario in which a developer uses Argo, EDEM, and Knowledge Depot in developinga simple Travel Expense Form application written in Java, monitor its usage, and review the datafrom the monitoring.
12
The interface shown in Figure 5 represents applications used to report travel expenses for reim-bursement purposes. The layout of the interface is based on certain tacit, cognitive assumptionsabout the tasks and the users of the form. For example, the grouping and order of fields reflectsdevelopers’ expectations about how users mentally organize the requested data and the sequencein which it is most convenient or efficient to provide it.
Figure 5:Travel Voucher Application
In order to test some of these assumptions, a developer uses the EDEM Agent Editor to specify anExpectation Agent that will trigger whenever a user begins editing the “Address Section” of theTravel Expense Form, see Figure 6. The developer uses the Agent Editor to express interest indetecting when the user begins to edit the “ZIP” text field. The developer then adds this simpleevent specification to an Expectation Agent that will trigger whenever a user begins editing the“Address,” “City,” “State,” or “ZIP” fields in the Travel Expense Form.
13
Figure 6: Agents Editor
The agent specification consists of: (1) a name, (2) an activate pattern (an event pattern which, ifsatisfied, causes the agent to be activated), (3) a cancel pattern (an event pattern which, if satisfied,causes the agent to cancel evaluation of its activate pattern, (4) timing constraints, (5) a conditionto be checked once the activate pattern has been satisfied, (6) an action to be performed if theagent is activated while the condition is TRUE, (7) and a flag indicating whether the agent shouldcontinue checking after successfully triggering once.
Figure 7 presents a display of the state of the event monitors when a user has edited the “Traveler”field and then shifted input focus to the “Address” field. This display is primarily for our develop-ment and debugging. Note that the LOST_EDIT event is not generated for the “Traveler” fielduntil the user actually begins editing another field in the form. This is an example of how EDEMprovides events at a higher level of abstraction than the window system events generated by Java.In this fashion, an agent centered on detecting when editing has been completed in a particularfield does not need to be concerned with monitoring every other field in the interface.
The agent defined in this example generates a single event (“Address Started”) as soon as any ofthe fields in the “Address Section” has been edited. This agent can then be used in conjunctionwith other agents that detect when other sections have been started and/or finished. The developercan thus compare the actual order that users complete the sections and fields in the Travel ExpenseForm with the order that was originally expected.
14
Figure 7: Monitoring Events
Figure 8: Organizing Information for Review in Knowledge Depot
15
Feedback from agents must be reviewed and analyzed to inform future design decisions. To thisend, we have integrated EDEM with the Knowledge Depot which collects and organizes Expecta-tion Agent reports for later review. Figure 8 shows several categories of data collected fromExpectation Agents monitoring the Travel Expense form. These categories can then be used tolink back into the appropriate (of affected) architectural elements in the Argo design of the system(Figure 9), thus providing an indication of where changes to the system will need to occur.
A software developer can use the Argo design environment to specify the components of theTravel Voucher Application. In future work, we seek to make the feedback from agents linkdirectly to these design components.
Figure 9: Architecture of the Travel Voucher Application in Argo
Figure 10: Process Model for Constructing the Architecture of an Application
16
The Argo Software Design Environment
Argo is our software architecture design environment illustrated in Figures 1, 9, and 10. It is basedon the cognitive theories of reflection-in-action, opportunistic design, and comprehension andproblem solving (see the earlier section, “Software Design Environments” on page 6). Argo offersbasic CASE tool features for entering an architectural model and applying code generating trans-formations. Furthermore, Argo contains critics that automatically provide design feedback, a userinterface for browsing design feedback, a process model to guide the architect and help controlcritics, and multiple coordinated design perspectives.
Our current Argo implementation support critics that run in a background thread of control to con-tinuously analyze a software architecture as it is being manipulated. Critics are made up of ananalysis predicate, a design feedback item, and various attributes used to determine if the critic istimely and relevant. Criticism control mechanisms are objects that define Argo’s policies on whento apply individual critics to a given part of the architecture. For example, one criticism controlmechanism selects critics that are relevant to stated design goals, while another selects critics thatare timely (relevant to design decisions that the architect is currently considering). Argo associatescritics with specific types of design materials (i.e., elements of the architect model) rather thanstoring all critics in a central knowledge-base. This allows us to keep Argo’s kernel simple andflexible, and will soon allow up to load models and their associated critics over the Internet asneeded.
Argo presents design feedback to the architect through a “to do list” metaphor. Items on the “to dolist” indicate an issue that the architect must consider before the design is finalized. Critics unob-trusively add and remove items as design problems arise and are resolved. When the architectselects an item an explanation of the issue and possible resolutions is displayed and the “offend-ing” elements of the architecture are highlighted to help build the mental context needed toresolve the issue. Furthermore, the architect can send email to the expert who authored the criticto open a dialog and draw the architect into the organizational context needed to resolve someissues. In future work we will address recording these design issues and resolutions as a form ofdesign rationale.
Perspectives are implemented as predicates that select a part of the architecture model to be dis-played in a given context. For example, one perspective may show software components and theircommunication relationships, while another perspective may show the relationship between com-ponents and operating system resources.
The Argo process model consists of a set of design materials for representing design tasks anddependencies between them. Since the process modeling domain is built on top of the Argo kernel(just as is the architecture modeling domain), we can apply Argo’s own facilities to editing, visu-alizing, and critiquing the design process. Each step in the design process indicates a set of designdecisions that the architect will consider during that step. As the architect progresses through theprocess model, process status information is used to infer which critics are timely.
Critics, criticism control mechanism, and perspectives are each implemented as Java booleanfunctions. This allows great freedom in expressing the analyses to be performed and permits us to
17
use standard Java development tools. In the future, we hope to integrate an expert system or othertechnologies that focus on the kinds of predicates that are most often needed. Support for end usercustomizability is also planned.
EDEM Software for Usage Data Collection
We have developed prototype software for detecting and resolving mismatches in usage expecta-tions by allowing developers to specify their expectations in terms of user interface events. Theseexpectations are encoded in the form of expectation agents that continually monitor usage of theapplication and perform various actions when encapsulated expectations are violated.
In order to support this functionality, expectation agents need access to events occurring in anapplication’s user interface. However, in order for expectations to be expressed at an appropriatelevel of abstraction, they need access to higher level events than are produced by the window sys-tem. As a result, expectation agents are implemented on top of an expectation-driven event moni-toring substrate (EDEM) that provides a multi-level event model. Aspects of the EDEM softwareare illustrated in Figures 6 and 7.
Other researchers have proposed event monitoring as a means for collecting usability data, how-ever, existing approaches suffer from one or more of the following limitations: (1) low-levelsemantics—events are captured and analyzed at the window system level, or just slightly above;(2) decontextualization—analysis is done post-hoc on raw event traces, potentially relevant con-textual cues are lost; (3) “one-way” communication—data flows from users to developers whomust then infer meaning, no “dialogue” is established to facilitate mutual understanding; (4) batchorientation—hypothesis formation and analysis is performed after large amounts of (potentiallyirrelevant) data have been collected, no means for hypotheses to be analyzed and action takenwhile users are engaged; and (5) privacy issues—arbitrary events are collected without anyexplicit constraints on the purposes of collection, no way to provide users with discretionary con-trol over what information is collected and what information is kept private.
The EDEM software addresses these issues and goes beyond existing approaches in supportinguser involvement in development. Our system provides: (1) a multi-level event model—allowingagents to compare usability expectations against actual usage at reasonable levels of abstraction;(2) contextualization—taking action and collecting information while users are engaged in usingthe application; (3) two-way communication—helping initiate dialog between users and develop-ers when breakdowns occur; and finally, (4) specializable monitoring and analysis—promoting ashift from batch-oriented data collection and analysis to hypotheses-guided collection and analy-sis.
The Knowledge Depot for Information Review and Project Awareness
In our scenario, groups need a mechanism for communicating about usability and other data,design decisions, and general progress. The Knowledge Depot, an enhanced version of GIMMe(Zimmermann, Lindstaedt, Girgensohn, 1997), captures email conversations, and other informa-tion, and organizes this information, allowing users to browse through the information to redis-cover (or learn for the first time) why different decisions were made, what problems were
18
encountered as a result of those decisions, and allowing the user to regain some of the context inwhich those decisions were made. The Knowledge Depot is illustrated in Figure 8.
The system organizes its knowledge around a set of topics defined over time by all users of thesystem. The topics frame is shown in Figure 8 as a hierarchical list of discussion items. A topic isfour things:
1. A phrase describing a concept, task or activity representing aspects of the group’s work;2. A place where people go to find information;3. A definition of the type of information the system looks for to determine whether something
belongs in the topic; and4. A destination that people will aim their messages at in order to have the message stored cor-
rectly for later retrieval.
A topic then might have a name like “Form Completion Sequence.” This says that any messagethat has “Form Completion Sequence” in the subject line will be captured in this topic, and peoplebrowsing for messages will know to look for email discussing the Form Completion Sequencewithin this topic. People having an email conversation about the Form Completion Sequence thenneed only put “Form Completion Sequence” in the subject, and CC the Knowledge Depot systemfor the information to be captured and put in an appropriate place. Furthermore, the hierarchicalorganization of topics allows a message that has “Form Completion Sequence Violation” in thesubject to enter the “Form Completion Sequence” topic, and then enter the “Violation” subtopic.
If a new type of discussion begins, any group member can create a new topic or subtopic to cap-ture the new type of discussion. If the terminology changes, users can change the definition of thetopics. For example, if there is a topic “NYNEX, collaboration” for discussing work done withNYNEX, and NYNEX changes its name to Bell Atlantic, the topic would simply be updated to“NYNEX, Bell Atlantic, collaboration”. This allows the organization of knowledge to evolve overtime as the group itself evolves.
The Knowledge Depot uses the same type of privacy mechanism as the Information Lens: infor-mation does not become publicly available unless explicitly emailed to a special user name (Mal-one et al. 1989). The Information Lens used the account “Anyone” to determine which knowledgeshould be publicly distributed, while Knowledge Depot uses a mail account named after the groupto capture mail that is to be archived in the group memory.
Although the Depot focuses on email, it can also process electronic bulletin boards, and users canupload formatted documents (i.e. PDF, HTML, GIF, etc.), which are all treated the same as mailmessages in how users are made aware of them. These documents can’t be searched by their con-tents the way mail and newsgroup messages can, but can still be accessed by browsing topics, andby giving the system queries by date, author and/or subject.
The Knowledge Depot has many similarities to a newsgroup, if the newsgroup is used as a placewhere people Cc mail so that it can be made available to non-recipients of the message, and sothat it can be stored for a time. The Knowledge Depot though is not constrained a single news-group with a linear list of messages, nor is it equivalent to a large number of newsgroups where
19
users have to determine which group (or groups) are most appropriate. The depot uses a flexibleset of topics where any user can at any time add a new topic, and based on the definition of thetopic, all existing messages will be checked to see if they belong in the topic (in addition to anyother topics that the message is already in). Potentially, the entire topic structure could beremoved, and replaced with a new topic structure, with messages automatically reorganizingthemselves to account for the new hierarchy. While a newsgroup acts as an oversized shared emailbox, Knowledge Depot acts as a shared database. The database is used to organize all of a groupsdiscussions and documents, and provides standard types of database queries and reorganizations.
Immediate, On-going Work Plan
Below are some immediate goals for the second year of the grant. These goals are
All Systems• Develop revised evaluation criteria for the software prototypes (Argo, EDEM, Knowledge
Depot).• Revise and enhance the technologies based on usage and feedback: there are plans to test
all three software prototypes (Argo, EDEM, Knowledge Depot) in industry (respectivelyat Rockwell, Lockheed-Martin, and NYNEX/Bell Atlantic).
• Continue to report on progress through technical reports, conference papers, and journalarticles.
Argo• Increase the critic analysis and the visualization support in the software design environ-
ment.• Investigate the application of the design environment architecture to new domains such as
requirements representations (CoRE method) and newer architectural specification lan-guages (UML–Unified Modeling Language).
EDEM• Refine the interface for developers defining usage expectations.• Assess the generality of the expectation agent model and, in particular, identify possibili-
ties for generalizing the expectation model to more than usability testing.
Knowledge Depot• Improve the integration with usability reports from EDEM.• Add more general searching of text and documents classified in the Knowledge Depot.
Personnel Supported on the Grant
The Principle Investigator is• David F. Redmiles, Assistant Professor
The grant supports one graduate research assistant throughout the year. However, this funding hasoccasionally been split between two PhD students:
• David Hilbert
20
• Michael Kantor
In addition, a third PhD student has contributed to the research being funded from an ARPA grantawarded to Professors Taylor and Redmiles. He is
• Jason Robbins
Finally, the grant benefits from the collaboration with NYNEX/Bell Atlantic in White Plains. Inparticular, the following personnel lend their time to this effort:
• Michael Atwood, Technical Director, Advanced Software Development EnvironmentsGroup
• Bea Zimmermann, Member Technical Staff, Advanced Software Development Environ-ments Group
Research Support of Senior Personnel
ARPA. Since the application for this proposal, the Principle Investigator was awarded a grantfrom the Advanced Research Projects Agency (ARPA). He is Co-Principle Investigator on thisgrant; Richard Taylor is the Principle Investigator. The title of the grant is “Open Technology forSoftware Evolution: Hyperware, Architecture, and Process.” The planned, three-year budget is$2,606,666. The starting date was October 1996.
This grant supports Taylor’s on-going effort in the area of software tools for large-scale, complexsystems. Redmiles’ contribution focuses on the application of cognitive theories of design, con-tinuing work he participated in at the University of Colorado on design environments. The grantalso provides another venue for applying the framework and tools which are the focus of this NSFgrant, namely the EDEM expectation agents system for supporting communication betweendevelopers and end users through expectation agents.
MICRO . The principle investigator also has submitted a one-year project proposal to the Univer-sity of California Microelectronics Innovation and Computer Research Opportunities (MICRO).The proposal is entitled “Applying Design Critics to Software Requirements Engineering.” Therequested budget is $36,000. The proposal describes the potential application of software designenvironments to the specific area of formal requirements.
List of Publications—Year 1
1. Robbins, J., Hilbert, D., Redmiles, D. Extending Design Environments to Software Architec-ture Design, Journal of Automated Software Engineering, to appear.
2. Kantor, M., Zimmermann, B., Redmiles, D. From Group Memory to Group AwarenessThrough Use of the Knowledge Depot, Technical Report UCI-ICS-97-20, Department ofInformation and Computer Science, University of California, Irvine, CA, May 1997.
3. Hilbert, D., Robins, J., Redmiles, D. Supporting Ongoing User Involvement in Developmentvia Expectation-Driven Event Monitoring, Technical Report UCI-ICS-97-19, Department ofInformation and Computer Science, University of California, Irvine, CA, May 1997.
21
4. Robbins, J., Hilbert, D., Redmiles, D. Using Critics to Support Software Architects, TechnicalReport UCI-ICS-97-18, Department of Information and Computer Science, University of Cal-ifornia, Irvine, CA, April 1997.
5. Shukla, S., Redmiles, D. Collaborative Learning in a Bug-Tracking Scenario, Workshop onApproaches for Distributed Learning through Computer Supported Collaborative Learning(Boston, MA), held in conjunction with the Conference on Computer Supported CooperativeWork (CSCW 96), Association for Computing Machinery, November 1996.
6. Robbins, J., Hilbert, D., Redmiles, D. Using Critics to Analyze Evolving Architectures, Sec-ond International Software Architecture Workshop (ISAW-2 — San Francisco, CA), held inconjunction with SIGSOFT’96: the Fourth Symposium on the Foundations of Software Engi-neering (FSE4), Association for Computing Machinery, October 1996.
7. Robbins, J., Hilbert, D., Redmiles, D. Extending Design Environments to Software Architec-ture Design, Proceedings of the 11th Annual Knowledge-Based Software Engineering(KBSE-96) Conference (Syracuse, NY), IEEE Computer Society, Los Alamitos, CA, Septem-ber 1996, pp. 63-72—Best Paper Award.
8. Robbins, J., Morley, D., Redmiles, D., Filatov, V., Kononov, D. Visual Language FeaturesSupporting Human-Human and Human-Computer Communication, Proceedings of the IEEESymposium on Visual Languages (Boulder, CO), IEEE Computer Society, Los Alamitos, CA,September 1996, pp. 247-254.
9. Robbins, J., Redmiles, D. Software Design From the Perspective of Human Cognitive Needs,Proceedings of the 1996 California Software Symposium (Los Angeles, CA), UCI IrvineResearch Unit in Software, Irvine, CA, April 1996, pp. 16-27.
List of Presentations—Year 1
1. Expectation-Driven Event Monitoring (EDEM), UC Irvine, CU Boulder, UMass Amherst,ARCADIA Consortium Meeting (Boulder, CO), presented by D. Redmiles, June 1997.
2. Supporting Ongoing User Involvement in Development via Expectation-Driven Event Moni-toring, CU Boulder, Meeting of the “Life-Long Learning and Design” Research Group (Boul-der, CO), presented by D. Redmiles, June 1997.
3. Argo: A Design Environment for Evolving Software Architectures, IEEE, Nineteenth Interna-tional Conference on Software Engineering (Boston, MA), presented by J. Robbins, May1997.
4. Supporting Evolutionary Design via Expectation Agents, DARPA, Design Rationale Cluster(EDCS DR) Meeting (Stone Mountain, GA), presented by D. Redmiles, March 1997.
5. Applying Usability Inspections to Web Page Design, UC Irvine, Bay Area Round Table(BART) Meeting (Palo Alto, CA), presented by D. Redmiles, February 1997.
22
6. Argo Software Architecture Design Environment, DARPA, Information Management Cluster(EDCS IM) Meeting (Manassas, VA), presented by D. Redmiles and D. Hilbert, October1996.
7. Using Critics to Analyze Evolving Architectures, ACM, Second International Software Archi-tecture Workshop (ISAW-2 — San Francisco, CA), held in conjunction with SIGSOFT’96:the Fourth Symposium on the Foundations of Software Engineering (FSE4), presented by J.Robins, October 1996.
8. Extending Design Environments to Software Architecture Design, IEEE, The 11th AnnualKnowledge-Based Software Engineering (KBSE-96) Conference (Syracuse, NY), presentedby J. Robbins, September 1996.
9. Visual Language Features Supporting Human-Human and Human-Computer Communica-tion, IEEE, The IEEE Symposium on Visual Languages (Boulder, CO), presented by J. Rob-bins, September 1996.
Web Site Guide
Project information and selected publications are accessible from the PI’s home page:
http://www.ics.uci.edu/~redmiles/
A direct link to the research funded under this grant is the following:
http://www.ics.uci.edu/pub/eden/
As noted earlier, some funding from DARPA is being leveraged in this work. A direct link toresearch funded under that project is the following:
http://www.ics.uci.edu/pub/edcs/
References
Abel, M. Experiences in an Exploratory Distributed Organization, in E.A. Galegher (Ed.), Intel-lectual Teamwork: Social and Technological Foundations of Cooperative Work, LawrenceErlbaum Associates, 1990, pp. 489-510.
Bogia, D. P., Kaplan, S. M.Flexibility and Control for Dynamic Workflows in the wOrlds Envi-ronment, Proceedings of the Conference on Organizational Computing Systems, 1995, pp.148-159.
Dourish, P., Bly, S.Portholes: Supporting Awareness in a Distributed Work Group, Proceedingsof the Human Factors in Computing Systems Conference (CHI ‘92), 1992, pp. 541-547.
Fischer, G.Cognitive View of Reuse and Redesign, IEEE Software, Special Issue on Reusability,
23
Vol. 4, No. 4, 1987, pp. 60-72.
Fischer, G.Domain-Oriented Design Environments, Journal of Automated Software Engineering,Vol. 1, No. 1994, pp. 177-203.
Greenbaum, J., Kyung, M. (Eds.).Design at Work: Cooperative Design of Computer Systems,Lawrence Erlbaum Associates, Hillsdale, NJ, 1991.
Guindon, R., Krasner, H., Curtis, B.Breakdown and Processes During Early Activities of Soft-ware Design by Professionals, in E.S. G.M. Olson, S. Sheppard (eds.), Empirical Studies ofProgrammers: Second Workshop, Ablex Publishing Corporation, Lawrence Erlbaum Asso-ciates, Norwood, NJ, 1987, 65-82.
Hayes-Roth, B., Hayes-Roth, F.A Cognitive Model of Planning, Cognitive Science, Vol. 3, No.1979, pp. 275-310.
Hilbert, D., Robbins, J., Redmiles, D.Supporting Ongoing User Involvement in Development viaExpectation-Driven Event Monitoring, Technical Report UCI-ICS-97-19, Department ofInformation and Computer Science, University of California, Irvine, CA, May 1997.
Isaacs, E., Tang, J., Morris, T.Piazza: a Desktop Environment Supporting Impromptu andPlanned Interactions, Proceedings of the ACM 1996 Conference on Computer SupportedCooperative Work (CSCW ‘96), 1996, pp. 315-324.
Kantor, M., Zimmermann, B., Redmiles, D.From Group Memory to Group Awareness ThroughUse of the Knowledge Depot, Technical Report UCI-ICS-97-20, Department of Informationand Computer Science, University of California, Irvine, CA, May 1997.
Kintsch, W., Greeno, J.G.Understanding and Solving Word Arithmetic Problems, PsychologicalReview, Vol. 92, No. 1985, pp. 109-129.
Lee, A., Girgensohn, A., Schlueter, K.NYNEX Portholes: Initial User Reactions and RedesignImplications, Proceedings of the International Conference on Supporting Group Work(GROUP ‘97—Phoenix, AZ), ACM, November 16-19, 1997, (in press).
Malone, T. W., Grant, K. R., Lai, K.-Y., Rao, R., & Rosenblitt, D. A.The Information Lens: AnIntelligent System For Information Sharing And Coordination, in M. H. Olson (Ed.), Tech-nological Support for Work Group Collaboration, Lawrence Erlbaum, Hillsdale, NJ,1989,pages 65-88.
Mantei, M. M., Baecker, R. M., Sellen, A. J.Experiences in the Use of a Media Space, Proceed-ings of the Human Factors in Computing Systems Conference (CHI ‘91), 1991, pp. 203-208.
Pennington, N.Stimulus Structures and Mental Representations in Expert Comprehension ofComputer Programs, Cognitive Psychology, Vol. 19, 1987, pp. 295-341.
24
Robbins, J., Hilbert, D., Redmiles, D.Extending Design Environments to Software ArchitectureDesign, Proceedings of the 11th Annual Knowledge-Based Software Engineering (KBSE-96) Conference (Syracuse, NY), IEEE Computer Society, Los Alamitos, CA, September1996, pp. 63-72—Best Paper Award.
Rumbaugh, J., Booch, G., Jacobson, I.Unified Modeling Language Reference Manual,Addison-Wesley, Reading, MA, 1997.
Schoen, D.The Reflective Practitioner: How Professionals Think in Action, Basic Books, NewYork, 1983.
Schoen, D.Designing as Reflective Conversation with the Materials of a Design Situation,Knowledge-Based Systems, Vol. 5, No. 1, 1992, pp. 3-14.
Visser, W.More or Less Following a Plan During Design: Opportunistic Deviations in Specifica-tion, International Journal of Man-Machine Studies, Vol. 33, No. 1990, pp. 247-278.
Wharton, C., Rieman, J., Lewis, C., Polson, P.The Cognitive Walkthrough Method: A Practitio-ner's Guide, in J. Nielsen, R. Mack (Eds.), Usability Inspection Methods, John Wiley &Sons, Inc., New York, 1994, pp. 105-140, Ch. 5.
Zimmermann, B., Lindstaedt, S., & Girgensohn, A.Growing Group Memories in the Workplace:A Field Study of GIMMe, Technical Report, NYNEX Science & Technology, White Plains,NY, 1997.
