ORIGINAL ARTICLE

D.B. Kaber Æ N. Segall Æ R. S. Green

K. Entzian Æ S. Junginger

Using multiple cognitive task analysis methods for supervisory controlinterface design in high-throughput biological screening processes

Received: 18 August 2005 / Revised: 17 January 2006 / Accepted: 30 January 2006� Springer-Verlag London Limited 2006

Abstract Cognitive task analysis (CTA) approaches arecurrently needed in many domains to provide explicitguidance on redesigning existing systems. This studyused goal-directed task analysis (GDTA) along withabstraction hierarchy (AH) modeling to characterize theknowledge structure of biopharmacologists in planning,executing and analyzing the results of high-throughputorganic compound screening operations, as well as thelab automation and equipment used in these operations.It was hypothesized that combining the results of theGDTA and AH models would provide a better under-standing of complex system operator needs and howthey may be addressed by existing technologies, as wellas facilitate identification of automation and systeminterface design limitations. We used comparisons of theGDTA and AH models along with taxonomies ofusability heuristics and types of automation in order toformulate interface design and automation functionalityrecommendations for existing software applicationsused in biological screening experiments. The proposedmethodology yielded useful recommendations forimproving custom supervisory control applications thatled to prototypes of interface redesigns. The approachwas validated through an expert usability evaluation ofthe redesigns and was shown to be applicable to the lifesciences domain.

Keywords Cognitive task analysis Æ Goal-directed taskanalysis Æ Abstraction hierarchy Æ Human–machineinterface design Æ High-throughput screening

1 Introduction

Historically, human factors professionals have beeninfrequently involved in the systems design process fromits early stages, such as system specification and con-ceptual design. However, they are frequently called in toretrofit existing systems in order to adequately accom-modate operator information processing needs, promotesituation awareness (SA), and enhance performance(e.g., redesign of nuclear power plant control rooms inresponse to the Three Mile Island incident 1979; Vicente2004). Cognitive task analysis (CTA), an analysis of theknowledge, thought processes, and goal structures ofcognitive tasks (Hollnagel 2003), has been used to designnew system interfaces, in cognitive engineering of hu-man–machine systems, to develop expert systems, foroperator selection, and for training purposes (Wei andSalvendy 2004). CTA can be a useful tool for identifyingoperator dynamic goal sets, factual knowledge stores,mental strategies, critical decisions, and SA require-ments as a basis for new human–machine interface de-sign, or for critiquing existing interfaces, processes, andsystems. However, there are few examples of CTAmethods being applied to existing systems redesign.

An example of CTA method use for new systemsdesign includes Riley and Endsley’s (2002) work withgoal-directed task analysis (GDTA) for analyzing mili-tary intelligence and logistics officer informationrequirements in battlefield course-of-action planningand logistics management. They used the results of thisanalysis to support development of design guidelines fornew logistics planning software for officers to supportfield combat operations. Bolstad et al. (2002) also usedthe GDTA methodology to analyze team SA require-ments of Army brigade officers. They projected how theresults of the analysis could be used as a basis for

D.B. Kaber (&) Æ N. Segall Æ R. S. GreenCognitive Ergonomics Laboratory, Department of IndustrialEngineering, North Carolina State University, Raleigh, NC, USAE-mail: dbkaber@eos.ncsu.eduTel.: +1-919-5153086Fax: +1-919-5155281

K. EntzianCenter for Life Sciences Automation, University of Rostock,Rostock-Warnemunde, Germany

S. JungingerAnalytical Instrument Group (AIG), e.V., Rostock-Warnemunde,Germany

Cogn Tech Work (2006)DOI 10.1007/s10111-006-0029-9

designing decision support tools to enhance team per-formance in tactical operating centers in coordinationand communication among troops. Endsley et al.(2003a, b) developed SA-oriented design, a process fordeveloping new systems with the goal of supportingoperators in achieving and maintaining good SA. Thedesign process involves creating a GDTA for SArequirements analysis. The SA requirements are thentranslated to a system design using SA-oriented designprinciples. Finally, the system is evaluated using SAmeasurement tools and changes are made as necessary.The SA-oriented design process was applied successfullyto the development of displays for Army command andcontrol operations. As another example of a CTAmethod used for new system design, Hajdukiewicz et al.(2001) used the abstraction hierarchy (AH) modelingapproach to represent an anesthesiologist’s work do-main—a patient—and provide a framework for sup-porting integrated medical informatics design. TheGDTA and AH methodologies provide useful infor-mation for defining new interface content, for support-ing operator understanding of how systems function,and for describing patterns of device use in certainoperations.

However, there is currently a paucity of CTA meth-ods that provide explicit guidance on redesigning exist-ing systems, although some tools can be used in theevaluation of such systems. For example, the criticaldecision method (CDM) (Klein et al. 1989) has beenused to evaluate existing expert systems in the militarydomain, but suggestions were not provided for reme-dying problems identified by the analysis. Few studieshave actually employed CTAs in redesign efforts; how-ever, those that have, report successful processes mea-sured by improved performance with new interfaces(e.g., Klinger and Gomes 1993; Lin et al. 1998). Forexample, Lin et al. (1998) used CTA to analyze thepsychological requirements of programming an existingpatient-controlled analgesia pump. They collected dataon the analgesia pump using bench tests (expert re-views), field observations and comments from nurses.This data were organized in function flow diagrams,decision-action diagrams, mapping of user input andpump display output, and an information requirementslist. The results of the CTA provided the researcherswith an understanding of the demands that program-ming tasks place on users and an indication of how theexisting interface failed to help users in effectively deal-ing with these demands. This analysis, along with anapplication of human factors guidelines (such as pro-viding consistent feedback), was used to design a newinterface for analgesia pumps. The redesigned interfacewas shown to be more effective and efficient than theexisting interface, while placing less workload demandson users.

In the domain of high-throughput screening (HTS;see High-throughput screening section) of biological andchemical compounds for new drug component devel-opment, human operators are subjected to many per-

formance and workload requirements in programmingexperiments, supervising automated systems, preventingerrors, and analyzing results. These requirements aremade more explicit in Table 1 and can serve as a basisfor systems design and selection of methods for analysis.Several CTA methods can potentially be used to en-hance existing interfaces in the HTS domain. These arealso listed in Table 1.

Referring to the table, it can be seen that the CDM isparticularly applicable to modeling dynamic taskscharacterized by high time pressure and informationcontent through the use of probes eliciting: (1) reasonsfor perceptual and conceptual discriminations, (2) typi-cality judgments, and (3) critical cues from domain ex-perts who have experienced non-routine incidents (Kleinet al. 1989). The GDTA is also well suited to complex,dynamic tasks. The result of the method, including a listof critical decisions and SA requirements, is useful foranalyzing domains in which operators are required tohave high SA to perform tasks effectively. The AHmethod of analyzing work domains is utilized to extractinformation requirements, constraining relationships,multivariate relationships, and means-end relationshipsthat can be used as a basis for interface design in tasksrequiring high human–automation coupling for optimalperformance. Finally, GOMS (goals, operators, meth-ods, selection rules) cognitive models describe the pro-cedural knowledge that an operator needs to have inorder to carry out tasks on a certain system (Kieras1999). Among the outcomes of this method are estimatesof task complexity and time to learn how to perform thetask. In this study, the GDTA and AH tools were se-lected for conducting a work domain analysis of HTSprocesses and making recommendations of system re-design guidelines to support human operator ease of useand task performance, as these methods comprehen-sively address the requirements in Table 1. Related tothis, the direct translation of CTA results to designguidelines has only recently received attention (Xu2005). Although the theoretical underpinnings of many

Table 1 Several cognitive task analysis methods appropriate forhigh-throughput screening

Operator requirements Applicable CTA methods

Manage high workload(time stress, dynamic environment)

CDMa, GDTAb

Achieve and maintainhigh SA

GDTAb

Synergize performance withautomated systems and robotics

AHc, GOMSd

Understand complex biochemicaloperations and process/automateddevice engineering needs

GDTAb, GOMSd

Process high informationcontent tasks and displays

CDMa, GDTAb

aCritical decision method (Klein et al. 1989)bGoal-directed task analysis (Endsley 1993)cAbstraction hierarchy (Rasmussen 1985)dGoals, operators, methods, selection rules (Card et al. 1983)

CTA tools, as well as methods for conducting them andrepresenting their outcomes, are usually well docu-mented, the formal procedure for applying the tools tonew interface and system design is often lacking.

1.1 Goal-directed task analysis

Goal-directed task analysis is an information require-ments assessment methodology developed by Endsley(1993) and originally demonstrated in the aviation do-main. GDTA has been used in many historical CTAs forSA requirements analysis in air traffic control (Endsleyand Rodgers 1994), supervisory control of advancedmanufacturing systems (Usher and Kaber 2000), and inanalyzing military command operations (Bolstad et al.2002). The method focuses on identifying operator per-ception, comprehension and projection requirements inperforming complex systems control. The results of aGDTA include lists of critical operator decisions and SArequirements that can be used as a basis for definingappropriate content of complex system informationdisplays, as well as for training program development,development of SA assessment measures, and operatorselection.

The general steps to conducting a GDTA includeidentifying the users’ major goals, identifying subgoalsto support overarching goals, identifying operationaltasks to achieve the subgoals, identifying questions aspart of decision-making in task performance, anddeveloping information requirements to answer thesequestions (Usher and Kaber 2000). This information iselicited from a domain expert in structured interviews.The experts are typically presented with a task scenarioand asked to mentally place themselves in the situationand to describe performance in the absence of anyexisting automated support systems. The analyst thencreates a goal tree (or outline) describing the informa-tion requirements, independent of the technology thatmay ordinarily be used (see Fig. 1). The analysis is based

upon operator goal states in the scenario and not onspecific states of the task environment or support sys-tems. For these reasons, GDTA is a good tool for sup-porting new systems designs to facilitate operatormanual control. These features of the GDTA also limitits potential for specifying design requirements forexisting interfaces, since the interface itself is not char-acterized as part of the analysis.

1.2 Abstraction hierarchy

Abstraction hierarchy modeling is the representation ofa work domain model in an event-independent mannerto promote operator ability to recover from unantici-pated error conditions in working with technology(physical devices, automation; Vicente 1999). AH mod-els are hierarchical, structural models consisting ofmultiple levels of abstraction (Rasmussen 1985). At thehighest level, the models define the purpose of technol-ogy in the work domain. The lowest level of an AHmodel represents the physical components of a system.In between, generalized functions of the system arepresented. Linkages among the levels represent how thepurpose of the system is implemented through specificdevices and they provide an explanation of why certaincomponents are needed to achieve a system purpose(Rasmussen et al. 1994). In addition, an AH of a workdomain can serve as a framework for describing thecontrol tasks required to maintain adequate systemoperation. Causes of errors are explained ‘‘bottom-up’’in the levels of abstraction, while reasons for propersystem functioning can be derived ‘‘top-down’’ from thefunctional purpose (Rasmussen 1985).

An AH model is typically presented using a grid ofthree columns and five rows (see Fig. 2). The columns(from left to right) present a part-whole decompositionof the work domain to systems, subsystems, etc. Therows (from top to bottom) present functional (abstrac-tion) decomposition of the system from the overall

1.0 Major Goal

1.1Sub-goal

1.2Sub-goal

1.3Sub-goal

Decision Decision Decision

SA Requirements:Level 3 – Projection

Level 2 – ComprehensionLevel 1 - Perception

SA Requirements:Level 3 – Projection

Level 2 – ComprehensionLevel 1 - Perception

SA Requirements:Level 3 – Projection

Level 2 – ComprehensionLevel 1 - Perception

Fig. 1 Goal tree model forGDTA (courtesy MelanieC. Wright)

purpose, through generalized functions, to the physicalcomponents supporting the functions. Means-end con-nections are presented across the rows.

The results of AH modeling can serve as a basis fordeveloping automation technology and interfaces forhelping operators manage system states, when coupledwith information on task strategies and operator deci-sion-making. Using an AH representation of a processcontrol system (PCS), as a basis for new interface design,has been shown to improve operator performance (Vi-cente 1992). AH models can also be utilized to developsystem user manuals and training programs to educateoperators on connections between system functions andinterface controls.

Similar to GDTA, although the AH may be appro-priate for guiding new interface development, it may notbe sufficient for supporting redesign of existing inter-faces. When an AH is used to describe an existing sys-tem, since the method does not focus on task events,system functional capabilities are not presented in aspecific context, making it difficult to identify relativelimitations and the need for design enhancements. Forexample, in the domain of life sciences, a wide variety ofdevices and automation are currently utilized in high-throughput molecular compound screening for drugdevelopment. Different robotic and analytic measure-ment technologies are used in various enzyme- orchemical-based tests and for the different steps withineach test (e.g., micro-culture plate handling/transporta-tion, liquid transfer and dilutions). Consequently, theextent to which an operator’s information needs for

carrying-out specific tasks are supported by existinginterface technology will also vary. Being an event-independent tool, the capability of an AH to revealdifficulties in specific task performance is limited. Themethod does not relate operator information needs toSA requirements for task performance using existingsystem features. Its efficiency as a reasoning mechanismmay also be constrained for this reason (Bisantz andVicente 1994).

This limitation was made evident in Xu’s (2005)study, the goal of which was to identify problems andneeds in automated flight decks. Xu developed an AH ofthe work domain—pilot–automation interaction (PAI).Four data sets (pilot surveys, PAI-related incidents, datafrom a study on automation-assisted pilot performance,and pilot mental model test data from the same study)were mapped onto the AH, according to the locations ofthe issues in the functional structure of the problemspace. The mapping of the data revealed several PAI-related problems (e.g., some pilots’ mental models werefound to be inaccurate and inefficient) with most pilotconceptual gaps occurring in higher-order functionalproperties areas. Existing interfaces were also found tobe lacking in terms of displaying higher-order functionalinformation. In Xu’s study, an AH alone was not suffi-cient to identify problems within the work domain; onlyin tandem with information from multiple data sets wasthe technique useful for finding automation limitations.Unfortunately, the results of this study were not used asa basis for interface design recommendations for existingaircraft automation.

Goal or purpose

“Highest level of abstraction in model”

General constraints affecting system goals“Abstract function level”

Generic functions of system

Components of system required to achieve functions

“Generalized function level”

“Physical function level”

(Includes identification of component parts.)

(Includes identification of system components.)

(Describes how objects of system relate to achieving higher- level functions.)

“Component level”(refer to components in descriptions)

Fig. 2 General form of anabstraction hierarchy model

Related to this work, Bisantz et al. (2003) definedlevels of automation (LOA), human roles, and initialdisplay prototypes for a US Navy surface combatant(ship) during the early, conceptual phase of its designprocess. They developed both AH models of a subsetof the ship’s systems and a series of cross-referencedmatrices that described system functions, operatorcognitive functions, system engineering decomposition,and display requirements to generate operator roledefinitions, information requirements, and generalinterface design recommendations. Again, the AHmodeling approach was combined with other taskanalysis methodologies to serve as a basis for newinterface design requirements.

2 Objectives and hypotheses

There are few descriptions of systematic approachesfor the translation of CTA results, such as GDTAs,AH models, decision ladders, etc. to system andinterface design guidelines to support human perfor-mance. Szczepkowski et al. (2005) demonstrated howCTA methods can be used to support work-centereddesign approaches. They showed how the applicationof CTA methods could be used to identify conditionsunder which certain automation capabilities may beused by operators to perform complex tasks. However,the study did not address the development of softwareor interfaces. Another study by Woods (2005) identi-fied basic requirements for cognitive work design andcriteria to determine when automation is needed. Thecriteria may also be used to assess existing systems.The need for translation of CTA outcomes to tangibletechnological design was identified. Beyond the needfor approaches to translating CTA results to designrequirements, methods like GDTA and AH have tra-ditionally targeted new systems design and have notresulted in explicit guidance on redesigning existingsystems.

In this research, a structured methodology wasproposed for the formulation of interface design andautomation function guidelines for existing systems,using the combination of results from GDTA and AHmodels. The methodology was applied to the domainof life science processes. It was expected that theintegrated GDTA and AH approach would provide acritical understanding of how complex system operatortask needs may currently be addressed by existingtechnologies. Furthermore, taking advantage of thefeatures of each tool was expected to allow for iden-tification of system shortcomings and future designenhancements. Using a combination of the tools wasexpected to provide a strong basis for design aimed atpromoting operator SA, performance and satisfaction.Thus, it was also hypothesized that a system rede-signed according to the proposed CTA methodologywould prove superior to its original counterpart in

usability/performance evaluations, as carried out byexperts in the life sciences domain.

3 Methods

3.1 High-throughput screening

In this research, we applied GDTA and AH modelingmethodologies to supervisory control of high-through-put (biological) screening (HTS) processes used in drugderivative development for the purpose of establishingnew process control interface design guidelines thatwould support operator SA and performance. Contem-porary HTS processes involve chemical-based assays oforganic and inorganic compounds for effects on humancellular functions, or enzyme reactions that are commonin cells (Entzian et al. 2005). These processes, which areconducted in sterile laboratory or clean room settings,have become highly automated in recent years (seeThurow et al. 2004). Automation increases the pace atwhich organisms can be tested for potential uses indevelopment of biocatalysts to be used in new drugdevelopment or in industrial products, such as enzymesfor detergents, etc. (Comfort et al. 2004).

The particular application we studied involved en-zyme-based (Trypsin) testing of marine compounds atthe University of Rostock (Germany), Center for LifeSciences Automation (CELISCA), to identify candidatecompounds useful for drug development. Completeautomation of this particular process includes integra-tion of an optimized robot for chemical analysis(ORCA) with analytical measurement devices on aprocess line (see Fig. 3 for a picture of one of the bio-logical screening lines at CELISCA). The robot is cus-tom designed for the screening process and can beconfigured on demand for specific experiments (see ro-bot manipulator (#4) in Fig. 3). The integrated devicesinclude: a bar coder for labeling micro-plates in whichcompounds are tested and for tracking the micro-plates;automated pipetting (liquid transfer) systems for fillingmicro-plates with enzyme substrates, test compoundextracts and other reagents; an incubator for bringingthe temperature of reactions on the micro-plates tonormal human body temperature; and an automatedmicro-plate reader for analyzing the enzyme activity, orcellular reaction, in each plate well (each culture) (e.g.,luminance or fluorescence tests of light reflected offsamples).

The ORCA, which is programmed, controlled andmonitored from a central PCS, transports micro-platesto and from the various workstations on the line. ThePCS also manages the coordination of machine activities(controlled by their own local controllers). Each line hasthe capacity to process several hundred compound ex-tracts as part of a single experiment (job). Typical pro-duction volumes include thousands of samples processedover 2–3 day periods.

One supervisory controller typically operates theHTS process by planning and programming the robotictasks. An additional lab technician works on deliveringmicro-plates, chemicals, plate labels, and pipetting re-sources (tip boxes, reservoirs, etc.) to the process line.Although there are some health and safety risks fortechnicians in such tasks, the operations are currentlyhandled by humans because existing automated guidedvehicle technologies are not considered to be reliable oraccurate enough for such material handling.

A typical HTS experiment poses a high cognitiveworkload for supervisors, who must keep track of thetiming of process steps, whether chemical reactions areoccurring safely, and whether robot motions are accu-rate. Robot material handling errors may occur in thescreening process or chemical reactions may not pro-gress as planned because of operator programming er-rors, leading to delays in reactions or the need to scrapan entire experiment. This is costly to the test facilitybecause many of the organisms being investigated areextremely rare and the extracts are expensive to develop.Therefore, operators may need to periodically intervenein the master robot control in order to prevent a linefrom ‘‘crashing’’. The process control interface mustprovide the capability for near real-time, closed-loopprogramming of process events based on the progress ofan experiment.

3.2 Development of GDTA model

A GDTA was developed to describe the detailed steps aspart of planning, controlling and analyzing the auto-mated assay of marine compounds for the potential toaffect human cellular functions (or enzymatic reactionstypically occurring in cells). Our approach to the GDTA

involved an analyst interviewing expert biopharmacol-ogists (at CELISCA) to describe knowledge structuresrelevant to supervisory control of the HTS line. Initially,procedural task analyses were performed on the pro-gramming of basic HTS methods using existing com-mercial off-the-shelf software (e.g., Beckman-Coulter’sSAMI� software). These analyses identified the generalsteps as part of the enzyme-based screening test. Beyondthis, the experts provided background information onthe basics of enzyme reactions and the use of micro-plates for conducting life science experiments.

These analyses were used as stepping-off points forthe GDTA, which focused on biopharmacologist adap-tation of a ‘‘bench-top’’ screening assay to a HTS line.The major goal of the biopharmacologist was identifiedas the discovery of compounds with the potential fordevelopment into drug derivatives. A subgoal as part ofadapting the bench-top method to the HTS line is toidentify which types of micro-plates are to be used in theassay. The specific tasks to this subgoal include, forexample, determining the best well configuration forsample plates and determining whether the volume oftest micro-plates is acceptable for the assay. Criticalquestions or operator decisions were also elicited. Forexample, in determining the best well configuration, thebiopharmacologists may ask, ‘‘Is micro-plate capacitycritical to this assay?’’, ‘‘Is access to the greatest wellvolume critical to the assay?’’ and ‘‘How can the amountof test compound required by the process be reduced?’’Finally, biopharmacologist SA requirements weredeveloped to answer these questions. The requirementstargeted the three levels of SA, perception (observationof the system and its environment), comprehension(understanding of the system and environment relativeto task goals), and projection (prediction of future sys-tem and environment states) identified in Endsley’s

Fig. 3 HTS robotic system forenzyme-based testing ofcompounds

(1995) model of SA. The SA requirements for the deci-sion whether access to the greatest well volume is criticalas part of the task of determining the best well config-uration, included: (1) the total volume of liquid to betransferred to micro-plates; (2) knowledge of the rela-tionship between well configuration and access to extractvolumes (e.g., ‘‘V’’-shaped wells in plates provide accessto the greatest ‘‘death’’ volume, as compared to flat-bottom wells).

Examples of questions asked by SA analysts of thebiopharmacologist during the knowledge elicitationphase of the GDTA are presented in Table 2. (Wepresent additional details on the overall goal structure ofbiopharmacologists in screening operations and specificSA requirements in the Results section.)

3.3 Development of AH models

Our AH modeling efforts targeted the physical deviceson the HTS line, the device user interfaces (proprietarycontrol software), and the line method editor softwarenecessary to carry out the Trypsin test. In the line setupat CELISCA, software developed by manufacturers ofeach HTS device (i.e., the bar code print and apply de-vice, micro-plate incubator, ORCA, automated pipett-ing device, and automated micro-plate reader) is used toprogram device methods. The devices on the line areintegrated through the PCS that includes a screening line‘‘method editor’’ application, which communicates withthe user interface of the proprietary device softwarethrough an ‘‘executive’’ software controller. The linemethod editor includes standard Windows action/con-figuration dialogs for all devices on the HTS line. Thesedialogs are used during system programming to set de-vice parameters, as part of a screening method, and tosend data to device software modules via the executivesoftware controller.

The AH models characterizing the physical devicesincluded a part-whole decomposition of each device tospecific components. A functional decomposition of thedevices also described the purpose of specific equipment,the constraints affecting this purpose, a device’s gener-alized functions, and its physical components. Forexample, the purpose of the pipetting device is to opti-mize liquid handling in biological analysis; its generalconstraints are accuracy and capacity in liquid transferamong labware; this is achieved using functions such aspipetting liquid from one micro-plate to another; andthe components for pipetting include a gripper, pipettingtools, mechanical tool holders, etc. (We provide moredetails on our AH models of physical devices andautomation in the Results section.)

Historical research has primarily applied the AHmethodology to physical devices (e.g., Rasmussen 1985).Only in contemporary work has AH modeling beenapplied to automation. As reported, Xu (2005) devel-oped an AH model of flight deck automation to analyzePAI. This is also a unique aspect of the method used inthe present study. Through the AH modeling process,connections between physical devices and automatedcontrol systems (or software) can be made by linking thefunctions, subsystems, and components of automationto those of the physical device. These types of connec-tions can be useful for providing knowledge of themeans-end relationships between levels in a model oracross automation and device models, as a basis forsystem state diagnosis. In addition, the visualization ofcausal relationships between automated componentsand functions can be important to operator training.For example, links between an AH model of the barcoder and the software used to control the device canclarify which parts of the bar coder are controlled byautomation, and to what degree. This may help show theuser the extent of the automation, which can be useful insystems with multiple modes and degrees of control.This approach to modeling the automation and thephysical device it automates can provide more infor-mation about the system, specifically the structure of theautomation and the control mechanisms employed.

We adapted the AH modeling methodology to rep-resent software for controlling devices on the screeninglines at CELISCA. In our automation models, the col-umns (from left to right; see Fig. 2 for reference) presentdecomposition of the control software, as a whole, topresentation of specific functions. The rows (from top tobottom) present a functional decomposition of the spe-cific application from the overall purpose, throughgeneric control functions, to the software options andsettings supporting functions. Although the lower rowsof an AH model are typically used to present physicalcomponents of a system, instead of identifying softwaresubroutines and the inner-workings of computer code,as a basis for control function explanation, we decidedthat from a user perspective, it would be more mean-ingful to reveal the interface features and optionsthrough which subroutines of the software are made

Table 2 Sample questions for GDTA knowledge elicitation

SA level Question

1. Perception What information within the environmentdo you need to attend to in order toaddress critical task decisions?

2. Comprehension How are your perceptions of system statesand the environment related to your goals?What information do you need to integrateto understand the meaning of stateof the system?What do certain pieces of informationmean with respect to your capabilityto execute a method?

3. Projection Based on your observations of statesof the system and environment,and understanding of what this informationmeans to achieving task goals, whatprojections can you make about futurestates of the system?What projections should you make aboutfuture states of the system and environment?

accessible. This also ultimately allowed for a clearerintegration of the AH automation models with the de-vice models from an operator perspective and identifi-cation of specific interface design problems relative tobiopharmacologist task performance. Similar to thedevice AH models, means-end connections were pre-sented across the rows of the automation models.

The method for developing the AH models in thecurrent study involved structured interviews with anexpert process engineer at CELISCA to elicit device andautomation purposes and functions. In addition, visitsto the HTS line were conducted in order to inspect theexisting technology. The components of the differentdevices, as well as the various software configurations,were documented and verified by the engineer for theanalysts.

3.4 Use of taxonomies to classify designrecommendations

In order to facilitate the process of making recommen-dations for supervisory control interface designenhancements, we used a taxonomy of human–computerinteraction design heuristics presented by Nielsen (1993)to identify aspects of usability important in good inter-face design and to classify our specific design recom-mendations for the HTS line automation. Nielsen’sheuristics include: (a) visibility of system status; (b)match between the system and the real world; (c) usercontrol and freedom; (d) consistency and standards indesign; (e) error prevention; (f) recognition of informa-tion versus requiring users to recall; (g) flexibility andefficiency of use; (h) using parsimonious design; (i)facilitating error detection and recovery; and (j) pro-viding accessible help utilities. The evaluation of existinginterface features and options was guided by theseheuristics, in order to ensure a comprehensive search forshortcomings that would limit usability and operatorperformance.

Furthermore, we referenced the model of types andLOAs by Parasuraman et al. (2000) for classifying anyrecommendations related to the underlying functionalityof the automation (software). Parasuraman et al. (2000)identified four general types of automation based onforms of human information processing in complexsystems control, including: (a) information acquisition;(b) information analysis; (c) decision-making; and (d)action implementation. The evaluation of existingautomation functionality was guided by considering theextent of computer assistance of operators in each aspectof information processing in the steps of the HTS pro-cess.

3.5 Generation of design recommendations

To begin the formulation of interface design guidelinesfor enhancing the existing software applications, all

goals included in the GDTA, which required task per-formance with the current software, were identified. TheAH automation models relevant to the identified goalswere then used to establish lists of interface actions thatoperators would need to perform to achieve the goals.The heuristics and LOA taxonomies were used to sys-tematically examine the existing software, as describedby the AH model, for features or functions not sup-portive of operator goals and components inadequatefor usability needs. For some tasks, it was intuitivelyobvious that no heuristic or LOA would be applicable.For example, in the present study, help documentationfor software used to control the automation on the HTSline at CELISCA was limited; consequently, the finalheuristic in Nielsen’s (1993) taxonomy, provision of helputilities, was often not relevant, in which case our searchproceeded to the next heuristic or LOA. Finally, whenconsideration of a heuristic or LOA led to identificationof an aspect of the existing software, which did notsupport the usability or functionality required by anoperator goal (according to the GDTA and interfaceactions identified based on the AH model), a designrecommendation was generated. In summary, the fol-lowing general steps can be taken to formulate designrecommendations using GDTA and AH models:

1. Identify goals in the GDTA that require interactionwith devices or automation.

2. Compile a list of (interface) actions necessary toachieve each goal using relevant AH models.

3. Examine the action list (and interface screenshots)using heuristics to find components that do not sat-isfy usability principles.

4. Examine the action list by considering levels ofinformation processing automation to find compo-nents that do not support operator goals.

5. When a mismatch between a certain interface featureor automation component and usability or func-tionality requirement is identified, formulate anappropriate design recommendation.

Figure 4 presents a flow diagram depicting the link-ages among the steps and the outcomes of this process.

3.6 Preliminary validation

In order to validate our proposed methodology, weredesigned one of the software interfaces used to controlthe HTS line under study—the action/configurationdialog for the barcode print and apply device—using thedesign recommendations generated from the processdescribed above. A prototype of the redesigned interfacewas constructed, and its usability was evaluated by ex-pert biopharmacologists and engineers at CELISCA. Inorder to assess the usability of the redesigned interface,as compared to the original interface, evaluators per-formed a common task of selecting and configuring abarcode label to apply to assay micro-plates, using bothinterfaces. They then completed a questionnaire previ-

ously developed for global assessment of systemsusability (Brooke 1996) for each interface and providedcomments on the advantages and disadvantages of theinterfaces. The new bar coder interface and the evalua-tion results are described below.

4 Results

Our application of the methodology described above toHTS operations at CELISCA for testing of biologicalcompounds resulted in: (1) a GDTA describing all bio-pharmacologist goals and tasks in carrying-out the HTSprocess; (2) AH models of all devices and control soft-ware on the HTS line; (3) a list of guidelines forenhancing the existing software interface designs andautomation functionality; and (4) preliminary proto-types of software interface redesigns for usability andperformance testing.

For the GDTA, we initially developed a compre-hensive outline of biopharmacologist goals, tasks, deci-sions and information requirements based on thestructured interviews. Subsequently, a collection ofhierarchical diagrams of the GDTA was created topromote ease of referencing by analysts and experts inverification of the goal structure. Figure 5 presents adiagram of the overarching goal and all major subgoals,as part of discovering compounds leading to drugderivative development. The complete GDTA included25 high-level goals, 20 subgoals, and a total of 88 tasksto these subgoals. On average, there were 4.4 tasks to

each subgoal and 2.2 critical decisions per task. Ananalysis of the SA requirements associated with thevarious biopharmacologist decisions revealed a total of228 unique pieces of task information. Figure 6 presentsin outline form the tasks, decisions, and information(SA) requirements for Goal 1.1.9.1 in the GDTA, whichdeals with the application and reading of barcode labelsduring the HTS process. This goal involves two tasks:integrating the bar coder into the process (in the PCSsoftware) and determining how it will be used during theassay. Two decisions must be made by the operatorwhen integrating the bar coder: what information will beincluded in the label and where the label will be appliedto a micro-plate. The information required to addressthese decisions is the code that will be used on the label.To achieve the second task, that of determining thefunctions of the bar coder, the operator must decidewhether a new barcode needs to be applied to micro-plates or whether an existing barcode is to be read. Inaddressing this decision, the operator needs to knowwhether a barcode is already present on the micro-plates(from the manufacturer or client) and what step is tofollow bar coding in the assay process. To facilitate datacollection (using the micro-plate reader), for example, itis first necessary to record the content of the barcodelabel.

The equipment for which AH models were con-structed includes the bar code print and apply device,automated pipetting system, incubator, ORCA, andautomated micro-plate reader. In addition, AH modelswere created for all proprietary software and action/

GDTA

Li

com are

nk of purpose of automation to equipment functions and

ponents. Description of how automation functions

implemented through software and the purpose of

interface features

Assess consistency with

usability heuristics

List of recommendations to

enhance automation

List of recommendations to

enhance interface

Integrate design

recommendations in database

Assess extent of applicability of

automation to operator goals

Compare outcomes of modeling

techniques

(Evaluate existing technology for

supporting operator SA and performance

through usability and functionality)

Critical decisions and

information requirements

AH Models Fig. 4 Flow diagram of designrequirements generationprocess for HTS domain

configuration dialogs, as part of the PCS software thatallowed for specific device configuration and manipula-tion. Although we developed AH models of the barcoder and incubator equipment as part of the overallCTA, this equipment has no proprietary control soft-ware. Specifically, AH models were created for thecontrol software of the pipetting device and micro-platereader, and for the PCS action/configuration dialogs forthe bar code device, incubator, micro-plate reader, andpipetting device.

The example HTS process Goal 1.1.9.1 (facilitatingmicro-plate labeling and reading) is supported by the barcoder device. Figure 7 presents the AH model for thebar coder. Following the links from the top of the AHdiagram down to the bottom, an operator can discoverhow the functions of micro-plate labeling and readingare implemented by the bar coder components. Fol-lowing the links from the bottom of the diagram to thetop, an operator can learn why the various subsystemsor components as part of the bar coder, such as the

1.Find “hits” among a huge number of

compounds as fast as possible and at low cost

1.1.Adapt “bench-top” method to High-

throughput Screening (HTS) line

1.2.Ensure results of

assay meet quality criteria

1.3.Optimize assay

method for efficiency

1.4.Conduct accurate

analysis of data from assay

1.5.Generate

understandable reports of results for

clients

1.2.1.Identify criteria for

quality factors

1.2.2.Calculate quality

statistics based on sample data

1.2.3.Compare test statistics with

criteria

1.2.4.Assess quality of experimental runs

1.3.1.Avoid bottlenecks

1.3.2.Determine optimal (plate) batch size

1.3.3.Establish optimal sequence of steps

in assay

1.3.4.Optimize pipetting

steps

1.3.5.Identify available work-in-process

storage space

1.4.1.Establish criterion activity level for identifying “hit”

1.4.2.Use basic statistical analysis to identify

and address outliers in data

1.4.3.Determine enzyme activity levels (e.g., Trypsin) (conduct

data analysis)

1.4.4.Determine

variation in enzyme activity levels

1.4.5.Decide whether

test compound is active

1.5.1.Address customer information needs for decisions on test compounds (i.e., to support

determination of whether further investigation of compounds is

needed)

1.1.1.Identify steps that need to be

performed as part of (automated version of) assay

1.1.2.Identify appropriate micro-plate

types for assay

1.1.3.Establish plate configuration to achieve statistically valid results

1.1.4.Identify automated devices to use

to perform steps of assay

1.1.5.Identify time critical steps as part of assay as basis for sequencing

steps

1.1.6.Identify feasible sequences of steps

in assay that allow for successful execution of experiment (i.e., Is

there any flexibility in sequence of steps in method?)

1.1.7.Adapt manual pipetting steps to

automated version of assay

1.1.8.Design measurement approach to facilitate analysis of inhibition of

enzyme activity by sample compounds

1.1.9.Develop program for assay method

using HTS line control software (e.g. Beckman Coulter-SAMI)

1.1.10.Ensure reproducibility of results

from HTS line

Fig. 5 High-level goals as part of HTS GDTA

1.1.9 Develop program for assay method using HTS line control software (e.g., Beckman Coulter(Note: In general, the assay method can be divided into three major steps, includinglabeling micro-plates, preparing sample plates, and conducting the enzyme-based test.)1.1.9.1 Facilitate plate labeling and reading:

T1 Integrate bar coder and reader into method.What type of information content is to be included in the label?What label position is to be used?

Need – Meaningful code representing plate content.T2 Determine functions of barcode device to be used during assay.

Does the assay require reading or print and apply functions?Need – Presence of barcode on plate.Need – Step in assay method that comes next (e.g. data collection)

SAMI):-

Fig. 6 Goal 1.1.9.1 andassociated tasks (in italics),decisions (in the form ofquestions), and informationrequirements

printer and the label paper feeder, are necessary for thespecific functions of micro-plate labeling.

The main screen of the PCS action/configurationdialog for the bar coder can be seen in Fig. 8. The AHmodel for this software is presented in Fig. 9. Followingthe links from the top of the diagram down to the bot-tom, an operator can discover how the software func-tions of labeling and reading micro-plates areimplemented through the interface controls and fea-tures. Following the links from the bottom of the dia-gram to the top, an operator can learn why the interfacefeatures, including selecting among label format optionsand selecting among options for micro-plate sites, existas part of the software to support the function oflabeling micro-plates.

Once the GDTA and AH model representations werecomplete, we compared them in order to relate bio-pharmacologist goal structures and critical decisions (aspart of the GDTA) to the purpose of the automatedsystems on the HTS line and functions and components(as part of the AH models). We used these comparisonsto formulate our interface design and automation

functionality recommendations for the existing softwareused in the screening process at CELISCA. These rec-ommendations were organized according to Nielsen’s(1993) usability framework and the framework of typesof automation proposed by Parasuraman et al. (2000).

Figure 10 directly relates the content of the GDTAfor the goal of facilitating micro-plate labeling andreading to the interface action list for use of the barcoder action/configuration dialog, based on the AHmodel. Table 3 presents the interface design recom-mendations aimed at enhancing the bar coder action/configuration dialog to better support the target goal offacilitating micro-plate labeling and reading. Forexample, the heuristic ‘‘match between the system andthe real world’’ calls for using terminology that isfamiliar to the user, versus using system-oriented terms(Nielsen 1993). In line with this heuristic, we recom-mended clarifying for operators the meaning of differentbarcode label formats, such as ‘‘Code 128’’. To assistbiopharmacologists in their decision-making, it wasrecommended that the software suggest to the user adefault label configuration for printing and applying a

Rotating feed motor

Feed reel for labels

Take-up reel for labelsFeed reel for foil

Take-up reel for foil

Reel position sensors

Process of system initialization

Process of printing bar code

Process of applying bar code

Process of reading bar code

Printer

Foil feeder

Vacuum gripper

Plate holder

Laser scanner

Process of verifying available paper/foil Process of moving plate

holder to home position Process of paper/foil feed

Process of applying thermal ink to labelProcess of removing label from backing paper Process of locating label position

and blowing label on micro-plateProcess of verifying label

Process of activating reader (scanner)

Process of moving plate for scanning Process of positioning

plate holder (begin/end)

Thermal print head

Vertical translation motor

Laser diode

Rotating (scan) mirror

Receiver (photo diode)

Vacuum pump

Rotation motor

Translational motor

Rotational control motor

Translational control motor

Label paper feeder

Label and read micro-plates

RS-232 C connection

CPU

RS-232C connection to PC

Apply and read controller

CPU

RS-232C connection to PC

RS232C to printer

and recognize, micro-plates

Manual entry keypad

LCD display

Specific configuration of bar coding system components

Assign ID to,

Fig. 7 AH model for bar coder device

new barcode, based on previously defined and storedlabel configurations for enzyme-based screening tests. Itwas also recommended that based on the presence of anexisting barcode on the micro-plate, the system wouldsuggest the appropriate action for handling the micro-plate (i.e., printing and applying a new barcode vs.reading an existing barcode). Similar recommendationswere formulated for all other goals in the GDTA thatpertained to the use of software or devices, as part of theautomated enzyme-based assay of compounds.

A new prototype interface for the bar coder action/configuration dialog was developed to address the rec-ommendations described in Table 3 and enhance itsusability (see Fig. 11). With the redesigned interface,users are able to select one of several different label typesand then drag-and-drop the label onto a rotatable pic-torial representation of a micro-plate in order to specifyits location. This is more efficient than programmingplate holder rotations and offsets using text entry fieldsfor proper label positioning. The interface then promptsthe user for the label content, based on previously storedlabels. This interface was considered to be more intui-tive, with the removal of obscure terminology andreorganization of functions, based on common operatortasks.

The original interface and the new prototype wereevaluated by five subject-matter experts (SMEs) atCELISCA, including biopharmacologists and processengineers, whose ages ranged between 30 and 40. Theywere asked to rank the interfaces, on a scale from 1(‘‘low’’) to 5 (‘‘high’’), in terms of ten characteristics(complexity, consistency, learnability, ease of use, etc.),and to comment on their shortcomings and advantages.Overall, they found the new interface to be better thanthe original interface, with average scores of 4.04 and3.24, respectively. Among the advantages of the newinterface were its intuitiveness and the fact that it wasmore interactive, as compared to the original interface.The main disadvantage that the SMEs noted for the newinterface was that the graphical depiction of the micro-plate was not considered realistic enough.

5 Conclusions

This research represents one of the first studies to pro-vide a systematic approach for translating CTA resultsinto design recommendations for existing systems. Italso presents a novel use of AH modeling for repre-senting automated control systems for physical devices.

Fig. 8 Bar coder action/configuration dialog inBeckman-Coulter’s SAMI (R)software

Manual string entry

Read strings from file

Manual entry of label

prefix and suffix with

auto indexing of string

Select among label

format options (Code

128, Code 39, text only) Select “Update”

option

Select among

options for plate

sites (A, B, C, D)

Define global

operation of device

Printing control

Label and apply

control

Reading control

Select type of bar codeStore type of

bar codeDefine content of bar code

Select label position

on microplateSet offsets for label height

(relative to top/bottom of plate)Set frequency of bar code

verification (or none)Select plate position

for reading Set parameters for

addressing reading errors Set parameters for verification errors

(reading previously labeled plates)

Label and read micro-plates

Print and apply

device software Elect use of internal

system transport ID

Select among action

options (Automation,

Read, Manual Use)

Manual entry of

position of code

on label (in mm)Select among “Read” or

“No-Read” options

Manual entry and store

number of read periods Select among reading error

handling options (log error,

flag error and stop) Select among verification error handling

options (i.e., errors during verification

of bar code on plate against system ID -

Ignore, or log and stop)

Control

mechanisms of

bar coder

Select (global)

action

Specific configuration of bar coder control software.

Fig. 9 AH automation model for bar coder action/configuration dialog

1.1.9 Develop program for assay method using HTS line control software (e.g., Beckman Coulter - SAMI): (Note: In general, the assay method can be divided into three major steps, including labeling micro-plates, preparing sample plates, and conducting the enzyme-

based test.) 1.1.9.1 Facilitate plate labeling and reading:

T1 Integrate bar coder and reader into method. What type of information content is to be included in the label?What label position is to be used?

Need – Meaningful barcode representing plate content.

T2 Determine functions of barcode device to be used during assay.

Does the assay require reading or print and apply functions?

Need – Presence of barcode on plate.

Need – Identification of step in assay method that follows bar coding (e.g., data collection)

Task 1: Define content of barcode options:

Manual string entry Read strings from file Manual entry of label prefix and suffix with auto indexing of string Elect use of internal SILAS transport ID

Select label position on micro-plate options: Side A Side B Side C Side D

Set offsets for label height options: Manual entry of position of code on label

Task 2: Select action options:

Automated Manual Read

Fig. 10 Relation of GDTA content to interface action lists based on AH model for Goal 1.1.9.1

The integrated CTA methodology proposed in thisstudy involved exploiting of the strengths of GDTA andAH modeling in order to generate interface design andautomation functionality recommendations for existingsystems. The approach has been shown to be applicableto the life sciences domain, specifically for supervisorycontrol of automated biological screening line systemsfor identification of potentially useful organic com-pounds in new drug development. A comprehensiveGDTA describing biopharmacologist goals, tasks, deci-sions, and information requirements (necessary toachieve the goal of enzyme-based testing of compounds)was compared with AH models describing the devicesand automation utilized in the screening process. High-level software design recommendations were generatedregarding system functionality by using the results ofthis comparison along with taxonomies of usabilityheuristics and types of automation. Low-level designguidelines were generated in the same way, with thepurpose of enhancing interface features. These designrecommendations revealed that operator task needswere not adequately addressed by existing technologies.

The recommendations were subsequently used to de-velop a new prototype, which domain experts found tobe superior to the original interface.

5.1 Caveats

There are several limitations to the work presented here.With respect to constructing cognitive work domainrepresentations, GDTA and AH modeling require sig-nificant time investments on the part of analysts andSMEs. It is difficult to develop an AH model and itsresult may depend on the expert’s knowledge andunderstanding of the system (Bisantz and Vicente 1994).However, these limitations hold true for many CTAtools, like the CDM (Klein et al. 1989).

It would have been possible to identify supervisorycontrol interface design limitations using Nielsen’s(1993) taxonomy of heuristics, alone. Heuristic evalua-tion is usually done by systematically examining inter-faces and evaluating their compliance with generalusability principles/design guidelines. This methodology

Table 3 Interface design and automation functionality recommendations for Goal 1.1.9.1

Tasks Recommendations Heuristics Levels of automation

T1 1. Avoid terminology that may be unfamiliarto users (e.g., Code 128, Side A)

Match between system and real world,recognition rather than recall

T1 2. Suggest (default) label content and position Recognition rather than recallT1, T2 3. Make action selection (Manual, Automated,

and Read) more salientVisibility of system status

T1, T2 4. Provide explanations of different optionsin dialogs (e.g., by right-clicking on the option)for inexperienced users

Recognition rather than recall,help and documentation

T1, T2 5. Prevent users from changing settings theycannot affect (e.g., process time estimate)

Error prevention

T1, T2 6. Combine rotate (micro-plate) carrier withmanual dialog (if never used separately)

Parsimonious design

T2 7. Based on location of bar coder icon in method,or presence of barcode on micro-plate, suggest(default) action (print and apply vs. read)

Decision-making

Fig. 11 Redesigned bar coderaction/configuration dialog

has been applied to interface design for other complexcontrol applications, like advanced aircraft piloting(Kaber et al. 2002), and not just conventional human–computer interaction (desktop computing) applications.It is a quick and useful technique for identifyingusability problems (Nielsen 1993). However, heuristicevaluation does not directly recommend solutions toidentified interface problems (Nielsen 1993). It is for thisreason that the comparison of the GDTA and AHmodels is crucial to support design recommendations forexisting systems in the context of specific tasks. Ouranalysis of biopharmacologist goals, tasks, decisions,and SA requirements led to design improvement sug-gestions that were goal-directed and that addressed theoperator’s decisions and information needs. The evalu-ation of the automation functionality decomposition(through AH modeling) served to create recommenda-tions that support the biopharmacologist in effective andnatural task performance and problem-solving (cf., Vi-cente 1999).

The heuristics and LOA taxonomies (Nielsen 1993;Parasuraman et al. 2000) that were employed to guideour search for interface design and automation func-tionality recommendations were selected due to theirparsimoniousness and robustness. However, as dis-cussed above, some heuristics were not applicable to theanalysis of the screening line control interfaces. Differenttaxonomies may be appropriate for different domains,or for analyses focused on specific issues (e.g., systemfunctionality problems or interface usability issues).Other taxonomies could be used to classify cognitivesystem design enhancements, based on the methodologywe have defined, such as Rasmussen’s (1983) skills, rules,knowledge taxonomy. Rasmussen’s (1983) taxonomyhas been used in the past to categorize interface usabilityproblems (Fu et al. 2002). However, this taxonomy isfocused on the behavioral aspects of cognitive process-ing of tasks, as compared to Nielsen’s (1993) taxonomywhich is focused on interface features, and could be usedto classify automation enhancement recommendations.Endsley and Kaber’s (1999) LOA taxonomy, which de-scribes the various degrees of automation of complextasks, could be used to classify automation functional-ity-related recommendations. The taxonomy describesspecific LOAs that would provide for a finer classifica-tion of functionality recommendations than thatachieved by using Parasuraman et al.’s (2000) taxon-omy. This taxonomy may also be suitable for makingrecommendations on whether system functions shouldbe allocated to the human operator or to automation,based on operator goal state information from theGDTA and characterization of automation capabilitiesin the AH models.

5.2 Future research

Several research directions related to the present workare worth pursuing in the future. First, it would be

helpful to further validate the methodology describedhere by applying our design recommendations to allexisting screening system technologies. That is, interfaceprototypes could be created for all applications andinterfaces used in HTS, by implementing our designrecommendations. CELISCA plans to use the results ofthis work as a basis for enhancing ‘‘home-grown’’ pro-cess control software interfaces to manage a chemicalscreening line in order to identify compounds that maybe useful for remediation of warfare agent dump sites inthe former German Democratic Republic. These pro-totypes could then be compared to the existing systeminterfaces in terms of operator performance, workloadand SA, using structured cognitive engineering methodssuch as user testing.

In addition to making use of the GDTA and AHmodels for modifying existing systems and interfaces,the CTA approach presented here has outcomes that arebeneficial to the operation of real screening systems. TheGDTA results can be used to structure training pro-grams for new operators, with the goal of helping themachieve and maintain good SA and facilitate perfor-mance. The AH equipment models can be used intraining on how each equipment function is imple-mented in the screening line and the purpose of thevarious subsystems and components. The AH automa-tion models can be used to teach line operators how eachdevice function is implemented through the software andthe purpose of the various interface features.

The application of this methodology to other workdomains needs to be investigated to demonstrate itsrobustness. It would be interesting to compare the typesof recommendations generated in domains similar toHTS operations in terms of cognitive task demands onoperators due to unstructured environments, dynamicworkloads, performance of multiple, simultaneous tasks,etc. Examples include aviation, air traffic control, andpatient care in intensive care units.

Acknowledgements This research was supported by a NationalScience Foundation (NSF) Information Technology ResearchGrant (no. 046852). Ephraim Glinert was the technical monitor forthe NSF. The views and opinions expressed in this paper are thoseof the authors and do not necessarily reflect the views of the NSF.We thank the University of Rostock and CELISCA for providingus with access to high-throughput biological screening systems andsupporting the research through allocation of biopharmacologistand process engineer time to the effort.

References

Bisantz AM, Vicente KJ (1994) Making the abstraction hierarchyconcrete. Int J Hum Comput Stud 40:83–117

Bisantz AM, Roth E, Brickman B, Gosbee LL, Hettinger L,McKinney J (2003) Integrating cognitive analyses in a large-scale system design process. Int J HumComput Stud 58:177–206

Bolstad CA, Riley JM, Jones DG, Endsley MR (2002) Using goaldirected task analysis with Army brigade officer teams. InProceedings of the 46th Annual Meeting of the Human Factors& Ergonomics Society. Human Factors & Ergonomics Society,Santa Monica, CA, pp 472–476

Brooke J (1996) SUS: a ‘quick and dirty’ usability scale. In: JordanPW, Thomas B, Weerdmeester BA, McClelland IL (eds)Usability evaluation in industry. Taylor & Francis, London, pp189–194

Card S, Moran T, Newell A (1983) The psychology of human–computer interaction. Erlbaum, Hillsdale, NJ

Comfort DA, Chhabra SR, Conners SB, Chou C-J, Epting KL,Johnson MR, Jones KL, Sehgal AC, Kelly RM (2004) Strategicbiocatalysis with hyperthermophilic enzymes. Green Chem6:459–465

Endsley MR (1993) A survey of SA requirements in air-to-aircombat fighters. Int J Aviat Psychol 3(2):157–168

Endsley MR (1995) Toward a theory of situation awareness indynamic systems. Hum Factors 37(1):32–64

Endsley MR, Kaber DB (1999) Level of automation effects onperformance, situation awareness and workload in a dynamiccontrol task. Ergonomics 42(3):462–492

Endsley MR, Rodgers MD (1994) Situation awareness informationrequirements for en route air traffic control. (Tech. ReportDOT/FAA/AM-94/27). Office of Aviation Medicine, UnitedStates Department of Transportation, Federal AviationAdministration, Washington, DC

Endsley MR, Bolstad CA, Jones DG, Riley JM (2003a) Situationawareness oriented design: from user’s cognitive requirementsto creating effective supporting technologies. In Proceedings ofthe 47th Annual Meeting of the Human Factors & ErgonomicsSociety. Human Factors & Ergonomics Society, Santa Monica,CA, pp 268–272

Endsley MR, Bolte B, Jones DG (2003b) Designing for situationawareness: an approach to human-centered design. Taylor &Francis, London

Entzian K, Allwardt A, Holzmuller-Laue S, Junginger S, Rod-delkopf T, Stoll N, Thurow K (2005) Automationslosungen furbiologische und chemische Screeningverfahren. ProceedingsGMA-Kongress ‘‘Automation als Interdisziplinare Heraus-forderung’’, Baden-Baden, June 7–8, pp 235–242

Fu L, Salvendy G, Turley L (2002) Effectiveness of user testing andheuristic evaluation as a function of performance classification.Behav Inf Technol 21(2):137–143

Hajdukiewicz JR, Vicente KJ, Doyle DJ, Milgram P, Burns CM(2001) Modeling a medical environment: an ontology for inte-grated medical informatics design. Int J Med Inform 62:79–99

Hollnagel E (2003) Prolegomenon to cognitive task design. In:Hollnagel E (ed) Handbook of cognitive task design. LawrenceErlbaum Associates, Mahwah, pp 3–16

Kaber DB, Tan K-W, Riley J (2002) Improved usability of aviationautomation through direct manipulation and graphical userinterface design. Int J Aviat Psychol 12(2):153–180

Kieras D (1999) A guide to GOMS model usability evaluationusing GOMSL and GLEAN3. University of Michigan, AnnArbor, Michigan

Klein GA, Calderwood R, MacGregor D (1989) Critical decisionmethod for eliciting knowledge. IEEE Trans Syst Man Cybern19(3):462–472

Klinger DW, Gomes ME (1993) Cognitive systems engineeringapplication for interface design. In Proceedings of the 37th

annual meeting of the Human Factors & Ergonomics Society.Human Factors & Ergonomics Society, Santa Monica, CA, pp16–20

Lin L, Isla R, Doniz K, Harkness H, Vicente KJ, Doyle DJ (1998)Applying human factors to the design of medical equipment:patient-controlled analgesia. J Clin Monit Comput 14:253–263

Nielsen J (1993) Usability engineering. Academic, BostonParasuraman R, Sheridan TB, Wickens CD (2000) A model of

types and levels of human interaction with automation. IEEETrans Syst Man Cybern 30(3):286–297

Rasmussen J (1983) Skills, rules, knowledge: signals, signs, andsymbols and other distinctions in human performance models.IEEE Trans Syst Man Cybern 13:257–267

Rasmussen J (1985) The role of hierarchical knowledge represen-tation in decision-making and system management. IEEE TransSyst Man Cybern 15:234–243

Rasmussen J, Pejtersen A, Goodstein L (1994) Cognitive systemsengineering. Wiley, New York

Riley JM, Endsley MR (2002) Computer-aided decision support: isit what the Army needs? In Proceedings of the 46th annualmeeting of the Human Factors & Ergonomics Society. HumanFactors & Ergonomics Society, Santa Monica, CA, pp 477–481

Szczepkowski M, Neville K, Popp E (2005) Application of a work-centered design method to support counterspace operations. InProceedings of the 49th annual meeting of the Human Factors& Ergonomics Society. Human Factors & Ergonomics Society,Santa Monica, CA

Thurow K, Gode B, Dingerdissen U, Stoll N (2004) Laboratoryinformation management systems for life science applications.Organic Process Research & Development 12.8 A-M

Usher JM, Kaber DB (2000) Establishing information require-ments for supervisory controllers in a flexible manufacturingsystem using goal-directed task analysis. Hum Factors ErgonManuf 10(4):431–452

Vicente KJ (1992) Memory recall in a process control system: ameasure of expertise and display effectiveness. Mem Cognit20:356–373

Vicente KJ (1999) Wanted: psychologically relevant, device- andevent-independent work analysis techniques. Interact Comput11:237–254

Vicente KJ (2004) The human factor: revolutionizing the way welive with technology. Vintage Canada, Toronto

Wei J, Salvendy G (2004) The cognitive task analysis methods forjob and task design: review and reappraisal. Behav Inf Technol23(4):273–299

Woods D (2005) Generic support requirements for cognitive work:laws that govern cognitive work in action. In Proceedings of the49th annual meeting of the Human Factors & ErgonomicsSociety. Human Factors & Ergonomics Society, Santa Monica,CA

Xu W (2005) A cognitive engineering approach for facilitatinganalysis of human–computer interaction in complex work do-mains. In Proceedings of the 49th Annual Meeting of the Hu-man Factors & Ergonomics Society. Human Factors &Ergonomics Society, Santa Monica, CA