Annual Report: 0426852 Page 1 of 9 Annual Report for Period: 10/2004 - 10/2005 Submitted on: 07/01/2005 Principal Investigator: Kaber, David B. Award ID: 0426852 Organization: North Carolina State U Title: ITR - (ASE + NHS) - (int): Intelligent Human-Machine Interface & Control for Highly Automated Chemical Screening Processes Project Participants Senior Personnel Name: Kaber, David Worked for more than 160 Hours: Yes Contribution to Project: David Kaber is an associate professor of industrial engineering. As the principal investigator, he conducted the initial research tasks at the University of Rostock (URO) and coordinated the efforts of the three project subteams at North Carolina State University (NCSU). Specifically, he addressed the goal-directed (cognitive) task analysis of human supervisory control of highly automated biological screening processes, which is to serve as a basis for the development of a cognitive model of screening line operator behavior. The task analysis is also to serve as a basis for interface design recommendations for existing software used by operators. Name: Chow, Mo-Yuen Worked for more than 160 Hours: Yes Contribution to Project: Mo-Yuen Chow is a full professor of electrical and computer engineering (ECE). As a co-principal investigator on the project, over the past year he directed the activities of the ECE subteam, including identifying approaches for modeling robotic-assisted biological screening processes and student prototyping of simulations. He has also directed the ECE student in formulating a model of network-control of multiple screening lines by a single human supervisor under error conditions by using adaptive bandwidth and resource allocation methods. Name: St. Amant, Robert Worked for more than 160 Hours: Yes Contribution to Project: Rob St. Amant is an associate professor of computer science (CS). As a co-principal investigator on the project, over the past year he directed the activities of the CS subteam. He has investigated various knowledge representation methods for extending the results of the task analysis. He also has worked to identify pathways/ compilers for converting cognitive model pseudo code to actual computational cognitive models. Name: Stoll, Regina Worked for more than 160 Hours: Yes Contribution to Project: Regina Stoll is an adjunct-associate faculty in the Department of Industrial Engineering at NCSU. Her permanent appointment is Director of the Institute for Work and Social Medicine at the URO and full professor in the School of Medicine. This past year she served as the liasson between NCSU researchers and URO scientists to support NCSU's completion of the initial research tasks as part of the ITR grant. She also collaborated with the principal investigator (Kaber) on the design and execution of experiments on physiological measures of operator workload in complex dynamic control task performance. Regina also served as the principal investigator on a parallel grant to the URO for study of, ôPhysio-ergonomic Optimized Human-machine Interfaces for Life Science Automation.ö
Transcript
Annual Report: 0426852
Page 1 of 9
Annual Report for Period:10/2004 - 10/2005 Submitted on: 07/01/2005
Principal Investigator: Kaber, David B. Award ID: 0426852
Organization: North Carolina State U
Title:ITR - (ASE + NHS) - (int): Intelligent Human-Machine Interface & Control for Highly Automated Chemical Screening Processes
Project Participants
Senior Personnel
Name: Kaber, David
Worked for more than 160 Hours: Yes
Contribution to Project: David Kaber is an associate professor of industrial engineering. As the principal investigator, he conducted the initial research tasks at the University of Rostock (URO) and coordinated the efforts of the three project subteams at North Carolina State University (NCSU). Specifically, he addressed the goal-directed (cognitive) task analysis of human supervisory control of highly automated biological screening processes, which is to serve as a basis for the development of a cognitive model of screening line operator behavior. The task analysis is also to serve as a basis for interface design recommendations for existing software used by operators.
Name: Chow, Mo-Yuen
Worked for more than 160 Hours: Yes
Contribution to Project: Mo-Yuen Chow is a full professor of electrical and computer engineering (ECE). As a co-principal investigator on the project, over the past year he directed the activities of the ECE subteam, including identifying approaches for modeling robotic-assisted biological screening processes and student prototyping of simulations. He has also directed the ECE student in formulating a model of network-control of multiple screening lines by a single human supervisor under error conditions by using adaptive bandwidth and resource allocation methods.
Name: St. Amant, Robert
Worked for more than 160 Hours: Yes
Contribution to Project: Rob St. Amant is an associate professor of computer science (CS). As a co-principal investigator on the project, over the past year he directed the activities of the CS subteam. He has investigated various knowledge representation methods for extending the results of the task analysis. He also has worked to identify pathways/ compilers for converting cognitive model pseudo code to actual computational cognitive models.
Name: Stoll, Regina
Worked for more than 160 Hours: Yes
Contribution to Project: Regina Stoll is an adjunct-associate faculty in the Department of Industrial Engineering at NCSU. Her permanent appointment is Director of the Institute for Work and Social Medicine at the URO and full professor in the School of Medicine. This past year she served as the liasson between NCSU researchers and URO scientists to support NCSU's completion of the initial research tasks as part of the ITR grant. She also collaborated with the principal investigator (Kaber) on the design and execution of experiments on physiological measures of operator workload in complex dynamic control task performance. Regina also served as the principal investigator on a parallel grant to the URO for study of, ôPhysio-ergonomic Optimized Human-machine Interfaces for Life Science Automation.ö
Annual Report: 0426852
Page 2 of 9
Post-doc
Graduate Student
Name: Segall, Noa
Worked for more than 160 Hours: Yes
Contribution to Project: Noa Segall is a full-time Ph.D. student in industrial engineering. She has worked as a half-time research assistant to David Kaberon this project. Her focus has been on the cognitive task analysis of human control of highly automated biological screening lines,abstraction hierarchy modeling of life sciences equipment and automation, and interface design recommendation for existingsoftware interfaces.
Name: Green, Becca
Worked for more than 160 Hours: Yes
Contribution to Project: Becca Green is a full-time Ph.D. student in industrial engineering. She has worked as a quarter-time research assistant to DavidKaber on this project. Her focus has been on the cognitive task analysis of human control of highly automated biological screeninglines, abstraction hierarchy modeling of life sciences equipment and automation, and interface design recommendation for existingsoftware interfaces.
Name: Li, Zheng
Worked for more than 160 Hours: Yes
Contribution to Project: Zheng is a full-time Ph.D. student in electrical and computer engineering. He has worked as a half-time research assistant toMo-Yuen Chow on this project. His focus has been on real-time resource allocation and scheduling for effective network-basedcontrol of multiple robot-assisted chemical screening lines under error conditions, and formulation of an approach to adaptivebandwidth and resource allocation.
Name: Vanijjirattikhan, Rangsarit
Worked for more than 160 Hours: Yes
Contribution to Project: Rangsarit (Jae) is a full-time Ph.D. student in electrical and computer engineering. He has worked as a half-time research assistantto Mo-Yuen Chow on this project. His focus has been on using petri nets as a basis for developing simulation models of highlyautomated biological screening lines. He has also worked with Stateflow simulation software for modeling screening lines.
Name: Horton, Thomas
Worked for more than 160 Hours: Yes
Contribution to Project: Thomas is a full-time Ph.D. student in computer science. He has worked as a half-time research assistant to Rob St. Amant on thisproject. His focus has been on identifying knowledge representation tools to serve as a basis for cognitive modeling of biologicalscreening line operator behaviors. He has also worked to identify computer programming languages for creating compilers totranslate pseudo-code used as part of knowledge representation methods to actual computational cognitive models.
Name: Williams, Lloyd
Worked for more than 160 Hours: Yes
Contribution to Project: Lloydd is a full-time Ph.D. student in computer science. He has worked as a half-time research assistant to Rob St. Amant on thisproject. His focus has been on identifying knowledge representation tools to serve as a basis for cognitive modeling of biologicalscreening line operator behaviors (e.g., SHAKEN). He has also worked to identify computer programming languages for creatingcompilers to translate pseudo-code, used as part of knowledge representation methods, to actual computational cognitive models(e.g., ACT-R).
Undergraduate Student
Technician, Programmer
Annual Report: 0426852
Page 3 of 9
Other Participant
Research Experience for Undergraduates
Organizational Partners
University of RostockThe URO has received a parallel grant through the German State of Mecklenburg- Vorpommern (SMV) for study of, ôPhysio-ergonomic Optimized Human-machine Interfaces for Life Science Automation.ö The main tasks as part of this grant include: (1) investigation of human physiological load/strain in use of highly automated life science processes; (2) development of effective data analysis interfaces for web- based access of screening process data and data on operator physiological responses; and (3) technical optimization of human-machine interaction for high- throughput screening (HTS) systems. These tasks are complimentary to the research steps being addressed by NCSU through this ITR grant. The URO grant commenced on 1/1/05 and also has a performance period of three years. This past year the URO provided NCSU researchers with complete access to facilities and personnel at the Center for Life Sciences Automation in Rostock-Warenmunde, Germany in order for NCSU to complete the cognitive task analysis on highly automated screening line operator performance. The results of NCSUÆs work on interface design recommendations for existing screening software applications will be fed to the URO for evaluation in terms of usability and loads imposed on operators. The design recommendations will also be used by the URO in developing web interfaces for databases on screening line processes. Finally, the computational cognitive model of supervisory control behavior in monitoring and running automated biological screening lines to be developed by NCSU will be shared with the URO for defining approaches to optimize human-machine interaction in HTS systems.
Other Collaborators or ContactsThe CS (computer science) subteam has been in consultation with researchers in cognitive modeling concerning the enhancement of the ACT-R cognitive architecture, specifically visual processing routines for more effective application to HRI tasks. Our contacts include Mike Byrne (Rice University), Mike Fotta (DNAmerican, Inc.), and Frank E. Ritter (Pennsylvania State University).
Activities and Findings
Research and Education Activities: (See PDF version submitted by PI at the end of the report)(See attached file.)
Findings: (See PDF version submitted by PI at the end of the report)(See attached file.)
Training and Development:The training opportunities that have resulted in association with the activities of the IE subteam include:
Annual Report: 0426852
Page 4 of 9
(1) familiarization of all graduate assistants on the project with CTA methodologies as well as the results that can be expected from the methods; (2) familiarization of expert biochemists and electrical engineers at CELISCA with CTA methods; (3) training of the IE research assistants on how to conduct a GDTA and to develop AH models; and (4) education of the students on how CTA may be used as a basis for redesign of existing complex system interfaces or for the design of new automation aids. The results of the AH modeling of the automated screening line equipment and control software could be used by CELISCA as a basis for developing training manuals for new line operators. As previously mentioned, the AH models of equipment facilitate operator understanding of how each function of the equipment is implemented and why the various subsystems or components exist as part of the system. With respect to the AH models on the process control software applications, as previously mentioned, operators in training can use the model representations to learn how specific device functions are implemented through the software and they can learn why the various interface features/options exist as part of the software. Beyond these training opportunities, it is possible to structure training programs for new screening operators based on the results of the GDTA in order to help operators achieve and maintain good situation awareness in supervisory control of robotic systems. The outcomes of the GDTA include the decisions to each screening process goal and the associated information requirements that an expert biopharmacologist considers important. Based on the goal states of an operator, they can be trained to know what decision need to be made when and where to look for specific information in the environment to allow them to make accurate decisions (i.e., develop good SA for performance). (This is a training opportunity that could easily be supported by the outcomes of this project, but such efforts are beyond the scope of the proposal.) The training opportunities that have resulted from the ECE subteam activities include the following: (1) By using the Petri Net and SIMULINK model developed to represent the HTS lines at CELISCA, we can support process engineers in formulating and testing new ideas on how to further automate the biological screening assays, particularly to address faulty device or material states. (2) The process simulation model can also be used to support testing of designs of new automation aids for screening process operator control and fault diagnosis and resolution. The simulation mode will allow process engineers to learn what approaches work or donÆt work. (3) The process simulation model can also be used to support training of the project research assistants on effective interface design for information presentation on multiple simultaneous complex processes. The students on the IE subteam will work to prototype process control interfaces that effectively deliver to a biochemist the output information from the HTS line simulation model. (4) Finally, the two graduate students who are supported in the ECE subteam have been using the project research as examples for educating other graduate students in a Mechatronics course (ECE556) and a Control Systems course (EC716). The training opportunities that have resulted from the CS subteam activities include the following: (1) Familiarization of CS graduate assistants on the project with concepts in knowledge acquisition and planning representations to support the process. (2) Training of the CS graduate assistants in the use of a semi-automated tool for
Annual Report: 0426852
Page 5 of 9
knowledge acquisition (SHAKEN), in particular for domain knowledge related to screening tasks. (3) Training of the CS graduate students in the construction of basic cognitive models, via the tutorial materials available for the ACT-R architecture. Beyond these opportunities, one of the research assistants on the CS subteam (Lloyd Williams) will undergo an M.S. thesis defense in August, 2005.
Outreach Activities:The IE subteam has nothing to report on outreach activities at this time. The ECE subteam has nothing to report on outreach activities at this time. There have been no outreach activities from the CS subteam.
Journal Publications
Kaber, D. B., Segall, N. & Green, R., "Using multiple cognitive task analysis methods for supervisory control interface design in high-throughput biological screening processes", Theoretical Issues in Ergonomics Science, p. , vol. , ( ). In preparation
St. Amant, R., Horton, T. E., and Ritter, F. E., "Model-based evaluation of expert cell phone menu interaction", ACM Transactions onComputer-Human Interaction, p. , vol. , ( ). Accepted
St. Amant, R., Kukreja, U., Ritter, F. E., Boyce, C. W., and Janssen, C., "Cognitive Modeling and Human-Robot Interaction: RemoteNavigation", IEEE Transactions on Systems, Man, and Cybernetics., p. , vol. , ( ). Submitted
Ritter, F. E., Van Rooy, D., St. Amant, R., and Simpson, K., "Providing user models with direct access to computer interfaces: An exploratorystudy of a simple human-robot interface", Journal of Cognitive Engineering and Decision Making, p. , vol. , ( ). Submitted
Books or Other One-time Publications
Kaber, D. B., "Human-centered design for human-robot interaction", (2005). Conference Proceedings Paper, PublishedEditor(s): J. Sinay, P. Mondelo, K. L. Saarela, W. Karwowski & M. MattilaCollection: CAES ?2005 Proceedings (CD-ROM)Bibliography: Kosice, Slovak Republic, May 25-28
Vanijjirattikhan, R., Li, Z., Chow, M-Y. & Kaber, D. B., "Timed Petri Net modeling and simulation of high throughput biological screening lines", (2006). Conference Proceedings Paper, In preparationCollection: IECON06Bibliography: IEEE
Li, Z., Vanijjirattikhan, R., Chow, M-Y. & Kaber, D. B., "Adaptive bandwidth allocation and sampling rates in control of multiple high throughput biological screening lines", (2006). Conference Proceedings Paper, In preparationCollection: ICRA06Bibliography: IEEE
St. Amant, R., and Wood, A. B., "Tool use for autonomous agents", ( ). Conference Proceedings Paper, AcceptedCollection: National Conference on Artificial IntelligenceBibliography: Pittsburgh, PA. AAAI Press
Annual Report: 0426852
Page 6 of 9
St. Amant, R., Riedl, M. O., Ritter, F. E., and Reifers, A., "Image Processing in Cognitive Models with SegMan", ( ). Conference Proceedings Paper, AcceptedCollection: Proceedings of HCI InternationalBibliography: Las Vegas, NV
Horton, T. E., and St. Amant, R. A., "A taxonomy of presentation and manipulation techniques for human- robot interfaces", (2006). Conference Proceedings Paper, In preparationCollection: International Conference on Human-Robot InteractionBibliography: Publisher unknown at this time.
Web/Internet Site
URL(s):http://people.engr.ncsu.edu/dbkaber/NSF_ITR/Description:This year the project team created a website for exchanging information among the IE, ECE and CS subteams and to disseminate research results to our collaborators at the URO. The site acts as an informal repository for technical notes and intermediate results related to the CTA, cognitive modeling for the project, and the process simulation development. It is in an initial stage currently. Each subteam has developed its own home page to provide links to documents generated by the specific team. These pages can be accessed from the main project web page. The ECE and CS subteam pages are password protected (Username: rostock; Password: rostock). At this time, the majority of links across the pages of the site are to PDF files presenting: project activity descriptions, information on specific biological screening processes under study, team meeting notes, literature review summaries, cognitive task analysis results (e.g., GDTA hierarchy diagrams and AH models), draft Petri Net models of the HTS lines at CELISCA, conceptual diagrams of elements of the initial Stateflow simulation to model screening lines, mathematical models of the network resource allocation problem in single operator control of multiple screening lines, and results of SHAKEN translation (graphical knowledge representation) of GDTA results.
Other Specific Products
Product Type:
Software (or netware)
Product Description:SIMULINK-based simulation of HTS line and integrated robotics systems at CELISCASharing Information:The ECE subteam will share the process simulation model with automation engineers working at the URO and CELISCA via a project FTP site. It is expected that CELISCA engineers will be able to use the simulation in formulating and testing new ideas on how to further automate the biological screening assays, particularly to address faulty device or material states. Based on actual process control data, the simulation will be further developed to model healthy and faculty operating conditions.
Contributions
Contributions within Discipline:
Annual Report: 0426852
Page 7 of 9
The activities of the IE subteam have served to advance industrial engineering, specifically in the area of human factors. No prior research has applied AH modeling to the life sciences domain for representing automated systems. Furthermore, no other research has used GDTA and AH modeling in combination as a basis for determining the extent to which existing screening process automation technologies support operator decision making and information requirements, or to formulate design recommendations for new operator process control interfaces. The activities of the CS subteam have served to advance research in cognitive modeling and artificial intelligence, from a computer science perspective. We expect our work on translation between cognitive task analysis and a common planning representation will contribute toward an integration of research in task analysis and artificial intelligence. Our work on translation between planning and a cognitive modeling formalism will similarly provide a bridge between cognitive modeling and AI, an interdisciplinary area that is promising but largely unexplored. Results will begin to appear toward the end of the summer of 2005.
Contributions to Other Disciplines: All the research activities and findings of the various subteams are expected to contribute to the discipline of lab automation/life sciences automation. In general, this project is to develop a human-centered approach to the design of operator interfaces and adaptive control technologies for managing HTS processes in drug derivative development, which incorporate advanced robotics and automated devices. This design approach may be formalized through future publications and disseminated to the life sciences automation community to support the development of human-centered lab systems. The human factors methods used to represent biopharmacologist knowledge structures and to describe the configurations and functions of automated devices used in biological screening processes may be adopted by the life sciences automation community for analysis of human interaction with automation in other types of chemical processes, biocatalysis applications, and microarray-based research. The Petri Net modeling techniques and Stateflow simulations that the ECE subteam is applying to the HTS screening lines at CELISCA may also be used to model other types of life science processes involving human-automation interaction. These methodologies can be used to create test-beds for assessing new automation aids and control technologies, limiting the need to conduct experiments with actual operators and systems. Finally, the CS subteam has demonstrated that graphical knowledge representation methods, like SHAKEN, can be useful approaches to extending the results of existing CTA methodologies to support computational cognitive model development. In this way, our project is defining a pathway from CTA to cognitive modeling using accepted tools. This may serve to streamline the process of modeling and predicting operator behaviors in interacting with complex automated systems, specifically in life science processes, in order to evaluate various system interface design alternatives, etc.
Contributions to Human Resource Development: This project is providing specialized training for IE, ECE and CS graduate students in applying advanced research methodologies to the domain of human interaction with life sciences automation. The students are developing theoretical understandings of CTA, and systems and user modeling methods through project work at NCSU. They are also becoming familiar with life science processes via hands-on lab experiences at the URO. It is expected that the students will develop rare and valuable skill sets for working in the life sciences automation industry.
Contributions to Resources for Research and Education:
Annual Report: 0426852
Page 8 of 9
This collaborative project motivated the development of a larger Partnership for International Research & Education (PIRES) proposal by NCSU and the URO, which was submitted to the NSF OISE (Office of International Science & Engineering) this past March. The Partnership proposal included research projects to expand the collaboration of the two universities across the disciplines of chemical engineering, computer science, electrical and computer engineering and industrial engineering. Specifically, we planned to develop an integrated biocatalysis discovery process distributed across the two institutions by using existing resources for microarray- based research at NCSU and automated analytical chemistry processes at the URO. Related to this, a collection of bioinformatics research projects was proposed, including development of advanced lab process sensor networks and data fusion technologies, distributed process control modeling, CTA applied to biochemist planning of biochemistry processes and data analysis, cognitive modeling of supervisory control behavior in automated biocatalysis processes, and advanced statistical analysis and decision tool development for supporting the integrated discovery process. The proposal also included an education plan with undergraduate and graduate research exchange programs, as well as international summer short- course programs, and joint supervision of doctoral student dissertation projects by NCSU and URO faculty. If NCSU and the URO are successful in receiving this funding, NCSU is committed to developing a new Human-Automation Interaction (research) Center (HAI-C) for center-to-center collaboration with CELISCA at the URO. The proposed development plan is for the faculty and resources of the NCSU CS Intelligent Interfaces, Multimedia and Graphics Lab, the ECE Advanced Diagnosis, Automation and Control Lab, and the IE Cognitive Ergonomics Lab to be integrated to form the new center. These labs uniquely complement each other in terms of solving research problems in automation of life science processes, real-time control of network/distributed systems, automated system state monitoring, and intelligent user interface design. As part of the PIRES proposal, the NCSU College of Engineering committed to synergizing the various lab resources in the new HAI-C. The present ITR project and the potential PIRES grant represent foundational funding for the development of the center. With respect to our contribution to institutional and information resources of a much more limited scope, we have continued with service to our respective fields. St. Amant has been invited to serve as a senior program committee for the 2006 ACM Conference on Intelligent User Interfaces, as well an associate chair for the CHI Notes category of the 2006 ACM Conference on Human Factors and Computing (CHI). Over the past year he has reviewed for the International Journal of Human-Computer Studies and for the IEEE Conference on Visualization. Kaber was recently appointed as a track editor for the Journal of Cognitive Engineering & Decision Making, specifically ôStudies in Simulation & Synthetic Environments.ö He was also invited to serve on the editorial board of the journal of Theoretical Issues in Ergonomics Science.
Contributions Beyond Science and Engineering: It is expected that the results of this project will lead to the prototyping of new operator interface technologies and adaptive automated control system technologies for supervision and management of HTS processes (biological/chemical). It is possible that these technologies could be refined and produced by small biotechnology instrumentation companies in the Rostock, Germany and Raleigh, NC areas. For example, the Analytical Instrument Group (AIG), eV. is a small biotechnology company with offices in both Rostock and Raleigh, which may be interested in commercializing technologies based on the results of this ITR project. In this way, the products of the research might serve to create new jobs in the States and Germany and to increase economic growth.
Special Requirements
Annual Report: 0426852
Page 9 of 9
Special reporting requirements: None
Change in Objectives or Scope: None
Unobligated funds: less than 20 percent of current funds
Animal, Human Subjects, Biohazards: None
Categories for which nothing is reported:
1.1. Adapt “bench-top” method to High-throughput Screening (HTS) line (i.e., plan automated version of assay):1.1.3. Establish plate configuration to achieve statistically valid results:
T1 Ensure no edge effects in assay process (e.g., thermal effects on wells at edges of plates).Do I need to include empty wells at perimeter of plate?
Need - Number of sample concentrations to be included on plate.Need - Cost of leaving plate wells empty.
Figure IE.1. Portion of GDTA hierarchical outline for adaptation of screening assay to automated line.
Figure IE.2. GDTA hierarchical diagram presenting goals of adaptation of screening assay to automated line.
Figure IE.3. GDTA diagram on subgoal of establishing micro-plate configuration.
Figure IE.4. AH model for barcode print and apply device.
Figure IE.5. AH model of software for controlling the barcode print and apply device.
1.1.9.4 Facilitate test plate incubation
Task 1 steps:Preparation (heating) of plates:• Input/select minimum time, Input/select
maximum time– Input time estimate (manual entry)
Apply micro-environment to test plates:• Process of setting and verifying heating
system state/temperature– Use touch-keypad to set temperature
Control of plate incubation processes:• Setting and activating plate chamber heat
– Manual start-up or auto activation
Integrate incubator into method
What is the incubation duration for the plate?
What is the temperature of the incubator microenvironment required by the assay?
Knowledge of “bench top” assay, as documented in literature
Figure IE.6. Comparison of GDTA content on facilitating micro-plate incubation with AH model of incubator control software.
GDTA
AH models of equipment and
automation
UI design suggestions
Existing interface
New interfaces
SHAKEN translation
(bypass GOMS?)
ACT-R
User testing
Petri nets
Stateflow simulation
NominalError modes
Resource allocation schemes
Task/interface representation
Figure IE.7. Current project elements and (data) interconnections.
Figure CS.1. Aggregation and precedence relations in a plan-based task representation.
Figure CS.2. SHAKEN representation of task structure (root node only).
Figure CS.3. SHAKEN representation of task structure (expanded).
Figure CS.4. SHAKEN representation of task structure (further expanded).
Figure CS.5. SHAKEN representation of task structure (single node expanded).
;;; THIS SAVE FROM CMAP OF: _Ensure-assay-quality5223 ;;; KM REPRESENTATION OF THIS "PROTOTYPE"
Figure CS.6. SHAKEN output (partial): Text representation of task structure.
A.1
The major research activities of the IE (industrial engineering) subteam (Kaber, Segall, andGreen) in this ITR project have included:
(1) Conducting a review of literature on contemporary cognitive work analysis methods;(1) Conducting a cognitive work analysis (CWA) on supervisory control of automated biological
screening lines for new drug derivative development using structured interviews, goal-directed task analysis (GDTA) and abstraction hierarchy (AH) modeling approaches;
(1) Formalizing the results of the GDTA using hierarchical diagrams and present the results ofAH modeling of screening line equipment and automation using means-end diagrams;
(1) Making comparison of the results of the GDTA and AH modeling approaches to develop anunderstanding of how high-throughput screening operator needs are currently addressed (ornot) by existing automation and information display technologies;
(1) Using the results of the GDTA and AH modeling analyses as a basis for formulatinginterface design guidelines to enhance existing interactive software applications used insupervisory control of automated biological screening processes and to serve as a basis forprototyping interactive applications to address screening process goals for which automation/software does not currently exist.
(1) Planning for activities as part of Year 2 of the project, including: (a) development of highlyusable interface prototypes for interactive screening process control software; (b)development of a computational cognitive model of supervisory control behavior inscreening process management; and (3) application of the cognitive model to control aStateflow simulation of a HTS line through the interface prototypes.
The major research activities of the ECE (electrical and computer engineering) subteam (Chow,Vanijjirattikhan, and Li) in this ITR project have included:
(1) Formulating and modeling an IP (Internet Protocol)-based remote real-time systemmonitoring and supervisory control capability for simultaneous operation of multiple HTSlines under “healthy” as well as faulty operating conditions using timed Petri net.
(1) Developing a MATLAB/SIMULINK based simulator to represent the dynamics of masterrobots on the multiple HTS lines under healthy and faulty operating conditions.
(1) Investigating and developing an adaptive bandwidth and sampling rate allocation algorithmfor simultaneous operation of multiple HTS lines under healthy as well as faulty operatingconditions.
(1) Investigating how IP network delays and bandwidth constraints affect the adaptivebandwidth and sampling rate allocation and develop new methodologies to compensate forIP-induced negative effects.
(1) Planning for integration of the MATLAB/SIMULINK simulation with the computationalcognitive model to be developed by the CS subteam, specifically how the model will exploitthe simulation of the adaptive bandwidth and sampling rate allocation algorithm for effectivesimultaneous control of multiple HTS lines.
(1) Planning to support the human-machine interface design activities of the IE subteam in thecoming year by linking the MATLAB/SIMULINK simulation to prototype control interfacedisplays for presenting simulation output to operators in a realistic manner.
A.2
The major research activities of the CS (computer science) subteam (St. Amant, Williams, andHorton) in this ITR project have included:
(1) Surveying tools and concepts for automated translation between cognitive modelingformalisms at different levels of abstraction, to facilitate modeling of screening tasks.
(2) Formalizing the translation between an AI planning domain language (discussed below) andthe language of a cognitive modeling architecture, ACT-R, at the language (syntax andsemantics) level.
(3) Extending the capabilities of the ACT-R architecture, in the area of visual processingroutines, to support practical analysis of HRI systems, in particular user interfaces forscreening tasks.
(4) Performing a survey of interaction mechanisms and presentation techniques for existingapproaches to HRI.
(5) Planning for integration of tools and formalisms with interface design activities of the IEsubteam, plus the simulation work of the ECE subteam.
F.1
With respect to the IE subteam activities, the following results have been generated:
(1) Literature survey of cognitive work analysis methods
Recent books and journal and conference proceeding manuscripts on the use of work domainanalysis (WDA) for supporting systems design and training were reviewed. These studiesincluded:
Hollnagel, E. (2003). Handbook of Cognitive Task Design. Mahwah, NJ: Lawrence Erlbaum.Mazaeva, N. & Bisantz, A. M. (in review). On the representation of automation using a work
domain analysis. Submitted to Theoretical Issues in Ergonomics Science.Naikar, N., et al. (accepted). The development of a coherent approach to work domain analysis:
Methodology. To appear in Proceedings of the 49th Annual Meeting of the Human Factorsand Ergonomics Society. Santa Monica, CA: Human Factors and Ergonomics Society.
Szczepkowski, M., et al. (accepted). Application of a work-centered design method to supportCounter-space operations. To appear in Proceedings of the 49th Annual Meeting of theHuman Factors and Ergonomics Society. Santa Monica, CA: Human Factors andErgonomics Society.
Woods, D., et al. (accepted). Generic support requirements for cognitive work: Laws that governcognitive work in action. To appear in Proceedings of the 49th Annual Meeting of the HumanFactors and Ergonomics Society. Santa Monica, CA: Human Factors and ErgonomicsSociety.
Xu, W., et al. (accepted). A cognitive engineering approach for examining interaction inautomated flight decks. To appear in Proceedings of the 49th Annual Meeting of the HumanFactors and Ergonomics Society. Santa Monica, CA: Human Factors and ErgonomicsSociety.
The book by Hollnagel reviews a broad range of cognitive task analysis (CTA) methods andprovides information on some application domains. The research by Mazaeva and Bisantzpresents novel applications of the AH model methodology to systems automation. It alsopresents work on representations of human-automation interaction using decision laddermethods. The work by Naikar et al. provides a comprehensive method for performing WDA,including problem and constraint identification. The work by Szczepkowski et al. demonstrateshow CTA methods can be used to support work-centered design approaches, but the study doesnot address the development of software or interfaces. The research is helpful from theperspective of using CTA methods to identify conditions under which certain automationcapabilities may be useful to operators performing complex tasks. The study by Woods et al.identifies basic requirements for support functions for cognitive work and criteria to determinewhen automation is needed. The criteria may also be used to assess existing systems. Finally, thestudy by Xu et al. presents an application of WDA in order to identify automation needs incomplex systems. This study also applied AH modeling to identify gaps in operatorunderstanding of systems, limitations of existing system interfaces for supporting completeoperator knowledge of system states, and operator training needs. The study supports ourapproach of using multiple CTA methods, such as GDTA and AH modeling to identify criticalneeds in HTS equipment and automation design for operators. Unfortunately, across all of thestudies reviewed, no approaches were defined for systematic translation of CWA or WDA results
F.2
to system/interface design guidelines to support human performance. This is the case for workmodels constructed based on CTA, AH modeling, decision ladder methods, etc.
(2) Cognitive work analysis using GDTA and AH modeling
We used multiple CTA techniques to describe the knowledge structure of biopharmacologists inplanning, executing and analyzing the results of HTS operations, as well as the lab automationand equipment used to facilitate these operations. GDTA has been used in many prior cognitivetask analyses (CTAs) (e.g., Endsley, 1993; Endsley & Rodgers, 1994; Usher & Kaber, 2000;Bolstad, Riley, Jones & Endsley, 2002) and focuses on identifying operator situation awarenessrequirements in performing complex systems control. AH modeling has historically been used incomplex work domain analyses (e.g., Rasmussen, 1985; Rasmussen et al., 1994) and has beenfound to be an effective tool for revealing how automated system processes and functions arefacilitated through specific components and to provide explanations of why certain componentsare needed to achieve system purposes. The results of a GDTA include lists of critical operatordecisions and information requirements that can be used as a basis for defining appropriatecontent of complex system information displays. The results of AH modeling can serve as a basisfor developing system user manuals and training programs to educate operators on connectionsbetween automated control functions and software system functions as well as interface featuresand options.
Our approach to the GDTA involved an analyst interviewing an expert biopharmacologist todescribe knowledge structures relevant to the use of a highly automated, HTS line at the Centerfor Life Sciences Automation (CELISCA) of the URO in order to test organic compounds thatmay have the potential to serve as bases for drug derivatives used in future cancer medications,virus medications, etc. Initially, procedural task analyses were performed on biopharmacologistdevelopment of basic HTS methods using existing commercial-off-the-shelf software. Theseanalyses identified the general steps as part of an automated enzyme-based assay of compoundsfor the potential to affect human cellular functions. Beyond this, the expert provided backgroundinformation on the basics of enzyme reactions and the use of micro (culture) plates forconducting life sciences experiments. The analyses were used as stepping-off points for theGDTA, which focused on biopharmacologist adaptation of a “bench-top” version of a screeningassay to the HTS line in the CELISCA laboratory. The GDTA: (a) identified the major goal ofthe biopharmacologist in the planning, execution and analysis of results of the enzyme-basedscreening process (i.e., to discover marine compounds leading to drugs); (b) identified sub-goalsthat are supportive of this overall goal; (c) identified the specific tasks to achieve the sub-goals;(d) created critical questions aimed at addressing decision-making in the HTS operation; and (e)developed biopharmacologist situation awareness requirements to answer these questions.
The general approach taken to the AH modeling in the context of HTS operations includeddeveloping models of both the physical devices as part of an HTS line and the software used tocontrol the devices. In the current line setup at CELISCA, proprietary software developed by themanufacturers of each HTS device (e.g., barcode print and apply devices, micro-plate incubators,optimized robots for biological sample analysis (ORCA), automated pipetting (liquid transfer)devices (e.g., Biomek2000, BiomekFX), and automated plate readers) is used to program devicemethods. The devices on the line are integrated through a central process control system that
F.3
includes a screening line “method editor” application, which communicates with the userinterface of the proprietary device software through an “executive” software controller. Theexecutive also communicates via ActiveX with the device specific process control softwaremodules. The executive provides automated control of the entire HTS line. The line methodeditor includes Action/Configuration dialogs for all devices on the HTS line. These dialogs areused during system programming to set device parameters, as part of a screening method, and tosend data to device software modules via the executive.
Our AH modeling in this context targeted the physical devices on the screening line, the deviceuser interface (proprietary control software), and the line method editor software. At the highestlevel of abstraction, the AH models define the purpose of the automation in the work domain.The lowest level of a model represents the physical components of the system. In between arethe generalized functions of the automation. Linkages among the levels represent how thepurpose of the automation is implemented through specific devices and they provide anexplanation of why certain components are needed to achieve a system purpose (Rasmussen,Pejtersen & Goodstein, 1994).
For example, an AH model for the barcode printer and reader (on the screening line) identifiesthe purpose of the system as assigning identifications to micro-plates and recognizing platelabels during an assay process. The general constraints affecting the purpose of the device (i.e.,any constraining functions) include the aspects of labeling and reading. The generic functions forthe device include the processes of system initialization, printing, applying bar codes, andreading bar codes. These generic processes can be broken down into component processes. Forplate label reading, component processes include activating the scanner, moving the micro-platefor scanning and positioning the plate holder as part of the bar coding device. With respect tophysical components and functions of the bar coder, the subsystems for printing, applying andreading include a label-paper feeder, a foil feeder, a print head, a vacuum gripper, a laserscanner, the micro-plate holder, and a manual entry keypad and LCD. These subsystems can alsobe broken down into components. Finally, the AH model identifies means-end connectionsamong the generic and component processes, the device subsystems and components, the genericprocesses and subsystems, and the component processes and components of the device. Throughthese connections an operator can discover how each function of the equipment is implementedand they can learn why the various subsystems or components exist as part of the system.
Our application of GDTA and AH modeling to HTS operations at CELISCA for testing ofbiological compounds, as part of new drug derivative discovery, has resulted in: (a) a goalstructure for biopharmacologists, along with lists of critical HTS decisions and SA requirements;and (b) AH models of all devices on the existing HTS line (the bar coder, incubator, ORCA,Biomek2000, and Fluostar plate reader) and control software for an enzyme-based assay ofmarine organism compounds.
The formalized representations of these results are presented in the next section.
F.4
References:
Bolstad, C. A., Riley, J. M., Jones, D. G., & Endsley, M. R. (2002). Using goal directed taskanalysis with Army brigade officer teams. In Proceedings of the 46th Annual Meeting of theHuman Factors and Ergonomics Society (pp. 472-476). Santa Monica, CA: Human Factorsand Ergonomics Society.
Endsley, M. R. (1993). A survey of SA requirements in air-to-air combat fighters. InternationalJournal of Aviation Psychology, 3(2), 157-168.
Endsley, M. R. & Rodgers, M. D. (1994). Situation awareness information requirements for enroute air traffic control. (Tech. Report DOT/FAA/AM-94/27). Washington, DC: Office ofAviation Medicine, United States Department of Transportation, Federal AviationAdministration.
Rasmussen, J. (1985). The role of hierarchical knowledge representation in decision-making andsystem management. IEEE Transactions on Systems Man and Cybernetics, 15, 234-243.
Rasmussen, J., Pejtersen, A. & Goodstein, L. (1994). Cognitive Systems Engineering. New York:Wiley.
Usher, J. M. and Kaber, D. B. (2000). Establishing information requirements for supervisorycontrollers in a flexible manufacturing system using goal-directed task analysis. HumanFactors & Ergonomics in Manufacturing, 10(4), 431-452.
(3) Formalization of GDTA and AH modeling results
The results of a GDTA are typically presented using a hierarchical outline or diagrams. Weinitially developed a detailed outline of biopharmacologist goals, tasks, decisions andinformation requirements based on the structured interviews. Figure IE.1 shows a small portionof the GDTA hierarchical outline. The overarching goal for the analysis was biopharmacologistadaptation of a “bench-top” version of a screening assay to an HTS line in a state-of-the-artlaboratory at CELISCA. (It is important to note that specialized biological screening tests aredeveloped by scientists and published in the biotechnology literature. These tests are thenadopted by research labs or pharmacology companies for automated screening of crude organiccompounds, synthetic compounds etc., as to there potential to influence enzymatic processesnaturally occurring in human cells. The specialized test procedures published in the literature are“bench-top” versions; that is, they describe how to manually perform a particular screening test(assay). The work of CELISCA focuses on the integration and use of lab automation forconducting such screening operations in a high-throughput manner.) A subgoal to thisoverarching goal is the need to establish a configuration, or layout, for a micro (culture) platethat ensures accurate and reliable test results (see line 1.1.3). One of the tasks to this subgoal is toensure there are no “edge effects” in plate processing, or that liquid solutions in the micro-plateare not influenced by environmental variations due to their proximity to the edge of a plate. Thecritical decision for the biopharmacologist as part of this task is listed in the outline along withspecific information needs on the experiment and production constraints.
The entire hierarchical outline resulting from the GDTA was then translated into a collection ofhierarchical diagrams to promote ease of referencing by analysts and experts in ensuring theresults were correct. Figure IE.2 presents a hierarchical diagram of the overarching goal for theanalysis and all major subgoals. Figure IE.3 presents a diagram of the information on subgoal
F.5
1.1.3, as discussed above. It is important to note that the information requirements listed in thelowest block of the diagram can be used as a basis for defining screening line control systemsoftware interface content in order to support effective operator performance. All GDTAdiagrams produced as part of this research were subsequently reviewed by the expertbiopharmacologist, working with the project team, to verify accuracy. The CS subteam has usedthe GDTA diagrams as a basis for generating alternate forms of biopharmacologist knowledgerepresentations (e.g., SHAKEN software representations)(see activities and findings for the CSsubteam). It is expected that such representations will be useful for beginning development of acomputational cognitive model of biopharmacologist behavior in planning, executing andanalyzing the results of HTS operations during the second year of the project.
An AH model is typically presented using a grid of 3 columns and 5 rows (see Figure IE.4 forthe model on the bar coder device). The columns (from left to right) present decomposition ofthe system (as a whole) to presentation of the component processes and physical devices. Therows (from top to bottom) present functional decomposition of the system from the overallpurpose through generalized functions to the physical components supporting the functions. Themeans-ends connections are also presented across the rows (see lines in Figure IE.4).
This research also involved adapting the AH modeling methodology to representation ofautomation or software for controlling devices on the screening lines at CELISCA. In theautomation models, the columns (from left to right) present decomposition of the controlsoftware, as a whole, to presentation of specific functions. The rows (from top to bottom) presentfunctional decomposition of the specific application from the overall purpose through genericcontrol functions to the software options and settings supporting the functions. Although thelower rows of an AH model are typically used to present physical components of a system,instead of identifying software subroutines and the inner-workings of computer code as a basisfor control function explanation, we decided that from a user perspective, it would be moremeaningful to reveal the human-interface display features and options through which thesubroutines of the software are made accessible. This also ultimately allowed for a clearerintegration of the AH automation models with equipment/device models from an operatorperspective. Means-ends connections are also presented across the rows of the automationmodels.
Figure IE.5 presents the AH model for the control software for the bar code device as part of thescreening line method editor (the Action/Configuration dialog) referred to earlier. The highestlevel of the model identifies the purpose of the software as controlling the mechanisms of the barcoder. The abstract function of the bar coder is repeated in this model. The generic functions ofthe control software are identified as: defining global operation of the device, printing control,label and apply control, and reading control. These generic control processes of the software canbe broken down into component functions. For example, the printing control function involvesselecting the type of bar code, storing this information, and defining the content of the bar code.At the next level in the model, the interface display features and options, made available to usersthrough the software dialog, are identified. For example, the options required to specify the barcode content include using manual string entry, reading strings from a file, auto indexing oflabels, and use of the internal process control system ID for a micro-plate. Means-endconnections are presented among the generic and component functions, and the component
F.6
functions and interface features. Following the links (lines) from the top of the diagram down tothe bottom, an operator can discover how each automation function is implemented through thesoftware. Following the links from the bottom of the diagram to the top, an operator can learnwhy the various interface features/options exist as part of the software.
(4) Comparison of GDTA and AH results
Once the GDTA and AH model representations were complete, the results were compared torelate biopharmacologist goal structures and critical decisions (as part of testing processes) to thepurpose of the automated systems on the HTS line and functions and components. We reviewedthe decisions and information requirements for each goal as part of planning, execution andanalysis of HTS operations and compared them with the software functions currently available tousers for screening process control as well as the specific interface content. (The specificquestions asked by analysts in this work (to motivate operator goal state and automationcapability comparisons) have yet to be formally documented.) We then identified components ofexisting (software) systems that were not supportive of specific operator goals and might beunnecessary for process control (from the biopharmacologist perspective). Beyond this, thecomparisons of GDTA and AH model results allowed us to identify components of softwareinadequate for supporting operator system state/situation awareness and decision makingprocesses. (The outcomes of this step have yet to be formally documented.) In general, theapproach taken here is expected to be superior to historical research (e.g., Endsley, 1993)presenting the use of GDTA as a standalone analytical tool for identification of existing systemshortcomings and future design needs. The use of GDTA and AH modeling in combination inthis project is a novel approach to CWA and represents a publishable result.
(5) Interface Design Recommendations based on CTA results
In this activity, the IE subteam used the comparisons of the GDTA and AH model results toformulate interface design and automation function recommendations for the existing softwareapplications used in screening process control at CELISCA (e.g., the screening line methodeditor). These recommendations were organized according to a usability framework and aframework of types of automation. Specifically, we used a taxonomy of human-computerinteraction design heuristics presented by Nielsen (1993) to classify our interface designrecommendations. Nielsen’s heuristics concern the following: (a) the visibility of system status;(b) the match between the system and the real world; (c) user control and freedom; (d)consistency and standards in design; (e) error prevention; (f) facilitating recognition ofinformation through interfaces versus requiring users to recall; (g) facilitating flexibility andefficiency of use; (h) using parsimonious design; (i) facilitating error detection and recovery; and(j) providing accessible help utilities. We also made reference to Parasuraman et al. (2000)model of types and levels of automation for classifying any recommendations related to theunderlying functionality of the automation (software). They identified four general types ofautomation based on forms of human information processing in complex systems control,including: (a) information acquisition; (b) information analysis; (c) decision making; and (d)action implementation.
F.7
As an example, another subgoal in the GDTA hierarchy (developed through thebiopharmacologist interviews) is to facilitate micro (culture) plate incubation during screening ofcompounds for reactions with an enzyme. The micro-plates are typically brought to human bodytemperature in order to promote the generalizability of experiment results to the development ofdrug derivatives for human consumption. Figure IE.6 presents the tasks as part of automatingmicro-plate incubation on the existing screening line at CELISCA and the decision andinformation requirements of the operator (see left side of figure). The incubator must beintegrated into the overall automated screening method. This is done through the method editorsoftware mentioned previously and the operator must decide on the incubation duration andtemperature for the assay. In order to do this, the operator must fully understand the manualmethod of performing the assay. The right side of Figure IE.6 presents the specific steps theoperator must complete at the incubator device and with the control software (based on thecontent of the AH models developed earlier). This includes inputting the incubation duration andsetting and verifying the temperature, which can be done manually or through automation. Thesesteps are currently accomplished using different software dialogs or by using a keypad at thedevice.
The operator interface design recommendations made for this screening process goal included:(a) allow operator access to temperature setting at the same time they are setting the incubationduration through an integrated application dialog; and (b) eliminate redundancy in methods forincubator configuration for screening by requiring the operator to only use manual entry via thedevice keypad or time and temperature setting via the control software. These recommendationswere classified as addressing the user control and freedom and error prevention heuristics inNielsen’s taxonomy. One additional automation function recommendation was made for themethod editor software to suggest incubation temperatures and durations to an operator based ondata on similar experiment configurations stored in an existing process database. Thisrecommendation was classified as representing decision aiding automation in the model byParasuraman et al. (2000).
This approach and the preliminary results will also be used as a basis for formulating designrecommendations for prototyping futuristic supervisory control interfaces that might be used byHTS operators for programming, monitoring, controlling, and analyzing multiple screening linesperforming simultaneous assays, all varying in terms of the enzymes being used and thecompounds being tested.
References:
Nielsen, J. (1993). Usability Engineering. Academic Press, Boston, MA.Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000). A model of types and levels of
human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics,30(3), 286-297.
(6) Project Planning Activity
The objective of this activity was to conceptually define the collaboration among the threeNCSU subteams in the project during Year 2. The proposed project schedule included
F.8
development of the cognitive model of screening process supervisory controller behavior inSpring 2005 with model validation to occur during summer 2005. This work was to be followedby completion of a Stateflow simulation of the existing CELISCA automated biologicalscreening line in Fall 2005 along with prototyping of new interactive information displays tosupport operator process control in Spring 2006. At this point in time, NCSU is taking anintegrated and parallel approach to these project activities, which is expected to accelerate thepace of knowledge discovery through the research grant.
Figure IE.7 presents a diagram of the various analyses to be conducted, and models to bedeveloped, as part of the project, as well as how information is to pass between these elements.The GDTA and AH modeling elements are complete and the IE subteam is in the process offormalizing the user interface (UI) design suggestions. At the beginning of the Spring 2005semester, the overall project team elected to focus the cognitive modeling effort on operatorbehavior in addressing the screening process goal of ensuring the results of an assay meet qualitycriteria (see Subgoal 1.2 in Figure IE.2). The CS team has completed a SHAKEN translation(graphical knowledge representation) of the GDTA result as a basis for future ACT-R modeldevelopment (see additional details in the report of the CS team activities and findings). It isexpected that use of the SHAKEN tool may allow us to bypass GOMS (Goals, Operators,Methods, Selection Rules) model coding as a basis for ACT-R computational cognitive modeldevelopment. The CS team plans to work with the expert biopharmacologist from CELISCA inlate summer 2005 to begin the initial cognitive model development.
At the same time this effort is to occur, the ECE team will seek to finalize a prototypeMATLAB/SIMULINK based simulator of the HTS line at CELISCA. The ECE team research onPetri nets for process modeling has served as a basis for defining elements of a Stateflowsimulation. The prototype simulator will represent all devices included in the biologicalscreening at CELISCA. The URO has provided NCSU with time estimates and variances for allscreening process steps to facilitate the simulation development. The simulation is expected tomodel micro sample plate preparation and micro test plate screening. It will also model thefunctioning of advanced algorithms for adaptive allocation of communications bandwidth andprocess resources in single operator, simultaneous control of multiple screening lines via a labnetwork.
The ECE simulator of the automated screening process will be linked to the prototype processcontrol software interfaces to be developed by the IE team. That is, the output of the simulationwill be presented through the interfaces. The cognitive model developed by the CS team willthen be applied to the task and device simulations as a basis for evaluating various interfacedesign alternatives for supporting supervisory control of screening processes.
The overall objectives of the project and the steps to achieving the proposed outcomes remainthe same. However, this revised approach allows for a much more integrated effort that resolvesany serial processing dependencies, which may have existed in our original plan.
F.9
With respect to the ECE subteam activities, the following results have been generated:
(1) Literature Survey on System Modeling using Petri Nets and Resource Allocation
Categories of Petri Nets, related with various application areas, were identified. We haveidentified the existence of high-level Petri Nets, such as timed Petri Nets and colored Petri Nets.These tools are considered to be appropriate for the HTS line modeling because Petri Nets caneasily incorporate HTS operation times under different healthy and faulty operation conditions,as well as the resources allocation attributes for analyses and modeling.
An extensive literature survey was also conducted on the resource allocation area. We identifiedtwo relevant scheduling methods for multi-task adaptive bandwidth allocation problems withdirect applicability to our problem formulation. (This research has only recently been formallydocumented.) The two methods include real-rate scheduling models and feedback elasticscheduling models.
(For additional information, please refer to the document entitled, “A summary of a literaturesurvey on resource allocation problems and a possible solution: Real-rate scheduling,” availableat: http://www.adac.ncsu.edu/protected/itr.htm (password-protected: username "rostock";password "rostock") .)
(2) Petri Nets and Modeling the HTS Line
Based on the literature review, Petri Nets were considered to be suitable tools for the HTS linemodeling work. Unlike queuing network models, which focus on resource utilization, Petri Netmodels can reflect several important model properties, such as process synchronization,asynchronous events, concurrent operations, and conflicts or resource sharing. These propertiesare indispensable in modeling the robotics lines in HTS. Petri Nets can also be successfullyapplied for modeling and analysis in several areas such as communication protocols,manufacturing, automation processes, and failure prediction.
(For additional information, please refer to the document entitled, “Trypsin test procedure Petrinets model (Phase I),” available at: http://www.adac.ncsu.edu/protected/itr.htm (password-protected: username "rostock"; password "rostock") .)
(3) Formulation of multi-HTS line control as an adaptive bandwidth allocation problem
For this project, a network-based control system will be modeled for management of multipleHTS lines. The model problem for this project is a single supervisory controller managing foursubsystems (robotic screening lines). The structure of the model task will be considered indeveloping the network-based control system.
As a first step in this activity, the ECE subteam elected to model the process of micro sampleplate preparation as part of the overall automated biological screening process. In the processcontrol problem formulation, the HTS lines are modeled as subsystems that can simulate healthyand faulty conditions with process time impacts.
F.10
The ECE subteam is currently working towards mathematical formulation of the process controlproblem based on the definitions of the structure of the physical subsystems as part of the actualHTS lines at CELISCA. A sampling rate scheduling approach has been taken to modelingcommunications bandwidth allocation in the network process control model.
(For additional information, please refer to the document entitled, “Problem formulation on theresource allocation problem as part of NSF ITR project,” available at:http://www.adac.ncsu.edu/protected/itr.htm (password-protected: username "rostock"; password"rostock") .)
(4) SIMULINK and ACT-R Integration
The ECE subteam has collaborated in the planning of the integration of the various elements ofthe overall project, including linkage of the screening process simulation to the prototype controlinterfaces to be developed by the IE subteam as well as the cognitive model of supervisorycontroller behavior to be created by the CS subteam. It is important for the various projectsubteams to begin planning these integrations early in the project in order to ensure that suitabledata exchanges formats are defined and can be used to facilitate connections among the variousmodels.
Work by the CS subteam on the project has been to a large extent driven by results produced bythe IE and ECE subteams. The main goal of the CS subteam is to provide a link between tasks inthe screening process and how they are supported in terms of computer interfaces andinformation sources. The CS research thus straddles two disciplines: computational cognitivemodeling and intelligent user interfaces. Work has proceeded along two main lines, one aimed atbuilding practical tools for modeling HRI tasks in the screening domain, and the other atextending the theoretical scope of models for HRI. With respect to the CS subteam activities, thefollowing results have been generated:
(1) Formalizing the relationship between Cognitive Task Analysis and cognitive modeling.
A critical issue for cognitive modeling work in the domain of screening is providing a well-grounded translation between the natural language used in Cognitive Task Analysis and the moreprogram-like languages used in cognitive models such as ACT-R. We have chosen a translationprocess that involves an intermediate step: first encoding specifications of CTA concepts into awell-understood AI formalism for reasoning about action, and then translating the formalizedconcepts into a specification of a cognitive model. The reason for choosing a planning languageis that planning formalisms have been studied extensively as models for action, and there exists arelatively large body of theoretical literature describing the capabilities and limitations ofdifferent languages for representing actions.
For our intermediate AI formalism we have chosen PDDL, the Planning Domain DefinitionLanguage (Ghallab et al., 1998). PDDL supports the representation of relationships betweenactions and the environments in which they can be carried out. These relationships generally take
F.11
the form of knowledge about the conditions that must hold for an action to be possible, and thechanges that an action produces on the environment. PDDL is in general use in the AI planningcommunity and has been exercised in planning competitions and the development of practicalplanning systems. We have significant experience using PDDL to represent knowledge aboutdependencies in user interfaces. PDDL has so far proved an appropriate choice for representingCTA tasks. There has been to date only a brief exploration of the relationship between planningconcepts and CTA; the translation process that we are developing will constitute a novel anduseful finding once complete.
References:
Ghallab, M., Howe, A., Knoblock, C., McDermott, D., Ram, A., Veloso, M., Weld, D., andWilkins, D. (1998). PDDL---The Planning Domain Definition Language. AIPS-98 PlanningCommittee. Online document. http://citeseer.ist.psu.edu/article/ghallab98pddl.html
(2) Formalizing the theoretical relationship between AI planning and cognitive modeling.
A computational cognitive model is essentially a computer program that, given an appropriatespecification of a problem-solving procedure, can carry out cognitive operations to execute theprocedure. Modern cognitive models are based on architectures, such as ACT-R (Anderson andLebiere, 1998), that generalize over tasks and problem domains. As with task analysisrepresentations, languages for cognitive modeling tend not to be designed to allow theoreticalanalysis as representations of action. For example, given a set of primitive operations, is itpossible to combine them to carry out a specific task? This kind of analysis is possible inplanning representations but not, in general, in cognitive modeling languages. Our work hasmade progress toward allowing cognitive models to be analyzed in more general terms than hasbeen possible in the past.
Thus, given a PDDL description of a task analysis model, we are working toward translation ofthis description into a form that can be executed by a cognitive modeling architecture, inparticular, ACT-R. Our work here has been based on a system developed in our laboratory lastyear, G2A, which translates models developed in an abstract language (GOMS) into ACT-Rprimitives (St. Amant et al., 2004). Over the past several months we have extended G2A bymaking it platform independent, ensuring that it is compatible with the latest release of the ACT-R architecture in 2005, and generalizing its translation capabilities. Our goal, which will not beachieved until later this year, is to enable G2A to translate from PDDL into ACT-R primitives;we have made significant progress on the path toward this goal.
References:
Anderson, J. R., and Lebiere, C. (1998). The atomic components of thought. Mahwah,NJ:Erlbaum.
St. Amant, R., Freed A., and Ritter, F. E. (2004). Specifying ACT-R models of user interactionwith a GOMS language. Cognitive Systems Research 6(1): 71-88.
F.12
(3) Building tools to support modeling and model translation.
One of the reasons we have settled on PDDL as an intermediate language is that tools exist tofacilitate the encoding of task knowledge into the language. SHAKEN, developed at SRI, is sucha tool (Clark et al., 2001). SHAKEN is a Web-based interactive system that allows a user toexpress knowledge about a domain in a structured form. An example of the type of taskknowledge to be represented is given in Figure CS.1, which shows a hierarchy of tasks forensuring assay quality. Tasks are described in a semi-structured language appropriate for inputby hand into SHAKEN; links in the hierarchy represent precedence and compositionrelationships.
We have partially encoded a cognitive task analysis model in SHAKEN, producing output inPDDL form, shown approximately in graphical form in Figure CS.1. Results can be seen inFigures CS.2 through CS.5.
We begin with the results of a Goal-Directed Task Analysis (GDTA). Starting from thehierarchical outline representation of a task, as given by GDTA, we use the SHAKEN rapidknowledge formation tool to encode the information. Figure CS.2 shows a reduced graphicalrepresentation of the screening process goal of, “Ensure results of assay meet quality criteria,”with only the root node of the goal representation visible. Figure CS.3 shows the graph furtherexpanded. Figure CS.4 shows another level expanded, and CS.5 shows how individual nodes canbe expanded, removing the global context. It is important to note that each node within the graphexists as an independent entity with its own processes. These nodes are then combined torepresent the overall task. Once the entire task has been represented within the SHAKENsoftware, we then use the software to output Lisp-based code that is close to PDDL in syntax.The completed graph for, “Ensure results of assay meet quality criteria,” produces the codeshown in Figure CS.6. To our knowledge this is the first encoding of a model developed usingCTA into a planning representation. One of the issues we face is that the SHAKEN interface isnot flexible enough to support the rapid entry and modification of knowledge in this domain; thisremains an area for current and future work.
References:
Clark, P., Thompson, J., Barker, K., Porter, B., Chaudhri, V., Rodriguez, A., Thomere, J.,Mishra, S., Gil, Y., Hayes, P. and Reichherzer, T. (2001). Knowledge Entry as the GraphicalAssembly of Components. Proceedings of the First International Conference on KnowledgeCapture (K-Cap-2001), pp. 22-29.
(4) Cognitive architecture extensions: Attention and visual processing.
One of the most difficult parts of developing a cognitive model is evaluation. Beyond experimentdesign, there is a significant practical problem that must be solved: Is the environment withwhich the cognitive model interacts sufficiently similar to the environment that humans interactwith on the tasks of interest? The natural solution is to have cognitive models interact with thesame environment (in our case, the same user interfaces) used by humans. Our past work has tosome extent solved this problem, but not in general for tasks that require real-time monitoring
F.13
and detailed modeling of vision and attention. In our most recent work, we have developedextensions to the ACT-R architecture that enhance the visual routines that ACT-R uses toprocess screen images, mainly by including a component representing a visual focus of attentionmore detailed than in the default architecture. (This work has been carried out and continues withcollaborators mentioned above: Byrne, Fotta, and St. John.) The added flexibility will make itmuch easier to connect ACT-R models with screening process control interfaces, for evaluationand refinement, as part of this ITR project.
(5) Models of interaction for HRI.
Another important component of our effort to understand user behavior in HTS tasks isaccounting for the ways that different interaction mechanisms and presentation techniques caninfluence performance. This perspective on user interaction is at a higher level of abstractionthan cognitive modeling, and produces complementary information. We are currently performinga literature survey of interaction techniques that have been applied and evaluated in human-robotinterfaces. Because HRI is a relatively new field, most approaches to supporting interaction withrobots are based on conventional desktop metaphors: windows, buttons, and so forth. Anecdotalexperiences suggest that many of the heuristics for conventional application design are lessappropriate for monitoring and controlling physical robots. We expect that our completed surveywill inform and extend the results of the IE subteam, as described above.
