Theorising Learning from Incidents: A Human-Machine

Systems PerspectiveLing Rothrock

The Harold and Inge Marcus

Department of Industrial and Manufacturing Engineering

The Pennsylvania State University

University Park, PA 16802

University of Aberdeen, 11-12 June 2014

Ling Rothrock

EMPLOYMENT HISTORY: 

Associate Professor, The Pennsylvania State University, 2008-present

Assistant Professor, The Harold & Inge Marcus Department of Industrial & Manufacturing Engineering, The Pennsylvania State University, 2002-2008  

Assistant Professor, Department of Biomedical, Industrial, and Human Factors Engineering, Wright State University, Dayton, OH, 2000-2002

Research Scientist, Army Research Laboratory, Human Research and Engineering Directorate, FT Huachuca Field Office, AZ, 1998-2000.

Officer, United States Army, FT Bliss, TX, 1996-1998.

FIELD INSTITUTION DEGREE DATEIndustrial Engineering Georgia Institute of Technology Ph.D. 1995Industrial Engineering Georgia Institute of Technology M.S. 1992Applied Mathematics Florida Institute of Technology B.S. 1990

PhDs

Jung Hyup Kim, 2013, Assistant Professor, University of Missouri. 

Namhun Kim, 2010, Assistant Professor, Ulsan National University of Science and Technology 

Jing Yin, 2009, Consultant, The Ironside Group, Inc.

Damodar Bhandarkar, 2008, Senior HF Engineer, Pritney Bowes 

Hari Thiruvengada, 2007, UX Design Manager, Honeywell, Inc. 

Sungsoon Park, 2007, Principal Consultant, Samsung SDS 

Dongmin Shin, 2005, Associate Professor, Hanyang University

Purpose of Visit

The objectives of my sabbatical leave are to improve my professional skills through working with notable researchers in my field; applying my skills toward challenging human-machine problems in refinery process control; and create case studies to give undergraduate and graduate students an appreciation for current problems in process control.

Department of Industrial and Manufacturing Engineering at Penn State University

World’s first industrial engineering department founded in 1909

Students: ~450 undergraduates; ~70 MS; ~70 PhD

32 faculty members

Research Areas Human Factors – ergonomics, human centered design, human-

computer interaction Manufacturing – distributed systems and control, design Operations Research – applied probability and stochastic systems,

optimization, game theory, statistics and quality, simulation Production, supply chain, health systems engineering, service

engineering

Consider Advanced Process Control

ASM Examples drawn from the Abnormal Situation Management (ASM) Consortium

Categories

0 3 6 9 12 15 18 21 24

Frequency

Defective Installation

Failure to Follow Procedure/Instruction

Failure to Recognise Problem

Inadequate/Incorrect action

Inadequate Work Practices

Inadequate or No Procedure

PeopleandWorkContextFactors

EquipmentFactors

Defective Equipment

Equipment Design Flaw

Equipment/Mechanical Failure

ProcessFactors

Operation Beyond Original Design Limits

Process Design Flaw

Source: ASM Consortium

Sources of Plant Disturbances

Presented by N Kosaric at 2005 Defect Elimination Conference

Causes of Equipment Failure

Source: ASM Consortium

Causes of Process Upsets

RISK OF HUMAN ERROR

40%

Equipment Failure Other

20%

Human Error

40%

Causes of Plant Disturbances

HUMAN FACTORS

TOOLS

MACHINES

SYSTEMS

TASKS

JOBS

WORK ENVIRONMENT

“Failure to Adequately Inform and Engage the

Human-in-the-Loop in Automated Processes.”

Source: ASM ConsortiumThe Fundamental Problem

Key Learning from ASM Projects

Typical analyses that focus on just root causes are insufficient for identifying systemic improvement opportunities: Root causes explain ‘why’ something occurred, not ‘what’

occurred in terms of failures Root causes are general and not specific enough to drive

continuous improvement – details are buried in incident report No effective methods for aggregating root cause details across

incidents for systemic analysis of problems and improvements

IncidentEvent NEvent 2Event 1 Event N+1

RootCause

‘Why’ event occurred

Missing ‘What’ went wrong

RootCause

How aggregate details withinand across incidents?

Models

Research Process to Improve Learning from Incidents

ScenarioHuman-in-the-loop (HITL) Simulation

Platform

Database

Implement dynamic events to test hypotheses

Data logging includes operator interaction, system states, and required activities

Enable data access to measure performance and inform model building

Validate findings in context

Subject-MatterExperts

Extend findings to industryCall Centers

Command and Control

Hypothesis and Experimental Design

Formulate hypothesis based on incidents and causal factors

Construct platform to simulation domain

Construct computational models of human performance and judgment

Process Control

HITL Performance Measurement: Windows of Opportunity

A construct that specifies a functional relationship between a required situation and a time interval that specifies availability for action.

FalseAlarm

MissCorrect Rejection

Situation No SituationRequired Required

Environment

Action

No Action

Response

Correct

Incorrect

Early On-time Late

A B C

D E F

G

H I

Human-in-the-loop (HITL) Simulation

Platform

Windows using Temporal Logic

Given an action , b, and a time window, w,

we define 6 predicates, M, such that,

situationmeet not does if 0

win specified situationmeets if Iw b

bb

1)(1

wtoward relevant not is if 0

wtoward relevant is if Iw b

bb

1)(2

)(]1)([1||),(|| 1,

1sijwTTji OwIthatsuchiiffwM

i bb

)(]1)([1||),(|| 1,

2sijwTTji UwIthatsuchiiffwM

i bbA

B

)(]1)([1||),(|| 1,

3sijwTTji CwIthatsuchiiffwM

i bbC

]1)([]0)([1||),(|| 21,

4 jwjwTTji iiIIthatsuchiiffwM bbbD E F

)0)((,1||)(|| 2,

5 jwTTj iIiiffM bbG

)0)((,1||)(|| 2,

6 jwTTi iIjiffwM bH

Rothrock, L., & Narayanan, S. (Eds.). (2011). Human-in-the-loop Simulations: Methods and Practice. London: Springer-Verlag.

Extension to Team Research

Output files A Output files B Output files C

Phase 3: Analyze Team/Individual Performance Integrated

Output files

PerformanceAnalyzer

Tool

Team/ Individual member performance

Phase 2: Run Simulation

Team Role A

Each member communicates with the other using speech and internal messaging system

TeamRole B

TeamRole C

Script Maker

ScenarioPhase 1: Scenario Generation

Experimentation Strategy

Human-in-the-loop (HITL) Simulation

Platform

No Training (NT) Team Coordination Training

(TCT)Task Delegation Training (TDT)

1. No specific training is imparted.

2. Team members are provided with information on the definition of team coordination and task delegation.

3. No specific tasks are delegated to each operator role.

1. Team Coordination is emphasized during training.

2. Team members are instructed on how to achieve effective coordination via demonstration of good and bad practices.

3. No specific tasks are delegated to each operator role.

1. Task Delegation is emphasized during training.

2. The operator’s display is split into two distinct areas and is designated to each of the two roles. Operators monitor and perform actions within the designated area, while passing information pertaining to the other area onto their teammate.

3. Specific tasks are delegated to each operator role based on competencies and operator capabilities.

Research Question: Is one form of training superior?

Database

Rothrock, L., Cohen, A., Yin, J., Thiruvengada, H., & Nahum-Shani, I. (2009). Analyses of Team Performance in a Dynamic Task Environment. Applied Ergonomics, 40(4), 699-706.

Teamwork Dimension (Dependent Variables)Task Type Teamwork

DimensionResponsibilities for operator roles

Aircraft Information Coordinator (AIC) Sensor Operator (SO)

PrimaryInformation Exchange

Request Visual Identification (VID) report and pass it to other teammates.

Evaluate incoming sensor signals.

Correlate sensor signal to a particular aircraft.

Transmit the correlated sensor signal.

Backup Communication Operators did not use speech

channels for communication (not considered).

Operators did not use speech channels for communication (not considered).

PrimaryTeam Initiative/

Leadership

Vector Defensive Counter Air (DCA) within 256 Nautical Miles (NM) from ownship.

Vector DCA outside 20 NM from ownship.

Vector DCA outside danger zones. (Vectoring of DCA is done by changing its speed, course and altitude)

Issue level one warning to hostile aircrafts.

Issue level two warning to hostile aircrafts.

Issue level three warning to hostile aircrafts.

BackupSupporting Behavior

Assign identification to unknown aircrafts.

Assign missiles to hostile aircrafts.

Engage missiles upon hostile aircrafts.

Assign identification to unknown aircrafts.

Error Correction Change the identification of

incorrectly identified aircrafts. Change the identification of

incorrectly identified aircrafts.

Experimental Design

78 participants from a major university (in 39 teams) randomly received one of three training conditions (between-subjects design)

Each participant trained on 6 10-min scenarios and then tested on 2 10-min scenarios (high and low workload)

Three training conditions (no training, team coordination training, task delegation training) used varying in type of presentation (nothing, reading material, or video)

Team Performance Assessment

Environment

Operator Response

Team HITL

Simulation

Truth Maintenance

System

Relative Accuracy

Index (RAI)

Latency Index

(LI)

Information Exchange

Communication

Supporting Behaviour

Team Initiative/ Leadership

Teamwork dimensions

Ontime-Correct( )1Relative Accuracy Index (RAI)

mi

im

The Statistical Model

consider logistic regression as the model for analyzing data in which the dependent variable is a proportion:

that can also be expressed as:

Which is a particular case of the Generalized Linear model, in which linear regression models are extended to the exponential family of distributions that includes both the normal and the binomial distributions.

1

1

exp( )( )

1 exp( )

J

ij jji i i J

ij jj

XE Y p

X

1log

1

Jiij jj

i

pX

p

The Statistical Model (cont.)

Experiment designed to evaluate the effect of a certain type of training on RAI

Since the dependent variable (RAI) is a proportion, the suitable distribution for modeling it, is the binomial distribution

RAI was measured for each one of the two team members, at two stress levels (Low/High), where each team belonged to one of three training groups (NT, TCT, TDT).

For each of the 39 teams, divided randomly among the three types of training, there are four dependent measures of RAI since each team member (SO and AIC) has two outcome measures, corresponding to high and low levels of stress.

Findings

In IE condition TCT training significantly improved performance Negative correlation between AIC and SO under

stress

In SB condition Effects of stress more pronounced Activities involved require longer key sequences

and, under stress, fewer identifications were made

Findings (cont.)

In TI/L condition Absence of DCA activities suggesting limited

cognitive resources Participants in TCT condition outperformed

those in NT or TDT conditions

Implications of Work

Quantitative assessment of teamwork through combination of teamwork and task dimensions with time windows

Provides insight into the impact training methods on performance

Transition to opportunities in process control systems