Center for Risk and Reliability
A Hybrid Technique for Organizational
Safety Risk Analysis
Reliability Eng. ProgramMechanical Eng. Department
Presented at
International Conference on
Probabilistic Safety Assessment and Management (PSAM 9)
May18-23 2008, Hong Kong, China
Zahra Mohaghegh
Ali Mosleh
Center for Risk and Reliability
University of Maryland
Center for Risk and Reliability
Motivation
• Major system failures with significant
contributions from human & organizational
factors
– Chernobyl
– Columbia and Challenger space shuttle accidents
• In the quest to achieve 80% reduction in aviation
accidents, US FAA has recognized
“organizational factors” as one of the most
critical components
Center for Risk and Reliability
Key Questions
• What are the organizational “factors” that
affect system risk?
• How?
• To what degree?
Center for Risk and Reliability
The Research Approach
5. Key Gap… Leading to
New Research
1. Classical PRA
2. Approaches to Include
Organizational Factors
In PRA
3. Organizational
Accident Causation Theories
4. Quantitative
Organizational Safety Risk
Frameworks
6. Principles of Org. Safety Risk Analysis 7. Proposed Hybrid Technique
for
Org. Safety Risk Analysis
8. Realization Of Modeling Principles in SoTeRiA
9. An Example Application of SoTeRiA in Aviation
10. An Example Application of Hybrid technique In
Aviation
Center for Risk and Reliability
Quantitative Approaches
• Static : Variations of Influence Diagrams
(e. g., BBN), Process Models and Logic
Models – MACHINE (Embrey, 1992)
– SAM (Pate-Cornell, 1996)
– Omega Factor Model (Mosleh & Golfeiz, 1999),
– ASRM (Luxhoj, 2004)
– “Causal Modeling of Air Safety” (Roelen et al., 2003)
• Dynamic: e.g, use of “System Dynamics”– Mousang (2004)
Center for Risk and Reliability
Key Gap
• In the absence of a comprehensive theory, or at least a set of principles rooted in theory, all models look equally good, or equally poor, with very little basis to discriminate, and build confidence.
• This research focused on improving the theoretical understanding of relation between characteristics of organizations and their (system) safety outputs
Center for Risk and Reliability
Modeling Principles
Problem Definition, Metrics &
Scope
(A ) Defining the Unknown of Interest
(B ) Safety & other Org. Performance
(C ) Safety Performance & Deviation
Level of Analysis (D ) Multilevel framing
Factors / Elements (E ) Basic building block
(F ) Factor Level
(G) Factor Selection
(H) Measurement methods
(I ) Role of Perception
(J ) Factor Interdependencies
(K) Multidimensional measurement perspective
Relations • Multidimensional links
• Dynamic characteristics
Boundaries & Assumptions (N) Depth of causality and level of detail
(O) Generalizablity
Characteristics (P) Hybrid
Center for Risk and Reliability
SoTeRiA (Socio-Technical Risk Analysis )
Industrial &
Business
Environment
Social and Political
Culture and Climate
organizational
vision, strategy
and
goals
Regulatory
Environment
Safety
Culture
Organizational
Culture
Organizational
Structure&
Practices
Org. Safety
Structure &
Practices
Group
Climate
Group
Safety
Climate
SCT
Business
Outcome
System
Risk
Unit
Process
Model
Organizational
Climate
Org. Safety
Climate
Psyc.
Safety
Climate
Individual
PSFs
Emergent Process
(Leadership/supervision,
social interaction & homogeneity )
Physical Environmental Factors (e.g., Harsh Weather Condition)
Industrial &
Business
Environment
Social and Political
Culture and Climate
organizational
vision, strategy
and
goals
Regulatory
Environment
Safety
Culture
Organizational
Culture
Organizational
Structure&
Practices
Org. Safety
Structure &
Practices
Organizational
Structure&
Practices
Org. Safety
Structure &
Practices
Group
Climate
Group
Safety
Climate
Group
Climate
Group
Safety
Climate
SCT
Business
Outcome
System
Risk
Unit
Process
Model
Organizational
Climate
Org. Safety
Climate
Organizational
Climate
Org. Safety
Climate
Psyc.
Safety
Climate
Individual
PSFs
Emergent Process
(Leadership/supervision,
social interaction & homogeneity )
Physical Environmental Factors (e.g., Harsh Weather Condition)
Center for Risk and Reliability
Menu of Modeling Techniques
AND AND
0 1
1
0 1
+
3
20 1
Cut-Sets
Probabilityof Sequence
ESD
FTAND AND
0 1
1
0 1
+
3
20 1
Cut-Sets
Probabilityof Sequence
ESD
FT
Techniques for Technical SystemTechniques for Technical System(e.g., Groen et al., 2002)(e.g., Groen et al., 2002)
A1
A2
B2
B1
C1 C2 D1 D2A
B
C D
Latent Variable Measured variable
RegressionRegression--based Causal Modelingbased Causal Modeling(e.g., (e.g., BollenBollen, 1989), 1989)
Activity
Transformation Inputs
Controls/ Criteria
Resources
OutputsActivity
Transformation Inputs
Controls/ Criteria
Resources
Outputs
Process Modeling TechniquesProcess Modeling Techniques(e.g., (e.g., HeinsHeins, 1993, 1993 )
A
B1
X
B2
Y
B3
Z
Bayesian Belief Network( BBN)Bayesian Belief Network( BBN)
Center for Risk and Reliability
Menu of Modeling Techniques
A1
A2
A4
A31
A
C1
C2
C4
C3
C
A1
A2
A4
A31
A
C1
C2
C4
C3
C
Configurational approach(e.g. Meyer et al., 1993)
Quantitative
Qualitative
A
B
C
D
E
Quantitative
Qualitative
A
B
C
D
E
QQ-BBN(Wang, 2007)
Population
BirthDeath
Birth Rate Death Rate
System Dynamics(e.g. Sterman, 2000)
Human Reliability Techniques
(e.g., Chang & Mosleh , 2004)
•THERP
•IDAC
•NARA
Center for Risk and Reliability
Hybrid Technique for Organizational Safety Risk
SD
Environment
QQ-BBN
ESD/FT
MethodStock and flow
Diagram
HCL(Mosleh
et al. , 2005)
technical systems
•process modeling
•regression-based
models
•configurational
characteristics
•human model
deterministic relations
dynamic integration
Center for Risk and Reliability
Hybrid Technique for Organizational Safety Risk
BB
N
FT
ESD
SD
SMPs
Human
Action
Human
Action
TeamPerformance(e.g. Maintenanceteamperformance)
Team Perfo
rmance
Model
Failure due toother causes
System 1
Human
Action
System 1
OmegaFactor
Resources
ResourcesResources
Initiatingevent
S
S
F
Failure due to
organizational factors
(e.g. maintenance error)
System Dynamics Model of
Org.
Center for Risk and Reliability
Financial
performanceTechnical System Risk Model
Quality of Maintenance
Maintenance Unit Process Model
Regulatory Auditing System
Individual-level PSFs
Weather
Condition
Emergent
Process
Maintenance Group Safety Climate
Organizational
Safety Structure & Practices
Organizational Safety Culture
A
B
C
D
EF
G
H
I
J
K
L
M
SoTeRiA –based Aviation Maintenance Model
Center for Risk and Reliability
Technical System Risk
Engine
Flight crew action
S
S
F
Initiating
Event
Engine Failure
Internal Engine
Failure
Engine failure
Maintenance
related
Engine
mismanagement
By crew
Engine failure by
External factors
Center for Risk and Reliability
MFinancial
performanceTechnical System Risk Model
Quality of Maintenance
Maintenance Unit Process Model
Regulatory Auditing System
Individual-level PSFs
Weather
Condition
Emergent
Process
Maintenance Group Safety Climate
Organizational
Safety Structure & Practices
Organizational Safety Culture
A
B
C
D
EF
G
H
I
J
K
L
Implementing Implementing SoTeRiASoTeRiA in Aviation Maintenancein Aviation Maintenance
Center for Risk and Reliability
Return to Service
Inspection
Maintenance Quality
Outsource MX In-house MXThrough Flights/ Daily
ChecksLayer (1)
Direct
Activities
4thParty
CTRS Actions
Technicians
Actions
CAMP
Proc.
Tools, Equipment
Parts/m
aterial
MX.
Facility/ Base
Inspector
Actions
4thParty
CTRS Actions
RISI
Procedures
Layer (1-1)
Outsourced/
In-house
Outsource
technicians
MX.
Schedule
ADS/ CMR/
STC/SBs
Manuals
Records &
Reporting
Inspection
Program
Repair
Proc.
Locally Produced
T/E
Purchased/
Produced
Purchased
Tools, Equipment
Operator produced
Parts
Purchased Parts/
Materials
Purchased/
Produced
Layer (1-1-1)
Layer (1-1-2) Eng.
Proc.
SFAR
Proc.
Maintenance Unit Process ModelMaintenance Unit Process Model
Adapted from Eghbali (2006 )
Center for Risk and Reliability
Financial
performanceTechnical System Risk Model
Quality of Maintenance
Maintenance Unit Process Model
Regulatory Auditing System
Individual-level PSFs
Weather
Condition
Emergent
Process
Maintenance Group Safety Climate
Organizational
Safety Structure & Practices
Organizational Safety Culture
A
B
C
D
EF
G
H
I
J
K
L
M
Implementing Implementing SoTeRiASoTeRiA in Aviation Maintenancein Aviation Maintenance
Center for Risk and Reliability
“Training” in System Dynamics Environment
Total technicions Experience
Increase in experience
f rom hiring
Loss of experience
f rom attrition
Increase on the job experience
Training
Saf ty experience gained
on job
Target experience
Experience gap
Av erage technicions experience
Average technicions experience
Total Number of Quits
Hiring
Ref erence experience
Relativ e ev perience
of technicions
Average experience
of technicions hired
Time to prov ide training
Relative technicion commitment
Total technicions
Relative management
commitment to safety
Rookie Quit
Senior Quit
T raining
Based in Part on Cooke (2004)
Center for Risk and Reliability
“Hiring Model” (System Dynamics Environment)
~
Ef f ect of Relativ e technicion
Commitment on quit
Senior quit f raction
Rookies SeniorsHiring
Up to speed Senior Quit
Rookie Promotion Time
~
Effect of Relative technicion
Commitment on quit
Rookie Quit
demand
Rookie quit f raction
Gap in Number of technicions
Target technicion Demand
Technician working
hours per y ear
Total technicions
Relative management
commitment to safety
Hiring Time
Relative technicion commitment
Staf f ing
Based in Part on
Cooke (2004) & Cabe(1998)
Center for Risk and Reliability
Financial
performanceTechnical System Risk Model
Quality of Maintenance
Maintenance Unit Process Model
Regulatory Auditing System
Individual-level PSFs
Weather
Condition
Emergent
Process
Maintenance Group Safety Climate
Organizational
Safety Structure & Practices
Organizational Safety Culture
A
B
C
D
EF
G
H
I
J
K
L
M
Implementing Implementing SoTeRiASoTeRiA in Aviation Maintenancein Aviation Maintenance
Center for Risk and Reliability
Human Reliability Model
(in System Dynamics Environment)
Total technicions
EPC time pressure
EPC moral
EPC experience
GTT
Time Pressure
demand Time Available
Regulatory allowed working hours
technicion error Probability
~
APOA experience
~
APOA moral
~
APOA time pressure
Relative evperience
of technicions
Relative technicion commitment
Teh Error Probability
input to IRIS
Human Reliability
The module is built partially based on Nuclear Action Reliability Assessment (NARA) (Kirwan, et.al., 2004)
Center for Risk and Reliability
Technician Commitment(in System Dynamics Environment)
A modified version of Cooke’s (2004) model
Center for Risk and Reliability
Financial
performanceTechnical System Risk Model
Quality of Maintenance
Maintenance Unit Process Model
Regulatory Auditing System
Individual-level PSFs
Weather
Condition
Emergent
Process
Maintenance Group Safety Climate
Organizational
Safety Structure & Practices
Organizational Safety Culture
A
B
C
D
EF
G
H
I
J
K
L
M
SoTeRiASoTeRiA in Aviation Maintenancein Aviation Maintenance
Center for Risk and Reliability
Management Commitment Model(in System Dynamics Environment)
• The basic modeling follows Sterman 2000
• Describes the process of groping toward a proper quantity
• Assumes an initial level of commitment which later changes according to safety and financial pressures applied to organization
Management Commitment to safety
Change in Management
Commitment to Safety
refrence technicion error
Target management
commitment to safety
Time to change manaegment
commitment to safety
Pressure to change management
commitment to safety
Effect of relative
tech error
on management commitment
to safet
Relative
technicion error Probabilitytechnicion error Probability
Safety priority
exponent
effect time
Effect of financial pressure on
management commitment to safety
Z Score
FP effect time
Financial priority
exponent
Management drive to prioritize
financial situation over safety
managemnet commitment
Partially referred to Cooke (2004)
Center for Risk and Reliability
Financial
performanceTechnical System Risk Model
Quality of Maintenance
Maintenance Unit Process Model
Regulatory Auditing System
Individual-level PSFs
Weather
Condition
Emergent
Process
Maintenance Group Safety Climate
Organizational
Safety Structure & Practices
Organizational Safety Culture
A
B
C
D
EF
G
H
I
J
K
L
M
SoTeRiASoTeRiA in Aviation Maintenancein Aviation Maintenance
Center for Risk and Reliability
Financial Stress Model
(in System Dynamics Environment)
• “Altman's score” model has been employed As a measure of financial distress
• “Z score” is a linear combination of some financial ratios available on afirm’s balance sheet
Relative
technicion error Probability
Relative Risk
Z Score
Increase rate
in ZDecrease rate
in Z
Z Increasing Multiplier
Z Decreasing Multiplier
Increase
Decrease
Financial Pressure Module
Center for Risk and Reliability
Integration of Software Tools
FT
SD
ESD
MX
BBN
SD
BBN
FT
ESD
Tech. error
probability
Aircraft
Airworthiness
Top event
probability
Risk
STELLA IRIS
Center for Risk and Reliability
Example Causal Loop
Management
Commitment to
Safety
Training
Attrition
Average
Technician
Experience
Technician
Error
Probability
+ +
+ -
+
+
+
-
+
+
+
+
Center for Risk and Reliability
Example Causal Loop:
Safety & Profitability
Management
Commitment
To Safety
Technician
Commitment
To Safety
Technician
Error
Probability
Financial
Pressure
-
+
-
-
+
+
-
+
Center for Risk and Reliability
Typical Output: Financial Stress as a Trigger Point
2:06 PM Fri, Jul 06, 2007Page 1
0.00 4.00 8.00 12.00 16.00 20.00
Years
1:
1:
1:
2:
2:
2:
3:
3:
3:
0
1
1
0
0
1
1: Management Commitment to safety 2: technicion error Probability 3: Z Score
11 1
1
1
1
2
2
2
2
2 2
3
3
3
3
3
3
Center for Risk and Reliability
Typical Output :A period of Low-error Stability
2:16 PM Fri, Jul 06, 2007Page 1
0.00 4.00 8.00 12.00 16.00 20.00
Years
1:
1:
1:
2:
2:
2:
3:
3:
3:
0
1
1
1: Management Commitment to safety 2: technicion error Probability 3: Personal tech Commitment to Safety
1
1
1
1
1
1
22
22
2
2
3
3
3
3
3
3
Center for Risk and Reliability
Typical Output: Total Risk Over 15 Years
1.005500000000000E-08
1.006000000000000E-08
1.006500000000000E-08
1.007000000000000E-08
1.007500000000000E-08
1.008000000000000E-08
1.008500000000000E-08
1.009000000000000E-08
1.009500000000000E-08
1.010000000000000E-08
1.010500000000000E-08
1.011000000000000E-08
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Time
Risk
Series1
Center for Risk and Reliability
Concluding Remarks
� Introduced a 4-layer hybrid dynamic framework for causal modelling of organizational safety risk
� Integration of deterministic (e.g. SD ) and probabilistic (e.g., BBN, ESD, and FT ) modelling methods
� Flexible risk-informed decision making tool with explicit consideration of
� Dynamic effects, such as time lags between decisions and outcomes, and feedbacks such as the impact of incidents on worker awareness and attention to safety
� The uncertain nature of the impact of organizational factors on human performance
� The impact of human performance on the systems and evolution of
risk scenarios
Center for Risk and Reliability
Acknowledgment
� The work described in this paper was in part supported by the US Federal Aviation Administration.
Center for Risk and Reliability
Example Causal Loops:
Management
Commitment to
Safety
Training
Attrition
Average
Technician
Experience
Technician
Error
Probability
- -
+ -
-
-
+
-
+
+