Can we predict safety culture?Safety modelling and analysis of
organizational processes in air traffic
Can we predict safety culture?Safety modelling and analysis of
organizational processes in air traffic
Sybert Stroeve, Alexei Sharpanskykh, Barry Kirwan
EUROCONTROL CARE Innovative III Project
vrije Universiteit amsterdam
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 2
ContentsContents
Safety culture
Organizational modelling
Model results and validation
Conclusions
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 3
Safety in complex organizations Safety in complex organizations
Human1
Human3
Human2
System1
System4System2
System3
Accident
Human6Human4 Human5System5
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 4
Organizational & Safety culture Organizational & Safety culture
Values: What is important
Beliefs: How things work
Norms: The way we do things around here
Organizational culture
Safety of operations
Safety cultureOrganizational culture aspects that impact safety
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 5
Safety culture measurement Safety culture measurement
What is important?
How do things work?
What is the way you do things around here?
Safety culture survey
Safety culturesurvey data
Causal relations with organizational structures and processes?
How to (structurally) improve safety culture?
Effect on safety risk?
Remaining questions / Research aim
?
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 6
Case & SourcesCase & Sources
Case� Safety culture issues in safety occurrence reporting at a specific Air Navigation Service Provider (ANSP-3)
Sources for model development� Safety culture issues identified by EUROCONTROL at ANSP-1&2� Formal organizational documentation of ANSP-3� Interviews at ANSP-3 and Eurocontrol HQ� Agent-based organizational modelling method� Sociological & psychological literature
Sources for model validation� Results of EUROCONTROL safety culture questionnaires at ANSP-3� Results of EUROCONTROL safety culture workshops at ANSP-3
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 7
ContentsContents
Safety culture
Organizational modelling
Model results and validation
Conclusions
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 8
Organizational modelling framework(Sharpanskykh, 2008)Organizational modelling framework(Sharpanskykh, 2008)
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 9
Organization-oriented view (example)Organization-oriented view (example)
Safety
Investigator
Safety
Manager
Safety Investigation Unit
Level 1
Level 2
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 10
Performance oriented view (example)Performance oriented view (example)
………
Level of details of (incident/ hazard) notification reports
It is required to maintain a sufficient level of details of (incident/hazard) notification reports
18.2
Proficiency level of incident investigators
It is required to maintain a high proficiency level of incident investigators
18.1
Timeliness and quality of the incident investigation
It is required to maintain timeliness and a high quality of the incident investigation
18
Performance indicatorGoalN
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 11
Process-oriented view (example)Process-oriented view (example)
Create a
notification report
Investigation of
an occurrence
Begin
begin_or(or1)
Report
occurrence?
Yes
Preliminary processing
of a notification report
begin_or(or3)
Decision
positive?
Making decision about
the investigation
necessity
Discussion of the
intermediate occurrence
investigation results
Distribute the final
safety occurrence
assessment report
end_or
(or4)
No
Yes
end_or
(or3)
No
End
Making decision
on the occurrence
reporting
Update an interim safety
occurrence assessment
report
begin_or(or2)
Report provided
to SIU?
Making decision about
the notification report
assignment
Preliminary
assessment of
an occurrence
Yes
Implementation of
safety
recommendations
end_or
(or2)
No
begin_or(or4)
Recommenda-
tions endorsed?
Yesend_or
(or1)
No
Responsibilities for task Investigation of an occurrence
Execution and making technological decisions: Safety Investigator roleMonitoring, consulting and making managerial decisions: Safety Manager role
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 12
Agent-oriented viewAgent-oriented view
Controller
Agent 1
Controller
Agent 4
Controller
Agent 3
Controller
Agent 2Supervisor
Agent
Manager
Agent
Internal
model
Environment
(external world)
observation
action
communication
actionobservationMulti-agent organization
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 13
Internal agent perspectiveInternal agent perspective
Communication
Observation
Communication
Action
beliefs
knowledge
Information attitudes
desires
commitments
intentions
Pro-attitudes
Characteristics
needs
decisionmaking
skillspersonality
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 14
Agent’s information and pro-attitudesAgent’s information and pro-attitudes
Dynamic internal states: Temporal Trace Language
Relations between internal states: Causal Networks
Commitment to safety
of a controller
Priority of safety-related
goals in the role
description
Perception of
commitment to safety of
management
Perception of
commitment to safety of
team
Influence of a controller
on safety activities
Maturity level w.r.t. ATC
task
e1
e2
e3
e4
e5
e6
w1
w2
w3
w4
w5
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 15
Agent’s decision making: Vroom’s expectancy theoryAgent’s decision making: Vroom’s expectancy theory
Report an
occurrence Own group approval
Contribution to
organizational safety-
related goals
Self-esteem, self-
confidence, and self-
actualization needs
Administrative reprimand
Improvement of safety
Material reward Extrinsic needs
E12
E13
E14
E15
I21
I32
V1
V2
V5
V4
Social interaction needs
Own group appreciation of
the action
Management approval V3
Management appreciation of
the action
E16
I43
I51
Intrinsic needs
I64
I65
I35
I45
Decrease of own professional
status in own group
Decrease of own professional
status in management’s
opinion
E17I72
I75
E18
I83
I85
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 16
Model input (examples) Model input (examples)
Evidence input variables (23)� e1: Priority of safety-related goals in the role description � e4: Influence of a controller on safety activities � e7: Sufficiency of the amount of safety investigators � e10: Sufficiency and timeliness of training for changes � e19: Self-confidence for ATC task � e35: Intensity of informal interactions in the team of controllers� e61: Individualism index of a controller
Weights (60)� Couplings between evidence variables in causal networks
Other parameters (115)� Task durations� Decision making parameters� Environment
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 17
Model outputModel output
Perceived management commitment to safetyI7
Perceived supervisor commitment to safetyI6
Perceived team commitment to safety I5.2
Commitment to safety of controllersI5.1
Quality monthly safety overview reportsI4.1
Quality of safety occurrence assessment reportsI3.1
Quality of processed notification reports I2.1
Reporting quality by controllersI1.1
Safety culture indicatorIndex
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 18
ContentsContents
Safety culture
Organizational modelling
Model results and validation
Conclusions
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 19
Validation PhasesValidation Phases
Model
Questionnaire
1 Safety culture indicators
Organizational information
Model
Questionnaire
2 Safety cultureindicators
Organizationalinformation
Model
Questionnaire3
Organizationalinformation Sensitivity analysis
Workshops
Factors &Recommendations
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 20
Results of Phase 1Results of Phase 1
MediumHighPerceived management commitment to safetyI7
LowHighPerceived supervisor commitment to safetyI6
MediumHighPerceived team commitment to safety I5.2
MediumHighCommitment to safety of controllersI5.1
MediumHighQuality monthly safety overview reportsI4.1
MediumMediumQuality of safety occurrence assessment reportsI3.1
-HighQuality of processed notification reports I2.1
MediumHighReporting quality by controllersI1.1
SurveyModelSafety culture indicatorIndex
Model
Questionnaire
Safety culture indicators
Organizational information
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 21
Updates of model input valuesin Phase 2 (examples)Updates of model input valuesin Phase 2 (examples)
I have the opportunity to provide input in the ATC systems development or acquisition process.
We are consulted about changes to the technical/engineering system that impact on the
way we do our work0.500.70
Our opinion and input into safety assessments are actively sought after.
Influence of a controller on safety activitiese4
The other people in the organization do not understand the safety roles we fulfil
Safety is a responsibility shared throughout the organization 0.700.90
Balancing safety against the other requirements of my job is a challenge
Priority of safety-related goals in the role
descriptione1
NewOld
Value of variableRelated survey questionsDescriptionVariable
Model
Questionnaire
Safety cultureindicators
Organizationalinformation
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 22
Results of Phase 2Results of Phase 2
MediumMediumPerceived management commitment to safetyI7
LowMediumPerceived supervisor commitment to safetyI6
MediumMediumPerceived team commitment to safety I5.2
MediumMediumCommitment to safety of controllersI5.1
MediumMediumQuality monthly safety overview reportsI4.1
MediumMediumQuality of safety occurrence assessment reportsI3.1
-HighQuality of processed notification reports I2.1
MediumMediumReporting quality by controllersI1.1
SurveyModelSafety culture indicatorIndex
Model
Questionnaire
Safety cultureindicators
Organizationalinformation
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 23
Major Organizational Factors (Phase 3)Major Organizational Factors (Phase 3)
Sufficiency and timeliness of training for changes8
Developed and implemented Safety Management System (SMS)7
Influence of a controller on safety activities6
Availability of reliable and ergonomic technical systems for controllers
5
Priority of safety-related goals in the role description4
Sufficiency of the number of safety investigators3
Level of development of managerial skills of supervisors2
Sufficiency of the number of controllers1
Major Organizational Factors
Model
Questionnaire
Organizationalinformation
Sensitivity analysis Major Organizational
Factors
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 24
Major Organizational Factors (Phase 3)Major Organizational Factors (Phase 3)
GoodSufficiency and timeliness of training for changes8
GoodDeveloped and implemented Safety Management System (SMS)7
GoodInfluence of a controller on safety activities6
GoodAvailability of reliable and ergonomic technical systems for
controllers5
SomePriority of safety-related goals in the role description4
GoodSufficiency of the number of safety investigators3
GoodLevel of development of managerial skills of supervisors2
SomeSufficiency of the number of controllers1
AgreementMajor Organizational Factors
Model
Questionnaire
Organizationalinformation
Sensitivity analysis
Workshops
Major Organizational
Factors
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 25
Recommendations (Phase 3)Recommendations (Phase 3)
1. More involvement of controllers in safety assessment for development of new systems and procedures
2. Improve workload of controllers by developing explicit rules for balancing safety and capacity in nominal and non-nominal conditions
3. Improve the quality of management by supervisors
4. Improve coherence and communication in the safety management system
5. Improve the communication about and training for changes
Model
Questionnaire
Organizationalinformation Sensitivity analysis Factors &
Recommendations
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 26
Comparison of model-based and survey-based recommendationsComparison of model-based and survey-based recommendations
The model-based recommendations are all consistent with those of the survey study
The survey-based recommendations reflect the larger detail in the organizational context as emerged in the survey workshops and are thus more specific
Additional recommendations identified in the survey study for aspects out-of-the-scope of the model, e.g.� On-the-job-training� Learning processes at the Engineering department
Model
Questionnaire
Organizationalinformation Sensitivity analysis
Workshops
Factors &Recommendations
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 27
ContentsContents
Safety culture
Organizational modelling
Model results and validation
Conclusions
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 28
Can we predict safety culture?Can we predict safety culture?
Valid predictions of safety culture indicators were achieved on the basis of the model and the survey questionnaire input data
Valid predictions of major organizational factors influencing safety culture and consistent recommendations were achieved on the basis of a sensitivity analysis
The agent-based organizational modelling approach can be an added value in the safety culture survey method� The model provides a structured and formally grounded means for analysis and improvement of safety culture
� Sensitivity analysis helps identifying key factors� The modelling approach is expected to support enhancing questionnaires and preparing safety culture survey workshops
Yes we can!
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 29
Beyond our validated caseBeyond our validated case
The value of organizational modelling� Systematic analysis of organizational structures and processes � Agent-based versus traditional methods (system dynamics, operation research)
� Emergent behaviour at various aggregation levels (individuals, units)
Enhance application and scope for safety culture modelling� Modelling and assessment of other ANSPs� Broadening the scope of the organizational model� Modelling the effect of national culture on safety culture � Modelling goal-directed organizational change processes
Relation between safety culture and safety performance� Agent-based model of safety culture influencing operational performance
� Risk assessment by agent-based model of operational performance
EUROCONTROL Innovative Research Workshop, Brétigny-sur-Orge, France, 1 December 2009 30
Questions / DiscussionQuestions / Discussion