System SafetyM12 Safety Cases and Arguments V1.2
Matthew Squair
UNSW@Canberra
24 March 2016
1 Matthew Squair M12 Safety Cases and Arguments V1.2
1 Introduction
2 Overview
3 Methodology
4 But do safety cases work?
5 Limitations, advantages and disadvantages
6 Conclusions
7 Further reading
2 Matthew Squair M12 Safety Cases and Arguments V1.2
Introduction
1 Introduction
2 Overview
3 Methodology
4 But do safety cases work?
5 Limitations, advantages and disadvantages
6 Conclusions
7 Further reading
3 Matthew Squair M12 Safety Cases and Arguments V1.2
Introduction
Learning outcomes
Understand what a safety case is
Be able to critically review the content and argument of a safety case
Be able to structure and prepare the content of a safety case
Understand the strengths and weaknesses of the technique
4 Matthew Squair M12 Safety Cases and Arguments V1.2
Overview
1 Introduction
2 Overview
3 Methodology
4 But do safety cases work?
5 Limitations, advantages and disadvantages
6 Conclusions
7 Further reading
5 Matthew Squair M12 Safety Cases and Arguments V1.2
Overview
Overview
“The Nimrod safety case process was fatally undermined by a generalmalaise: a widespread assumption... that the Nimrod was ’safeanyway’ (because it had successfully flow for 30 years) and the task ofdrawing up the safety case became essentially a paperwork and’tickbox’ exercise.”
— C. Haddon Cave, The Nimrod Review
6 Matthew Squair M12 Safety Cases and Arguments V1.2
Overview
Overview
Safety cases
Originated in the British chemical industry CIMAH regulations
Applied to oil industry after the Piper Alpha oil rig fire
Applied to UK Rail after Clapham junction accident
Have become part of the EU safety culture
Embedded in various safety standards
DEF-STAN 00-56DEF (AUST) 5679Australian DMO SAMS FrameworkCMMI SAFE+IEC 61508
7 Matthew Squair M12 Safety Cases and Arguments V1.2
Overview
Overview
Despite it’s prevalence there are serious concerns about it’s practicalapplication [Haddon-Cave 2009] and theoretical underpinnings
We’ll look at the theory and application of safety cases with a focus onarguments in the context of acquisition
We’ll also discuss the problems and limitations of safety cases
8 Matthew Squair M12 Safety Cases and Arguments V1.2
Overview
How is a safety case different to MIL-STD-882?
A MIL-STD-882 system safety program
Is acquisition focused (customer-supplier)
Addresses proximal (system) causes of accidents
Safety Assessment Report is analogue ’ish’ to a safety case
A Safety Case
Can be operation (operator-regulator)
Convince a regulator the plant is safe to operate (WHS)
Can be acquisition developed (DEF STAN 00-56 )
Can be goal (more usual) or rule/standard based*
*Safety cases have traditionally formed part of goal (performance) basedsafety regimes
9 Matthew Squair M12 Safety Cases and Arguments V1.2
Overview
Why do it?
Various reasons
You may need a tool to manage operational safety
You may wish to reduce liability risk
The regulator may require as a ’permit to operate’
You may want to structure and organise safety documentation
You may want to communicate system risk to stakeholders
Be clear about the purpose
Different stakeholders may mean very different things when it comes tosafety cases, be clear about your purpose and who it serves when youprepare one
10 Matthew Squair M12 Safety Cases and Arguments V1.2
Overview
Key definitions
Safety argument. A safety argument is a clear, comprehensive anddefensible argument that explains how the available evidence supports theoverall claim of acceptable safety within a particular context [Kelly 1998]
Safety case. A safety case is a structured argument, supported by a bodyof evidence, that provides a compelling, comprehensible and valid casethat a system is acceptably safe for a given application in a givenenvironment (i.e a context) [MOD (UK) 2007]
Safety case report. The physical artifact(s) that presents the safetyargument and case. Normally the safety case report is not a standalonedocument and will refer out to supporting evidence.
11 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology
1 Introduction
2 Overview
3 Methodology
4 But do safety cases work?
5 Limitations, advantages and disadvantages
6 Conclusions
7 Further reading
12 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology
Methodology [Bishop, Bloomfield 1998]
1 Identify safety requirements
2 Identify system architecture and outline the safety case
3 Assessment (preliminary) of concept design safety trades
4 Progressive elaboration of the design & safety case in parallel
5 Integrate into final safety case
6 Plan for long-term support infrastructure
7 Review and approval8 Long-term monitoring and audits
of areas of concernof support processesto gather field evidence to support assumptions
9 Revise to reflect system and context changes
13 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology Contents of a safety case
Contents
Contains at a minimum[Kelly 1998]:
Supporting evidence on which the case is based, because argumentwithout evidence is unfounded
A high level argument, because evidence without argument isunexplained
May include a number of separate sub-argumentsA convergent conclusion as to the acceptability of the system
A meta-argument as to why the argument and evidence should bebelieved because both evidence and argument can be faulty[Hawkins et al., 2011]
Is the totality of the safety evidence NOT just a safety case report
Structure and organisation is essential to achieve clarity
14 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology Contents of a safety case
Toulmin’s model of practical arguments
Current practices in formal safety argument are based on the practicalargument model [Toulmin 1958]
Focuses on the justification aspects of arguments rather than inferential.Argument parts consist of facts (evidence), conclusions, warrants, backingand qualifiers
The warrant is why it’s considered to move from the fact to the conclusion
The rebuttal is a legitimate constraint that may be placed on theconclusion drawn
Backing is evidence introduced if the warrant on the face of it is notcredible
15 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology Contents of a safety case
Toulmin’s model (cont’d)
16 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology Contents of a safety case
A small philosophical quibble
The problem is that Toulmin developed his model so that one couldanalyse an argument, that is argument is used in the verb sense
Safety arguments tend to inherently skew to an advocacy position, and therebuttal part of Toulmin’s model gets overlooked, that is in safetyarguments the word argument is used as a noun
From there it is a small step to the narrative fallacy e.g. presenting all thatgood data that the system is safe
Of course there’s very little evidence of rare catastrophic events becausethey’re, well, rare...
17 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology Contents of a safety case
A small philosophical quibble
The problem is that Toulmin developed his model so that one couldanalyse an argument, that is argument is used in the verb sense
Safety arguments tend to inherently skew to an advocacy position, and therebuttal part of Toulmin’s model gets overlooked, that is in safetyarguments the word argument is used as a noun
From there it is a small step to the narrative fallacy e.g. presenting all thatgood data that the system is safe
Of course there’s very little evidence of rare catastrophic events becausethey’re, well, rare...
17 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology Formal notations
The problem with words
For larger safety arguments, there is the risk that the amount of words willobscure the argument. One solution is to use ‘semi-structured prose’,where standard terms (evidence, claim, strategy, justification etc) arehighlighted
Example
SFAIRP satisfied argument. The argument establishes the claim (c1),that the system design satisfies the so far as is reasonably practicablecriteria in the context of a definition of what is constitutes reasonablypracticability. To establish the top claim, two sub-claims (c1, c2) areestablished: (c1) all identified hazards have been eliminated, or their riskreduced as low as is reasonably practicable and (c2) that the residual riskis not unacceptably high.
18 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology Formal notations
The problem with words (Pt II)
Other techniques can be applied [Holloway, 2008]:
Use formatting of paragraphs, indenting and numbering.
Mathematical proof (given, by layout) format supported by tabularstatement/reason(s) pairs, John Rushby takes this further
LISP programming language format
19 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology Formal notations
Graphical notations for safety arguments
Two formal graphical notations are available:
Goal Structuring Notation (GSN). Developed by Kelly & others,there is a GSN community standard
Claims, Arguments, Evidence (CAE). Developed by Bishop &others, supported by Adelard’s Safety Case Editor tool
Both are graphical in nature to assist in clarity of argument
Both are based on Toulmin’s practical argument structure
Clarity does not denote soundness
The use of one particular notation or another does not infer any greater orlesser soundness upon the actual worth of the argument
20 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology Formal notations
Graphical notations for safety arguments
GNS versus CAE notation
21 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology Developing the safety case
Developing the safety argument (GSN notation)
1 Establish top level goals (customer/statutory)
2 Record the stakeholders for the goals
3 Define derived requirements (standards, codes etc)
4 Establish (3) as goals (or constraints) and link to top goals
5 Break down the top level goals into sub-goals
6 Show how design & analysis decisions meet goals via strategies
7 Record the decisions as they are made
8 Justify strategies
Evidence versus argument
Evidence without argument is unexplained, argument without evidence isunfounded
22 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology Developing the safety case
Example fragment of a safety argument in GSN notation
23 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology Developing the safety case
Dealing with scale and complexity
GSN has been extended in reason years to include
Safety case modules. Allow the partitioning of cases into moreeasily managed modules and module interfaces (systems of systemsapproach)
Safety case patterns. Standardised templates to encourage re-use ofsuccessful arguments [Kelly, McDermid 1997]
24 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology Developing the safety case
Example modular safety case
Figure: Eurocontrol RVSM pre-implementation safety case
25 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology Developing the safety case
Example modular safety case (cont’d)
Figure: Eurocontrol RVSM Implementation module26 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology Developing the safety case
Safety case patterns
Figure: Safety pattern: functional safety argument
27 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology Maintaining the safety case
Safety case maintenance
In theory, a safety case should be maintained till system retirement
Example
The Long Term Safety Review of the U.Ks Magnox reactors, quoted in[Kelly 1998] found that lack of maintenance to the original safety case had causedit to become inconsistent with current plant design and operations. The reviewfurther found that adding to and re-evaluating a safety case that has become outof date with respect to current safety standards was problematic
In practice, unless effort is expended to maintain the case it rapidly fallsout of date
A commitment to maintain requires regulatory & corporate buy in
For some facilities (such as nuclear) the system life may be up to acentury, longevity of evidence becomes a problem
28 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology Maintaining the safety case
Safety case maintenance
One of the biggest challenges is maintaining the safety case in the face ofsystem changes
We would like to use the safety case to assess changes for safety impact
We also have to repair the case after a change has been made, hopefully ina cost effective fashion
A graphical safety argument with traceability structures is invaluable forthese purposes [Kelly, McDermid 2001]
29 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology Challenging the safety case
Safety arguments as scientific hypothesis
The best tool that we have for differentiating between a good theory and abad one is the scientific method:
our hypothesis is that our system is safe
the argument is why we think this is justified
in science a justifiable hypothesis is not considered proven
in science the hypothesis is then challenged by others
but with safety argument is this (ever) the case?
The safety case as ’proof’ fallacy
An unchallenged safety case is essentially an appeal to authority argument,authority in this case being how impressive the report is
30 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology Challenging the safety case
So how do we challenge a safety case?
Four broad avenues of attack:
Deconstruction
Refutation
Disconfirming evidence
And...
proof by construction that is have an accident or near miss (notrecommended)
The above might seem a lot but (for example) a claim that the likelihoodof a LOCA accident is 10−9 per reactor year is a very strong statement,and strong statements demand strong proof surely?
31 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology Challenging the safety case
So how do we challenge a safety case?
Four broad avenues of attack:
Deconstruction
Refutation
Disconfirming evidence
And... proof by construction that is have an accident or near miss (notrecommended)
The above might seem a lot but (for example) a claim that the likelihoodof a LOCA accident is 10−9 per reactor year is a very strong statement,and strong statements demand strong proof surely?
31 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology Challenging the safety case
So how do we challenge a safety case?
Four broad avenues of attack:
Deconstruction
Refutation
Disconfirming evidence
And... proof by construction that is have an accident or near miss (notrecommended)
The above might seem a lot but (for example) a claim that the likelihoodof a LOCA accident is 10−9 per reactor year is a very strong statement,and strong statements demand strong proof surely?
31 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology Challenging the safety case
Deconstruction
Based on the work of french philosopher Jacque Derrida on the theory ofmeaning (and it’s inherent indeterminacy) and his use of it in critiquingphilosophical arguments [Armstrong, Paynter 2002]
Derrida’s view on arguments
An argument is defined by what it ignores and the perspectives it opposes(explicitly or implicitly)
32 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology Challenging the safety case
Deconstructionist technique
Develop a counter argument that seems warrantable and use this toexpose the internal flaws and contradictions in the original case
1 Reversal. Reverse the argument, ignore how warranted the original is& look for warrantable counter-arguments
2 Displacement. Compare the relative warrantedness of both3 Evaluate the three possible end states
The original argument is found to need revisionThe counter argument is found to need revisionThey both turn out to be equally compelling1
1Due to the limits of deductive closure33 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology Challenging the safety case
Deconstruction (Class exercise)
Modelling software reliability
Argument. Software failures occur randomly because of the random nature ofinputs from the environment that trigger latent faults and that we can applyclassical reliability techniques.
What might be a warrantable counter argument, or arguments?
34 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology Challenging the safety case
Refutation of argument [Greenwell et al. 2006]
Challenge the specific arguments on the basic of fallacious argumentstructures and refute them
35 Matthew Squair M12 Safety Cases and Arguments V1.2
Methodology Challenging the safety case
Disconfirming evidence
Challenge the evidence with disconfirming evidence
Based on Karl Popper’s concept of the science project as one of trying todisconfirm theories not confirm them
Consider
Quality of the evidence provided (pool size, outlier handling, magicbullet approaches)
Hazard control coverage metrics (is the argument vulnerable)
Independence and dissimilarity of evidence sources
Then go out and gather strongly disconfirming evidence that targets thegaps
36 Matthew Squair M12 Safety Cases and Arguments V1.2
But do safety cases work?
1 Introduction
2 Overview
3 Methodology
4 But do safety cases work?
5 Limitations, advantages and disadvantages
6 Conclusions
7 Further reading
37 Matthew Squair M12 Safety Cases and Arguments V1.2
But do safety cases work?
Practical and theoretical problems with the approach
A number of of significant safety cases have been reviewed, and problemsfound with them
Magnox reactor safety review
Haddon enquiry into the Nimrod disaster
Ladkin analysis of the EUROCONTROL RVSM safety case
Knight analysis of Opalinus Clay Nuclear repository safety case
None of these were minor projects, so it appears that even when great careshould be taken, flawed arguments still appear
The theoretical problem is that for high consequence systems thelikelihood must be very, very low and we must have a very high faith in theargument that this is so. Do we?
38 Matthew Squair M12 Safety Cases and Arguments V1.2
Limitations, advantages and disadvantages
1 Introduction
2 Overview
3 Methodology
4 But do safety cases work?
5 Limitations, advantages and disadvantages
6 Conclusions
7 Further reading
39 Matthew Squair M12 Safety Cases and Arguments V1.2
Limitations, advantages and disadvantages
Limitations of the method
Limitations
Relies upon correspondence between safety argument and safety case
Relies upon peoples ability to reason and argue effectively, there’s nota lot of evidence that people are actually good at this
40 Matthew Squair M12 Safety Cases and Arguments V1.2
Limitations, advantages and disadvantages
Advantages
Advantages are that
Is almost mandatory if working in a goal based regulatory environment
Is invaluable in organising the safety program documentation ’tail’
Can promote thought and discussion, if used appropriately
Can provide a change safety impact assessment capability in service
41 Matthew Squair M12 Safety Cases and Arguments V1.2
Limitations, advantages and disadvantages
Disadvantages
Disadvantages are that it
Can become over time, another tick the box exercise
Is vulnerable to the narrative fallacy
Has a tendency to become an advocacy piece
Is very hard to review effectively without formal training
Can become an administrative burden that is perpetually chasing thesystem
42 Matthew Squair M12 Safety Cases and Arguments V1.2
Conclusions
1 Introduction
2 Overview
3 Methodology
4 But do safety cases work?
5 Limitations, advantages and disadvantages
6 Conclusions
7 Further reading
43 Matthew Squair M12 Safety Cases and Arguments V1.2
Conclusions
Conclusions
Safety cases emerged out of the political and industrial landscape ofEngland in the late 1970’s, they reflect a particular societal viewpoint onboth who should be responsible for managing major hazards should bemanaged and therefore how they should manage them.
They are in the end another tool, neither an end in themselves nordemonstrably the only way to assure the safety of complex systems.
Their current demonstrated deficiencies perhaps more demonstrate thedifficulty humans have in arguing rigorously and logically, than any specificlimitations of the method
44 Matthew Squair M12 Safety Cases and Arguments V1.2
Further reading
Bibliography
[Armstrong, Paynter 2002] Armstrong, J. M. and Paynter, S. P. (2002). Safe Systems:Construction, Destruction and Deconstruction. In: Redmill, F. and Anderson, T.(eds.), Current Issues In Safety Critical Systems, pp. 63-76, Springer-Verlag, Berlin.
[Bishop, Bloomfield 1998] Bishop, P. G. & Bloomfield, R. E. (1998). A Methodologyfor Safety Case Development. In: F. Redmill & T. Anderson (Eds.), IndustrialPerspectives of Safety-critical Systems: Proceedings of the Sixth Safety-criticalSystems Symposium, Birmingham 1998.
[DoD (US) 1993] DoD (US) (1993) Standard Practice for System Safety (1993) USDept of Defense Standard MIL-STD-882C, 19 January 1993.
[Greenwell et al. 2006] Greenwell, W. S, Holloway, M., C. Knight, J.C., (2006) ATaxonomy of Fallacies in System Safety Arguments, Proceedings of the 2006International System Safety Conference.
[Haddon-Cave 2009] Cave, C.H. (2006) An Independent Review Into the Broader IssuesSurrounding the Loss Of The RAF Nimrod MR2 Aircraft XV230 In Afghanistan in2006, The Stationary Office, Tech. Rep., 2006
45 Matthew Squair M12 Safety Cases and Arguments V1.2
Further reading
[Hawkins et al., 2011] Hawkins, R., Kelly, T., Knight, J. and Graydon, P. (2011) Anewapproach to creating clear safety arguments, in Proc. SafetyCritical SystemsSymp., Feb. 2011.
[Holloway, 2008] Safety case notations: Alternatives for the non-graphically inclined?, In3rd IET International Conference on System Safety, The Institutions of Engineeringand Technology, Birmingham, UK, Oct. 2008.
[Kelly, McDermid 1997] Kelly T, McDermid J. (1997) Safety case construction andreuse using patterns. In: Proc. 16th Intl. Conf. Computer Safety, Reliability, andSecurity (SAFECOMP97). New York, 1997.
[Kelly 1998] Kelly, T.P., (1998) Arguing Safety, A Systematic Approach to ManagingSafety Cases, Doctoral Thesis, Dept of Computer Science, University of York 1998.
[Kelly, McDermid 2001] Kelly T, McDermid J. (2001) A systematic approach to safetycase maintenance. Reliability Engineering and System Safety 2001;71(3):271-284.
[MOD (UK) 2007] UK MoD (2007) Defence Standard 00-56 Issue 4: Safetymanagement requirements for defence systems, HMSO.
[Toulmin 1958] S. E. Toulmin, S.E., (1958) The Uses of Argument, CambridgeUniversity Press, 1958.
46 Matthew Squair M12 Safety Cases and Arguments V1.2