PowerPoint Presentation

An Analysis of Challenges in Safety Certification and Implications for Traceability ResearchMehrdad Sabetzadeh

Interdisciplinary Centre for Security, Reliability and Trust (SnT)University of LuxembourgApril 13, 20151Traceability deeply connected with safety certificationTraceability mandated by virtually all safety standardsAny form of safety assessment requires clear links between hazards, risks, and mitigationsEvolution and reuse of safety certification materialManagement of certification activitiesfor example, certification progress monitoring22Background: Safety certificationPurpose: To provide confidence that a system is safe for operationthat is, the system does not pose undue harm to people, property or the environmentCertification often distinguished from other types of safety assessment in that:it is based on standardsit is often done by 3rd-party licensing and regulatory bodies33Process-centered certificationFocused on ensuring that the development process complies with prescriptive guidelinesRationale: Following a good process results in a good productMost safety standards are prescriptiveExamples: IEC 61508 (meta-standard), DO-178B (Avionics), EN50129 (Railways), ISO 26262 (Automotive)4Product-centered certificationFocused on demonstrating that a given system meets its safety objectives (goals)Product guilty until proven innocent! [Jackson et al. 2007]Product-centered certification is an emerging topicGradually being adopted by standardsFor example, goal-based standards in the maritime industry5[Jackson et al. 2007] Software for Dependable Systems: Sufficient Evidence?6SafetyevidenceOverall safety goals for system Goal decomposition (satisfaction arguments)supportsupports

Was the evidence built according to the prescribed process?

Does the evidence justify the satisfaction of systems safety goals (requirements)?Process-CenteredProduct-CenteredSafety / Assurance Case[Kelly & Weaver 2004][Kelly & Weaver 2004] The goal structuringnotationa safety argument notation.The traceability challenge7SafetyevidenceOverall safety goals for system Goal decomposition (satisfaction arguments)supportsupports

Keeping track of how the evidence artifacts were built and the trace links between the artifactsMaintaining the link betweensafety goals (claims),safety arguments,and safety evidenceWhat is safety evidence?Depends on safety standards being usedAbstract taxonomy?Answered through a Systematic Literature Review (SLR)Survey of 218 peer-reviewed studies from 4963 studies published between 1990 and 2012 [Nair et al. 2014]8[Nair et al. 2014] An extended systematic literature review on provision of evidence for safety certification.

Evidence taxonomy9

9Challenges in safety certification(and relationship to traceability)10Challenges in academic literature11

From [Nair et al. 2014] 11Priorities and challenges: a practitioners perspective Based on three surveys in alarge EU project on safety certification (Open Platformfor EvolutioNary Certificationof Safety-critical Systems)Baseline survey (15 responses, 1 per consortium member)Market research survey (85 valid responses from practitioners)Public online survey (52 valid responses)12

OPENCOSS13

Most frequent evidence types used for certification (baseline survey)14

[de la Vara et al. 2012 (a)] Baseline for the evidence management needs of the OPENCOSS platform (Deliverable 6.1) Organization of certification documentation (baseline survey)15

Markup (with cross references)Tool-generatedDatabase15Approach for traceability management (baseline survey)16

Name conventionsSatisfaction arguments16Traceability links maintained(baseline survey)17

17Top 3 certification challenges(from market research)Effort, cost, complexity, inconsistency, bureaucratic (paperwork) (60.7%) Change management (evolving standards, evolving products), differences between national and international standards (21.4%) Rigidity, lagging market and technology (17.9%)18[de la Vara 2012 (b)] Towards a model-based evolutionary chain of evidence for compliance with safety standardsOnline survey results19

Respondents role,level of experience, andlevel of involvement in safety certification projects[Nair et al. 2015] Evidence management for compliance of critical systems with safety standards: A survey on the state of practiceProcess evidence types (frequencies)20

Product evidence types(frequencies)21

Change impact analysis strategies across different projects22

Usage frequency of evidence traceability techniques23

Challenges24

Notable discrepancies between state of the art and state of practice(related to traceability)Graphical notations for argumentation have low adoption in industry despite being well-researchedPossible explanations: tool usability issues, lack of technology transfer, lack of alignment with industrial realitiesChecklists pervasive in industry but not adequately exploited in research activitiesPossible explanations: checklists simple with limited research utility, genuine research gap25Do we need better techniques to deal with natural language?From the trenches

Safety Certification in theMaritime and Energy Domain

ModelME! (2009 2011)26Some personal observations about traceability in safety-critical systemsNo clear knowledge of what traceability information to collect and whyStandards open to interpretation on nearly everything including traceabilityTraceability information models tend to get too complex (the prefect traceability graph problem)Change analysis and evolution still largely manualTraceability information scattered across several tools Traceability to expert judgment and rationaleLimited visibility into 3rd party components2727Overarching challengesCertification too much being a matter of negotiationSoftware still takes a backseat to hardware although this is fast changingBusiness considerations may be conflicting with common sense28Some technical work related to traceability and motivated by safety-critical applications30Writing best practices and automated checking of best practicesDocument vocabulary analysisChange impact analysis for natural language documentsUsing UML profiles for tracing evidence artifactsModel-based design inspections based on traceability information models and design slicingModel-Driven EngineeringNatural LanguageProcessing[Nejati et al. 2012] A SysML-based approach to traceability management and design slicing in support of safety certification[Panesar-Walawege et al. 2013] Supporting the verification of compliance to safety standards via model-driven engineering[Arora et al. 2014] Improving requirements glossary construction via clustering[Arora et al. 2013] Automatic checking of conformance to requirements boilerplates via text chunking[Sabetzadeh et al. 2013] A goal-based approach for qualification of new technologiesGoal-based probabilistic assessment of safety risks[Arora et al. 2015] Change impact analysis for natural language requirements: An NLP approach30Change impact analysis31Requirements (more generally evidence) constantly changing!Every time a change occurs, we need to determine which other requirements (evidence artifacts) may be impacted

Approach32

Solution CharacteristicsAccounts for the phrasal structure of requirements33The mission operation controller shall transmit satellite status reports to the user help desk document repository.user help desk, Deleteduser document repository, Added Addresses a gap in industrial settings where there is often no reliable information other than the text of the requirementsTool Support34

Validated in two industrial case studies34ConclusionFor the conclusion I have only two slides. One about impact and the other about future work35Factors (non-exhaustive) to consider intraceability solutions for safety-critical systemsPrevalent use of natural language in evidence documentsIncreasing use of models (e.g., Simulink, UML, BPMN)Analysis goals (purpose) pursued from traceability, particularly with respect to evolution, monitoring, and governanceAvoidance of traceability overloadSmarter traceability information models capable of dynamic trace derivation (as opposed to static creation of traces)Industry impactFocus on most important evidence artifacts (biggest bang for the buck)3636Challenge in Evidence provisionNMedianUnim-portantOf little ImportanceModerately ImportantImportantVery Important

Determination of confidence in evidence to support a particular claim about system safety48Important0% (0)2.1% (1)20.8% (10)39.6% (19)37.5% (18)

Compliance demonstration for systems whose compliance has not been previously demonstrated48Important2.1% (1)4.2% (2)14.6% (7)41.7% (20)37.5% (18)

Need for providing arguments to show how evidence meets the requirements/objectives of a safety standard49Important2% (1)0% (0)18.4% (9)46.9% (23)32.7% (16)

Provision of adequate process information as evidence for the whole development and V&V process48Important0% (0)4.2% (2)18.8% (9)43.8% (21)33.3% (16)

Suitability and application of safety standards50Important2% (1)6% (3)22% (11)32% (16)38% (19)

How to effectively create and structure safety cases48Important4.2% (2)4.2% (2)20.8% (10)35.4% (17)35.4% (17)

Compliance demonstration for new technologies49Important0% (0)10.2% (5)20.4% (10)34.7% (17)34.7% (17)

Provision of evidence for systems that reuse existing components/subsystems49Important2% (1)8.2% (4)16.3% (8)42.9% (21)30.6% (15)

Determination and decision upon the information that can be provided as evidence47Important0% (0)6.4% (3)23.4% (11)44.7% (21)25.5% (12)

Existence of problems which, based on your experience, are exclusive to the application domain selected and do not arise in others48Important4.2% (2)6.3% (3)25% (12)33.3% (16)31.3% (15)