Automated traceability to assist
DO-254 certification
Andy Nicol, Principal Firmware Engineer
2
©
2016 S
ele
x E
S L
td –
All
rights
reserv
ed
Agenda
• Company Overview.
• Firmware Engineering group overview.
• DO-254 and the problem of traceability.
• What is ReqTracer and how does it help?
• Example outputs.
• Example use case.
• Pros and cons of ReqTracer.
• Summary.
• Questions.
3
©
2016 S
ele
x E
S L
td –
All
rights
reserv
ed
Finmeccanica Airborne & Space Systems Divison
Overview
Integrated Networking Solutions for Netcentric Capabilities.
Sensors & Systems for Homeland Protection, Homeland Defence, ATC/ATM, VTMS.
Mission Critical Systems and Defensive Aids Systems.
4
©
2016 S
ele
x E
S L
td –
All
rights
reserv
ed
Firmware Engineering at Finmeccanica Edinburgh
• The Firmware Engineering discipline is responsible for providing FPGA
expertise to a wide range of projects and products within the Radar and
Advanced Targeting business based in Edinburgh.
• Designs include:
• Radar antenna control.
• Control of aircraft self-protection systems.
• Implementation of radar and image processing algorithms.
• Control of electro-optic turrets.
• Projects have traditionally required designs ‘to the principles of’ DO-254.
• Increasing numbers needing full compliance and certification.
5
©
2016 S
ele
x E
S L
td –
All
rights
reserv
ed
DO-254 Compliance
• Increasing number of customer requests for DO-254 ‘compliance’/certification.
– ‘Compliance’ is a vague term.
• What do we focus on ?
– Difficulties for ongoing or legacy projects.
• High cost/effort for projects where DO-254 wasn’t a customer
requirement at start-up.
• DO-254 focuses on:
– Requirements traceability.
– Verification/validation.
• Automation may offer part of solution on both fronts:
– Reduce errors.
– Cut timescales.
6
©
2016 S
ele
x E
S L
td –
All
rights
reserv
ed
Flow-down / Traceability Problem
Customer Requirements
System Requirements
FPGA Requirements
FPGA Test Requirements
FPGA Testbench
FPGA Design
Test Results
7
©
2016 S
ele
x E
S L
td –
All
rights
reserv
ed
Traceability Problem
• Traceability of requirements documentation well understood.
• Tools such as IBM Rational DOORS manage direct and derived
requirement flow-down from customer requirements to sub-system
requirements.
• Allows (relatively) simple traceability from sub-system requirements back
to customer requirements.
• Traceability from requirements to code or to simulation results has
traditionally been more difficult.
• Problem becomes extremely complex for anything more than the simplest
designs.
• Manual processes are very time consuming and error prone.
8
©
2016 S
ele
x E
S L
td –
All
rights
reserv
ed
What is ReqTracer?
• Mentor Graphics tool which provides an interactive method to trace and
analyse requirements.
– Complete traceability is achieved by linking high level specifications
through to design implementation and verification results.
– Linkage is achieved across a number of different document types.
– Produces a range of customisable reports throughout the design process.
– Interfaces to other key Mentor tools such as HDS.
– Can be used to facilitate DO-254 certification.
9
©
2016 S
ele
x E
S L
td –
All
rights
reserv
ed
How Does ReqTracer Help?
• Requirements traceability:
– From source requirements (DOORS/Word etc.).
– To implementation (HDS/VHDL etc.).
– To verification (Modelsim/Questa).
– Coverage reports.
– Simulation assertions.
• Removes need for compliance spreadsheet:
– Manually intensive.
– Error prone.
– Applied to varying degrees depending on project.
10
©
2016 S
ele
x E
S L
td –
All
rights
reserv
ed
How Does ReqTracer Help?
• ReqTracer reports can be used as evidence of:
– Total coverage, derived requirements, uncovered requirements.
– Impact analysis of individual requirement changes.
– Overall ‘quality’ of the project.
• Helps meet DO-254 by:
- automating tracability.
- reporting and analysing data.
- tracking changes in requirements (the standard mandates changes are
managed effectively).
- ensuring links between requirements, the implemented code and the
verification tests and results.
11
©
2016 S
ele
x E
S L
td –
All
rights
reserv
ed
ReqTracer Environment – Management View
12
©
2016 S
ele
x E
S L
td –
All
rights
reserv
ed
ReqTracer Environment – Project Overview
13
©
2016 S
ele
x E
S L
td –
All
rights
reserv
ed
ReqTracer Environment – Coverage Analysis View
14
©
2016 S
ele
x E
S L
td –
All
rights
reserv
ed
ReqTracer Environment – Impact Analysis View
15
©
2016 S
ele
x E
S L
td –
All
rights
reserv
ed
Verification Plan
• A spreadsheet listing all testing to be carried out, providing the link between
the requirements from DOORS to the coverage from ModelSim/ Questa.
• Created with the aid of the Mentor Graphics Questa Plug-in for OpenOffice.
• Links are directly to assertions/other coverage options present in the UCDB
file.
16
©
2016 S
ele
x E
S L
td –
All
rights
reserv
ed
Questa/ModelSim – UCDB Files
• It is possible to demonstrate that the requirements have been implemented
correctly by saving the simulation results as a UCDB file.
• This file type captures any source of coverage data produced by verification
tools.
– In Questa & ModelSim this is used to store code coverage, assertion data
and functionality coverage.
• Multiple UCDB files can be merged together.
– This means results from multiple small testbenches can be merged to form
a system level coverage result.
– This could vastly cut down the simulation time for a top level testbench.
• Simulation results can also be merged with a verification plan to show that the
testing criteria has been met.
17
©
2016 S
ele
x E
S L
td –
All
rights
reserv
ed
Questa/ModelSim – UCDB Example
• Example of a previously saved UCDB file, code coverage is active,
assertions are present in testbench.
• Assertion hits are 100%.
• In this state, we can look at the success rate of code coverage, but it
does not show these results in the context of the project and its
requirements.
18
©
2016 S
ele
x E
S L
td –
All
rights
reserv
ed
Questa/ModelSim – UCDB Example cont.
• Merging the spreadsheet testplan together with the simulation results
produces the following UCDB:
• Coverage data now has context – shows how much of the testplan has been
successfully verified by the simulation behaviour.
19
©
2016 S
ele
x E
S L
td –
All
rights
reserv
ed
Reports During Design Development
• ReqTracer can provide a range of reports throughout the design & verification
process including:
• Analysis Results.
Complete summary of project & coverage of all documents.
• Traceability Matrix.
Lists upstream-downstream coverage links between two or more
documents.
• Downstream Impact Analysis.
Provides traceability information for specific high level requirements.
Particularly useful for illustrating the impact of changing requirements
on the design.
20
©
2016 S
ele
x E
S L
td –
All
rights
reserv
ed
ReqTracer Pros
• Automates linking of multiple file types.
• Significantly reduces the level of manual effort required.
• Provides multiple graphical views of results.
• A picture is worth a thousand words when showing results to
management…
• Adds value at multiple stages of the life cycle, not just at the close-out /
certification stage.
• Generates a number of custom reports which show the “quality” of your
project, quoting linkage, derived reqs, any attributes you’ve added.
• Can create regular snapshots of a project showing how it has changed
over time.
• show requirements added/modified/deleted.
• addition of new design and test code.
21
©
2016 S
ele
x E
S L
td –
All
rights
reserv
ed
ReqTracer Cons
• Links are static.
• Once a link is created, the requirement could change without flagging the
design and/or test as being out of date.
• Regular expression tool is clunky and unintuitive.
• Relatively high learning curve required to get started.
• Improved integration with Mentor’s HDL Designer (our main development
environment) would be helpful.
22
©
2016 S
ele
x E
S L
td –
All
rights
reserv
ed
Summary
• Clear tracking of requirements to hardware implementation is vital to safety
critical design and helps improve design quality of any complex FPGA / ASIC
design.
• ReqTracer automates and significantly simplifies this process.
• ReqTracer’s various visualisations and reports add value at all project stages.
• Tracks project quality during development.
• Gathers full traceability information during design close down.
• Most importantly – provides required information for DO-254 certification.
23
©
2016 S
ele
x E
S L
td –
All
rights
reserv
ed
Any Questions?