Post on 15-Feb-2022
transcript
How to comply with ISO26262 efficiently – the case for highly automated testing
By Steve Barriault
Technical Sales & Marketing Manager –
Asia
© Vector Software Inc, all rights reserved.
Company introduction
• Founded in 1990 in Rhode Island by embedded engineers
• First release of VectorCAST in 1994
• VectorCAST provides unit and integration testing, as well as system test
coverage tools that are uniquely automated
• These tools can be run on a host, a simulator or directly on a target
• Have worldwide offices and representation
Georgia
Rhode Island
Arizona
Japan
Korea
China
France
Italy
London
Israel
SwedenNetherlands
India
Software testing under ISO 26262
• ISO 26262 is IEC 61508 adapted to the needs of
the Automotive Industry
• Adopts a similar approach to software testing and
code coverage requirements to other, longer-lived
standards (such as DO-178B)
• The challenge: meet its requirements in an
industry where deadlines are coming up much
faster than in Aerospace
– In order to achieve success, efficiency will be key!
Our experience with standards
• Our roots are in the Aerospace industry, where our
15+ year of experience enabled us to work with all
the leading organizations
• Our Automotive business is expanding quickly, with
a growing list of companies using VectorCAST
Tests in ISO 26262B
y S
co
pe • Unit test
• Integration test
• System testB
y G
oa
l • Requirements-based testing
• Interface testing
• Fault injection test
• Resource usage test(*)
Me
tric
Activitie
s • Structural code coverage
• Test-requirement association
(*) Source: Table 12 and 15:
Methods for software unit/integration testing
Requirement-base testing
• Ensures that the software fulfill its mission
– Sometimes called functional test
• Strongly recommended for all levels of ASIL, for both unit
and integration testing
• Stubbing can be performed to enhance your ability to test
low-level requirements in isolation
– They “replace” your existing code so you can better control your
inputs and outputs in the code
– But stubs can take a long time to be generated with scripting-based
tools
– With VectorCAST, the stubs are automatically generated in seconds,
with no user input whatsoever
Requirement-base testing
• The link between the requirement and the test case should
ideally be documented
– In VectorCAST, it can be. Our unit and integration tests can be linked
to specific requirements.
– The test case data (PASS|FAIL) that demonstrate requirements can
be uploaded to a requirement management system such as DOORS
VectorCAST
Test cases
DOORS®
requirements
Req. 1
Req. 2
Req. 3
Test 1
Test 2
Test 3
Execution on
host,
simulator
or target
PASS/FAIL
PASS/FAIL
PASS/FAIL!
External interface test
• External Interface Testing is a subset of functional testing.
• Highly recommended for all ASIL level, both unit and
integration
• It verifies that:
– Functions sent data out in the appropriate format and delivery
mechanism
– Functions that receive data in the appropriate format perform
correctly
– That the behavior when receiving data that is not formatted directly is
known
• Can also be tested by VectorCAST
Other types of test
• Fault injection test:
– Voluntarily inject arbitrary faults to test safety mechanisms
(ex: by corrupting values of variables)
– Recommended for unit/integration testing,
strongly recommended ASIL D (and C in integration)
– In VectorCAST, can provide test cases that have faulty values and
verify that the defensive code gets invoked
• Resource usage test:
– Often only doable on target or at least simulator
– Recommended for unit/integration testing
– Strongly recommended ASIL D
– Our superior degree of target integration can also help you do some
of this, but perhaps not all
Generating test case values
• Based on requirements
– Strongly recommended for all ASIL, unit/integration test
• Equivalence classes:
– This method may be used to partition possible input values of
external interfaces
– Strongly recommended ASIL B, C and D
– VectorCAST has a facility to generate automatically such partitioned
test cases
• Error guessing:
– Here, the tester tries to test errors that are suspected to be error
prone
– Only recommended all ASIL
– Also easily possible in VectorCAST
Generating test case values
• Analysis of boundary values
– Try values approaching, at, or crossing the boundaries, including out
of range value
– Can mean the type range or the functional range
– Strongly recommended for ASIL B, C and D
• VectorCAST has extensive tools to do this
– Auto-generation of MIN-MID-MAX test cases for all the extreme
variable type values
– Import from CSV functional range values – and execution of these in
test cases
Code coverage
• Lets you know when you have been “testing enough”
• Different criteria that require more or less test cases to
achieve
• VectorCAST supports all three criteria recommended by
ISO 26262 (and the “other criteria” – function/call coverage)
Sta
tem
ent • One test case
minimum to execute one line of code
Bra
nch • At decision
point, both TRUE and FALSE to be executed M
CD
C • All operands must independently affect the outcome of the condition
How coverage criteria stack up
• Statement
– Line of code executed at least once
• Branch
– Both the TRUE and FALSE
branches are executed
• MC/DC
– All operands can independently
affect the outcome
if((a || b) && c)
13© Vector Software, all rights reserved
if((a || b) && c) T F
1 test case required
2 tests cases required
if((a || b) && c) RESULT
F F T F
T F T T
F T T T
F T F F
At least n+1 test case
required
ab
c
Knowing what needs
to be done!
Green: Fully covered
(good)
Red: Not covered
(bad)
Orange: Partially
covered (?)
Statement
+
Branch
+
MCDC
+/- critical software
very critical software
What you get - Automaticity
• Unit test environments are generated automatically:
– All drivers and stubs generated with NO user input
• Constructing a test case is done through point and click OR CSV
– NO scripting of ANY kind
• Both Black Box and White Box are allowed
• Execution on target is done 100% automatically
– You click a button, and the rest is done for you
• Execution reports are generated 100% automatically
• Both GUI and command line are available
• Re-running any test environment in regression mode is automatic, even
if the test cases are modified, or if the underlying code is modified
– Regression testing is completely automated
What you get - Flexibility
• Full, guaranteed support for C/C++ of ANY complexity
• Users can control the value of ALL parameters, return values, global
data and data coming from stubs – even for pointers, exceptions, etc.
• Can test individual values, special values (NAN, positive infinity, etc),
range of values, list of values, even call code to generate Monte Carlo-
style of test cases
• Can create complex test cases that set state machines and test their
transition from one state to another (compound test cases)
• Creation of test cases from CSV
• Automatic test case generation based on basis path analysis, MIN-MID-
MAX, and more so as to give a leg up during structural coverage
• Code coverage is displayed in an easy-to-understand way
Other capabilities you get…
• The ability to test libraries – even if you don’t have access to the code!
• The capability to test as you develop (agile development), or even to first
write test cases before writing code (test-driven development)
• The debugger can be used to control test case execution (so the tool
becomes a test vector generator for debugging too)
• The ability to perform timing calculations, in some conditions can also
be done while other processes from the OS are executing
• The ability to stub library functions, if desired…
• … and much more
Quite simply, you are getting a complete test bench that
enables you to comply with ISO 26262 efficiently, so you
can still meet demanding deadlines!
What about tool qualification?
• ISO 26262 Part 8, Section 11 mentions tool qualification– The objective of the qualification of software tools is to provide evidence of
software tool suitability for use when developing a safety-related item or
element, such that confidence can be achieved in the correct execution of
activities and tasks required by ISO 26262
• Depends on how critical the tool’s reliability is to the quality
of the code
• Process more flexible than DO-178B, but if you need tool
qualification, Vector Software has a long experience of
providing this service
– We can provide you documents demonstrating the tool performance
as adequate in your environment: same compiler version, board,
debugger, and tool version
Conclusion
• ISO 26262 contains a number of recommendations that
have been proven effective in other industries– It does not reinvent the wheel but builds on a rich heritage to customize an
unique standard to the needs of the Automotive industry
• Unit/integration testing and code coverage can be very time-
intensive, which no one in this industry can afford!
• The only way to meet ISO 26262, increase code quality and
still meet your deadlines is to invest in test automation
– Fortunately, VectorCAST tools have both the automation and
flexibility required for you to achieve ISO 26262 compliance in a
timely manner
Questions?