1© 2015 The MathWorks, Inc.
Verification and Validation of
High-Integrity Systems
Chethan CU, MathWorks
Vaishnavi HR, MathWorks
2
Growing Complexity of Embedded Systems
Engine Management
Transmission Control
Forward Camera
Electric Power Steering
Smart Junction Box
Smart Junction Box
Battery Management
Propulsion Motor Control
DC/DC Converter
Stability Control
Infotainment
HVAC Control
Navigation
Instrument Panel
Vehicle-to-Vehicle
Vehicle-to-
Infrastructure
Short-Range Radar
Ultrasonic Sensor
Long-Range Radar
Stability Control
AirbagEmergency Braking
Automatic Parking
Adaptive Cruise Control
All-Wheel Drive
Active Damping
4-Wheel Steer
Back-up Camera
Body Control Module
Tire Pressure Monitor
Voice Recognition
Adaptive Front
Lighting
Power Window
Power Seat
Keyless Entry
Power Liftgate
E-Call
2000 2015Lines of Code
16 M
2-3M
6 M
Siemens, “Ford Motor Company Case Study,” Siemens PLM Software, 2014
McKendrick, J. “Cars become ‘datacenters on wheels’, carmakers become software companies,” ZDJNet, 2013
3
Model-Based Design, Verification and Validation
RequirementsExecutable
Specification
Model used for
production code
generation
Simulink Models
C/C++
Generated code
Verified code
ready for
target
deployment
Target
Requirement
based Model
Standards
Compliant
Model
Design Error
free and
Functionally
correct Model
Code
Generation
ready Model
Simulink Models
4
Key Takeaways
▪ Author, manage requirements in Simulink
▪ Early verification to find defects sooner
▪ Automate manual verification tasks
▪ Workflow that conforms to safety standards
▪ Static Source code verification
High Level
Design
Detailed
Design
Coding
Integration
Testing
Unit
Testing
Verified & Validated
SystemSystem
Requirements
5
Poor Requirements Management
Sources: Christopher Lindquist, Fixing the Requirements Mess, CIO Magazine, Nov 2005
Why do 71% of Embedded Projects Fail?
6
Requirements
Challenge with Traditional Development Process
Specification C/C++
Hand code
7
Simulink Models for Specification
Requirements C/C++Executable
Specification
Hand code
8
Complete Model Based Design
Code
Generation
RequirementsExecutable
Specification
Model used for
production code
generation
Simulink Models
C/C++
Generated code
9
RequirementsExecutable
Specification
Model used for
production code
generation
Simulink Models
C/C++
Generated code
Model Based Design Verification Workflow
Component
and system
testing
Equivalence
testing
Equivalence
checking
Review and
static analysis
10
Challenges with Requirements
Where are
requirements
implemented?
How are
they tested?
Is design and
requirements
consistent?
RequirementsExecutable
Specification
Model used for
production code
generation
Simulink Models
C/C++
Generated code
11
Gap Between Requirements and Design
RequirementsExecutable
Specification
Model used for
production code
generation
Simulink Models
C/C++
Generated code
12
Simulink Requirements
Author
Track Manage
13
Requirements Editor
14
Requirements Editor
15
Import
Import Requirements from External Sources
IBM Rational DOORS
Simulink Requirements EditorMicrosoft Word
16
Requirements Perspective
17
Requirements Perspective
18
Link Requirements, Designs and Tests
REQ 3.1 ENABLING CRUISE CONTROL
Cruise control is enabled
when …..
19
Link Requirements, Designs and Tests
REQ 3.1 ENABLING CRUISE CONTROL
Cruise control is enabled
when …..
ENABLE SWITCH DETECTION
If the Enable switch is
pressed ……
Derives
20
Link Requirements, Designs and Tests
REQ 3.1 ENABLING CRUISE CONTROL
Cruise control is enabled
when …..
ENABLE SWITCH DETECTION
If the Enable switch is
pressed ……
Implemented
By
Derives
21
Link Requirements, Designs and Tests
Verified
By
Test Case
x
REQ 3.1 ENABLING CRUISE CONTROL
Cruise control is enabled
when …..
ENABLE SWITCH DETECTION
If the Enable switch is
pressed ……
Implemented
By
Derives
22
Track Implementation and Verification
Passed
Failed
No Result
Missing
Verification Status
Implemented
Justified
Implementation Status
Missing
23
Respond to Change
If the switch is pressed and the counter reaches 50then it shall be recognized as a long press of the switch.
If the switch is pressed and the counter reaches 75then it shall be recognized as a long press of the switch.
ImplementsOriginal Requirement
Updated Requirement
24
Verify Design to Guidelines and Standards
Is the design
built right?
Is it too
complex?
Is it ready
for code
generation?
Requirement
based Model
Standards
Compliant
Model
Design Error
free and
Functionally
correct Model
Code
Generation
ready Model
Simulink Models
Requirements C/C++
25
Automate verification with static analysis
Check for:
• Readability and Semantics
• Performance and Efficiency
• Clones
• And more……Model Advisor Analysis
Requirement
based Model
Standards
Compliant
Model
Design Error
free and
Functionally
correct Model
Code
Generation
ready Model
Simulink Models
Requirements C/C++
26
Generate reports for reviews and documentation
Model Advisor Analysis Model Advisor Reports
Requirement
based Model
Standards
Compliant
Model
Design Error
free and
Functionally
correct Model
Code
Generation
ready Model
Simulink Models
Requirements C/C++
27
Navigate to Problematic Blocks
Requirement
based Model
Standards
Compliant
Model
Design Error
free and
Functionally
correct Model
Code
Generation
ready Model
Simulink Models
Requirements C/C++
28
Guidance Provided to Address Issues or Automatically Correct
Requirement
based Model
Standards
Compliant
Model
Design Error
free and
Functionally
correct Model
Code
Generation
ready Model
Simulink Models
Requirements C/C++
29
Built in checks for industry standards and guidelines
• DO-178/DO-331
• ISO 26262
• IEC 61508
• IEC 62304
• EN 50128
• MISRA C:2012
• CERT C, CWE, ISO/IEC TS 17961
• MAAB (MathWorks Automotive Advisory Board)
• JMAAB (Japan MATLAB Automotive Advisory Board)
Requirement
based Model
Standards
Compliant
Model
Design Error
free and
Functionally
correct Model
Code
Generation
ready Model
Simulink Models
Requirements C/C++
30
Configure and customize analysis
Requirement
based Model
Standards
Compliant
Model
Design Error
free and
Functionally
correct Model
Code
Generation
ready Model
Simulink Models
Requirements C/C++
31
Static
Analysis
Checks for standards and guidelines are often performed late
RequirementsExecutable
Specification
Model used for
production code
generation
Simulink Models
C/C++
Generated code
Rework
32
Static
Analysis
Edit-Time
Checking
Shift Verification Earlier With Edit-Time Checking
RequirementsExecutable
Specification
Model used for
production code
generation
Simulink Models
C/C++
Generated code
• Highlight violations as you edit
• Fix issues earlier
• Avoid rework
33
Find Compliance Issues as you Edit with Edit-Time Checking
34
Assess Quality with Metrics Dashboard
• Consolidated view of
metrics
• Size
• Compliance
• Complexity
• Identify where problem
areas may be
35
Grid Visualization for Metrics
▪ Visualize Standards
Check Compliance
– Find Issues
– Identify patterns
– See hot spots
Red: Fail
Orange: Warning
Green: Pass
Gray: Not run
Legend:
36
Detect Design Errors with Formal Methods
▪ Find run-time design errors:• Integer overflow
• Dead Logic
• Division by zero
• Array out-of-bounds
• Range violations
▪ Generate counter example to reproduce error
Requirement
based Model
Standards
Compliant
Model
Design Error
free and
Functionally
correct Model
Code
Generation
ready Model
Simulink Models
Requirements C/C++
37
Prove That Design Meets Requirements
▪ Prove design properties using formal requirement models
▪ Model functional and safety requirements
▪ Generates counter example for analysis and debugging
Requirement
based Model
Standards
Compliant
Model
Design Error
free and
Functionally
correct Model
Code
Generation
ready Model
Simulink Models
Requirements C/C++
38
Functional Testing
Does the
design meet
requirements?
Is it functioning
correctly?
Is it
completely
tested?
Requirement
based Model
Standards
Compliant
Model
Design Error
free and
Functionally
correct Model
Code
Generation
ready Model
Simulink Models
Requirements C/C++
39
Test Case
Main Model
Systematic Functional Testing
AssessmentsInputs
Test Sequence
Signal Builder
MAT file (input) MAT file (baseline)
Test Assessment
MATLAB Unit Test
and more! and more!
Excel file (input) Excel file (baseline)
Test Harness
40
Manage Testing and Test Results
41
Coverage Analysis to Measure Testing
Simulink• Identify testing gaps
• Missing requirements
• Unintended Functionality
• Dead Logic
Stateflow
Generated Code
Coverage Reports
42
Test Case Generation for Functional Testing
▪ Specify functional test
objectives– Define custom objectives that signals
must satisfy in test cases
▪ Specify functional test
conditions– Define constraints on signal values to
constrain test generator
Test Condition
Test Objective
Test Objective
43
Model-Based Design, Verification and Validation
RequirementsExecutable
Specification
Model used for
production code
generation
Simulink Models
C/C++
Generated code
Verified code
ready for
target
deployment
Target
Requirement
based Model
Standards
Compliant
Model
Design Error
free and
Functionally
correct Model
Code
Generation
ready Model
Simulink Models
44
Equivalence Testing
RequirementsExecutable
Specification
Model used for
production code
generation
Simulink Models
C/C++
Generated code
Is the code
functionally
equivalent to
model?
Is all the
code tested?
45
Equivalence Testing
▪ Processor in the Loop (PIL)
– Numerical equivalence, model to target code
– Execute on target board
Benefits
▪ Re-use tests developed for model to test code
▪ Collect code coverage
▪ Generate artefacts for IEC 61508, ISO 26262,
EN 50128, and DO-178 certification
▪ Early verification and defect detection
RequirementsExecutable
Specification
Model used for
production code
generation
Simulink Models
C/C++
Generated code
Target
Board
▪ Software in the Loop (SIL)
– Show functional equivalence, model to code
– Execute on desktop / laptop computer
Desktop
Computer
PIL
SIL
46
C/C++
Static Code Analysis
RequirementsExecutable
Specification
Model used for
production code
generation
Simulink Models
C/C++
Generated code
Is interface
between
generated and
other code fully
tested?
Is integrated
code free of
run-time
errors?
Other code
Is the code
compliant
to MISRA?
The Generated Code is integrated
with Other Code (Handwritten)
47
Static Code Analysis with Polyspace
▪ Code metrics and standards
– Comment density, cyclomatic complexity,…
– MISRA and Cybersecurity standards
– Support for DO-178, ISO 26262, ….
▪ Bug finding and Code proving
– Detect bugs and security vulnerabilities
– Prove absence of runtime errors
– Check data and control flow of software
Results from Polyspace Code Prover
48
Code Proving with Polyspace
49
Qualify tools with IEC Certification Kit and DO Qualification Kit
▪ Qualify code generation and verification products
▪ Includes documentation, test cases and procedures
BAE Systems Delivers DO-178B Level A Flight
Software on Schedule with Model-Based Design
KOSTAL Asia R&D Center Receives ISO 26262
ASIL D Certification for Automotive Software
Developed with Model-Based Design
50
Summary
1. Author and manage requirements within Simulink
2. Find defects earlier
3. Automate manual verification tasks
4. Reference workflow that conforms to safety standards
5. Static Code verification using Polyspace
51
MathWorks V&V Product Capabilities
Simulink Requirements* (New in R2017b)Requirements
Simulink Check* (New in R2017b)Standards Compliance
Simulink TestTesting
Simulink Design VerifierFormal Verification
Simulink Coverage* (New in R2017b)Coverage Analysis
Polyspace Bug Finder, Polyspace Code ProverStatic Code Analysis
Simulink TestSIL, PIL
* Customers with Simulink V&V licenses will automatically receive these new products
52
KOSTAL Asia R&D Center Receives ISO 26262
ASIL D Certification for Automotive Software
Developed with Model-Based Design
ChallengeDevelop automotive electronic steering column lock
software and certify it to the highest-level functional
safety standard
SolutionUse Model-Based Design to design, implement, and
verify the application software via back-to-back PIL
testing required for ISO 26262 ASIL D certification
Results▪ Development and certification time cut by 30%
▪ 80% of errors identified in modeling phase
▪ PIL test framework for ISO 26262 established
“Using Model-Based Design to design, implement, and
verify our software for the highest functional safety standard
enabled our team to save costs, increase efficiency, and
ensure software quality. Without Model-Based Design,
more engineers would be needed to complete the project in
the same time frame.”
– Cheng Hui, KOSTAL
Kostal’s electronic steering column lock
module.
53
Miele Proves Absence of Run-Time Errors in Control
Software Across Its Entire Product Line
ChallengeMaintain a reputation for producing quality appliances
and other products by minimizing defects in the
control software
SolutionIntegrate Polyspace Code Prover and Polyspace Bug
Finder into the development process to prove the
absence of run-time errors in the software and
enforce standard coding rules
Results▪ Hundreds of source files analyzed daily
▪ Developer focus on core functionality enabled
▪ Reusable, trusted components proven free of
run-time errors
“We have embedded static code analysis with Polyspace
products deeply into our quality assurance processes. It is
much better to find run-time errors as development begins than
to find them at the end of development—or worse, after the
product is delivered.”
- Stefan Trampe, Miele
The Miele Center Gütersloh in Germany.
54
Learn More
Visit MathWorks Verification, Validation and Test Solution Page:
mathworks.com/solutions/verification-validation.html
55
Training ServicesExploit the full potential of MathWorks products
Flexible delivery options:
▪ Public training available in several cities
▪ Onsite training with standard or
customized courses
▪ Web-based training with live, interactive
instructor-led courses
More than 48 course offerings:
▪ Introductory and intermediate training on MATLAB, Simulink,
Stateflow, code generation, and Polyspace products
▪ Specialized courses in control design, signal processing, parallel computing,
code generation, communications, financial analysis,
and other areas
www.mathworks.in/training
56
Verification and Validation of Simulink Models
This one-day course describes techniques for testing Simulink model behavior against system requirements.
Topics include:
▪ Identifying the role of verification and validation in Model-Based Design
▪ Creating test cases for Simulink models
▪ Analyzing simulation results to verify model behavior
▪ Automating testing activities and managing results
▪ Formally verifying model behavior
▪ Automatically generating artifacts to communicate results
57
Polyspace for C/C++ Code Verification
This two-day course discusses the use of Polyspace Bug Finder™ andPolyspace Code Prover™ to prove code correctness, improve softwarequality metrics, and ensure product integrity.
Topics include:
▪ Creating a verification project
▪ Reviewing and understanding verification results
▪ Emulating target execution environments
▪ Handling missing functions and data
▪ Managing unproven code (color-coded in orange by Polyspace® products)
▪ Applying MISRA C® rules
▪ Reporting
58
Thank You!