V1.0 | 2019-03-21

The Vector Tool Strategy for Software Testing

The Future is Software!

2

1. Motivation

2. Test Concepts

3. Conclusion

Agenda

3

…

Automotive

Avionics

RailwayMedical

Industry 4.0

Smart Home

The Future is Software!

Motivation

Embedded System

Time

1886 1920 20251980

Innovation & high added value

Combustion Engine

Mechanics

Electronics

Software

4

Cyber Physical System

Control System

e.g. E/E-System, IoT-System

Cyber Physical Systems

Motivation

SUT

Subsystem 1 of the SUT

SWC A

SWC C

SWC B

Subsystem 2 of the SUT

Subsystem 3 of the SUT

Sensors / Actuators

Physical Environment

Connection to mechanics/electric: engine, heating device, …

Software System Environment

• other ECUs or devices

• software in the cloud

• other SW components

Cyber Physical System:

Electrical or mechanical system connected to the real world controlled by software (algorithms)

Physical Environment:

Mechanics/electric connected to the control system via sensors and actuators

Control System:

Software controlling and monitoring the cyber physical system

Software System Environment:

Software not in the focus of the system to be implemented and tested

System under Test:

Software system to be developed and tested, built out of subsystems and software components

5

Challenges and Goals

Motivation

Test the software

early in the develop-

mentprocess

Test the software

independent of real

hardware and HIL test

rigs

Support testing of

highly distributed

systems

Test components

indep. of availability

of other components

Highly parallel testing

Seamless tool chain and test design

methods

6

1. Motivation

2. Test Concepts

3. Conclusion

Agenda

7

Deliver Incrementally

Test Concepts

Ris

k

Ris

k

TimeTime

Minimum Viable

Product

Release Cycle Release Cycles / Iterations

Continuous Feedback

“Walking on water and developing software from a specification are easy if

both are frozen.”

Edward V. Berard

8

Solution:

Software quality gates

Automated at each stage T(x)

Isolate: Target each developer commit – T(0) – “Time Zero”

Verify: Functionality, quality metrics, checks for completeness, traceability, …

Merge

Software Quality Gates

Test Concepts

Release branch

Implement feature

Commit changes to repository

Merge

Software qualitygate

T(x+1)Release branch

T(x)

Dev branch

: T(0): dev desk: → T(1): feature branch → T(2): dev branch: → T(3): release branch

9

Hybrid Test Design: Test Scripts & Models

Test Concepts

SUT

Test Environment

Control System

data flow

control flow

Test Script Models

Physical Environment

&

Software Environment

data flow

Control the models

Parameterization

Variant selection

Fault injection

Test script to verify the behavior of the SUT

Sequentially stimulate SUT

Observe SUT

Models to integrate the SUT into its “natural” environment

Asynchronous interaction between models and SUT

Models may often be very simple, only testing relevant part are modelled

10

SW Environment

& Plant Models

Test Scripts

Functional system tests, unit tests, …

Decomposition and Testing

Test Concepts

SUT

Subsystem 1 Subsystem 2

SWC A

SWC C

SWC B

SWC D

SWC E

Software Environment

Model

Physical Environment

Model

Simulated environment / simulated SUT SW

Real SUT SW running on virtual platform

SWC C

SWC E

SWC D

SWC B

SWC A

“Divide and Conquer” Approach:

SUT is build out of subsystems and software components (features) → divide

When decomposing/partitioning the SUT in a concrete test setup, only a single component or subsystem will be tested → conquer

Integration workflow for SWC A:

Step 1 – full decomposition

Test SWC A in isolation (decomposed SUT)

Other parts of SUT (SWC B, C, D, E) are simulated appropriately

Simulate system and plant environment appropriately

Step 2 – subsystem integrated

Test subsystem 1 (decomposed SUT)

Subsystem 2 is simulated appropriately

Simulate system and plant environment appropriately

Step 3 – SUT fully integrated

Test the whole SUT

No parts of SUT are simulated

Simulate system and plant environment appropriately

11

Interface to the simulation and testing environment

Virtual Execution Environment

Test Concepts

Virtual Execution Environment

SUT

Test Environment

Test Script Models

SUT:

All kind of software

Windows, Linux

ROS, POSIX, …

Virtual execution environment:

Deploy and execute SUT in a virtual environment

Hosted

on PC hardware

in a virtual machine

in the cloud

12

Change-Based Testing

Test Concepts

SUT Source Code

Change-Based Testing:

Source code traceability: Which source code is covered by which test case?

Which test cases are affected by a source code change?

Only test what has changed

reduce test time

find errors faster

improve testing efficiency

Test Cases(unit, integration and system tests)

13

SWC A

Subsystem 2

Subsystem 1

Software SUT

Integration Workflow

Test Concepts

Software SUT

Subsystem 1

SWC A

Isolated Component Subsystem Integrated Fully Integrated SW

SWC C

SWC B

…

SWC A

SWC C

SWC A

SWC C

SWC E

SWC D

SWC B

SWC A

14

Vector Testing Solution – Test Tools for all Test Phases

Test Concepts

System Validation

HW/SW Integration

Test

SWIntegration

Test

SW Unit Test

SWImplementation

So

ftw

are

Syste

m

SWSystem

Test

Lin

k t

o R

equir

em

ents

White-Box Testing + Code Coverage on Host & Target

Black-Box Testing + Code Coverage on Host

Black-Box Testing + Code Coverage on Target

VectorCAST/C++, VectorCAST/RSP

CANoe, vTESTstudio, vVIRTUALtarget, VectorCAST/QA

CANoe, vTESTstudio, VT System, VectorCAST/QA

Para

mete

r and V

ari

ant

Handling

Change-B

ased

Testing

Tra

ceability

Hybri

d T

est

Desig

n, D

ecom

positio

n

…

Vir

tual Execution E

nvir

onm

ents

15

1. Motivation

2. Test Concepts

3. Conclusion

Agenda

16

How to Meet Challenges and Goals

Conclusion

Challenges and Goals: Solution:

Software quality gates

Hybrid test design: test scripts & models

Decomposition: Divide and Conquer

Virtual execution environment

Change-Based Testing

One consistent tool chain for all test levels

17 © 2019. Vector Informatik GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.0 | 2019-03-21

Author:Katranski, UteVector Germany

For more information about Vectorand our products please visit

www.vector.com