Quality, Reliability, and (Model-Based) Testing of Embedded...

Quality Reliability

and (Model-Based) Testing

of Embedded Systems

Jan Tretmans

TNO ndash Embedded Systems Innovation Eindhoven

Radboud University Nijmegen

system

under

test

pass fail

TorXakis

model

Consumer Electronics

Medical Systems

Research

Applied Technologies

Research cooperation with leading Dutch

high-tech multinational industries amp SMErsquos

Research cooperation with all Dutch

universities with embedded systems research

Industrial

Network

Academic

Network

Research cooperation in EU projects

TNO

ESI

RU

3

Embedded Systems

Quality amp Testing

4

Embedded Systems

or What do Dykes De Ruyter and

Wafer Scanners have in common

5

What do Dykes De Ruyter and


6



7



8



9



Trend Software in Embedded Systems

10

100

time

rela

tive

eff

ort

SWelectronics

mechanics

physicschemistry etc

1970 2000

[Progress 2006]

Quality of Embedded Systems

Software is brain of system

bull software controls connects monitors

almost any aspect of ES system behaviour

bull majority of innovation is in software

Software determines

quality and reliability

of Embedded System

bull often gt 50 of system defects

are software bugs

11

Quality Software

12

Challenges for Embedded Systems Testing

bull Some Trends Issues and Challenges

for Quality and Testing of Embedded Systems

bull Implications for

Model-Based Testing

ndash Not scientific based on

(subjective) observation

ndash More questions than answers

13

systemsystemtest

subsystemsubsystemtest

componentcomponent test

modulemoduletest

1 multi-disciplinarity

2 complexity

3 connectivity

4 component-based

development

5 change variability and

evolvability

6 uncertainty

bull Combination of physicsmechanicselectronics hellip with computersoftware

bull Requires various expertise

bull Testing such combinations requires simulation

1 Multi-disciplinarity

14

bull Virtualization

ndash models to simulateemulate physical and environment parts in testing

ndash intelligent stub in-the-loop testing

ndash because real system is expensive infeasible dangerous

too slow too fast cannot produce error scenarios hellip


15

16

2 Complexity

Testing effort grows exponentially with system size

x [09]

y [09]

x [09]

x [09]

y [09]

z [09]

10 ways that it can go wrong

10 combinations of inputs to check





Testing cannot keep pace with development

combinatorial explosion of required testing effort

Machine with 300 parameters

bull 2 300

= 10 90

different configurations

bull atoms on earth = 10 50

atoms in known universe = 10 80

Completely testing lsquo + rsquo for 32-bit Int

bull 2 32

2 32

= 10 19

test cases

bull 1 nsec test = 585 years of testing

17

2 Complexity


bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


18

2 Complexity

3 Connectivity

bull Blurring boundaries of systems everything connected

bull Systems-Of-Systems

ndash Dynamically connected systems

ndash Not under own control

bull Software is glue

ndash with internal and external world

bull Testing

ndash what is SUT

bull Virtualization

ndash which systems are available

for testing

ndash which systems must be virtualized

bull Dynamics

ndash run-time testing and integration

19

4 Components Decompose ComplexityDecomposition

Focus of system

architecting

Composition

Focus of

integration

Creating

20

system

legacy

4 Components Integration Challenges

Components come from anywhere

21

4 Integration of Boeing 787

22

5 Change Variability Evolvability

Different customers want different products

ndash different platforms contexts

ndash product lines

ndash variability

ndash mass customization

Combinatorial explosion

which varieties

shall be tested

and when

23

5 Configurations


Importance of regression testing

ndash risk destroy something

which worked previously

changes in systems(-of-systems)

after delivery

run-time testing and health-monitoring

Products evolve ever faster

continuous delivery

deliver fast deliver often

test fast test often

test automation model-based testing

25

Paradoxes

bull software does not suffer from

wear and tear but software is

repaired and patched more

often than any other system

part that does suffer from it

bull late changes are a source of

bugs yet software is presumed

to be flexible so late changes

are still often made

6 Uncertainty

bull Sometimes you donrsquot know hellip

ndash testing a search engine

weather forecast hellip

ndash systems-of-systems

big data

bull Sometimes you donrsquot want to know hellip

ndash no details

ndash abstraction

ndash particular view

Uncertainty of

test outcomes amp oracles

ndash non-determinism

probabilities constraints

coin

button

alarm button

coffee

26

check -20o lt= temp lt= 40o

Trends amp Challenges

multi

disciplinarity

complexity

connectivity

quality

characteristics

change

variability

evolvability

uncertainty

27

components

embedded

systems

quality

amp testing

challenges

28

(Model-Based) Testing

Checking or measuring

some quality characteristics

of an executing software object

by performing experiments

in a controlled way

wrt a specificationtester

specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking

walk-throughdebugging

certification

CMM

quality controlrequirement management

software process

Quality There is more than Testing

verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31

MBT Example Modelscoin

button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

3 Keyword-Driven Testing

high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing

4 Model-Based Testing

36

MBT next step in test automation

bull Automatic test generation

+ test execution + result analysis

bull More longer and diversified test cases

more variation in test flow and in test data

bull Model is precise and consistent test basis

unambiguous analysis of test results

bull Test maintenance by maintaining models

improved regression testing

bull Expressing test coverage

model coverage

customer profile coverage

MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing

with Labelled Transition Systems

and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive

MBT Labelled Transitions Systems

SUT

conforms to

model

40

p p = x LU p x

out ( P ) = x LU | p x pP | p p pP

Straces ( s ) = ( L ) | s

p after = prsquo | p prsquo

Conformance ioco

i ioco s =def Straces (s) out (i after ) out (s after )

s is a Labelled Transition System

i is (assumed to be) an input-enabled LTS

41


Intuition

i ioco-conforms to s iff

bull if i produces output x after trace

then s can produce x after

bull if i cannot produce any output after trace

then s cannot produce any output after ( quiescence )

Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)

out (i after ) out (s after )

non-determinism

uncertainty

under-specification

43

Example ioco Test Generation

i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive

Prove soundness and exhaustiveness

mIOTS

( tgen(s) m passes t )

m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS

SUT passes t mSUT passes t

pass fail

MBT with ioco is Sound and Exhaustive

45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT

test harness verdict

test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink

bull Axini Test Manager

bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG

bull Smartesting CertifyIt

bull Spec Explorer

bull StateMate

bull STG

bull Temppo

bull TestGen (Stirling)

bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport

bull Machine Readable Passport ( MRP E-passport )

bull with chip (JavaCard) contact-less

bull storage of picture fingerprints iris scan

bull access to this data protected by encryption and a new protocol

bull few years ago released in EU

Our job testing of e-passports

bull emphasis on access protocol

== exchange of request-respons messages

between passport and reader (terminal)

53

MBT for E-Passports Model

54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel

e-passportamp wireless

reader

pass fail

test runs

english

specifications

55

MBT for E-Passports Test Runs

56

bull Tested

ndash Basic Access Control (BAC)

ndash Extended Access Control (EAC)

ndash Active Authentication (AA)

ndash Data Reading

bull Tests up to about 2000000 test events

ndash complemented with manual tests

bull No error found

MBT for E-Passports Results

57

MBT Microsoft

bull Microsoft

ndash EU US anti-trust case make Windows more open for competitors

ndash producing documentation for Windows protocols

ndash check documentation wrt real product by model-based testing

documentation modelprotocol

SUTMBT

ndash 75 protocols 50 fte 10000 reqs average 1000 LoMmodel

ndash MBT tool Spec Explorer

ndash Productivity gain 42 and 34 wrt traditional testing

58

MBT Whorsquos Done It

bull MBT User Conference

ndash telecom automotive embedded not so much administrative yet

ndash many companies are experimenting pilot projects

ndash not off-the-shelf

ndash connection to other development tools is important issue

ndash flexibility maintainability is big plus

ndash sometimes combined with scrum agile

ndash not everything with MBT

+ gain (time cost coverage ) 10 - 40

MBT Next Step Test Automation

Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice

MBT State of the Art

60


bull promising emerging

bull a number of successful applications

bull many companies are experimenting

MBT State of Practice

bull lagging behind

Reasons

bull technical

bull tools

bull organizational

bull maturity of testing

bull educational

bull

MBT State for the Future

(for High-tech Embedded Systems)

bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data

MBT Technical Issue State + Data

connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62

MBT Next Generation Challenges

model

composition

abstraction

scalability

uncertaintynondeterminism

concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple

paradigms integration

test

selection

criteria

Model

Based

Testing

63

State of

the Art

MBT Toolsscalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data

abstractionconcurrency

parallelism

usage

profiles for

testing


64

Next Generation MBT TorXakis

scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail

TorXakis On-the Fly MBT

TorXakis

on-the-fly

MBT

Models

bull state-based control flow and complex data

bull support for parallel concurrent systems

bull composing complex models from simple models

bull non-determinism uncertainty

bull abstraction under-specification

66

Tool

bull on-line MBT tool

Under the hood

bull powerful constraintSMT solvers (Z3 CVC4)

bull well-defined semantics and algorithms

bull ioco testing theory

for symbolic transition systems

bull algebraic data-type definitions

TorXakis Overview

But

bull research prototype

bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example

Dispatcher-Processing System

dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4

69hellipexampsDispatchProcs

Example Dispatcher-Processing System


Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71

DisPro01-processortxs

dispatcher


Start

Finish

processor

72

DisPro02-dispatchtxs

Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish

Start Finish Start Finish

Example Two Parallel Processors

73

processor 1

processor 1 | | | processor 2

F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start

Finish Finish Finish

Start Finish Start Finish Start Finish Start Finish

Example Four Parallel Processors

75

ExampleFourParallelProcessors

76

parallelism

Example Dispatcher-Processing SystemStart

Finish

Start Start Start


Start Finish Start Finish Start Finish Start Finish Start Finish

processor(i)

parallel

composition

processors

=

processor(1) ||| processor(2) ||| processor(3) ||| processor(4)

77


Job Start

dispatcher

dispatcher

Job

Finish Finish Finish Finish


dispatch_procs

=

dispatcher |[ Start ]| processors

78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs

ExampleDispatcherProcessingSystem

80


Inputs Job1 Job2 Job3

Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish

FUNCDEF isValidJob ( jobdata JobData ) Bool

=

jobdatajobId gt 0

strinre ( jobdatajobDescr REGEX([A-Z][0-9]2[a-z]+) )

FUNCDEF gcd ( a b Int ) Int

=

IF a == b

THEN a

ELSE IF a gt b

THEN gcd ( a - b b )

ELSE gcd ( a b - a )

FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data

job JobData gcd ( jobx joby )

[[ isValidJob(job) ]]

83

More Complex DataTest data generation from XSD (XML)

descriptions with constraints

complex data

84

Demo Dispatcher-Processing System

85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87

bull Large complex systems-of-systems

bull heterogeneous evolving systems

bull large complex connected

bull complex data with combinatorial explosion

bull customer variations

bull parallelism uncertainty

bull compositionality

bull specifications not always available

bull Domain Specific Language

bull virtual or simulated SUT

model

SUT

pass fail

TorXakis


detecting more bugs faster and cheaper










model coverage


MBT for High-Tech Systems TorXakis

89

Model-Based Testing

Discussion

Test Selection

Test Selection

bull Exhaustiveness never achieved in practice

bull Test selection = select subset of exhaustive test suite

to achieve confidence in quality of tested product

ndash select best test cases capable of detecting failures

ndash measure to what extent testing was exhaustive coverage

bull Optimization problem

best possible testing within costtime constraints

90

Testing and Quality

Test cases

Cost

Quality-assurancecosts

Remaining-defectscosts

92

Test Selection Approaches

1 random

2 domain application specific test purposes test goals hellip

3 model code based coverage

ndash usually structure based

93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection

Extra (domain) information required

bull which test cases have high value

bull which errors are likely

bull which errors have high impact

bull what is the user customer doing

Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion

How to Get these Dhellip Models

bull Everybody wants models

bull Doing nice things with models

ndash Model-based testing

model checking

simulation

bull How to get these models

ndash in particular for

legacy third-party out-sourced

off-the-shelf components

bull Does the model correspond with

the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution

Testing Model-Based Testing

model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling

Test-Based Modeling Research

systemtest

execution

modelModel

Learner

Automatically learning a model of the behavior

of a system from observations made with testing

bull test-based modeling

bull automata learning

bull black-box

reverse engineering

bull observation-based

modeling

bull behavior capture

and test

bull grammatical inference

Learned Model of OCE Printer Module

model

SUT

pass fail

TorXakis

MBT

bull Is it the promising future

of software testing

bull Can we do without it

bull If not MBT what then

MBT for High-Tech Embedded Systems

103

Consumer Electronics

Medical Systems

Research

Applied Technologies

Research cooperation with leading Dutch

high-tech multinational industries amp SMErsquos

Research cooperation with all Dutch

universities with embedded systems research

Industrial

Network

Academic

Network

Research cooperation in EU projects

TNO

ESI

RU

3

Embedded Systems

Quality amp Testing

4

Embedded Systems



5



6



7



8



9




10

100

time

rela

tive

eff

ort

SWelectronics

mechanics


1970 2000

[Progress 2006]






Software determines


of Embedded System


are software bugs

11

Quality Software

12





Model-Based Testing




13

systemsystemtest



modulemoduletest


2 complexity

3 connectivity

4 component-based

development


evolvability

6 uncertainty





14

bull Virtualization






15

16

2 Complexity


x [09]

y [09]

x [09]

x [09]

y [09]

z [09]










bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


17

2 Complexity


bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


18

2 Complexity

3 Connectivity







bull Testing

ndash what is SUT

bull Virtualization


for testing


bull Dynamics


19


Focus of system

architecting

Composition

Focus of

integration

Creating

20

system

legacy



21


22




ndash product lines

ndash variability



which varieties

shall be tested

and when

23

5 Configurations






after delivery



continuous delivery




25

Paradoxes










6 Uncertainty





big data


ndash no details

ndash abstraction


Uncertainty of




coin

button

alarm button

coffee

26



multi

disciplinarity

complexity

connectivity

quality

characteristics

change

variability

evolvability

uncertainty

27

components

embedded

systems

quality

amp testing

challenges

28






in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

3

Embedded Systems

Quality amp Testing

4

Embedded Systems



5



6



7



8



9




10

100

time

rela

tive

eff

ort

SWelectronics

mechanics


1970 2000

[Progress 2006]






Software determines


of Embedded System


are software bugs

11

Quality Software

12





Model-Based Testing




13

systemsystemtest



modulemoduletest


2 complexity

3 connectivity

4 component-based

development


evolvability

6 uncertainty





14

bull Virtualization






15

16

2 Complexity


x [09]

y [09]

x [09]

x [09]

y [09]

z [09]










bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


17

2 Complexity


bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


18

2 Complexity

3 Connectivity







bull Testing

ndash what is SUT

bull Virtualization


for testing


bull Dynamics


19


Focus of system

architecting

Composition

Focus of

integration

Creating

20

system

legacy



21


22




ndash product lines

ndash variability



which varieties

shall be tested

and when

23

5 Configurations






after delivery



continuous delivery




25

Paradoxes










6 Uncertainty





big data


ndash no details

ndash abstraction


Uncertainty of




coin

button

alarm button

coffee

26



multi

disciplinarity

complexity

connectivity

quality

characteristics

change

variability

evolvability

uncertainty

27

components

embedded

systems

quality

amp testing

challenges

28






in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

4

Embedded Systems



5



6



7



8



9




10

100

time

rela

tive

eff

ort

SWelectronics

mechanics


1970 2000

[Progress 2006]






Software determines


of Embedded System


are software bugs

11

Quality Software

12





Model-Based Testing




13

systemsystemtest



modulemoduletest


2 complexity

3 connectivity

4 component-based

development


evolvability

6 uncertainty





14

bull Virtualization






15

16

2 Complexity


x [09]

y [09]

x [09]

x [09]

y [09]

z [09]










bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


17

2 Complexity


bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


18

2 Complexity

3 Connectivity







bull Testing

ndash what is SUT

bull Virtualization


for testing


bull Dynamics


19


Focus of system

architecting

Composition

Focus of

integration

Creating

20

system

legacy



21


22




ndash product lines

ndash variability



which varieties

shall be tested

and when

23

5 Configurations






after delivery



continuous delivery




25

Paradoxes










6 Uncertainty





big data


ndash no details

ndash abstraction


Uncertainty of




coin

button

alarm button

coffee

26



multi

disciplinarity

complexity

connectivity

quality

characteristics

change

variability

evolvability

uncertainty

27

components

embedded

systems

quality

amp testing

challenges

28






in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

5



6



7



8



9




10

100

time

rela

tive

eff

ort

SWelectronics

mechanics


1970 2000

[Progress 2006]






Software determines


of Embedded System


are software bugs

11

Quality Software

12





Model-Based Testing




13

systemsystemtest



modulemoduletest


2 complexity

3 connectivity

4 component-based

development


evolvability

6 uncertainty





14

bull Virtualization






15

16

2 Complexity


x [09]

y [09]

x [09]

x [09]

y [09]

z [09]










bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


17

2 Complexity


bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


18

2 Complexity

3 Connectivity







bull Testing

ndash what is SUT

bull Virtualization


for testing


bull Dynamics


19


Focus of system

architecting

Composition

Focus of

integration

Creating

20

system

legacy



21


22




ndash product lines

ndash variability



which varieties

shall be tested

and when

23

5 Configurations






after delivery



continuous delivery




25

Paradoxes










6 Uncertainty





big data


ndash no details

ndash abstraction


Uncertainty of




coin

button

alarm button

coffee

26



multi

disciplinarity

complexity

connectivity

quality

characteristics

change

variability

evolvability

uncertainty

27

components

embedded

systems

quality

amp testing

challenges

28






in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

6



7



8



9




10

100

time

rela

tive

eff

ort

SWelectronics

mechanics


1970 2000

[Progress 2006]






Software determines


of Embedded System


are software bugs

11

Quality Software

12





Model-Based Testing




13

systemsystemtest



modulemoduletest


2 complexity

3 connectivity

4 component-based

development


evolvability

6 uncertainty





14

bull Virtualization






15

16

2 Complexity


x [09]

y [09]

x [09]

x [09]

y [09]

z [09]










bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


17

2 Complexity


bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


18

2 Complexity

3 Connectivity







bull Testing

ndash what is SUT

bull Virtualization


for testing


bull Dynamics


19


Focus of system

architecting

Composition

Focus of

integration

Creating

20

system

legacy



21


22




ndash product lines

ndash variability



which varieties

shall be tested

and when

23

5 Configurations






after delivery



continuous delivery




25

Paradoxes










6 Uncertainty





big data


ndash no details

ndash abstraction


Uncertainty of




coin

button

alarm button

coffee

26



multi

disciplinarity

complexity

connectivity

quality

characteristics

change

variability

evolvability

uncertainty

27

components

embedded

systems

quality

amp testing

challenges

28






in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

7



8



9




10

100

time

rela

tive

eff

ort

SWelectronics

mechanics


1970 2000

[Progress 2006]






Software determines


of Embedded System


are software bugs

11

Quality Software

12





Model-Based Testing




13

systemsystemtest



modulemoduletest


2 complexity

3 connectivity

4 component-based

development


evolvability

6 uncertainty





14

bull Virtualization






15

16

2 Complexity


x [09]

y [09]

x [09]

x [09]

y [09]

z [09]










bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


17

2 Complexity


bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


18

2 Complexity

3 Connectivity







bull Testing

ndash what is SUT

bull Virtualization


for testing


bull Dynamics


19


Focus of system

architecting

Composition

Focus of

integration

Creating

20

system

legacy



21


22




ndash product lines

ndash variability



which varieties

shall be tested

and when

23

5 Configurations






after delivery



continuous delivery




25

Paradoxes










6 Uncertainty





big data


ndash no details

ndash abstraction


Uncertainty of




coin

button

alarm button

coffee

26



multi

disciplinarity

complexity

connectivity

quality

characteristics

change

variability

evolvability

uncertainty

27

components

embedded

systems

quality

amp testing

challenges

28






in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

8



9




10

100

time

rela

tive

eff

ort

SWelectronics

mechanics


1970 2000

[Progress 2006]






Software determines


of Embedded System


are software bugs

11

Quality Software

12





Model-Based Testing




13

systemsystemtest



modulemoduletest


2 complexity

3 connectivity

4 component-based

development


evolvability

6 uncertainty





14

bull Virtualization






15

16

2 Complexity


x [09]

y [09]

x [09]

x [09]

y [09]

z [09]










bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


17

2 Complexity


bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


18

2 Complexity

3 Connectivity







bull Testing

ndash what is SUT

bull Virtualization


for testing


bull Dynamics


19


Focus of system

architecting

Composition

Focus of

integration

Creating

20

system

legacy



21


22




ndash product lines

ndash variability



which varieties

shall be tested

and when

23

5 Configurations






after delivery



continuous delivery




25

Paradoxes










6 Uncertainty





big data


ndash no details

ndash abstraction


Uncertainty of




coin

button

alarm button

coffee

26



multi

disciplinarity

complexity

connectivity

quality

characteristics

change

variability

evolvability

uncertainty

27

components

embedded

systems

quality

amp testing

challenges

28






in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

9




10

100

time

rela

tive

eff

ort

SWelectronics

mechanics


1970 2000

[Progress 2006]






Software determines


of Embedded System


are software bugs

11

Quality Software

12





Model-Based Testing




13

systemsystemtest



modulemoduletest


2 complexity

3 connectivity

4 component-based

development


evolvability

6 uncertainty





14

bull Virtualization






15

16

2 Complexity


x [09]

y [09]

x [09]

x [09]

y [09]

z [09]










bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


17

2 Complexity


bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


18

2 Complexity

3 Connectivity







bull Testing

ndash what is SUT

bull Virtualization


for testing


bull Dynamics


19


Focus of system

architecting

Composition

Focus of

integration

Creating

20

system

legacy



21


22




ndash product lines

ndash variability



which varieties

shall be tested

and when

23

5 Configurations






after delivery



continuous delivery




25

Paradoxes










6 Uncertainty





big data


ndash no details

ndash abstraction


Uncertainty of




coin

button

alarm button

coffee

26



multi

disciplinarity

complexity

connectivity

quality

characteristics

change

variability

evolvability

uncertainty

27

components

embedded

systems

quality

amp testing

challenges

28






in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103


10

100

time

rela

tive

eff

ort

SWelectronics

mechanics


1970 2000

[Progress 2006]






Software determines


of Embedded System


are software bugs

11

Quality Software

12





Model-Based Testing




13

systemsystemtest



modulemoduletest


2 complexity

3 connectivity

4 component-based

development


evolvability

6 uncertainty





14

bull Virtualization






15

16

2 Complexity


x [09]

y [09]

x [09]

x [09]

y [09]

z [09]










bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


17

2 Complexity


bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


18

2 Complexity

3 Connectivity







bull Testing

ndash what is SUT

bull Virtualization


for testing


bull Dynamics


19


Focus of system

architecting

Composition

Focus of

integration

Creating

20

system

legacy



21


22




ndash product lines

ndash variability



which varieties

shall be tested

and when

23

5 Configurations






after delivery



continuous delivery




25

Paradoxes










6 Uncertainty





big data


ndash no details

ndash abstraction


Uncertainty of




coin

button

alarm button

coffee

26



multi

disciplinarity

complexity

connectivity

quality

characteristics

change

variability

evolvability

uncertainty

27

components

embedded

systems

quality

amp testing

challenges

28






in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103






Software determines


of Embedded System


are software bugs

11

Quality Software

12





Model-Based Testing




13

systemsystemtest



modulemoduletest


2 complexity

3 connectivity

4 component-based

development


evolvability

6 uncertainty





14

bull Virtualization






15

16

2 Complexity


x [09]

y [09]

x [09]

x [09]

y [09]

z [09]










bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


17

2 Complexity


bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


18

2 Complexity

3 Connectivity







bull Testing

ndash what is SUT

bull Virtualization


for testing


bull Dynamics


19


Focus of system

architecting

Composition

Focus of

integration

Creating

20

system

legacy



21


22




ndash product lines

ndash variability



which varieties

shall be tested

and when

23

5 Configurations






after delivery



continuous delivery




25

Paradoxes










6 Uncertainty





big data


ndash no details

ndash abstraction


Uncertainty of




coin

button

alarm button

coffee

26



multi

disciplinarity

complexity

connectivity

quality

characteristics

change

variability

evolvability

uncertainty

27

components

embedded

systems

quality

amp testing

challenges

28






in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

Quality Software

12





Model-Based Testing




13

systemsystemtest



modulemoduletest


2 complexity

3 connectivity

4 component-based

development


evolvability

6 uncertainty





14

bull Virtualization






15

16

2 Complexity


x [09]

y [09]

x [09]

x [09]

y [09]

z [09]










bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


17

2 Complexity


bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


18

2 Complexity

3 Connectivity







bull Testing

ndash what is SUT

bull Virtualization


for testing


bull Dynamics


19


Focus of system

architecting

Composition

Focus of

integration

Creating

20

system

legacy



21


22




ndash product lines

ndash variability



which varieties

shall be tested

and when

23

5 Configurations






after delivery



continuous delivery




25

Paradoxes










6 Uncertainty





big data


ndash no details

ndash abstraction


Uncertainty of




coin

button

alarm button

coffee

26



multi

disciplinarity

complexity

connectivity

quality

characteristics

change

variability

evolvability

uncertainty

27

components

embedded

systems

quality

amp testing

challenges

28






in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103





Model-Based Testing




13

systemsystemtest



modulemoduletest


2 complexity

3 connectivity

4 component-based

development


evolvability

6 uncertainty





14

bull Virtualization






15

16

2 Complexity


x [09]

y [09]

x [09]

x [09]

y [09]

z [09]










bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


17

2 Complexity


bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


18

2 Complexity

3 Connectivity







bull Testing

ndash what is SUT

bull Virtualization


for testing


bull Dynamics


19


Focus of system

architecting

Composition

Focus of

integration

Creating

20

system

legacy



21


22




ndash product lines

ndash variability



which varieties

shall be tested

and when

23

5 Configurations






after delivery



continuous delivery




25

Paradoxes










6 Uncertainty





big data


ndash no details

ndash abstraction


Uncertainty of




coin

button

alarm button

coffee

26



multi

disciplinarity

complexity

connectivity

quality

characteristics

change

variability

evolvability

uncertainty

27

components

embedded

systems

quality

amp testing

challenges

28






in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103





14

bull Virtualization






15

16

2 Complexity


x [09]

y [09]

x [09]

x [09]

y [09]

z [09]










bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


17

2 Complexity


bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


18

2 Complexity

3 Connectivity







bull Testing

ndash what is SUT

bull Virtualization


for testing


bull Dynamics


19


Focus of system

architecting

Composition

Focus of

integration

Creating

20

system

legacy



21


22




ndash product lines

ndash variability



which varieties

shall be tested

and when

23

5 Configurations






after delivery



continuous delivery




25

Paradoxes










6 Uncertainty





big data


ndash no details

ndash abstraction


Uncertainty of




coin

button

alarm button

coffee

26



multi

disciplinarity

complexity

connectivity

quality

characteristics

change

variability

evolvability

uncertainty

27

components

embedded

systems

quality

amp testing

challenges

28






in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

bull Virtualization






15

16

2 Complexity


x [09]

y [09]

x [09]

x [09]

y [09]

z [09]










bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


17

2 Complexity


bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


18

2 Complexity

3 Connectivity







bull Testing

ndash what is SUT

bull Virtualization


for testing


bull Dynamics


19


Focus of system

architecting

Composition

Focus of

integration

Creating

20

system

legacy



21


22




ndash product lines

ndash variability



which varieties

shall be tested

and when

23

5 Configurations






after delivery



continuous delivery




25

Paradoxes










6 Uncertainty





big data


ndash no details

ndash abstraction


Uncertainty of




coin

button

alarm button

coffee

26



multi

disciplinarity

complexity

connectivity

quality

characteristics

change

variability

evolvability

uncertainty

27

components

embedded

systems

quality

amp testing

challenges

28






in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

16

2 Complexity


x [09]

y [09]

x [09]

x [09]

y [09]

z [09]










bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


17

2 Complexity


bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


18

2 Complexity

3 Connectivity







bull Testing

ndash what is SUT

bull Virtualization


for testing


bull Dynamics


19


Focus of system

architecting

Composition

Focus of

integration

Creating

20

system

legacy



21


22




ndash product lines

ndash variability



which varieties

shall be tested

and when

23

5 Configurations






after delivery



continuous delivery




25

Paradoxes










6 Uncertainty





big data


ndash no details

ndash abstraction


Uncertainty of




coin

button

alarm button

coffee

26



multi

disciplinarity

complexity

connectivity

quality

characteristics

change

variability

evolvability

uncertainty

27

components

embedded

systems

quality

amp testing

challenges

28






in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103


bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


17

2 Complexity


bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


18

2 Complexity

3 Connectivity







bull Testing

ndash what is SUT

bull Virtualization


for testing


bull Dynamics


19


Focus of system

architecting

Composition

Focus of

integration

Creating

20

system

legacy



21


22




ndash product lines

ndash variability



which varieties

shall be tested

and when

23

5 Configurations






after delivery



continuous delivery




25

Paradoxes










6 Uncertainty





big data


ndash no details

ndash abstraction


Uncertainty of




coin

button

alarm button

coffee

26



multi

disciplinarity

complexity

connectivity

quality

characteristics

change

variability

evolvability

uncertainty

27

components

embedded

systems

quality

amp testing

challenges

28






in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103


bull 2 300

= 10 90





bull 2 32

2 32

= 10 19

test cases


18

2 Complexity

3 Connectivity







bull Testing

ndash what is SUT

bull Virtualization


for testing


bull Dynamics


19


Focus of system

architecting

Composition

Focus of

integration

Creating

20

system

legacy



21


22




ndash product lines

ndash variability



which varieties

shall be tested

and when

23

5 Configurations






after delivery



continuous delivery




25

Paradoxes










6 Uncertainty





big data


ndash no details

ndash abstraction


Uncertainty of




coin

button

alarm button

coffee

26



multi

disciplinarity

complexity

connectivity

quality

characteristics

change

variability

evolvability

uncertainty

27

components

embedded

systems

quality

amp testing

challenges

28






in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

3 Connectivity







bull Testing

ndash what is SUT

bull Virtualization


for testing


bull Dynamics


19


Focus of system

architecting

Composition

Focus of

integration

Creating

20

system

legacy



21


22




ndash product lines

ndash variability



which varieties

shall be tested

and when

23

5 Configurations






after delivery



continuous delivery




25

Paradoxes










6 Uncertainty





big data


ndash no details

ndash abstraction


Uncertainty of




coin

button

alarm button

coffee

26



multi

disciplinarity

complexity

connectivity

quality

characteristics

change

variability

evolvability

uncertainty

27

components

embedded

systems

quality

amp testing

challenges

28






in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103


Focus of system

architecting

Composition

Focus of

integration

Creating

20

system

legacy



21


22




ndash product lines

ndash variability



which varieties

shall be tested

and when

23

5 Configurations






after delivery



continuous delivery




25

Paradoxes










6 Uncertainty





big data


ndash no details

ndash abstraction


Uncertainty of




coin

button

alarm button

coffee

26



multi

disciplinarity

complexity

connectivity

quality

characteristics

change

variability

evolvability

uncertainty

27

components

embedded

systems

quality

amp testing

challenges

28






in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

system

legacy



21


22




ndash product lines

ndash variability



which varieties

shall be tested

and when

23

5 Configurations






after delivery



continuous delivery




25

Paradoxes










6 Uncertainty





big data


ndash no details

ndash abstraction


Uncertainty of




coin

button

alarm button

coffee

26



multi

disciplinarity

complexity

connectivity

quality

characteristics

change

variability

evolvability

uncertainty

27

components

embedded

systems

quality

amp testing

challenges

28






in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103


22




ndash product lines

ndash variability



which varieties

shall be tested

and when

23

5 Configurations






after delivery



continuous delivery




25

Paradoxes










6 Uncertainty





big data


ndash no details

ndash abstraction


Uncertainty of




coin

button

alarm button

coffee

26



multi

disciplinarity

complexity

connectivity

quality

characteristics

change

variability

evolvability

uncertainty

27

components

embedded

systems

quality

amp testing

challenges

28






in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103




ndash product lines

ndash variability



which varieties

shall be tested

and when

23

5 Configurations






after delivery



continuous delivery




25

Paradoxes










6 Uncertainty





big data


ndash no details

ndash abstraction


Uncertainty of




coin

button

alarm button

coffee

26



multi

disciplinarity

complexity

connectivity

quality

characteristics

change

variability

evolvability

uncertainty

27

components

embedded

systems

quality

amp testing

challenges

28






in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

5 Configurations






after delivery



continuous delivery




25

Paradoxes










6 Uncertainty





big data


ndash no details

ndash abstraction


Uncertainty of




coin

button

alarm button

coffee

26



multi

disciplinarity

complexity

connectivity

quality

characteristics

change

variability

evolvability

uncertainty

27

components

embedded

systems

quality

amp testing

challenges

28






in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103






after delivery



continuous delivery




25

Paradoxes










6 Uncertainty





big data


ndash no details

ndash abstraction


Uncertainty of




coin

button

alarm button

coffee

26



multi

disciplinarity

complexity

connectivity

quality

characteristics

change

variability

evolvability

uncertainty

27

components

embedded

systems

quality

amp testing

challenges

28






in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

6 Uncertainty





big data


ndash no details

ndash abstraction


Uncertainty of




coin

button

alarm button

coffee

26



multi

disciplinarity

complexity

connectivity

quality

characteristics

change

variability

evolvability

uncertainty

27

components

embedded

systems

quality

amp testing

challenges

28






in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103


multi

disciplinarity

complexity

connectivity

quality

characteristics

change

variability

evolvability

uncertainty

27

components

embedded

systems

quality

amp testing

challenges

28






in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

28






in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103





in a controlled way


specificationSUT

System Under Test

Testing

29

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

30

static analysis

reviewing model

checking


certification

CMM


software process


verification

QUALITY

testingtesting

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

system

pass fail

model-based

test

generation

test

execution

model

Model-Based Testing

MBT

next step in

test automation

+ test generation

+ result analysis

31


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103


button

alarm button

coffee

32

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

SUT

System Under Test

pass fail

1 Manual testing

1 Manual Testing

33

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

SUT

pass fail

test

execution

TTCNTTCNtest

cases

1 Manual testing

2 Scripted testing

2 Scripted Testing

34

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

SUT

pass fail

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing


high-level

test notation

35

test

scripts

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

1 Manual testing

2 Scripted testing

3 Keyword-driven

testing

4 Model-based

testing


36











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103











model coverage


MBT Benefits

detecting more bugs

faster and cheaper

SUT

pass fail

model-based

test

generation

test

execution

model

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

38

Model-Based Testing


and ioco

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

39

LTS

model

SUTbehaving as

input-enabled LTS

TTCNTTCNTest

cases

pass fail

LTS

test

execution

ioco

test

generation

set of

LTS tests

SUT passes tests

SUT ioco model

sound exhaustive


SUT

conforms to

model

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

40

p p = x LU p x




Conformance ioco




41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

41


Intuition






Conformance ioco

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

42

coffee

dime

quart

dimequart

dimequart

dime

choc

quart

tea

coffee

dime

tea

specification

model

Example ioco

dime

coffee

dime

choc

dime

tea

i ioco s =def

Straces (s)


non-determinism

uncertainty

under-specification

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

43


i ioco s

i fails t

coffee

dime

tea

specification

model

s

coffee

dime

tea choc

pass failpass fail

generated

test case

t

choc

dime

tea

implementationi

i ioco s =def

Straces (s)


44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

44

s LTS

SUT

i ioco s

test

tool

gen LTS

(TTS)

t SUT

SUT passes gen(s)

SUT comforms to s

soundexhaustive


mIOTS


m ioco s

Test assumption

SUTIMP mSUTIOTS

tTESTS


pass fail


45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

45

Model-Based Testing

Tools

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

46

system

model

SUT

TTCNTTCNTest

cases

pass fail

model-based

test

generation

test

execution

MBT A Tool

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

SUT

system

model

pass fail

model-based

test

generation

test

execution

MBT A Tool

verdict

test result

analysis

requirements

ideas

SUT

test harness

TTCNTTCNtest

cases

47

off-line

MBT

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

system

model

pass fail

model

based

test

generation

+

execution

MBT A Tool

SUT


test result

analysis

requirements

ideas

48

on-the-fly

MBT

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull fMBT

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBTsuite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull PyModel

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull StateMate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

49

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

bull AETG

bull Agatha

bull Agedis

bull Autolink


bull Conformiq

bull Cooper

bull Cover

bull DTM

bull Gst

bull Gotcha

bull Graphwalker

bull JTorX

bull MaTeLo

bull MBT suite

bull M-Frame

bull MISTA

bull NModel

bull OSMO

bull ParTeG

bull PhactThe Kit

bull QuickCheck

bull Reactis

bull Recover

bull RT-Tester

bull SaMsTaG


bull Spec Explorer

bull Statemate

bull STG

bull Temppo


bull TestGen (INT)

bull TestComposer

bull TestOptimal

bull TGV

bull Tigris

bull TorX

bull TorXakis

bull T-Vec

bull Uppaal-Cover

bull Uppaal-Tron

bull Tveda

bull

MBT Many Tools

50

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

51

Model-Based Testing

Applications

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

52

Electronic Passport

New Passport










53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

53


54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

54

SUT

TTCNTTCNTest

cases

model-based

test

generation

test

execution

system

model

model-based

test tool

java

drivers

adapter

state-basedmodel


reader

pass fail

test runs

english

specifications

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

55


56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

56

bull Tested




ndash Data Reading



bull No error found


57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

57

MBT Microsoft

bull Microsoft





SUTMBT




58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

58












Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103


Manual Testing

Scripted

Keyword-Driven

Model-Based Testing

59

state of research

state of practice


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103


60






bull lagging behind

Reasons

bull technical

bull tools

bull organizational


bull educational

bull



bull

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

61

money

button1 button2

coffee tea

n int

[ n 35 ] -gt [ n 50 ] -gt

with data

model

2 MB XML file

4 MB processed

XML file

2 MB with a

Error Messages

[ correct ][ not

correct ]

with large

data


connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

connectivity

systems-of-systems

quest

for

quality

model-driven

development

continuous

complexity

size

uncertainty

heterogeneous

components

62


model

composition

abstraction

scalability


concurrency

parallelism

state +

complex data

usage

profiles for

testing

link to

MBSD

multiple


test

selection

criteria

Model

Based

Testing

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

63

State of

the Art


link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing


64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

64


scalability

link to

MBSD


multiple


test

selection

criteria

model

composition

state +

complex data


parallelism

usage

profiles for

testing

SUT

TorXakis

model

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

65

system

model

SUT

TTCNTTCNTest

cases

pass fail


TorXakis

on-the-fly

MBT

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

Models






66

Tool


Under the hood






TorXakis Overview

But


bull poor usability

Current Research

bull scalability

bull test selection

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

TorXakisModel

67

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

68

Model-Based Testing

TorXakis Example


dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

dispatcher

input

jobs

processor

1

processed

jobs

processor

2

processor

3

processor

4




Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103


Start

job

processor

Finish

job

Start

Finish

processor

70


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103


Start

job

Finish

job

Start Finish

processor

state

transition

system

processor

Idle

Processing

71


dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

dispatcher


Start

Finish

processor

72


Job

Start Finish

processor

Job Start

dispatcher

|[ Start ]|

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

Start

Finish

Start

Finish



73

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

processor 1


F1

S1

0

1

2

F2

S2

0

1

2

F2

S1 S2

F1

F2

S2 S1

F1

F2

S2 S1

F1

00

01

22

10

20 11 02

21 12


processor 2

parallelism

74

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

Start

Finish

Start Start Start




75


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103


76

parallelism


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103


Finish

Start Start Start



processor(i)

parallel

composition

processors

=


77


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103


Job Start

dispatcher

dispatcher

Job



dispatch_procs

=


78

composition

DisPro03-procstxs


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103


Job Start

dispatcher

dispatcher

Job



dispatch_procs

= HIDE [ Start ]IN


NI79

abstraction

DisPro04-hidetxs


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103


80



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103



Job1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3

dispatcher

Job



uncertainty

no unique expected

result

81


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103


uncertainty

non-determinismJob1

J2

J3

F1F2

Finish3

F2

F2

F3

F3 F1

F1 F2

F2

F1

F1

F3

F3


F1

J2

F2

F2

F3

F3

J3F2

J3

F3

F2F1

82


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103


Start

job

processor

Finish

job

Start Finish


=

jobdatajobId gt 0



=

IF a == b

THEN a

ELSE IF a gt b



FI

FI

TYPEDEF JobData

= JobData

jobId Int

jobDescr String

x y Int

state + data



83



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103



complex data

84


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103


85

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

86

Model-Based testing

Discussion

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

MBT by TNO-ESI

87











model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

model

SUT

pass fail

TorXakis












model coverage



89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

89

Model-Based Testing

Discussion

Test Selection

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

Test Selection








90

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

Testing and Quality

Test cases

Cost



92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

92


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103


1 random




93

test a x

a x

ax

a x

100 50

transition coverage

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

94

Test Selection






Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

Test Selection

95

usage

profiling

risk

analysis

statistical

testingcombinatori

al testing

decision

tables

random

test

generation

code

coverage

requirement

coverage

mutation

testing

equivalence

partitioning

boundary

value

analysiscause-

effect

graphing

test

purposes

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

96

Model-Based Testing

Discussion





model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103




model checking

simulation






the real system

button

coffee

buttoncoin alarm

Models

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

system

pass fail

model-based

test

generation

test

execution


model

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103

system

pass fail

model-based

test

generation

test

execution

modelModel

Learner

Test-Based Modeling


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103


systemtest

execution

modelModel

Learner





bull black-box

reverse engineering


modeling


and test



model

SUT

pass fail

TorXakis

MBT


of software testing




103


model

SUT

pass fail

TorXakis

MBT


of software testing




103

model

SUT

pass fail

TorXakis

MBT


of software testing




103

103

Date post:	05-Jun-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Quality, Reliability, and (Model-Based) Testing of Embedded...

Documents