Event-B in a Nutshell

Test Data Generation

13th CREST Open Workshop12th-13th of May 2011, London

*) joint work with colleagues from DEPLOY project

Alin Stefanescu - University of Pitesti, Romania

SBT Challenges

Search-Based Software Engineering for Model-Based Testing

Test Generation Approaches

Finite Model Learning Conclusions

Page 2

[Event-B in a Nutshell]

Page 3

Event-B history

Jean-Raymond Abrial (1938- )  

Inventor of the Z and B formal methods.

Z – developed in the 70s

B – developed in the 90s, successfully deployed in industry

Event-B – born with the 21st century

Evolution of B for system level specification

Developement supported by French and European projects:

FP6 RODIN and FP7 DEPLOY

Page 4

DEPLOY project (2008-2012) – funded by FP7

DEPLOY :: Industrial deployment of advanced system engineering methods for high productivity and dependability using formal methods

4 industrial partners■ Bosch, Siemens, SAP, SSF

3 industrial service providers■ Systerel, ClearSy, Cetic

7 academic partners■ Newcastle, Aabo, Düsseldorf, ETH Zurich, ■ Southampton, Pitesti, Bucharest

http://www.deploy‐project.eu

Page 5

Rodin platform for Event-B

Extension of Eclipse IDE (Java-based)

Theorem proving as core technology

Many other Rodin Plug-ins ProB: animation, consistency and model-checking Animators (AnimB) Decomposition Modularisation Team-work Code generation UML-B etc.

Page 6

Event-B in a nutshell

■ State-transition model (like ASM, B, Z)■ set theory as mathematical language■ refinement as basic modeling approach

■ Contexts■ carrier sets (domains)■ constants■ axioms

■ Machines■ global variables■ invariants ■ events that update the variables

■ Events■ local parameters■ guards■ actions

ITEMS := CONTEXT {{it1}, {it34}, {it36}, {it67}, {it89}, {it11}, {it354}, {it876}, {it321}, {it333}, {it78}, {it787}, {it7878}, {it2342}, {it3453}, {it6786}, {it1232}, {it7765}, {it7098})

items : Powerset(ITEMS)

Event-B model

Page 7

[Test Generation Approaches]

Page 8

Test generation based on Event-B

We investigate search-based testing (SBT) techniques for Event-B.

Model-based testing (MBT) is a newly introduced topic in DEPLOY

priority topic for industrial partners like SAP

challenges due to the sheer size of the state space of real-life scenarios

Model-Based Testing (MBT)

Page 9

Future MBT plugin in RODIN

MBT Plug-in

University of Pitesti and University of Dusseldorf

Extra test

information

{ Model-checking }

Event-B model

{ Search-based }

{ Constraint-based }

Test cases

MBT Users

Tool developers:

Page 10

Test generation from Event-B

Event-B model

Global variables: var1, var2, var3, ...

Events ev1(p11,...), ev2(p21,...), ev3(p31,...), ...

ev2(..),ev5(..),...ev3()ev4(..),ev2(..),...ev4(..)........................ev3(..),ev7(..),...ev5(..)...............ev6(..),ev5(..),...ev8

1. Generate a set of tests(sequence of events with concrete param.)

2. Optimize test suite(according to some criteria) – if still needed

ev2(..),ev5(..),...ev3()ev4(..),ev2(..),...ev4(..)...ev3(..),ev7(..),...ev5(..)

SBT Opportunity!

Page 11

What is the explicit state space

Event-B model

Global variables: var1, var2, var3, ...

Events ev1(p11,...), ev2(p21,...), ev3(p31,...), ... ...

State Space of the Event-B model

... ......

(3,4,{a,b},...)

ev3(5)

States given by the values of global variables Transitions labeled by events with concrete parameters

Abstract machine

Page 12

Test generation from Event-B

SBT Opportunity!

State Space of the Event-B model

... ......

(3,4,{a,b},...)

ev3(5)

Approach 1:

Explore the state space using the

ProB model checker

state space explosion mainly due to data

Try: guide the search

Page 13

Test generation from Event-B – part II

SBT Opportunities!

State Space of the Event-B model

... ......

(3,4,{a,b},...)

ev3

Approach 2:

Explore state space ignoring the data

(i.e. local parameters)

Problem 1: still large state space

Then: construct approximations of state space

up to depth K using finite automata

Try 1: using machine learning and static analysis

Try 2: using evolutionary algorithms?

Problem 2: infeasible sequences

Try 1: constraint solving for path feasibility

Try 2: test data generation with metaheuristics

Page 14

[Search-Based Testing Challenges]

Page 15

More details in: A. Stefanescu, F. Ipate, R. Lefticaru, C. Tudose. Towards Search-Based Testing for Event-B Models. To appear in Proc. of 4th International Workshop on Search-Based Software Testing (SBST), 2011.

Let’s take a look at some of specific challenges for Event-B...

Page 16

No explicit state space

Fact Event-B has no explicit states like the EFSMs no control state (as in EFSMs)

Problem Large (possibly infinite) state space testing coverage criteria must be defined only recent work addressing SBT for EFSMs

Possible ideas: coverage of all events (or a given subset of them) or coverage of all test

paths of length < K many other coverages possible, so industrial guidance is needed consider the class of Event-B models with a special state variable (see

industrial use cases from SAP, SSF, Bosch and UML-B models)

Page 17

Non-numerical types

Fact Event-B is based on set theory set relations, powersets, functions, set comprehensions, products, records, etc. Complex structured data (e.g. business domain)

Problem fitness functions in literature mostly defined for numerical types

Possible solutions design new fitness functions for set-based (non-numerical) types efficient encoding of mixed non-numerical/numerical test data

Page 18

Hierarchical models

Fact Event-B supports different types of hierarchy refinement from abstract to concrete levels model decomposition modularity most industrial models use some sort of hierarchy (due to size)

Problem no much previous work on SBT addressing hierarchical models

Possible ideas: adapt existing work on test selection for hierarchical state machines use the existing ProB model checker that can partially deal with hierarchy

Page 19

Non-determinism

Fact Event-B has different types of non-determinism :| or :∈ operators (e.g. x :∈ {item1, ..., item20}) non-deterministic choice of the event to be executed when several enabled non-deterministic choice of parameters (ANY construct) non-deterministic initialisation of variables satisfying the set of invariants

Problem no much previous work on SBT addressing non-deterministism

Possible ideas: devise fitness functions that improve the chance of choosing a given path in a

non-deterministic model (under certain assumptions) make the non-determinism visible (model instrumentation)

Page 20

[Test Data Generation]

Page 21

Generating test data for a path

Problem Given one path of events, provide the test data (event parameters) that

enables the execution of the path.

Approach genetic algorithms encoding of sets into binary genes mixed choromosomes (numerical and binary genes)

More details in:I. Dinca, A. Stefanescu, F. Ipate, R. Lefticaru, C. Tudose. Test Data Generation for Event-B Models using Genetic Algorithms. In Proc. of 2nd International Conference on Software Engineering and Computer Systems (ICSECS'11). CCIS Series, vol. 181, pp. 76-90, Springer, 2011.

Page 22

Test data generation with genetic algorithms

Simulator (ProB)

Fitness evaluation

ITEMS := CONTEXT {{it1}, {it34}, {it36}, {it67}, {it89}, {it11}, {it354}, {it876}, {it321}, {it333}, {it78}, {it787}, {it7878}, {it2342}, {it3453}, {it6786}, {it1232}, {it7765}, {it7098})

items : Powerset(ITEMS)

Encoding of variables

Mutation

Selection

Crossover

Event-B modelitems

…0 1 1 0 1 0

Step 1

• Purchase

Step 2

• ValidateLarge

Step 3

• CheckSpecial

0 1 1 0

1 0 1 0 1 0 0 1

0 1 1 0

0 1 1 1

1 0 10

Fitness functions

“Chromosome”

End?

Page 23

Fitness functions for one path

fitness := approach level + normalized branch level

Classical Tracey’s objective for numerical types New objective functions for set types

Page 24

Examples from the benchmark

Page 25

Statistical results

Statistical comparison of

Genetic Algorithms (GA) and

Random Testing (RT)

on 18 paths covering 5 Event-B models

using statistical test like t-test and U-test

And (of course) the winner is:

® GA performs significantly better than RT on most paths

Note: We are currently evaluating constraint-solving (mature for Event-B). It It seems to be quicker for small to medium path (with exceptions).

Page 26

[Finite Model Learning]

Page 27

Generating finite models from Event-B

Problem There is no explicit state space of an Event-B model

Approach Finite automata learning (adapted L* algorithm ) Aproximation through cover automata K-bound on the length of executions Use finite automata for conformance test generation

More details in:F. Ipate, I. Dinca, A. Stefanescu:Model Learning and Test Generation for Event-B using Cover Automata. Submitted to SEFM’11.

Page 28

First experiments

Preliminary approach

Approximation through cover automata

for bound l Incremental -> fits very well with model

refinements Minimal finite automata

Sometimes difficult to find counterexamples (to the approximation)

Scales for medium size models:

... ......

ev3

Bound l

SBT Opportunity?!

Page 29

[Conclusions]

Page 30

Opportunities for Search-Based Techniques

To wrap-up opportunities:

■ Test suite minimisation with multi-objective optimisation

■ Test data generation for one path with search-based algorithms

■ Construct finite models with evolutionary algorithms

■ Combine ProB model-checker with meta-heuristics

■ Combine ProB constraint-solver with meta-heuristics

■ Experiment with different search algorithms (PSO, ACO, SA,...)

To be answered until end of project (April 2012):

Which of the above work good in practice?