Budapest University of Technology and EconomicsDepartment of Measurement and Information Systems

Exploratory Analysis of the Performance of a Configurable CEGAR Framework

Ákos Hajdu1,2, Zoltán Micskei1

1Budapest University of Technology and Economics,

Department of Measurement and Information Systems2MTA-BME Lendület Cyber-Physical Systems Research Group

24th Minisymposium of DMIS, 31.01.2017.

1

Background – Formal verification

2

Real-life system

Formal model Formal requirement

Verification: explore states

CEGAR

Safe Counterexample

Abstraction Refinement

¬(Red Ʌ Green)

Motivation

Configurable CEGAR framework

o Different algorithm configurations

o Different kinds of models

Which is the “best” configuration?

Preliminary experiment and evaluation

3

Á. Hajdu, T. Tóth, A. Vörös, and I. Majzik, “A configurable CEGAR framework withinterpolation-based refinements,” in Formal Techniques for Distributed Objects,Components and Systems, ser. LNCS. Springer, 2016, vol. 9688, pp. 158–174.

Variables of the problem

Input variables: model

o System type (Hardware/PLC)

o Name

o Number of variables

o Size

Input variables: configuration

o Domain of abstraction (Pred./Expl.)

o Refinement strategy (Craig itp./Seq. itp./Unsat core)

o Initial precision (Empty/Prop.)

o Search strategy (BFS/DFS)

4

Variables of the problem

Output variables

o Is the model safe

o Execution time

o Number of refinement iterations

o Size of the ARG (Abstract Reachability Graph)

o Depth of the ARG

o Length of the counterexample (cex)

5

Measurement procedure

18 input models

o 12 hardware (benchmarks from HWMCC)

o 6 PLC (from a particle accelerator)

20 algorithm configurations

Repeated 5 times

Timeout 480 s

1800 measurement points, 1120 successful

6

Research questions

RQ1: Overall, high level properties

RQ2: Effect of individual input parameters

RQ3: Influence of input parameters on output

Validity

o External: representative input models

o Internal: repetitions, dedicated machine

7

RQ1: Overall, high level properties

8

Many outliers

Small IQR

RQ1: Overall, high level properties

9

Average execution time (ms, log scale)

Easy problems Varying difficulty

High success rate

Single configuration, but short time

PredSeq. Itp.Prop.DFS

RQ2: Effect of individual input parameters

10

Explicit value abstraction more efficient for PLCs

Execution time (ms)

RQ2: Effect of individual input parameters

11

Number of iterations

Less iterations with seq. itp.

Large difference for some PLCs

RQ3: Influence of input parameters on output

12

Predicate domain bad for PLCs

Predicate domain good for hardware

Explicit domain with Craig itp. good in general

Conclusions CEGAR framework

o Different configurations

o Different systems

Preliminary results

o Different configurations are moresuitable for different tasks

o Connections between input andoutput variables

Future work

o Improving the framework

o Further analysis, heuristics

13

inf.mit.bme.hu/en/members/hajdua