October 29, 2004 SAVCBS’04 Presented by: Gaoyan Xie

CTL Model-checking for Systems with Unspecified Components

Gaoyan Xie and Zhe Dang

School of Electrical Engineering and Computer Science, Washington State University, Pullman, WA 99164, USA

SAVCBS’04

Motivation

Challenges to the quality assurance of component-based systems: Externally obtained components may be new

sources of system failures Safely upgrading a deployed component is

extremely difficult Example:

The Ariane 5 disaster was caused by reusing certain software module from Ariane 4 without sufficiently testing it in the new environment

SAVCBS’04

Two Dimensions of The Possible Solutions

Methodologies Compositional approachesconventional, used most often

Bottom-up, from components to systems Decompositional approachesnot many can be automated

Top-down, from systems to components Techniques

Software Testing A trial and error technique Integration/system testing is often needed at the system level

Model-checking: A formal and usually complete technique The assume-guarantee paradigm is often used for system-level

verification

SAVCBS’04

Testing a Component in Isolation

Before it is released By component developers By third-party labscomponent certification Difficulty:

A component may target a wide scope of deployment environments, which may not be tried out

When it is integrated into a system By system developers Expensive:

A component is usually built with multiple sets of functionalities Infeasible:

The state space of a component’s interface may be too large to be thoroughly tested

SAVCBS’04

Testing A Component-based System (CBS)

Integration/system testing Difficulties:

Components within a CBS may run concurrently Even inside a component may have multi-threads running

concurrently, and Concurrency is notoriously difficult to test Locating the source of a bug is not easy!

Feasibility: A component may be used for dynamic upgrading a running

system, which may not be stopped for testing

SAVCBS’04

Model-checking Checks whether a finite-state system is a model of a

given temporal formula Requires a complete and formal specification of the

finite-state system Not always possible: design details or source codes of

components in a CBS are generally unavailable The assume-guarantee paradigm is often used for

system-level verification A compositional approach Requires an assumption for the environment of a module

SAVCBS’04

The Research Problem

A different perspective How to ensure that a component functions

correctly in a specific deployment environment (which is called a host system)

A model-checking problem Given a host system M with an unspecified

component X and a CTL formula f, check whether f is a model of (M, X):

(M, X)= f

SAVCBS’04

Basic Ideas

Goal Seek a definite answer (no approximation) to the

model-checking problem, i.e., (M, X)= f Solution

Automatically deduce a sufficient and necessary condition over the unspecified component

Check the condition through testing the unspecified component

SAVCBS’04

The System Model A system Sys= (M, X) consists of a host system M

and an unspecified component X The host system M is a well-specified finite state

transition system The unspecified component X can be viewed as a

deterministic Mealy machine Internal detail of X is unknown Interface (input/output symbols) and an upper bound for

the number of states in X are known An implementation of X is available for testing

M and X run concurrently and communicate with each other by synchronizing over X’s input/output symbols

SAVCBS’04

An Example

s0 s1 s2

s4

msg? send/yes

ack/yes

send/no

ack/yes

msg?

msg?

Commdata

ack

yes

no

Host system M

Check: starting from state s0, is state s3 reachable? (M, Comm), s0= E[true U s3]

s3s3

SAVCBS’04

Check: starting from state s0, is state s3 reachable? M, s0= E[true U s3]

s0 s1 s2

s4 s3

Transition system M

An Examplecontd.

SAVCBS’04

Check: starting from state s0, is state s3 reachable? M, s0= E[true U s3]

Search the graph to see whether there is a path between s0 and s3

s0s1s4s3 is such a witness path

s0 s1 s2

s4 s3

Transition system M

An Examplecontd.

SAVCBS’04

s0 s1 s2

s4 s3

msg? send/yes

ack/yes

send/no

ack/yes

msg?

msg?

Commsend

ack

yes

no

Test Comm to see whether the communication trace along the path can be accepted by Comm:

Will “send/no, ack/yes” be a successful test case for Comm?

Host system M

Check: starting from state s0, is state s3 reachable? (M, Comm), s0= E[true U s3]

Search the graph to see whether there is a path between s0 and s3

Is s0s1s4s3 still a witness path?

An Examplecontd.

SAVCBS’04

s0 s1 s2

s4 s3

msg? send/yes

ack/yes

send/no

ack/yes

msg?

msg?

Infinite number of paths could be witness path: s0s1s2s1s4s3, s0s1s2s3, s0s1s2s1s2…s3,…

Infinite number of communication traces need to be tested: “send/yes send/no ack/yes”, “send/yes ack/yes”, “send/yes send/yes…

ack/yes”, …

Commsend

ack

yes

no

Host system M

An Examplecontd.

SAVCBS’04

s0 s1 s2

s4 s3

msg? send/yes

ack/yes

send/no

ack/yes

msg?

msg?

Use an annotated directed graph Gh (called witness graph) to represent all those communication traces

The original model-checking problem is: True, if one of the communication traces in Gh is a successful test case for Comm False, if none of the communication traces in Gh is a successful test case for Comm

s1 s2

s4 s3

send/yes

ack/yes

send/no

ack/yes

Host system M Witness graph Gh

An Examplecontd.

SAVCBS’04

An Examplecontd. How to handle communication traces with infinite

length? A result from traditional black-box testing:

An equivalent internal structure of a black-box can be recovered through testing if only an upper bound for its number of states is known

This result implies that Only communication traces with bounded length need to be

tested The bound can be calculated from the number of states

in M and the upper bound for the number of states in Comm

SAVCBS’04

Our Algorithm The original CTL model-checking algorithm for

checking M= f: Goes through a series of stages. A subformula h of f is processed at each stage:

h is added to the labels of a state s if h is true at s Returns true if s0 is labeled with f, or false otherwise

Our algorithm adapts the original algorithm to handle systems with unspecified components.

It follows a similar structure, except that during each stage for subformula h, the labeling of a state s is far more complicated

SAVCBS’04

Our Algorithm—contd. A state s could be labeled with

1 if h is true at s and the truth has nothing to do with communications A witness graph if h is an CTL operator and the truth of h at s depends

on communication traces in the graph A logical combination of witness graphs if h is a logic combination of

other formulas and the truth of h at s depends on communication traces in the graph

For each CTL operator in h, a witness graph is constructed and associated with an ID (which begins from 2)

A state is labeled with an ID expression (constructed from IDs and logic connectives , )

A labeling function Lh is returned upon the completion of processing h

SAVCBS’04

Our Algorithm—contd. The algorithm outline:

ProcessCTL is the lableling process, it recursively handles the five cases of f: , , EX, EU, and EG by calling the procedures Negation, Union, ProcessEX, ProcessEU, and ProcessEG respectively

Procedure CheckCTL(M, X, s0, f)

Lf := ProcessCTL(M, f);

If s0 is labeled by Lf Then

If Lf (s0) = 1 Then Return true; Else

Return TestWG(X, reset, s0, Lf (s0)); Endif Else Return false; EndifEnd Procedure

The maximal length of communication traces to be tested is bounded by (k·n·m2), where k is the number of CTL operators in the formula f, m is the upper bound for the number of states in the unspecified component X, and n is the number of states in the host system M

SAVCBS’04

Another Example Check: starting from the initial state s0, whenever the system reaches state s2, it would

eventually reach s3; i.e., check

(M, X), s0= AG(s2 AF s3 ) to be true

s0 s1 s2

s4 s3

msg? send/yes

ack/yes

send/no

ack/yes

msg?

msg?

Commsend

ack

yes

no

Host system M

Taking a negation of the original problem, it’s equivalent to checking (M, X), s0= E[true U (s2 EG s3 )] to be false

SAVCBS’04

Check (M, X), s0=E[true U(s2EGs3)]

Step 1. Atomic subformula s2 is processed by ProcessCTL, which returns a labeling function L1 = {(s2, 1)}

s0 s1 s2

s4 s3

msg? send/yes

ack/yes

send/no

ack/yes

msg?

msg?

Commsend

ack

yes

no

Host system M

SAVCBS’04

Step 1. Atomic subformula s2 is processed by ProcessCTL, which returns a labeling function L1 = {(s2, 1)}

s0 s1 s2

s4 s3

msg? send/yes

ack/yes

send/no

ack/yes

msg?

msg?

Commsend

ack

yes

no

Host system M

Step 2. Atomic subformula s3 is processed by ProcessCTL, which returns a labeling function L2 = {(s3, 1)}

Check (M, X), s0=E[true U(s2EGs3)]

SAVCBS’04

Step 1. Atomic subformula s2 is processed by ProcessCTL, which returns a labeling function L1 = {(s2, 1)}

s0 s1 s2

s4 s3

msg? send/yes

ack/yes

send/no

ack/yes

msg?

msg?

Commsend

ack

yes

no

Host system M

Step 2. Atomic subformula s3 is processed by ProcessCTL, which returns a labeling function L2 = {(s3, 1)}

Step 3. Subformula s3 is processed by Negation, which returns a labeling function L3 = {(s0, 1), (s1, 1), (s2, 1), (s4, 1)}

Check (M, X), s0=E[true U(s2EGs3)]

SAVCBS’04

Step 4. Subformula EG s3 is processed by ProcessEG, which constructs a witness graph G1 = <N, E, L3> with an ID 2 and returns a labeling function L4 = {(s0, 2), (s1, 2), (s2, 2)}

s0 s1 s2

s4 s3

msg? send/yes

ack/yes

send/no

ack/yes

msg?

msg?

The host system M

s0 s1 s2send/yes

Witness graph G1 with ID 2

Check (M, X), s0=E[true U(s2EGs3)]

SAVCBS’04

Step 4. Subformula EG s3 is processed by ProcessEG, which constructs a witness graph G1 = <N, E, L3> with an ID 2 and returns a labeling function L4 = {(s0, 2), (s1, 2), (s2, 2)}

s0 s1 s2

s4 s3

msg? send/yes

ack/yes

send/no

ack/yes

msg?

msg?

The host system M

s0 s1 s2send/yes

Witness graph G1 with ID 2

Step 5. Subformula s2 EG s3 is processed by Negation and Union, which finally return a labeling function L5 = {(s2, 2)}

Check (M, X), s0=E[true U(s2EGs3)]

SAVCBS’04

Step 6. The formula E[true U (s2 EG s3)] is processed by procedure ProcessEU, which constructs a witness graph G2 = <N, E, Ltrue, L5> with an ID 3 and returns a labeling function Lf = {(s0, 3), (s1, 3), (s2, 3), (s3, 3), (s4, 3)}

s0 s1 s2

s4 s3

msg? send/yes

ack/yes

send/no

ack/yes

msg?

msg?

The host system M

s0 s1 s2

s4 s3

send/yes

ack/yes

send/no

ack/yes

Witness graph G2 with ID 3

Check (M, X), s0=E[true U(s2EGs3)]

Lf (s0) 1, this indicates that the truth of the original problem depends on communications. Then we shall see whether Lf (s0) can be evaluated to be true

SAVCBS’04

Evaluate ID Expression Lf

(s0) Evaluating Lf (s0) is a testing process with test-cases generated from the two

witness graphs (details can be seen in the paper) Essentially, we are testing whether some communication trace (of bounded

length) with m consecutive symbol pairs “send yes” is a successful test-case for the unspecified component X.

s0 s1 s2

s4 s3

msg? send/yes

ack/yes

send/no

ack/yes

msg?

msg?

Commsend

ack

yes

no

The host system M

Errata: on page 6, in the paragraph above “Note.”, “… two consecutive symbol pairs …” should be “… m consecutive symbol pairs… ”

SAVCBS’04

Summary A decompositional approach

Reduces a system verification problem to a testing problem over the unspecified component

A hybrid approach Combines model-checking with software testing techniques

Advantages Stronger confidence about the reliability of the system Customization of testing over a component with respect to particular

system requirement Reuse of intermediate verification results Automatic carrying out of the whole process Complete and sound algorithm with an appropriate bound

SAVCBS’04

Related Work Specification-based testing [Heitmeyer et. al. 1999], generate test-cases

from counter-examples produced by a model-checker Black-box checking [Peled et. al. 1999], test a single black-box against

a given LTL formula Automatic assume-guarantee [Giannakopoulou et. al. 2004], automatic

generation of assumptions of a component, uses a less expressive formalism (LTS) for properties

Automatic deduction of model-checking conditions [Fisler et. al. 2001], has false negative

[Xie & Dang 2004] LTL algorithms for systems with only one, finite-state unspecified

component (FATES’04) An automata-theoretic and decompositional approach to testing a

system of concurrent black-boxes (submitted)

SAVCBS’04

Ongoing Work Decompositional LTL model-checking Less-restricted system model

Asynchronous communications Multiple unspecified components Unspecified components with infinite state

space Case studies

CORBA applications