UniTesK:Model Based Testing in Industrial Practice

Victor KuliaminAlexander PetrenkoAlexander KossatchevIgor Burdonov

Institute for System Programming

Moscow, Russia

Outline

Origin of UniTesK Method UniTesK Manifesto UniTesK Solutions Case Studies

Origin of UniTesK Method

1994 – 1996ISP RAS – Nortel Networks project onfunctional test suite development for Switch Operating System kernel Few hundreds of bugs found in the OS kernel, which

had been in use for 10 years KVEST technology

About 600K lines of Nortel code tested by 2000

BUT: Failed to be introduced in Nortel processes

UniTesK Test Construction

System under Test

Behavior Model Testing Model

Coverage Model

Test InputBehavior Correctness Checking

Specification Development

Test Development Process

Test Quality Criteria

Requirements

Interface Definition

Scenario Development

System under Test

Mediator Development

Test Execution

Behavior Specifications

Test Scenarios

Mediators

Test Reports

Design

Test Results Analysis

Engineering Problems

How to simplify transformation of requirements into formal specifications?

How to minimize human involvement in test development, but make it possible in necessary points?

How to support a large variety of problem domains and test completeness criteria?

How to decouple tests and implementation, but keep them working together?

How to measure testing quality without implementation?

To make easier their usage for ordinary software engineers

To make relation between them more clear

To increase productivity, but not to loose flexibility

To make possible early start of test development

To make specifications and tests more reusable

UniTesK Manifesto

Maximum simplification of MBT techniques Automation where possible Accommodation to current industrial

practice Integration with widely used tools Integration with widely used languages

Tools

J@T CTesK Ch@se

UniTesK Test Architecture

Oracle

Test action construction

Test action construction

System under Test

Test Engine

Test Action Iterator

Specification

✕ Mediator

Test Scenario

Example

Client Data Management System: A number of clients Clients can be related as parent-subsidiary

Client can have only one parent, but may have several subsidiaries

No cycles along parent-subsidiary linksWhen parent is removed, all its subsidiaries

should be removed

Example : Interface

class ClientManager{ Client addClient ( String name ); boolean removeClient ( Client client );}

class Client{ boolean addSubsidiary ( Client client );}

Example : Client Dataclass Client{ String name; ClientSpecification parent = null; HashSet children = new HashSet(); invariant ParentChildLink() { if(parent != null && !parent.children.contains(this)) return false; Iterator j = children.iterator(); while(j.hasNext()) if(((Client)j.next()).parent != this) return false; return true; }

invariant NoCyclesAlongLinks() { Client r = this; do { r = r.parent; if(r == this) return false; } while(r != null); return true; }}

Example : addClient() Methodclass ClientManager{ specification Client addClient ( String name ) updates clients.? { post { if(name == null || clients.containsKey(name)) { // Client cannot be created return addClient == null && clients.equals(@clients.clone()); } else { // Client can be created HashMap oldClients = (HashMap)clients.clone(); oldClients.remove(name); return addClient != null && addClient.name.equals(name) && addClient.parent == null && addClient.children.isEmpty() && clients.get(name) == addClient && oldClients.equals(@clients.clone()); } } }}

Test Coverage Criteria

branch “Single branch”

return

post {

}

if(a || b) branches “Single branch”

predicates ( a || b ) !( a || b )

disjuncts a !a && b !a && !bbranch “Single branch”

branches “Single branch”

disjuncts a !a && !b

predicates ( a || b ) !( a || b )

!a && b

if(a || b) if(a || b)

class ClientManager{ specification Client addClient ( String name ) updates clients.? { post { if(name == null || clients.containsKey(name)) { branch "Client cannot be created"; return addClient == null && clients.equals(@clients.clone()); } else { branch "Client can be created"; HashMap oldClients = (HashMap)clients.clone(); oldClients.remove(name); return addClient != null && addClient.name.equals(name) && addClient.parent == null && addClient.children.isEmpty() && clients.get(name) == addClient && oldClients.equals(@clients.clone()); } } }}

Example : addClient() Method

Automaton from Coverage Goals

states

parameters operation domain

1

2

3coverage goals

Implicit Automata Description

Implicit specifications cannot be resolved To decrease effort they should not be resolved Huge automata can be described shortly

Test Scenarios

User can describe an automatonimplicitly in the form of Test Scenario

Functions: Provide the current state identifier Provide the next admissible stimulus in

this state

Test Scenario : Statescenario class ClientManagerTestScenario{ ClientManager target;

AbstractGenState getGenState() { IntToIntMapGenState genstate = new IntToIntMapGenState(); int n = 0, m = 0; Iterator j = objectUnderTest.clients.keySet().iterator(); while(j.hasNext()) { String s = (String)j.next(); n = Integer.parseInt(s); if(((Client)target.clients.get(s)).parent == null) genstate.setValue(n, 0); else { m = Integer.parseInt(((Client)target.clients.get(s)).parent.name); genstate.setValue(n, m); } } return genstate; }}

Test Scenario : Stimuliscenario class ClientManagerTestScenario{ scenario boolean addClient() { iterate(int i = 0; i < numberOfClients; i++; ) { String name = (i == 0)?(null):("" + i); if( target.precondition_addClient( name ) ) target.addClient( name ); } return true; }

scenario boolean removeClient() { iterate(int i = -1; i < numberOfClients; i++; ) { Client client = (i < 0)? ClientMediator.create(new ClientImpl("1")) : (Client)target.clients.get(("" + i)); if( target.precondition_removeClient( client ) ) target.removeClient( client ); } return true; }}

Test Execution : First StageTest

Engine

Test Scenario

Factorization

Testing Concurrency : Modeling

?a

?{a,b}

?b

?{a,a}?{b,b}?{a,a,a}?{a,a,b}

How to model responses on all possible multistimuli?

Plain concurrency axiom :reaction on multistimulus is the same as on some sequence of its stimuli

Asynchronous Reaction Specification

specification reaction UDPPacket UDPResponse ( ){ pre { return !ModelUDPPackets.isEmpty ( ); } post { return (@ModelUDPPackets.clone()).contains ( UDPResponse ) && !ModelUDPPackets.contains ( UDPResponse ); }}

Providing Concurrent Inputs

s11

System under Test

s21

s31

s12

s32

s13

s22s23

s14

s24

s33

Multisequence is used instead of sequence of stimuli

Collecting Asynchronous Reactions

Reactions form a partially ordered set

System under Test

r33r32

r12

r23r22

r11

r21

r31

Time

11

12

21 31

22 32

23

33

Evaluation of Reactions

Stimuli Reactions

Plain concurrency axiom

✕

UniTesK Test Architecture, 2

Oracle

Test input construction

System under Test

Test Engine

Test Action Iterator

Mediator

Synchronization Manager

Reaction Collector

Serializer

✕

✕

UniTesK Tools

J@T (C++ Link)Java (C++) / NetBeans, Eclipse (plan)

CTesKC / Visual Studio 6.0, gcc

Ch@seC# / Visual Studio .NET 7.1

OTKSpecialized tool for compiler testing

Technology Transfer

Forms Training Pilot projects Joint projects Project supervising

Successful transfers Lanit-Tercom

CTesK, Feb 2002 Luxoft (IBS group)

J@T, Jul 2003 Intel

OTK, Nov 2003

Case Studies

IPv6 implementations - 2001-2003 Microsoft Research Mobile IPv6 (in Windows CE 4.1) Oktet

Intel compilers - 2001-2003 Web-based client management system in bank Enterprise application development framework Billing system

References

1. V. Kuliamin, A. Petrenko, N. Pakoulin, I. Bourdonov, and A. Kossatchev. Integration of Functional and Timed Testing of Real-time and Concurrent Systems. Proc. of PSI 2003. LNCS, Springer-Verlag, 2003.

2. V. Kuliamin, A. Petrenko, I. Bourdonov, and A. Kossatchev. UniTesK Test Suite Architecture. Proc. of FME 2002. LNCS 2391, pp. 77-88, Springer-Verlag, 2002.

3. A. K. Petrenko, I. B. Bourdonov, A. S. Kossatchev, V. V. Kuliamin. Experiences in using testing tools and technology in real-life applications. Proceedings of SETT’01, India, Pune, 2001

4. I. B. Bourdonov, A. S. Kossatchev, V. V. Kuliamin. Using Finite State Machines in Program Testing. "Programmirovanije", 2000, No. 2 (in Russian). Programming and Computer Software, Vol. 26, No. 2, 2000, pp. 61-73 (English version)

5. I. Bourdonov, A. Kossatchev, A. Petrenko, and D. Galter. KVEST: Automated Generation of Test Suites from Formal Specifications. Proceedings of World Congress of Formal Methods, Toulouse, France, LNCS, No. 1708, 1999, pp. 608-621

6. http://www.ispras.ru/groups/rv/rv.html

Contacts

Victor V. Kuliamin Alexander K. Petrenko

E-mail: kuliamin@ispras.ru, petrenko@ispras.ru109004, B. Kommunisticheskaya, 25Moscow, RussiaWeb: http://www.ispras.ru/groups/rv/rv.htmlPhone: 007-095-9125317Fax: 007-095-9121524

Thank you!