Au

g./S

ept.

200

8Algebriac Testing

1

Automated Testing of Software Components Based on Algebraic

Specifications-- Method, Tool And Experiments

Hong ZhuDept. of Computing and Electronics,

Oxford Brookes University, Oxford, OX33 1HX, UK

Email: hzhu@brookes.ac.uk

Au

g./S

ept.

200

8

2Algebriac Testing

Outline

Background Motivation Related works Overview of the approach

Specification language CASOCC Testing tool CASCATEmpirical evaluationConclusion and future work

Au

g./S

ept.

200

8

3Algebriac Testing

Challenges in testing components Components often have no user interface

Developers spend as much time in writing test harness excessive overhead, inadequacy of testing, low effectiveness

Components are usually delivered as executable code without the source code and design information

White-box testing, model-driven testing methods not applicable contains no instrumentation

Internal behaviour observation and test adequacy measurement are virtually impossible

Existing approaches to the problems Self-testing, e.g. (Beydeda, 2006): yet to be adopted by the

industry Specification-based testing: design-by-contract, FSM, etc.

Au

g./S

ept.

200

8

4Algebriac Testing

Algebraic specification

A signature: a set of sorts and a set of operators

on the sortsA set of axioms:

in the form of conditional equations

Spec NAT Sorts: nat;Operators: zero: -> nat; succ: nat -> nat;

Axioms: zero succ(x);succ(x) = succ(y) => x=y;

End NAT

Algebraic specification (AS) emerged in the 1970s. In the past three decades, it has developed into a mature formal method.

Au

g./S

ept.

200

8

5Algebriac Testing

Basic idea of algebraic testing

S: stack, n: integer,

S.push(n).height() = S.height()+1

By substituting constants into variables, we can generate test cases

A ground term corresponds to a sequence of procedure/ method/ operation calls

Checking the equivalence between the values of the left and right hand sides is to check the correctness of test results

Au

g./S

ept.

200

8

6Algebriac Testing

Related works

Work /SystemEntities tested

Test case generationResult

checking

DAISTSGonnon et al. 1981

ADTManually scripted ground terms substituted into axioms

Manual program

LOFTGaudel et al. 1991

ADT

All ground terms up to certain complexity substituted to axioms; random values assigned to variables of predefined sorts.

Observation contexts

ASTOOT / LOBASDoong & Frankl,1994

ClassRewriting ground terms into their normal forms using the axioms

Manual program

DaistishHughes & Stotts 1996

ClassManually scripted ground terms substituted into axioms

Manual program

TACCLEChen et al. 1998,

2001

Class, Cluster

Ground normal forms substituted in axioms + path coverage

Observation context

Au

g./S

ept.

200

8

7Algebriac Testing

Overview of the proposed approach

Sorts to represent all types of software entities: ADT, Class, Component

Test case generation: Composition of observation contexts and axioms

with ground normal forms substituted into non-primitive variables and random values for primitive variables

Test oracle: Direct checking since test cases are checkable

Au

g./S

ept.

200

8

8Algebriac Testing

CASOCC specification languageSpec Stack observable F; import Int, String; operations creator

create: String->Stack; constructor

push: Stack,Int ->Stack;

transformer pop: Stack->Stack;

observer getId: Stack->String; top: Stack->Int;

height: Stack->Int; vars

S: Stack; n: Int; x: String;

axioms1: create(x).getId() = x;2: findByPrimaryKey(x).getId() = x;3: create(x).height() = 0;4: S.push(n) = S;

if S.height() = 10;5: S.pop() = S; if S.height() = 0;6: S.push(n).pop() = S;

if S.height() < 10;7: S.push(n).top() = n;

if S.height() < 10;8: S.push(n).height() =S.height()+1;

if S.height() < 10; 9: S.pop().height() = S.height()-1;

if S.height() >0;end

Au

g./S

ept.

200

8

9Algebriac Testing

Behavioural semantics and observable sorts

Definition 1. (Observable sort)In an AS <S, E>, a sort s is an observable sort implies that there is an operation _ == _ : ssBool such that for all ground terms t and t’ of sort s,

E ( (t == t’) = true) E ( t=t’ ). An algebra A (i.e. a software entity) is a correct implementation of an observable sort s if for all ground terms t and t’ of sort s,

A (t=t’) A ( (t == t’) = true)

Pre-defined sorts of Java primitive classes and data types are observable.

Au

g./S

ept.

200

8

10Algebriac Testing

Well founded formal specifications

Let U be a set of specification units in CASOCC and S be a set of sorts. For each sort sS, there is a unit UsU that specifies the software entity corresponding to sort s.

Let be the importation relation on S.

Definition 2. (Well founded specifications)

A sort s S is well founded if s is observable, or for all s’ in the import list of Us, s’ is an observable sort, or s’ is well founded.

A specification U is well founded if and only if the importation relation is a pre-order on the set S of sorts, and all sorts sS are well founded.

Au

g./S

ept.

200

8

11Algebriac Testing

Well-structured formal specificationsDefinition 3. (Well structured specifications)

A specification U in CASOCC is well structured if it satisfies the following conditions.(1) It is well founded;

(2) For every user defined unit Us U,

(a) there is at least one observer in Us;

(b) for every axiom E in Us, if the condition contains an equation =’, we must have, where s’ is the sort of terms and ’.

A practice implication: for all sorts there are finite lengths of observable contexts.

Au

g./S

ept.

200

8

12Algebriac Testing

Observation context

Definition 1. (Observation context)A context of a sort c is a term C with one occurrence

of a special variable of sort c. The value of a term t of sort c in the context of C, written as C[t], is the term obtained by substituting t into the special variable .

An observation context oc of sort c is a context of sort c and the sort of the term oc is. To be consistent on notations, we write _.oc: cs to denote an observation context oc.

An observation context is primitive if s is an observable sort. In such cases, we also say that the observation context is observable and call the context observable context for short.

Au

g./S

ept.

200

8

13Algebriac Testing

Form of observation context

The general form of an observation context oc:

_.f1(...).f2(...).....fk(...).obs(...)

where f1, ..., fk are transformers of sort sc,

obs is an observer of sort c, f1(...), ..., fk (...) are ground terms.

A sequence of observation contexts oc1, oc2, …, ocn , where _.oc1: cs1, _.oci: si-1si, i =2,…,n, can be composed into an observation context _.oc1.oc2. ….ocn.

Example: _.pop().pop().height()

Au

g./S

ept.

200

8

14Algebriac Testing

Checkable test casesTest cases:

a test case is a triple <T1, T2, C>, where T1 and T2 are ground terms C (optional) is a ground term of Boolean sort.

It means that values of T1 and T2 should be equivalent if C evaluates to True.

Definition 2. (Checkable test cases)

A test case T1=T2, [if C] is directly checkable (or simply checkable), if and only if

(a) the sort of terms T1 and T2 is observable, and

(b) the sort of equations in C is observable, if any.

written T1=T2, [if C]

Au

g./S

ept.

200

8

15Algebriac Testing

Test case generation algorithm (Skeleton) Input:

Spec s: CASOCC specification unit of the main sort;Sigs s1, s2, …, sk: The signature of imported sorts; TC: A subset of axioms in s (* the axioms to be tested *);vc: Integer (*complexity upper bound of variables*);oc: Integer (*complexity upper bound of observation contexts*) ;rc: Integer (* the number of random values to be assigned to variables of primitive sorts*)

BeginStep 1: InitialisationStep 2: Generate normal form terms for non-primitive variablesStep 3: Generate random values for primitive variables Step 4: Substitute normal forms into axiomsStep 5: Substitute random values into test cases Step 6: Compose test case with observation contextStep 7: Output test set

End

Au

g./S

ept.

200

8

16Algebriac Testing

Properties of the test case generation algorithmTheorem 1. The test case generation algorithm will always terminate if the specification is well founded. Theorem 2. The test cases generated are checkable, i.e. for all test cases <t1=t2; if c> generated by the algorithm, t1, t2 and c are of primitive or observable sorts. Theorem 3. The test cases are valid. That is, if the specification is well-structured and the observable sorts satisfy the constraints in Definition 1, we have the following properties.

(a) The program correctly implements the specification with respect to the behavioural semantics of algebraic specifications implies that the evaluation of t1 and t2 using the program give equivalent results provided that c is evaluated to be true.

(b) If the evaluation of t1 and t2 gives non-equivalent values in an implementation when c is evaluated to true, then there are faults in the program.

Au

g./S

ept.

200

8

17Algebriac Testing

Testing tool CASCAT

CASOCC Spec Parser

Test Case Generator

Test Result Evaluator

Component Spec in

CASOCC

Test Driver

Test Cases

J2EE Component Deployed on JBoss Platfo

rm

Test Report

CASCAT Tool

Au

g./S

ept.

200

8

18Algebriac Testing

Au

g./S

ept.

200

8

19Algebriac Testing

Experiment 1: Evaluation of effectiveness The experiment process

Selection of subject components: from well established public sources.

Development of formal specification: based on the document and source code.

Test case generation: automatically by the CASCAT tool from the specification.

Validation of formal specification. The subject component is checked against its formal specification by

executing the components on the test cases using the CASCAT tool. Fault injection: used MuJava. Eliminate equivalent mutants: manually examined Test execution.

A mutant is classified as fault detected if at least on of the axioms of the component is violated or the execution is terminated abnormally.

Au

g./S

ept.

200

8

20Algebriac Testing

The subjects: 1) Single Component Subjects

Subject/Component #C #M #L Src Type

Bank 4 4 322 [28] Stateful

Stock 3 5 250 [30] BMP

Product 5 15 278 [31] CMP

Cart 5 6 226 [28] Stateful

Count 3 2 101 [31] Stateful

LinkedList 4 8 155 [32] Stateful

Math 3 7 142 [34] Stateless

Au

g./S

ept.

200

8

21Algebriac Testing

Subject/Component #C #M #L Src Type

CollegeCourse 3 5 367 [28] BMP

Student 3 5 330 [28] BMP

Sales

Customer 3 7 350 [28] BMP

Order 5 7 1082 [28] BMP

SalesRep 3 5 375 [28] BMP

WarehouseStorageBin 3 5 360 [28] BMP

Widget 3 4 291 [28] BMP

GangsterGangster 6 27 214 [27] CMP

ReadAhead 4 7 234 [27] Stateless

SafeDriver

RateTable 3 16 292 [29] CMP

Register* 4 19 293 [29] CMP

RateQuote 3 5 341 [29] Stateful

GeneralInfo* 4 2 290 [29] Stateless

Average 3.7 8.6 315 -- --

Std Dev 0.9 6.3 196.4 -- --

The subjects: 1) Multiple Components Subjects

Au

g./S

ept.

200

8

22Algebriac Testing

Results of the experimentSubject/Component #M #NEM #D Rate%

Single component subjects

Bank 42 37 23 62.2

Stock 23 22 13 59.1

Product 18 14 14 100.0

Cart 8 8 7 87.5

Count 13 9 8 88.9

LinkedList 33 33 32 97.0

Math 98 73 73 100.0

Multiple component subjects

CollegeCourse 24 24 14 58.3

Student 25 25 15 60.0

Sales

Customer 40 40 27 67.5

Order 57 53 34 64.2

SalesRep 24 24 14 58.3

WarehouseStorageBin 46 44 27 61.4

Widget 34 28 20 71.4

GangsterGangster 9 8 8 100.0

ReadAhead 35 33 31 93.9

SafeDriverRateTable 48 36 36 100.0

RateQuote 272 220 202 91.8

Total 849 731 598 81.8

Average 47.2 40.6 33.2 79.0

Std Dev 59.9 47.7 44.9 17.6

Au

g./S

ept.

200

8

23Algebriac Testing

Main findings (1) The fault detecting ability is not sensitive to the scale of

the subject under test. (Correlation coefficient =0.20)

Au

g./S

ept.

200

8

24Algebriac Testing

Main findings (2) The fault detecting ability decreases only slightly when

testing multiple component subjects.

Au

g./S

ept.

200

8

25Algebriac Testing

Main findings (3) The method consistently detects significantly more faults in

session beans than in BMP entities beans despite that entity beans are usually much less complex than session beans.

This statement is supported by T-Test.

Au

g./S

ept.

200

8

26Algebriac Testing

Main findings (4) The development of axioms was less difficult than we expected. There is a simple pattern of axioms for entity beans despite their

differences in semantics. Component #Crts #Con/Trans #Obs #Axioms Time

Bank 1 1 2 6 60m

Cart 2 2 2 4 50m

Count 1 0 1 1 35m

Course 2 1 2 6 50m

Customer 2 2 3 4 60m

Gangster 2 5 20 19 90m

GeneralInfo 1 0 1 1 30m

LinkedList 1 2 5 16 100m

Math 1 1 5 9 60m

Order 2 0 5 5 60m

Product 2 3 10 14 70m

RateQuote 1 0 4 4 50m

RateTable 2 7 7 14 80m

ReadAhead 1 0 6 6 75m

Register 2 8 9 17 90m

SalesRep 2 1 2 3 40m

Stock 2 1 2 5 65m

StorageBin 3 0 2 4 40m

Student 2 1 2 4 40m

Widget 2 0 2 4 50m

Average 1.75 1.75 4.60 6.85 82.25m

Au

g./S

ept.

200

8

27Algebriac Testing

Is algebraic testing practical?

Cost of algebraic testing: Writing algebraic specification Deploy the component to component platform, such as JBoss Generation of test cases (automated by tool) Review of test report (checking correctness is done

automatically)

How expensive is writing algebraic specifications? Is writing algebraic specification learnable? What skill and knowledge are required to write

algebraic specifications?

Au

g./S

ept.

200

8

28Algebriac Testing

How to write algebraic specifications (1) the description of the signature

the identification of the operations, e.g. The signature of the operations can be derived from the type definitions of the

methods given in the source code. the classification of operations

Creators: create instances of the software entity and/or initialise the entity. They must have no parameters of the main sort, but result in the main sort.

Constructors: construct the data structure by adding more elements to the data. A constructor must have a parameter of the main sort and results in the main sort. It may occur in the normal forms if the axioms are used as term rewriting rules.

Transformers: manipulate the data structure without adding more data. Similar to constructors, a transformer must have the main sort as its parameter and results in the main sort. However, it cannot occur in any normal forms.

Observers: enable the internal states or data in the software entity to be observed from the outside. Observers must have a parameter of the main sort but result in an imported sort.

Au

g./S

ept.

200

8

29Algebriac Testing

How to write algebraic specifications (2) the determination of the axioms

For each setter setX(v) (set the value of attribute X to v), s,v. (s.setX(v).getX = v), If pre-condsetX(v)

s,v. (s.setX(v).getY = s.getY), where X Y.

For each getter getX (get the value of attribute X)s,v. (s.[getX] = s),

For each creator C(x1,x2,…, xn) x1,…, xn. . C(x1,…, xn).getXi = xi, If pre-cond C(x1,…, xn)

For each constructor and transformer F(x), s,x. (s.F(x).getX) = (x, s.getX), if pre-condF(x),

For each operation P(x) that involves more than one parts A and B:

s,x,y. (s. [A.P (x)].B = s.B.Q(y)), if pre-condA.P(x)

Au

g./S

ept.

200

8

30Algebriac Testing

Experiment 2: Cost of writing algebraic spec Subjects:

Students of computer science (35, year 3) Mathematics course: Advanced University Mathematics Part A and Part

B and Discrete Mathematics. Programming courses: C++ Programming, Java Programming and Data

Structure No exposure to formal methods

Mathematics Courses

Programming Courses

All Courses

Average 76.44 74.59 75.51

StandardDeviation

9.67 7.34 7.83

Table 1. Statistic Data of Student Capability

Au

g./S

ept.

200

8

31Algebriac Testing

Distribution of capabilities

Au

g./S

ept.

200

8

32Algebriac Testing

Math Courses Prog Courses All Courses

Average 76.44 74.59 75.51

StDev 9.67 7.34 7.83

Au

g./S

ept.

200

8

33Algebriac Testing

Process of the experimentLesson 0:

Introduction to formal methods. Lesson 1:

Introduced to algebraic specification and the CASOCC specification language. An example formal specification of stacks A brief introduction to software component for the first class test The first class test (individual and independent)

Lesson 2: Sample answer to the class test question 1 A brief introduction to software component for the second class test The second class test (also individual and independent)

Lesson 3: Sample answer to the second class test question A brief introduction to software component for the third class test The third class test (also individual and independent)

Lesson 4: Sample answer to the previous class test A brief introduction to software component for the final class test The final class test

Au

g./S

ept.

200

8

34Algebriac Testing

Marking scheme of class testsCorrectness of the answer: 50%.

It is assess according to the correctness of the signature and axioms in the student’s work.

Minor syntax errors that can be detected by CASCAD tool is deduced by 20%

Incorrect axioms were given no marks. Completeness of the axiom system: 50%.

It is assessed according to the coverage of the operations by the axioms.

The coverage of each operation was given the equal number of marks.

Au

g./S

ept.

200

8

35Algebriac Testing

Recording times

The time that each student took to complete the class test was recorded in the experiment. The students were given no limit on the time to

complete the class tests. The students were asked to hand in their work as

soon as possible. The students were briefed about the function and the

interface of the component before started to work on the class test question.

The time taken to write the algebraic specification excludes the time to understand the components.

Au

g./S

ept.

200

8

36Algebriac Testing

Components Used in Class Tests

Component Function Source Number of Operations

Numerical operations on complex numbers

Java programming textbook [1]

13

Bank account operations J2EE Tutorial [2] 12

Gangster database operations

J2EE Tutorial 16

Operations on Linked Lists

Classic textbooks, and also [17]

8

[1] 周长发 , Java数值计算算法编程，电子工业出版社， 2007. [2] Bodoff, S. et al. 2004. The J2EE Tutorial, 2nd Edt., Pearson 2004.

Au

g./S

ept.

200

8

37Algebriac Testing

Main Findings 1Is writing algebraic specifications learnable?

Au

g./S

ept.

200

8

38Algebriac Testing

Changes in the distributions of scores

Au

g./S

ept.

200

8

39Algebriac Testing

Distributions of Grades in Class Tests

Grade Average Test 1 Test 2 Test 3 Test 4

A 8 1 25 31 33

B+ 23 0 4 2 0

B 4 3 5 2 2

C 0 20 0 0 0

F 0 11 1 0 0

The students’ learning experience is not hard. They attained the knowledge and skill of algebraic specification in just a few lessons.

Au

g./S

ept.

200

8

40Algebriac Testing

Main Findings 2How expensive to write an algebraic specification for a software component?

Au

g./S

ept.

200

8

41Algebriac Testing

Changes in the distributions of times

On average a student took about half hour to complete the writing of an algebraic specification for a typical software component.

Au

g./S

ept.

200

8

42Algebriac Testing

Main Findings 3

Does writing algebraic specification need good mathematical skills?

Average Times

StDev of Times

Average Scores

StDev of Scores

Mathematics Courses

-0.44 0.34 0.10 -0.10

Programming Courses

-0.52 0.41 0.46 -0.44

All Courses

-0.52 0.40 0.28 -0.27

Correlation Coefficients

Au

g./S

ept.

200

8

43Algebriac Testing

Cluster analysis

We divide the students into the following four groups and calculated their scores in class tests.

P>M: More capable of programming than mathematics. P<M: More capable in mathematics than programming P~M High: Equally capable of programming and mathematics P~M Lower: Equally incapable of programming and

mathematics

Avg. Math

Avg. Prog

Avg. All

Avg. Score

Avg. Final Score

#Stds

P>M 69.78 77.44 73.61 90.82 96.91 6

P<M 84.31 74.92 79.62 85.44 95.97 13

P~M (higher)

77.50 77.83 77.67 87.23 96.46 8

P~M (lower) 67.58 68.67 68.13 85.49 95.50 8

Au

g./S

ept.

200

8

44Algebriac Testing

The students’ performances in class tests are more closely related to their programming capability than to mathematics knowledge and skills.

Notes: the link between students’ performance in class

tests and programming capability should be interpreted as their capability of learning algebraic specification rather than their final attainment.

the link is not strong since the absolute values of the correlation coefficients are in the range from 0.41 to 0.52.

Au

g./S

ept.

200

8

45Algebriac Testing

Main Findings 4

Is writing algebraic formal specifications a job only for the most capable?

Au

g./S

ept.

200

8

46Algebriac Testing

Writing algebraic specification must be the job of the most capable programmers. However, there is a potential bias.

The average scores and average times contain the results of the first and second class tests. Thus, they do not reflect the situation after the students completed their training.

Au

g./S

ept.

200

8

47Algebriac Testing

Au

g./S

ept.

200

8

48Algebriac Testing

(a) Relationship between final class test score and programming capability

y = -0.0002x2 - 0.0148x + 98.405 (3)

(b) Relationship between final class test times and programming capability

y = 0.0017x2 - 0.4558x + 56.043 (4)

After taking three lessons and class tests, the students are capable of writing algebraic specifications of almost equal quality, but the most capable ones took slightly less time.

Writing algebraic specifications can be a job for any well trained software developer rather than just for the few most capable ones.

Au

g./S

ept.

200

8

49Algebriac Testing

General conclusions of experiment 2 Conclusion 1 (Learnability): Writing algebraic specification is

learnable for ordinary software developers. Conclusion 2 (Independence of mathematical skills) The

knowledge and skill of programming is more important than mathematics to writing algebraic specifications of software components.

Conclusion 3 (Cost efficiency): Writing algebraic specification can be as cost efficient as programming in high level programming languages.

Conclusion 4 (Equality in performance): Writing algebraic specification can be a skill of every well trained software developer. Although their efficiency in writing algebraic specifications depends on their capabilities, there should be no significantly different from each other on the quality.

Au

g./S

ept.

200

8

50Algebriac Testing

Limitations of the conclusions 1

The conclusions are only applicable to writing algebraic formal specifications.

They do not necessarily imply that writing formal specifications in other formalisms has the same properties.

Further research: to investigate whether the same claim can be made to other formalisms such as Z, Petri-nets, process algebras like CSP, CCS and -calculus, and labelled transition system in general.

A notable advantage of algebraic specification is that the syntax and semantics of axioms are simple and easy to understand. They use little mathematics notations.

Au

g./S

ept.

200

8

51Algebriac Testing

Limitations of the conclusions 2

The conclusions are only applicable to writing formal specification

They do not necessarily imply that other formal development activities have the same properties, such as reasoning about software properties, proving software correctness, deriving software using formal specifications.

These activities may well require much deeper understanding of the theories of formal methods and the semantics of formal specification languages. They may also rely on skills of using software tools that supports formal methods.

Au

g./S

ept.

200

8

52Algebriac Testing

Conclusion Summary:

A technique of automated component testing based on algebraic specifications.

A specification language CASOCC An algorithm to generate checkable test cases. An automated prototype testing tool CASCAT for EJB components.

Advantages: AS are independent of the implementation, thus suitable for components A high degree of automation No need of the availability and uses of the full set of axioms of all

constituent and dependent entities Can focus on a subset of functions and properties of the component A high fault detecting ability Scalable and practically usable Cost efficient

Au

g./S

ept.

200

8

53Algebriac Testing

Future work

More experiments with multiple components subjects

Extending the tool from testing EJB 2.0 component to EJB 3.0, i.e. to directly support message driven components

Extending the technique for testing web services and concurrent systems

Au

g./S

ept.

200

8

54Algebriac Testing

ReferencesGonnon, J., McMullin, P. and Hamlet, R., Data-Abstraction

Implementation, Specification and Testing, ACM TOPLAS 3(3), 1981, 211-223.

Bernot, G., Gaudel, M. C., and Marre, B., Software testing based on formal specifications: a theory and a tool, Software Engineering Journal, Nov. 1991, 387- 405.

Doong,K. & Frankl, P., The ASTOOT approach to testing object-oriented programs, ACM TSEM3(2),1994, 101-130

Hughes, M. and Stotts, D., Daistish: systematic algebraic testing for OO programs in the presence of side-effects. ISSTA’96, 53-61.

Chen, H.Y., et al., In black and white: an integrated approach to class-level testing of object-oriented programs, ACM TSEM 7(3), 1998, 250-295.

Chen,H.Y., Tse,T.H. & Chen,T.Y., TACCLE: a methodology for object-oriented software testing at the class and cluster levels, ACM TSEM 10(1), 2001, 56-109.

Au

g./S

ept.

200

8

55Algebriac Testing

Kong, L., Zhu, H., and Zhou, B. 2007. Automated Testing EJB Components Based on Algebraic Specifications. Proc. of COMPSAC’07, Vol. 2, 717-722.

Yu, B., Kong, L., Zhang, Y., and Zhu, H. 2008. Testing Java Components Based on Algebraic Specifications. Proc. of ICST’08 (April 2008), 190-199.