Testing

MSO 08/09, WP

Why testing ?

Obvious.

Still, how many of you actually test your assignment before submitting them?

How do you know if you have tested enough??

How much testing really cost?

Seemingly a lot, since there are plenty of software testing companies in NL; they seem to make good business!

2

Some numbers

US financial sector suffers 1.5 billion USD (2002) due to inadequate testing (US Nat. Inst. of Standards and Tech.)

Testing cost 30 – 70% of total development cost! (various studies)

Effort to find a bug (Ellims et all 2004, JTest)

at system testing 12 – 20 hrs (600 Eur per bug!)

at unit testing 0 – 3 hrs (90 Eur per bug)

3

Why is it so hard ?

Size

At the system level you can be dealing with millions LOCs!

OO adds a multitude of other complications.

Concurrency

4

Plan

Unit test, integration test, system test OO issues, concurrency issues Testing with JUnit Adequacy of your tests Assignment-7 Specification-based testing Regression testing

5

Dennis et al only discuss testing very globally

Taxonomy of testing

According to the "scope" : Unit testing Integration testing System testing

According to ... "other things" : Functional testing GUI testing Acceptance testing Performance testing Regression testing

6

Unit Testing

Testing each "program unit" in isolation.

Manageable size; finding bugs are easier at this level.

Widely supported by xUnit tools.

Automated tools e.g. T2.

We'll take a look at Junit.

7

Unit Testing

What is "unit" ? Options : method class package

Typically people take method as "unit"

But you should be aware that in OO methods can influence each other through their side effects on objects' states.

This implies that there are methods that should not be tested in isolation.

OO issues complicates testing

Intra-class side effect makes it difficult to test each method in isolation:

9

class Subscription { int price ; int n = 12 ;

public Subscription(p) { price = p }

public int pricePerMonth() { return price/n } public cancel() { price=0; n=0 }}

OO issues

We fix Subscription and find no more error. So, now we're sure that every client can safely call "pricePerMonth".

Watch out!! Inheritance adds another complication

10

class client {

m(Subscription s) { s.cancel() ; return s.pricePerMonth() }

class EvilSubscription extends Subscription {

public int pricePerMonth() { return 0/0 } ...

This implies that whenever you override a method, you need to test if you still respect the spec of the original method (Lipkov substitution).

Integration test

Test if a set of "units" (that logically belong together) functions expectedly.

But what is a "set of units" ? Class or package

Class level testing. Given a class C, check that

Lots of work to do by hand; use automated tool (T2).

11

Various sequences of calls to C's methods and updates to C's exposed fields do not violate C's class invariant and the specs of each method.

Integration test

Package level testing.

12

Subscription

price...

Customer

SubsctiptionPackage

classinv price 0Service

Potential problem: if there are fields in Subscription which are open to package-level access, but their values are constrained by a class invariant.

Implying you have to test that every method in the package repects the class invariant of every class in the package.

Concurrency Concurrent threads multiply the number of possible

scenarios to consider:

This implies you need lots of tests to cover all scenarios expensive!!

But even that is not possible without special tools: We can't control JVM to do a specific scheduling. This also implies that you can't replay an error in a

concurrent setup!

13

thread 1 : { price++ ; price++ }thread 2 : { price = 0 }

Will this satisfy : 0 price 2 ?

And multi-cores are coming! (shudder)

The famous V model of testing

14

Image is taken from Wikipedia.

V model of testing

15

This approach assumes a water-fall SDLC model.

SystemRequirement

SystemSpecifications

Package/classlevel

specifications

Implementation unit testing

integration testing

system testing

acceptancetesting

But more generally phases can run parallel.

System test System testing tests if a software as a whole functions as

expected.

Problem: at this level a software often has to be interacted through its (complex) GUI.

Testing through GUI is HARD to write and hard to automate!!

Alternative: split system testing to Functional system testing Separate GUI testing

16

Functional System Testing

You need a well defined interface (set of methods) through which we can directly access a software functionality (without going through the GUI).

We will test through this interface.

Ideally this interface is prepared all the way from your design phase.

17

Test Interface

GUI

Business Logic

Test Plan

For a large application you (obviously) need a test plan.

Because it involves lots of activities (and huge investment).

IEEE 829, test plan is a document:

1. describing scope, approach, resources, and schedules of testing activities.

2. identifying test items, features to test, testing tasks, who will do the tasks, risks, and contingency plans.

18

Test Plan

IEEE 829

c) Approach d) Test items e) Features to be tested; features not to be tested g) Pass/fail criteria h) Suspension and resumption criteria i) Test deliverables

Oh well, a test plan essentially: Defines the scope of the testing Explains how we do the testing Estimates the resources needed

19

(Unit) Testing with Junit

20

C Ctest

Junit

report

Java

public class Subscription {

private int price ; private int length ; ... public double pricePerMonth() { double r = (double) price / (double) length ; return r ; }}

Testing with Junit

21

C Ctest

Junit

report

Java

import org.junit.* ;import static org.junit.Assert.* ;

public class SubscriptionTest {

@Test public void test_if_pricePerMonth_returns_Euro() { Subscription S = new Subscription(200,2) ; assertTrue(S.pricePerMonth() == 1.0) ; }

@Test public void test_if_pricePerMonth_rounds_up() { Subscription S3 = new Subscription(200,3) ; assertTrue(S3.pricePerMonth() == (double) 0.67) ; }}

Read the Tutorial, linked from Opdracht-7.

JUnit with Eclipse

22

JUnit with Eclipse

23

Have we tested enough ?? In principle, no way to know that.

Still, we can use "coverage" as indication:

The precentage of "parts" of your code that are executed during your tests.

Keep in mind: 100% coverage does not imply absence of bugs!!

But low coverage is obviously not good.

24

Various concepts of "coverage"

Very abstract: Class coverage (have we tested all classes?) Method coverage (have we tested all methods?)

Code level: Line coverage Branch coverage Path coverage

25

Line, branch, and path coverage

Line coverage: percentage of lines executed by your tests. E.g. a test (1x) with

s.period = 0 s.price = 0

will give you 100% line coverage. Very weak. But in practice is used most

26

discount(Subscription s) { if (s.period > 12) d = 0.05 else d=0 if (s.price > 1000) d += 0.05 return d }

Line, branch, and path coverage

Branch coverage: percentage of branches (of "if", "while" etc) executed by your tests.

E.g. the tests (2x)

s1.price=0 s1.period=0s2.price=2000 s2.period=2000

gives 100% branch coverage.27

discount(Subscription s) { if (s.period > 12) d = 0.05 else d=0 if (s.price > 1000) d += 0.05 return d }

s.period > 12

s.price > 1000

But you still miss the "red" case!!

Line, branch, and path coverage

Path coverage: percentage of paths in the "control flow graph" that your tests pass through.

There are 4 paths. You will need 4 tests to cover them all. Much stronger. Complication with loop & recursion take e.g. "prime

paths".

28

discount(Subscription s) { if (s.period > 12) d = 0.05 else d=0 if (s.price > 1000) d += 0.05 return d }

s.period > 12

s.price > 1000

Measuring Coverge in Eclipse

29

Read the Tutorial, linked from Opdracht-7.

Assignment 7 : Foo Subscription System

30

Foo architecture in more detail

31

Foo architecture in more detail

32

Discount Abstract class DiscountStrategy

description() calcDiscount(customer, activeDiscountStrategies)

Two concrete implementations: Discount_5top Discount_10pack

TASK 1: test these two classes (unit testing) Verify against the specifications (of the above two methods) 100% (emma) coverage!

33

Problem with traditional tests

We need concrete data as inputs and oracles need manual effort.

This is very fragile; if we change the functionality a bit, you'll have to re-calculate them.

34

@Test public void test_if_pricePerMonth_returns_Euro() { Subscription S = new Subscription(200,2) ; assertTrue(S.pricePerMonth() == 1.0) ; }

@Test public void test_if_pricePerMonth_rounds_up() { Subscription S3 = new Subscription(200,3) ; assertTrue(S3.pricePerMonth() == (double) 0.67) ; }

Alternative: specification-based testing We'll write specifications in-code (in Java). Alternative: use JML or OCL. You'll need additional tools. Your need to train your programmers in them.

TASK 2 : (optional, 1.5 pt)

Write specifications for ApplicationLogic, and do specification-based testing.

We'll limit to the methods addCustomer and removeCustomer of ApplicationLogic.

35

Architecture

36

ApplicationLogic

ApplicationLogic_SPEC

addCustomerremoveCustomerclassinv

ApplicationLogic_SPEC_test

app int addCustomer(String name, String email) {

... // pre-condition

int result = app.addCustomer(name,email) ;

.... // post-condition

return result

}

Example of in-code specification : spec-1

37

Suppose this is the spec of addCustomer:

"Name and email should not be null nor empty (pre-cond). If name does not occur in the system, then add the customer, return +1. Else don't add, return -1."

int addCustomer(String name, String email) {

assertTrue(name != null && email != null && ! name.equals("") && ... )

boolean occur = false ; for (String n : getCustNames()) if (n.equals(name)) { occur = true ; break }

int result = app.addCustomer(name,email) ;

if (occur) assertTrue(result == 1) else assertTrue(result == -1) ;

return result }

Example of in-code specification : spec-2

38

ApplicationLogic

ApplicationLogic_SPEC

addCustomerremoveCustomerclassinv

app

Suppose we want to impose this class invariant on the application logic:

"The value of the variable NextCustId should be non-negative."

boolean classinv() {

assertTrue(app.nextCustID 0) ;

return true ;

}

Examples of the tests

39

import org.junit.*;import static org.junit.Assert.*;

public class ApplicationLogic_test {

@Test public void test1() { ApplicationLogic.reset() ApplicationLogic_SPEC applogic = new ApplicationLogic_SPEC()

assertTrue (applogic.classinv()) applogic.addCustomer("Pinky","Pinky@world.com") assertTrue (applogic.classinv()) applogic.addCustomer("Pinky","Pinky@world.com") assertTrue (applogic.classinv()) }}

Advatanges of spec-based testing

No (fragile) oracle! But we still have to manually write the test sequences and inputs. Can be

automated! e.g. with T2 You must combine this with pair-programming. Disadvantage:

writing specs takes time Incure maintenace cost.

40

public void test1() { ... assertTrue (applogic.classinv()) applogic.addCustomer("Pinky","Pinky@world.com") assertTrue (applogic.classinv()) applogic.addCustomer("Pinky","Pinky@world.com") assertTrue (applogic.classinv()) }

Regression test

The re-testing of a software that has been modified. Is a serious problem for large software may take days! Between versions, the changes are usually very limited (to

some modules). However, the impact may be global due to interactions

between modules. This forces you to test the entire software. That is, you need

a test set that will deliver full coverage over your entire software.

But if we just re-execute the whole set of tests collected so far, this will take too long!!

Challenge: how do we identify a good minimum set?

41