Progress in Testing Component-Based Software

Craig H. Wittenberg

Component Applications Group

Microsoft Research

Aug. 23, 2000 ISSTA 2000 2

Talk Outline

Our project

Demos

Component and composition

Progress in testing components

Areas for further research

Aug. 23, 2000 ISSTA 2000 3

High Level Goals

A system of flexible components for a family of products

Faster and cheaper application assembly and evolution

Reliable, efficient, and consistent applications working together

Aug. 23, 2000 ISSTA 2000 4

Octarine Demo

Prototype written in C

Looks familiar

Adding Bullets, Music

Followed by architectural drawings

Aug. 23, 2000 ISSTA 2000 5

Octarine Text Architecture

ISequence Story ISequence Text

ISequence Music

Table

Text

ISequence

Aug. 23, 2000 ISSTA 2000 6

Adding Bullets/Numbering

ISequence Story ISequence Text

ISequence Music

Table

Text

ISequence

ISequence Bullets/Numbering

ISequence

Text

Aug. 23, 2000 ISSTA 2000 7

Landscape Demo

More recentMany small componentsStatic and dynamic connectionsMultiple uses of: Text, Layout, Grid, Data pipes

Architecture for many UisSmall devicesNon-visual interfaces

Aug. 23, 2000 ISSTA 2000 8

Our Approach

Factoring and composition

Precise interface specs

Significant design and prototype step

Huge emphasis on testing

Control inter-component dependencies

Tools support (design, build, and run time)

Critical mass of designs, components, frameworks, large demos

Aug. 23, 2000 ISSTA 2000 9

Component

Key points: Contracts, Composition, Dependencies, Deployment

No observable global state

“A software component is a unit of composition with contractually specified interfaces and explicit context dependencies only. A software component can be deployed independently and is subject to composition by third parties.”

Szyperski, Component Software, pg. 34.

Aug. 23, 2000 ISSTA 2000 10

Component DLL

Binary package plus docsNo top-level APIsContains one or more classes Implementation of contracts only No implementation inheritance Realize behavior of component Instances of classes are connected Dependencies upon

other classes services from other instances

Aug. 23, 2000 ISSTA 2000 11

An interface has: Name (compile time and runtime) Methods with pre/post conditions Abstract model with invariants State table

Specified separately from implementationImplementation does the obvious and: maps model variables to internal state

Never changed once published

Precise Interface Specs a.k.a. Software Contracts

Aug. 23, 2000 ISSTA 2000 12

ILight Methods and Model

ILight, {2461A1A0-639E-11d2-874D-00AA0060FCA9}

TurnOn();TurnOff();SetIntensity(int desired);

Modelboolean on; int intensity; // range is 0..100.

Aug. 23, 2000 ISSTA 2000 13

ILight Pre/Post Conditions

TurnOn():Post: on == true.

TurnOff():Post: on == false.

SetIntensity(desired):Pre: desired >=0 && desired <= 100.Post: intensity == desired.

Aug. 23, 2000 ISSTA 2000 14

A Light

The Light class implements ILight

Contract-specific and implementation-specific model mapping

LightILight

Map ILight

model to Impl.

Aug. 23, 2000 ISSTA 2000 15

Record Filter

Filter processes one set of records into another

Record Filter

Plants

Filter Spec

IString

IRecordSetIRecordSet

Map IRecordSet model to

Impl.

Aug. 23, 2000 ISSTA 2000 16

Light Switch

Converts calls to push() into on() and off()

Light Switch

A lightIPushIt ILight

Map IPushIt model to

Impl.

Aug. 23, 2000 ISSTA 2000 17

Closure

Executes closure over parameters data or connected objects

ClosureParameters

IDoIt IUnknown

Map IDoIt model to

Impl.

……

…

Aug. 23, 2000 ISSTA 2000 18

Composition

Composites looks like components Can test small pieces Can also use large ones

Separate structure from algorithms Definition/analysis of relationships Eliminates explicit dependencies

No glue as usually meant connection glue is factored out algorithmic glue is encapsulated

Aug. 23, 2000 ISSTA 2000 19

Composition time

Structure may be decided at Design time

Fully static composition Parameterized composition

Runtime Dynamic composition described as changes to existing

structure May be downloaded as add-on software package

Haven’t fully explored tradeoffs; tending towards more runtime composition

Aug. 23, 2000 ISSTA 2000 20

Example of Composition

Button to turn on light

Each piece has one+ contracts

Button LightIButtonEvents

ILightButton-to-Light Adapter

Desc.: Button, Adapter+stuff, Light, B->A, A->L

IPushIt

Aug. 23, 2000 ISSTA 2000 21

Encapsulation of the Switch

Button LightIButtonEvents

ILightEdge Conn.

Button-to-Light Adapter

Edge Connector for lookup/connection(role)

Desc.: Button, Adapter+stuff, Light, B->A, A->L, EC(out B, in A)

IPushIt

Aug. 23, 2000 ISSTA 2000 22

Granularity of classes

Leaf classesUp to 1500 lines of C

Probably less in Java or C#1-30 classes per component DLL

Static compositesNormal range: 5-10 instancesPackaged like a leaf classConnection description very cheap

Aug. 23, 2000 ISSTA 2000 23

Testing Open Systems

Late compositionUsually after component shippedRobustness of composite related to piecesSystem testing may be done by the user Need to support in-the-field debugging

Unit testing is criticalMust strive for more than the 80% caseFinding evidence of correctness

Aug. 23, 2000 ISSTA 2000 24

Testing Components

Verify contract as implemented Check pre/post conditions Expose contract model variables Composites have contracts too

Verify contracts consumed Addresses some integration issues Fault injection

Test and measure the binary formAllow feedback to improve contract, impl., etc.

Aug. 23, 2000 ISSTA 2000 25

What is a robust component?

Beizer on statement and branch coverage: “This is the weakest criterion in the family: testing less than this for new software is unconscionable and should be criminalized.”For us: Contract coverage 100% coverage at the object level

Coverage of all branches in sub-expressions Ad hoc approach to selecting paths

Code review

Aug. 23, 2000 ISSTA 2000 26

Test Sequencer

Contract Scenarios

Contract Scenarios

Test Subject

State Mapping Objects

State Mapping Objects

Testing Framework

Dependencies

Pre/Post Checks

(delegator)

Pre/Post Checks

(delegator)

Aug. 23, 2000 ISSTA 2000 27

Testing a Light

ILight Scenarios

LightMap

ILight model to

Impl.

ILightPre/Post Checks

Test Sequencer

Aug. 23, 2000 ISSTA 2000 28

Testing a Switch

Switch Scenarios

SwitchMap

model to Impl.

SwitchPre/Post Checks

Test Sequencer

LightILight

Checks

Aug. 23, 2000 ISSTA 2000 29

Production Process

One small team per componentDeliverables include Code for one component DLLDocumentationAny new or updated tests

Different levels of hardnessFactored, versioned build systemAudit at end

Aug. 23, 2000 ISSTA 2000 30

Revisions

Bugs still happen

Usually modify and release new component

In rare cases, release in place

Aug. 23, 2000 ISSTA 2000 31

Cost of Development

KLOC #People #Months #Classes Ship

TestComposition RT 2 5 5 4 9Connection Helper 1 2 1 1 6Collections 3 6 6 6 45Test framework 1 24 8Deployment RT 3.5 26 31 42 63

Totals 145 43 53 131

KLOC / year “shipped” = 4.4

KLOC / class = 1.2

Aug. 23, 2000 ISSTA 2000 32

The Road to 100% Coverage

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

110.00%

9/21

/199

9

10/1

5/19

99

10/2

1/19

99

10/2

8/19

99

11/3

0/19

99

12/9

/199

9

12/1

4/19

99

12/2

0/19

99

1/7/

2000

1/13

/200

0

1/21

/200

0

1/28

/200

0

2/3/

2000

2/11

/200

0

Aug. 23, 2000 ISSTA 2000 33

Interesting Bugs

Code and test made same bug

Coverage of last branch yielded bugs, twice

Post-condition in test was incomplete

Several difficult MT bugs

Shutdown of complex structures

Aug. 23, 2000 ISSTA 2000 34

Future Work

Reducing Cost

Testing Composites

Increasing Coverage

Contract Coverage

Contract Specification

Aug. 23, 2000 ISSTA 2000 35

Reducing Cost

Generate pre and post condition delegators Involves specification in contract and in

implementation

Apply consistency checks to contractsSelect test cases based on coverage goalsTesting generics

Aug. 23, 2000 ISSTA 2000 36

Reducing Cost, cont.

Test specification language

Multi-threaded componentsTrack low level state and locks held for

detecting race conditions

Compiler-generated codeReducing noise in output

Use a GC runtime

Aug. 23, 2000 ISSTA 2000 37

Testing Compositions

Static ones are like a componentUse composition description to helpLook for new paths not executed by unit testDomain-specific GUI (composition of bits on screen) Network-based (different failure modes) Hardware/software composites

Define contract for composite and try to show correspondence with structure

Aug. 23, 2000 ISSTA 2000 38

Achieving More Coverage

Loops

More paths through cleanup code

Choosing other paths to measureE.g., D-U paths in code or in state diagram

Aug. 23, 2000 ISSTA 2000 39

Contract Coverage

Sequencing testsState walking; choosing paths is keyGenetic algorithm

Test case correlation Track variations generated from specs with

actual tests

Aug. 23, 2000 ISSTA 2000 40

Contract Specification Issues

Reentrancy, eventsNeed a description of the state of an

instance when an event handler is calledE.g., when button fires events to adapter

State mutation described in specCan use that to help direct MT testing

Aug. 23, 2000 ISSTA 2000 41

Related Work

Brooks, Wirth, Parnas: various

Gall: Systemantics

Petroski: Designs and Errors

Williams: Inheritance + QI

Meyer: Design by Contract

Kiczales: Open Implementation

Harel, Wegner: Turing revisited

Aug. 23, 2000 ISSTA 2000 42

Related Work, cont.

CMU, Nierstrasz: Composition

Gamma, et. al., Coplien: Patterns

D’Souza & Wills: Catalysis

Beizer, RST Corp: Testing

Beck and others: Extreme Prog.

Holmes, et. all: Synchronization

Szyperski: Beyond OOP

Aug. 23, 2000 ISSTA 2000 43

Summary

Complex, robust, evolving software systems

Component and composite approach

Contracts and testing

Need better tools!

http://research.microsoft.com/comapps