Post on 27-May-2015
transcript
Test Blueprints
Adrian Lienhard, Tudor Gîrba,Orla Greevy and Oscar Nierstrasz
Software Composition GroupUniversity of Bern, Switzerland
Thanks for the support:
Exposing Side Effects in Execution Traces to Suport Writing Unit Tests
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
Unit Testing
...unit tests are important!
...write tests first!
Everybody knows
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
Unit Testing
...unit tests are important!
...write tests first!
Fair enough, but what if ...the code exists already?...and you don’t understand it well?
Everybody knows
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
Unit Testing
Yet, writing unit tests for legacy systems is important.
“Tests are your life insurance”
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
Challenges of Testing Legacy Code
1. Selecting a unit to test choose unit of appropriate complexity
2. Creating a fixture instantiate and set up graph of objects
3. Executing the unit under test stimulate the fixture
4. Verifying the expected behavior assert side effects and return value
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
Idea
Use example runs of the program to extract test scenarios.
Perform a dynamic analysis to extract a test blueprint that shows:
- how to create a fixture for a selected unit- how to execute the unit- how to verify the expected behavior
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
Program InstrumentationExecution of existing tests
Analysis
object flow +execution trace data
coverage data
Example program execution
Execution Trace Test Blueprint
Tool
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
Test Blueprint
ASTChecker>>declareVariableNode:
FunctionScope>>addTemp:
TempVar class>>new
...initialization...
TempVar>>name:
TempVar>>scope:
KeyedSet>>add:
...library code...
...
...E
xe
cu
tion
Un
it
Test BlueprintExecution Trace
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
Test Blueprint
Legend
existing reference
new reference (side effect)
:Class existing instance
:Class new instance
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
Creating the Fixture
Legend
existing reference
new reference (side effect)
:Class existing instance
:Class new instance
1 fscope := FunctionScope new.
2 varname := ’x’.
Unit Test for addTemp: method
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
Executing the Unit under Test
Legend
existing reference
new reference (side effect)
:Class existing instance
:Class new instance
1 fscope := FunctionScope new.
2 varname := ’x’.
3 var := fscope addTemp: name.
Unit Test for addTemp: method
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
Verifying Expected Behavior
Legend
existing reference
new reference (side effect)
:Class existing instance
:Class new instance
1 fscope := FunctionScope new.
2 varname := ’x’.
3 var := fscope addTemp: name.
4 self assert: var class = TempVar.
5 self assert: var name = varname.
6 self assert: var scope = fscope.
7 self assert: (
fscope tempVars includes: var).
Unit Test for addTemp: method
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
Blueprint Examples
InstanceScope>>newMethodScope
IRBuilder>>add: FunctionScope>>lookupVar:
NonClosureScopeFixer>>acceptVarNode:
Program instrumentation
Execution of existing tests
Analysis (Section 6)
object flow +execution trace data
coverage data
Example program execution
Execution Trace Test Blueprint
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
Implementation
Aliases are created when an object is:
‣ instantiated
‣stored into or read from a field
‣stored into or read from an array
‣passed as argument/return value
dynamic model static model
Alias
Instance
*
0..1
*0..1
Method* 1
Class1*
*
1
creator
Activation
parent
receiver
Execution-
Unit
root
children
createdAliases
* 1
activations
*
1
Object Flow meta-model
The flow of an object in the system is tracked by the parent-child relationship of its Aliases
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
Implementation
Imported references = used state (->fixture)Exported references = side effects (->assertions)
Importedreferences
Exportedreferences
Execution Unit
object_1object_2object_3
t
w r
w r w
w
r
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
Initial Case Study I
- One developer produced 12 tests in 2h- Average fixture size: 5- Average side effects: 4
How straightforward is it to use the proposed tool? Target system: Insurance broker application (520 classes, 6 years old)
- Test Blueprint worked surprisingly well ...yet, sporadically resorted to consulting source code
- Insufficient support for selecting execution units
Observations
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
Initial Case Study II
- 72 identical- 5 additional- 12 missed ==>
How do tests written using our approach differ compared to conventional tests written by an expert? Content Management System (377 classes)
Study: selected 14 unit tests, removed all 84 assertions and then systematically rewrote them using our tool
verification that special objects are left unmodified
our approach does not detect unmodified objects that are worthwhile to verify!
Resulting assertions
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
Conclusions
Yes, we can write unit test for not well known code...by exploiting runtime information
Test Blueprints tell us...how the fixture and assertions have to look like
Difficulties are how to...select good execution units in the trace...initialize objects to bring them into the desired state
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
Conclusions
Yes, we can write unit test for not well known code...by exploiting runtime information
Test Blueprints tell us...how the fixture and assertions have to look like
Difficulties are how to...select good execution units in the trace...initialize objects to bring them into the desired state
Questions?