Universiteit Antwerpen
Software Testing
Chapter 2
2. Models
Universiteit Antwerpen
(Based on “Part II: Models” of Testing Object-Oriented Systems)
• Models • Why ? What ? How ?
• Combinational Models • Decision Tables: What ? How ? • Test Generation
• State Machines • What ? (variants: Mealy & Moore) • State Transition Tables (State-to-state, Event-to-state) • The FREE State Model (State ? Transition ?) • Test Generation (N+)
• Part III: Test Patterns
2. Models
!2
Universiteit Antwerpen
Testing Approaches
!3
(Smart) Poking Around
Brute Force
Systematic & Focused
Universiteit Antwerpen
Models are what distinguishes craft from engineering !
Testing is a search problem • trillions and trillions of possible input and state combinations
• only a few will reach, trigger and propagate bugs
Solutions ? • Brute force … does not work at this scale • Poking around … inefficient + false feeling of confidence.
• Model based = systematic, focused and automated • Systematic: we try every target combination (target ≈ model) • Focused: where are bugs likely ? (likely ≈ fault model) • Automated: more tests; more repeatable test (tools ≈ model)
Why Models ? (in Testing)
!4
Universiteit Antwerpen
What is a Model ? (in Testing)
TEST MODELS • subject
- e.g. implementation, component, … under test
• point of view/theory - required behaviour,
where are bugs likely ? • representation
- graphs (nodes & edges), - checklists
• technique - experience with
requirements, design, …
ENGINEERING MODELS • subject
- e.g. airplanes, buildings, weather, economics
• point of view/theory - e.g. gravitation, air-pressure,
law of demand and offer • representation
- wire frame image, blueprint - mathematical equations
• technique - skill and expertise of
modeller matters!5
coverage
Universiteit Antwerpen
(TEST) MODELS • Specify precisely
- what is allowed … what is NOT allowed
Different models representing various perspectives • necessarily
- complete - unambiguous - consistent - coherent
CARTOONS • sketching, refining,
documentation - initial analysis, design, …
• potentially - incomplete - ambiguous - inconsistent - not integrated
Cartoons vs. Models
!6
Universiteit Antwerpen
Scope: Testing, Validation, Checking, Verification
!7
meta-model
component representation
component implementation
required behaviour
observed behaviour
Consistency CheckingCompleteness
Checking
Responsability-
based Testing
Implementation-based Testing
Covered in this course !
Verification (checklists, proofs) [no execution]
Validation[Black box]
[White box]
Universiteit Antwerpen
(Based on “Part II: Models” of Testing Object-Oriented Systems)
• Models • Why ? What ? How ?
• Combinational Models • Decision Tables: What ? How ? • Test Generation
• State Machines • What ? (variants: Mealy & Moore) • State Transition Tables (State-to-state, Event-to-state) • The FREE State Model (State ? Transition ?) • Test Generation (N+)
• Part III: Test Patterns
2. Models
!8
Universiteit Antwerpen
Decision Table: example
!9
Condition Section Action Section
Variant # Claims # Insured Age
Premium Increase
Send Warning
Cancel
1 0 <= 25 50 No No
2 >= 26 25 No No
3 1 <= 25 100 Yes No
4 >= 26 50 No No
5 2 to 4 <= 25 400 Yes No
6 >= 26 200 Yes No
7 5 or more Any 0 No Yes
Implicit Assumptions • Logical operator to interpret a row ? (AND) • Can a person of 15 really be insured ? And what about 99 ?
Annual renewal of a hypothetical car insurance policy
Universiteit Antwerpen
Decision Table: alternative views
!10
Variant
1 2 3 4 5 6 7
Condition Section
Number of Claims
0 0 1 1 2-4 2-4 >= 5
Insured Age
<= 25
>= 26
<= 25
>= 26
<= 25
>= 26
Any
Action Section
Premium increase
50 25 100 50 400 200 0
Send warning
No No Yes No Yes Yes No
Cancel No No No No No No Yes
Column wise view
Decision Tree $50, no letter<= 25
Number of Claims
Age
Age
Age
Cancel
None
1
2-4
>= 5
$25, no letter>= 26
$100, send warning letter
$50, no letter
<= 25
>= 26
$400, send warning letter
$200, send warning letter
<= 25
>= 26
• easier to understand (and program)
• possibility to overlook a case
Universiteit Antwerpen
STEPS • 1) identify the decision
variables and conditions • n conditions ⇒ 2n variants
• 2) identify the resultant actions to be selected or controlled
• 3) identify which actions should be produced in response to particular combinations of conditions
• 4) derive the logic function • via truth table • validate completeness and
consistency
APPLICABLE ? • One of several distinct
responses is to be selected • … according to distinct cases of
input variables (finite !) • input cases are mutually
exclusive • response is independent of
• order of input variables • prior input or output
• prefer small tables (combinational explosion)
• typical for business rules, simple protocols (locks), …
Combinational Models: Basic Approach
!11 Universiteit Antwerpen
decision table with n conditions ⇒ 2n variants
usually fewer variants, only explicit variants are listed Implicit variants ? • don’t care: condition true or false, doesn’t change the action
- e.g.: when # claims >= 5, then age is “don’t care” condition • can’t happen: condition which is assumed never to become true
- e.g.: insured age < 16 or 18 (age when driving is allowed) - can sometimes occur when context changes (Ariane crash) - must be tested explicitly (sometimes is a “surprise”)
• don’t know: is an incomplete model - e.g. what happens with an age > 300 ?
can’t happen + don’t know (+ don’t care): typical source for bugs - specify resulting action anyway (defaults) - test for the result
Identify decision variable and conditions
!12
Universiteit Antwerpen
Deriving the logic function: truth table
!13
Logic function: only boolean variables as input and output • Expression consisting solely of AND, OR, XOR, NOT + conditions • Convert decision table to truth table; truth table into expression
Decision Variable Variant
Condition section
Condition 1 2 3 4 5 6 7
Number of claims 0 T T F F F F F1 F F T T F F F2-4 F F F F T T F
5+ F F F F F F T
Insured Age 25 or younger T F T F T F DC26 or older F T F T F T DC
Action section
Premium increase = 0 F F F F F F TPremium increase = 25 F T F F F F FPremium increase = 50 T F F T F F F
Premium increase = 100 F F T F F F F
Premium increase = 200 F F F F F T FPremium increase = 400 F F F F T F FSend warning F F T F T T FCancel F F F F F F T
Universiteit Antwerpen
Deriving the Logic Function: logic minimization
!14
Initial Development
Logic Minimization Technique +- upper limit, number of boolean variables
-- inspection 3
1950s KV matrix 5
1970s Cause-effect graph 8
1950-1960s Quine-McCluskey method 9
1960s Starner-Dietmeyer 14
1980s Exact minimization 20-25
1980s Heuristic minimization 20-25
• reduce logic function to compact but equivalent expression - fewer test cases are needed - faster testing (automated) - may reveal omissions or inconsistencies
Universiteit Antwerpen
• Incorrect value assigned to a decision variable • Incorrect or missing operator in a predicate • Incorrect or missing variable in a predicate • Incorrect structure in a predicate
- dangling else, misplaced semicolon, … • Incorrect or missing default case • Incorrect or missing action(s) • Extra actions • Structural error in decision table implementation
- falling through (i.e. a missing break in C++), incorrect value for an index or action selector
• Missing or incorrect class or method signature - when variants are implemented via polymorphism
• Generic errors - wrong version, incorrect or missing specification item, ambiguous
Decision Table: Fault Model
!15
No empirical models about
distribution of faults !
Universiteit Antwerpen
• All-Explicit Variants - Most intuitive approach - 1 test case for each explicit variant of all decision variables - example: 0 claims, 1 claim, 2-4 claims, 5+ claims,
25 or younger, 26 or older • Applicable
- decision tables with non-binary variables - weak for undefined “can’t happen” situations - weak when boundaries are undefined (⇒ “don’t know” situation)
⇒ Also apply non binary variable domain analysis
Test Generation: All-Explicit Variants
!16
Universiteit Antwerpen
Start from truth table (+ corresponding boolean function) • All-Variants: every variant in the table is tested once
- 2n possibilities: works only for smaller tables
• All-True: every variant that produces TRUE - equivalent to testing all minterms of logical function - inappropriate when important actions follow from false variants
• All-False: every variant that produces FALSE - equivalent to testing complement of all minterms of logical
function - inappropriate when important actions follow from true variants
• All-Primes: each prime implicant of boolean function at least once - i.e. each product term in a minimal sum-of-products form of
boolean expression All-true, all-false, all-primes may miss critical bugs !!
All-Variants, All-True, All-False, All-Primes
!17 Universiteit Antwerpen
Each-Condition/All-Condition • each variable is made true once, with all other variables being false
+ one test case where all variables are true (and logic) + one test case where all variables are false (or logic)
Binary Decision Diagram Determinants • Starts from the truth table
- Produces a binary-decision diagram (see book p. 157-158) - Map the diagram into a determinant table (see book p. 159-160)
Variable Negation • Starts from sum-of-products boolean formula
- unique true points: … - near false points: …
Non-binary Variable Domain Analysis • domain testing (including boundary values) on decision variables
More test generation techniques
!18
Universiteit Antwerpen
Decision Table Test Suite Size
!19
Test Strategy Test size, Binary Decision Variables
Test Size, Non-binary Decision Variables
Test Power
All-Variants 2n 2n * 4 n 100%
Basic Variable Negation 0.1 * 2n 0.1 * 2n * 4n 97%
Each Condition/All- Conditions m + ∑ L(pi) (m + ∑ L(pi)) * 4n ?
All BDD Determinants d d * 4n ?
All-True t t * 4n ?
All-False f f * 4n ?
All-Primes m m * 4n ?
All-Explicit x x * 4n ?
For non-binary decision variable we have an expression (A <= x <= B) • test x = A - 1, x = A + 1, x = B -1, x = B + 1
• # tests may become smaller for ranges where x are adjacent
Universiteit Antwerpen
Inclusion Hierarchy
!20
All-Variants
All-FalseAll-True
All-Primes
Each-Condition/ All-Conditions
Basic VariableNegation
BDDDeterminants
Universiteit Antwerpen
(Based on “Part II: Models” of Testing Object-Oriented Systems)
• Models • Why ? What ? How ?
• Combinational Models • Decision Tables: What ? How ? • Test Generation
• State Machines • What ? (variants: Mealy & Moore) • State Transition Tables (State-to-state, Event-to-state) • The FREE State Model (State ? Transition ?) • Test Generation (N+)
• Part III: Test Patterns
2. Models
!21 Universiteit Antwerpen
= system whose output is determined by both current and past input (with Decision tables, only current input influences result)
• identical inputs are not always accepted • identical inputs may produce different outputs
State machine consists of 4 building blocks • State: an abstraction that summarizes the past inputs • Transition: an allowable two-state sequence
• “accepting” and “resulting” state • Caused by an event; may result in an action
• Guarded transition: event + guard predicate • Event: an input (or an interval of time) • Action: the result or output that follows an event
Special states: initial state, current state, final state
What is a state machine ?
!22
Universiteit Antwerpen
MOORE State Machine • transitions are passive • states are active
- may have output action - every output action has at
least one state
MEALY State Machine • transitions are active
- may have output action - any output action may be
used in more than one transition
• states are passive
Variants: Mealy and Moore
!23
• Mathematically equivalent • MEALY is preferred for engineering design • UML allows both !
- Hybrids are bad for test design !
Universiteit Antwerpen
Two-player Squash: Mealy State Machine
!24
game started
player1 served player2 served
player1 won player2 won
p2_Start()/ simulateVolley()
p1_Start()/ simulateVolley()
p1_WinsVolley() [this.p1.score()< 20] / this.p1AddPoint() simulateVolley()
p2_WinsVolley() [this.p2.score()< 20] / this.p2AddPoint() simulateVolley()p1_WinsVolley()/
simulateVolley()
p2_WinsVolley()/ simulateVolley()
p1_WinsVolley() [this.p1.score()== 20] / this.p1AddPoint()
p2_WinsVolley() [this.p2.score()== 20] / this.p2AddPoint()
p1_IsWinner()/ return TRUE
p2_IsWinner()/ return TRUE
Universiteit Antwerpen
Two-player Squash: Moore State Machine
!25
game started
Player1 serving simulateVolley()
Player1 won p1_AddPoint()
Player2 won p2_AddPoint()
p2_Start()p1_Start()
Player2 serving simulateVolley()
p1_WinsVolley()
p2_WinsVolley()
Player1 continues p1_AddPoint() simulateVolley()
Player2 continues p2_AddPoint() simulateVolley()
Answering Player2 won return TRUE;
Answering Player1 won return TRUE;
p1_IsWinner()p1_IsWinner() p2_IsWinner()p2_IsWinner()
p1_WinsVolley() [this.p1_Score() < 20] p1_WinsVolley()
[this.p1_Score() == 20]
p2_WinsVolley() […]
p2_WinsVolley() [this.p2_Score() == 20]
p1_WinsVolley() p2_WinsVolley()p2_WinsVolley()
p1_WinsVolley()
Universiteit Antwerpen
state-to-state format • row = accepting state; column = resulting state • cell = transition = event (with guard) + action
event-to-state format • row = event (with guard); column = accepting state • cell = transition = action + resultant state
expanded state-to-state format • 2 state-to-state tables: 1 for events and 1 for actions • separates the transition function and output function
State Transition Tables
!26
Universiteit Antwerpen
Example: State-to-state
!27
CurrentState
Resultant State/Event/Action
Game Started
Player 1 Served Player 2 Served Player 1 Won Player 2 Won
Game Started
p1_start() p2_start()
simulateVolley() simulateVolley()
Player 1 served
p1_winsVolley() [p1 score < 20]
p2_winsVolley() p1_winsVolley() [p1 score == 20]
this.p1AddPoint(), SimulateVolley()
simulateVolley()
Player 2 served
p1_winsVolley() p2_winsVolley() [p2 score < 20]
p2_winsVolley() [p2 score == 20]
simulateVolley() this.p2AddPoint(), SimulateVolley()
Player 1 won
p1_IsWinner()
return TRUE
Player 2 won
p2_IsWinner()
return TRUE
Universiteit Antwerpen
Example: Event-to-state
!28
Current State/Action/Next State
Event Guard Game Started Player 1 Served Player 2 Served Player 1 Won Player 2 Won
p1_start() simulateVolley()
Player 1 Served
p2_start() simulateVolley()
Player 2 Served
p2_WinsVolley()
DC (don’t care)
simulateVolley()
Player 2 Served
p2_Score < 20
this.p2AddPoint() ,simulateVolley()
Player 2 Served
p2_Score == 20
this.p2AddPoint()
Player 2 Served
p1_WinsVolley()
… …
p1_IsWinner()
return TRUE
Player 1 Won
… …
Universiteit Antwerpen
Flattened Regular Expression (FREE) • class under test is flattened
• all inherited features are collapsed in class under test • assume “type-substitution” principle • instance of a class may be substituted by an instance of a subclass
• model behaviour as state machine
• restrictions to UML statechart notation • few changes to UML statechart notation (e.g., alpha/omega states)
• can be expressed in UML
• results in testable model • no ambiguities (among others precise definition of “state”) • automatic generation of test cases
The FREE State Model
!29 Universiteit Antwerpen
• A particular subset of a class combinational value set • Represented by a State Invariant (predicate) Class Account { private: AccountNumber number; Money balance; Date lastUpdate}
What is State ?
!30
Account.balance
Account.lastUpdate
Account.number
Open StateInactive State3.650.182.500.000
“states”
Universiteit Antwerpen
• state invariant defines a subset of legal values for a class • state invariant is same or stronger than class invariant • method postconditions + state invariants = all states
• don’t use hybrids (i.e. no mixing of Mealy and Moore)
• alpha state • null state: represent declared object before construction • accept only constructor, new, … (alpha transition) • is NOT the same as the UML initial state (solid circle)
• omega state • null state: reached after deletion, destruction, out of scope • accepted explicit or implicit destructors (omega transition) • is NOT the same as the UML final state (bulls-eye)
Implications
!31
alpha-omega cycle
Universiteit Antwerpen
Transition = • 1 state invariant for accepting state • 1 state invariant for resulting state • 1 associated event
• message sent to the class under test • interrupt or similar external control (timer, …)
• [optional guard expression] • predicate expression evaluated in the context of class under test
• 0 or more actions • message sent to a server object of the class under test • response provided by an object of the class under test
What is a Transition ?
!32
Universiteit Antwerpen
What should happen when a state machine in a given state receives an event not specified for this state ? • ignore
• standard semantics for state machines • unacceptable for testing purposes
• omitted • incomplete specification: extend the state machine
• illegal (or “impossible”) • illegal event = valid event, not acceptable for the current state
(e.g. “pop()” on an empty stack) • if accepted results in an illegal transition
Sneak path • beware: guarded transitions, what if the guard is FALSE ? • transition to an “Illegal event exception” state
Unspecified Event/State Pairs
!33 Universiteit Antwerpen
Response Matrix = Modified event-to-state table • ROWS: list events + guards
• unguarded event: 1 row • guarded event: 1 row for each unique event/guard combination
• one row for each truth combination of subexpressions + additional column for all sub expressions
• if event is sometimes guarded, then include a “don’t care” row • COLUMNS: list Accepting states • CELLS: list responses
• error codes for possible responses
Response Matrix
!34
Universiteit Antwerpen
0. Accept (perform specified transition) 1. Queue (queue event for subsequent evaluation and ignore) 2. Ignore 3. Flag (return a non zero error code) 4. Reject (raise an exception) 5. Mute (disable the source of events and ignore) 6. Abend (invoke abnormal termination and halt process)
Response Matrix: Possible Responses
!35 Universiteit Antwerpen
Example: Response Matrix
!36
Accepting State/Expected Response
Events Guards alpha A B … C omega
constructor --- --- --- 0 6 6 6 6 6
event 1 x == 0
DC --- --- --- --- 1 … 2 6
F --- --- --- 0 --- … --- ---
T --- --- --- 0 --- … --- ---
event 2 i > 10 k == max isReset()
DC DC DC --- 2 5 … --- 6
F F F --- --- --- … 5 ---
F F T --- --- --- … 0 ---
F T F --- --- --- … 0 ---
… … … …
destructor --- --- --- --- 0 0 0 0 2
if event is sometimes guarded, then include a “don’t care”
Universiteit Antwerpen
s states, e events, a actions gives (s x a) es possible implementations • 5 states, 2 events and 2 actions: 10 billion possibilities • only one correct implementation
Possible faults • missing or incorrect transition • missing or incorrect event • missing or incorrect action • an extra, missing or corrupt state • a sneak path (a message is accepted when it shouldn’t be) • an illegal message failure (unexpected message causes a failure) • trap door (implementation accepts undefined messages)
State Based Testing: Fault Model
!37
No empirical models about
distribution of faults !
Universiteit Antwerpen
• structure check-list (Table 7.5) • structurally complete and consistent
• state name check-list (Table 7.6) • avoid weak, ambiguous or inappropriate state names
• guarded transition check-list (Table 7.7) • check logic and structure of guard expressions
• flattened machine check-list (Table 7.8) • free of level-to-level conflicts • consistent behaviour for inheritance tree • reuse of superclass test suites
• robustness check-list (Table 7.9) • safe and correct behaviour under failure modes
Verify the State Model
!38
Universiteit Antwerpen
N+ Test Generation • Features
• exercise all implicit transitions to reveal sneaky paths • implementation must be able to report current state
state reporter methods (available ? encapsulation ?) • Steps
• Develop a FREE model + Derive response matrix • Generate round-trip test cases via Transition Tree (see p. 249)
cover all branches (= transitions) at least once identify parameter values, expected state and exception
• Generate sneak path test cases cover all branches with unspecified transitions verify response + resulting state
• Sensitize the transitions find input values for all guards
Test Generation
!39 Universiteit Antwerpen
Based on years of experience with state-based testing of hardware and telecommunications infrastructure
• All Transitions • every specified transition at least once
• All n-Transition sequences • every specified transition sequence of n events
• All round-trip Paths • every sequence of specified transitions beginning and ending in
the same state is exercised at least once • M-Length Signature
(when state reporter methods are absent) • signature = sequence of output actions unique for given state • verify whether we reached the given state by verifying signature
State Machine Coverage
!40
Universiteit Antwerpen
(Based on “Part II: Models” of Testing Object-Oriented Systems)
• Models • Why ? What ? How ?
• Combinational Models • Decision Tables: What ? How ? • Test Generation
• State Machines • What ? (variants: Mealy & Moore) • State Transition Tables (State-to-state, Event-to-state) • The FREE State Model (State ? Transition ?) • Test Generation (N+)
• Part III: Test Patterns
2. Models
!41 Universiteit Antwerpen
pattern = • generalized solution to a specific recurring (design) problem
• capture the essence of proven solutions
pattern template • includes name (design discussions) • context
• when to apply, when NOT to apply • forces: different interests pull towards various directions
• trade-offs • advantages and disadvantages in terms of forces
• known uses
(Test) Patterns
!42
Universiteit Antwerpen
• name • intent • context (incl. test scope) • fault model
• what kind of faults are the target ? • motivate: why reach faults + trigger and propagate failures
• strategy • how to design and implement the test suite • procedure, oracle, automation • example
• entry criteria • preconditions for using this pattern • when is the“Implementation Under Test” (IUT) test ready ?
• exit criteria • defines the results necessary to achieve an adequate test
Test Pattern Template
!43 Universiteit Antwerpen
(See also IEEE standard 829) • prepare a test plan
• what to test when • deliverables • stop criteria
• design the test suite • test suite hierarchy (i.e. V-model) • procedures for running the tests • prepare test cases (using appropriate test pattern)
• test the component • results for each test case: pass / no pass / unable to execute • store results (and follow-up action) for later reference
• prepare test summary report • Evaluate the operability of the IUT
Test Documentation
!44
Universiteit Antwerpen
• Models • Why ? What ? How ?
• Combinational Models • Decision Tables: What ? How ? • Test Generation
• State Machines • What ? (variants: Mealy & Moore) • State Transition Tables (State-to-state, Event-to-state) • The FREE State Model (State ? Transition ?)
alpha-omega cycle • Test Generation (N+)
• Part III: Test Patterns
Conclusion
!45 Universiteit Antwerpen
Patterns are NOT a good vehicle for • Analytical comparison (and contrasting) • Presenting a general model, theory, point of view • Defining terms + conceptual relationship • Lengthy and complex system of ideas
I want you to teach some of the material & serve as opponent for your colleagues
• Will force you to acquire deep understanding • In depth understanding of parts
is preferred over broad understanding of the whole
Patterns are not a Panacea
!46
Universiteit Antwerpen
Preparing slides for a 2-hour lecture takes 8 hours ! • scope of the lecture is known
• message (what do I want the audience to remember ?) • story (how will I sequence the material from known to unknown ?)
Avoid “death by bullet points” • use tables / schemas / animation / … • slides are a tool (not a goal in themselves)
Passionate lecturing • use personal anecdotes, examples, …
• Examples? Industrial Guest Lectures! • relate to your audience
Tips
!47