CS-C3180 Software Design and Modelling – 2020-11-12Lassi HaaranenBased on material created by Juha Itkonen

Software testing Part 2: Test design techniques

”All useful and practical test techniques are heuristic: they are shortcuts that might help, but they do not guarantee success. Because test techniques are heuristic, there can be no pat formulas for great testing. No single test technique will serve; we are obliged to use a diversified strategy. And to be consistently successful requires skill and judgment in the selection and application of those techniques.”

Bach and Schroeder, Pairwise testing: A best practice that isn’t

Software testing• Software testing is a creative, important, and

demanding activity if done systematically.• It requires different kinds of knowledge [1].

• Domain knowledge• System knowledge• Generic knowledge

• The better testers understand user needs and expectations, the better they can test the system.

3[1] Itkonen, Juha, Mika V. Mäntylä, and Casper Lassenius. 2014. “The Role of the Tester’s Knowledge in Exploratory Software Testing.” IEEE Transactions on Software Engineering 39 (5): 707–724.

Learning goals of today

4

Core concepts

Test strategy, Coverage and knowledge in testing, Test design techniques: operational variables modelling, combinatorial testing, scenario-based testing

After passing this course, the student is able to…

Basic Can describe the purpose of software testing.Can explain the meaning of the core concepts.

Good Can define test strategy for a simple software system.Can systematically design tests using design models (RE, DM and SAD) and apply basic software test design techniques.Can assess test coverage and explain the meaning of used coverage criteria.

Very good Can apply domain knowledge to guide and reason the test design decisions.Can critically assess and improve design models (RE, DM, SAD) from the viewpoint of testing.Can analyse the contribution of software testing to the other development activities (RE, DM, SAD).

Overview of this lecture• Last week

• What is the purpose of software testing?• What is the contents of test strategy?

• This week: Test design techniques• operational variables modelling • combinatorial testing • scenario-based testingThese test design techniques will be applied in Assignment 4.

5

Test Strategy: Essential contents

SUT

COVERAGE

TECHNIQUES

KNOWLEDGE

THE SYSTEM

≠

6

QUALITY CRITERIA

Testing needs various models• Tester cannot rely on one

source only

• Different models and sources together draw the picture of the system

• Testing can utilize the design models, but needs also specific modelling for the testing purposes

Tester’s Model???

Use Cases

User Stories

Design ModelsTacit Knowledge, Experience

Testing Techniquesand Heuristics

Technical environments

Domain Model

SoftwareArchitecture

Users’ expectations

7

Testing needs details• Tests deepen the details of requirements

– Examples of the logic– Special cases and error handling– …

• Test related details might be less interesting from the requirements viewpoint

– Covering all combinations of input values– Boundaries and extreme cases– …

• In test design, the relevant details need to be documented and modelled– Not the obvious things– Not all the knowledge that a tester would need – tester should be a competent person

• Do not program people: If it is easy to program, automate it.– Repetitive actions, straightforward checks– Exact comparison of large volume of data– …

8

Testing Techniques – Systematic Coverage for Testing

• Increase: effectiveness, efficiency, repeatability, visibility, and measurement of testing

• White-box testing– Internal software structure used for test design– Try to maximize execution path coverage– Works best for unit testing

• Black-box testing – focus of this course– Requirements and features used for test design– Try to maximize feature and input coverage– Works best for system and acceptance testing

9

Testing levels – Systematic Coverage for Testing• Acceptance testing “determines whether a system satisfies its

acceptance criteria by checking desired system behaviors against the customer requirements” [2]. In this course, customer requirements are called as user requirements.

• System testing is concerned with testing the behavior of an entire system and is usually considered appropriate for testing quality requirements and external interfaces to other applications [2].

• Integration testing is the process of testing the interactions among software components, and modern integration strategies are typically architecture-driven [2].

• Unit testing is the process of testing individual software components such as functions or methods within an object [1].

10

Further reading[1] Sommerville 1 (2016) Sofware Engineering, 10th edition, Pearson.[2] Bourque P and Fairley R E (2014) SWEBOK® V3.0: Guide to the Software Engineering Body of Knowledge, IEEE Computer Press, pp.4.5-4.6.

Equivalence Partitioning and Boundary Value Analysis

• Partition (divide) the inputs and outputs into equivalent sets from the viewpoint of the system features.

• Inside one partition the behavior of the system is assumed to be identical.• Faults tend to appear near boundaries between the partitions.

invalidinvalid

0 1 12 13

valid

11

Three examples of test design techniques

1. Modeling operational variables and relations• Equivalence partitioning and boundary value analysis• Modeling operational relation for use cases and user stories

2. Combinatorial Testing• Covering the combinations of interacting variables

3. Scenario-Based Testing• Simulating realistic and plausible use of the system

12

BREAKOUT ROOM (5 min)

What have you learned about test designtechniques in your prior experience?

13

Requirements based testing byModeling Operational Variables and Relations

14

Testing technique 1

Use cases and testing• If the requirements are modelled as use cases,

it is good to utilize them in testing

• Description of the functionality• Use case as a test oracle: Verifying against user requirements• Validating the use case design

• Test design works as a review for use cases

• Revealing ambiguities and missing details• Finding defects and problems

15

Use Case is not Test Case or is it?• Use cases describe typical scenarios

• With common exceptions• Use cases are not complete specifications

• Testing only the conditions and details that are mentioned in use case is usually not enough

• Requirement models and test models have different purpose• Requirements emphasize clarity and conveying the understanding• Tests emphasize the uncommon scenarios, exceptions and failure

situations• Tests aim at coverage and details – need to extend use cases• One use case leads to several test cases

• Testing is about details, combinations, and prioritization.

16

Steps for Test Design based on Requirements1. Identify relevant operational variables2. Define values of these variables

• Equivalence Partitioning and Boundary Value Analysis can be used for each variable

• Classification Tree for organizing the variables3. Model operational relations

• All relevant input and output relationships – variants• Making relationships explicit

4. Implement the test cases (not focus of this course)

Based on “Extended Use Case Test Design Pattern”, chapter in Robert V. Binder. Testing Object Oriented Systems, Addison Wesley, 1999.

17

1. Identify relevant operational variables• Factors that affect the end result or the flow of actions of

the use case and user story– Each variable itself or in combination with other variable(s)

• Operational variables model how the operation of the feature may vary– Explicit input and output variables – Data that the system processes (databases)– Internal states of the system– Preconditions– Environmental conditions that result in different behavior– etc.

18

Finding the operational variables• What factors affect the outcome of

the use case and user story– Outcome = output, system state, and

flow of actions• What choices the user can make

that affect the outcome– Inputs, actions, order of actions, …

• What conditions affect the outcome– Internal states, data, configurations,

etc.– External and environmental

(pre)conditions• What if preconditions are not met?

19

Entered card pin code

Amount of the withdrawAmount of 50€ notes in the machine

Amount 20€ notes in the machine

Balance of the customer’s bank account

ATM example: operational variables

Right level of the operational variables• Operational variables must be defined on the level of

use cases and user stories• Too detailed variables lead to unbearable amount of details and tests

• Variables should model the variations that affect the outcome of the use case

• If an input into a single field does not affect the end result and flow of actions, it is not modeled as an operational variable.

• “input validity”, could be an operational variable that affects the outcome

20

2. Define values of the operational variables• Possible values of operational variables

• Equivalence classes• Boundaries and restrictions• Exceptional, invalid values and error situations

• What are the relevant values of the operational variables• Classify the values into classes that affect to the outcome of the use

case and user story• Consider relevant exceptions, special cases, and error situations

• Focus on the classes of inputs, not on actual values– E.g. “negative number” instead of “-5”

21

Example of the value classes

Entered bank card pin code Correct, wrong, empty

Amount of the withdraw 0, n*20, n*50, n*50+m*20, otherAmount of 50€ notes in the machine Plenty, few, none

Amount 20€ notes in the machine Plenty, few, none

Balance of the customer’s bank account Less than the withdraw, more or equal to the withdraw

ATM example: Values of the operational variables

22

Operational variables Value classes

3. Model operational relations• Relationships of the operational variables and the

outcomes• All distinct outcomes are modeled

Operational variablesinputs, conditions,

states, data, events

Outcomesoutputs, system state,

flow of actions

23

Use case scenariosare variants• Use case has different flows of

actions = different scenarios• Scenarios can lead to different

end result– or the scenarios just present

different flow of actions• For example, “user can cancel

the operation at any phase” – Can require many tests

Start of use case

End of use case

Basic scenario

Alternative scenario 1 Alternative scenario 4

Alternative scenario 2 Alternative scenario 3

24

ATM example: Operational relations

• Each row represents one variant.• Each variant has a different outcome.

pin code withdraw 50€ notes

20€ notes Account balance

Outcome

correct n*20 - plenty Equals or more Right amount of money withdrawn

correct n*50, where n is odd

none plenty Equals or more User advised to select amount that can be delivered as 20€ notes

correct > 0 - - Less than withdraw

User is informed about insufficient balance

wrong - - - - User is informed about wrong pin code and code is re-requested

empty - - - - Pin code is re-requested

… … … … …

25

Operational variables

How to find the expected outcome (oracle)?• Documents; RE, models, specifications, application

domain …• Discuss with domain experts• Collaboration with customers and users• Existing system• etc.

26

Example: Elevator Emergency CallUse Case Diagram

Passenger

Serviceperson

Making Emergency Alarm Call

Dispatching Job

Reporting

Elevator Communications System

Helping the Passenger Out of the Elevator

Service center operator

27

Example: Elevator Emergency CallUse Case Description 1/2▪ Use Case: Making Emergency Alarm Call▪ Summary: An entrapped elevator passenger pushes the emergency alarm button

in order to get help. A service center operator receives the emergency alarm call and informs the passenger that a serviceperson will come and let the passenger out of the elevator.

▪ Actors: Passenger and service center operator▪ Preconditions: An elevator has stopped between floors and there is a passenger

in the elevator. ▪ Goal of the primary users: The passenger wants to get out of the elevator

safely and as fast as possible.

28

Example: Elevator Emergency CallUse Case Description 2/2

Basic sequence:Step 1: The passenger presses the emergency alarm

button for 3s. or longer.Step 2: The service center operator gets a visible

notification about the emergency alarm call on the screen with an optional audio signal.

Step 3: The service center operator accepts the emergency alarm call to be handled.

Step 4: The system opens voice connection between the service center operator and the passenger.

Step 5: The system indicates the service center operator and the passenger that voice connection is open.

Step 6: The system guides the service center operator what information to ask from the passenger.

Step 7: The service center operator informs the system that the emergency alarm call is correct.

Exceptions:Step 1: If an entrapped passenger does not

push the alarm button long enough (more than 3 seconds), the system alerts the passenger with voice announcement.

Step 7: If the passenger has pressed the emergency alarm button by accident, the service center operator informs the system that the emergency alarm call is false. The system resets the emergency alarm call.

Postconditions:The entrapped passenger knows that service center operator will contact a service person who will be helping the passenger out of the elevator safely and as soon as possible.

30

Example: Elevator Emergency CallOperational Variables

31

Example: Elevator Emergency CallOperational Variables and Values documented as a Classification Tree

– What if passenger presses when not between floors?– Or when the elevator is moving? The moving variable is missing – is it relevant?– When there is no passenger in the elevator, is the situation relevant? Is it useless variable?

32

Button press

Elevator position

Passenger inside Situation Doors Operator status

Operator audio Outcome

Long Between Yes Emergency Closed Available EnabledThe system opens audio connection and informs the passenger and the operator

Long Between Yes Emergency Closed Low pri. task MutedThe system informs the passenger that the alarm was received and alerts the operator. Operator accepts the task and system opens the audio connection.

Long Between Yes Emergency Closed Available MutedThe system informs the passenger that the alarm was received and alerts the operator. System unmutes and opens the audio connection.

Long On floor Yes Emergency Closed Available Enabled ?

Long - Yes False alarm - Available Enabled

The system opens audio connection and informs the passenger and the operator. Operator confirms from the passenger that the alarm was false and operator resets the alarm.

Long - Yes False alarm -Serving emergency Enabled ?

Short - Yes - - - - System informs passenger - Between No Emergency - Available - ?

Example: Elevator Emergency CallOperational Relation

What would you do if you don’t know the answers to these questions?

BREAKWe'll continue at 13:15

33

Combinatorial Testing

34

Testing technique 2

From operational variables and values tocombinations• Analysis of the variables and values produces a set of

independent “interesting” values for the input variables• Relevant equivalence classes, boundaries and exceptions

• The next step towards more exhaustive testing is to analyze the interactions between the variables -> The combinations of the values

35

Combinatorial explosion

❑ Comprehensive combinatorial analysis leads to a very large number of combinations.

❑ Testing all combinations of the above values leads to 1728 combinations (3*2*2*2*2*3*4*3).

36

Coverage criteria for combinatorial testing• Base choice coverage

• Select a single (typical) base test case• Change the base case by varying single variable at a time

• Each-choice (1-wise) coverage• Every value of each variable is covered with some test case

• Pair-wise (2-wise) coverage• Every possible pair of values of any two variables is covered

• t-wise coverage• Every possible combination of values of any t variables is covered

• All combinations coverage• All possible combinations of values of all N parameters are covered

37

Example: Base choice coverage of three operational variables

38

Test caseElevator position Button press Doors

1 Between Long Closed, cannot open2 On floor Long Closed, cannot open3 Between Short Closed, cannot open4 Between Repeat Closed, cannot open5 Between Long Open6 Between Long Closed, can open

• Select a single (typical) base test case• Vary a single variable at a time

Example: Each choice (1-wise) coverage of three variables

39

Test case

Elevator position Button press Doors

1 Between Long Open2 Between Short Closed, cannot open3 On floor Repeat Closed, can open

• Select a smallest number of test cases that include all variable values

• Vary all or many variables simultaneously

Example: Pair-wise coverage of three variables 1/3

40

Test case Elevator position Button press Doors

1 On floor Long Open2 On floor Short Closed, cannot open3 On floor Repeat Closed, can open4 Between Long Closed, can open5 Between Short Open6 Between Repeat Closed, cannot open7 - Long Closed, cannot open8 - Short Closed, can open9 - Repeat Open

• Cover all value pairs with small number of test cases

Example: Pair-wise coverage of three variables 2/3

41

Test case Elevator position Button press Doors

1 On floor Long Open2 On floor Short Closed, cannot open3 On floor Repeat Closed, can open4 Between Long Closed, can open5 Between Short Open6 Between Repeat Closed, cannot open7 - Long Closed, cannot open8 - Short Closed, can open9 - Repeat Open

• Cover all value pairs with small number of test cases

Example: Pair-wise coverage of three variables 3/3

42

Test case Elevator position Button press Doors

1 On floor Long Open2 On floor Short Closed, cannot open3 On floor Repeat Closed, can open4 Between Long Closed, can open5 Between Short Open6 Between Repeat Closed, cannot open7 - Long Closed, cannot open8 - Short Closed, can open9 - Repeat Open

• Cover all value pairs with small number of test cases

43

Example: Pair-wise coverage of three variables

Test case Elevator position Button press Doors

1 On floor Long Open2 On floor Short Closed, cannot open3 On floor Repeat Closed, can open4 Between Long Closed, can open5 Between Short Open6 Between Repeat Closed, cannot open7 - Long Closed, cannot open8 - Short Closed, can open9 - Repeat Open

• Cover all value pairs with small number of test cases

9 / 18 COMBINATIONS

(2x3x3=18)

44

Numbers for combination strategies• All combinations

– 2x2x2x2x2x4x5x3x2 = 3840

• 1-wise coverage– Max(2,2,2,2,2,4,5,3,2)= 5

• Pair wise coverage = 24

Combinatorial testing cannot replace domain knowledge• You should understand how the features work

• How things interact and work together• What inputs affect to certain features and outputs

• Combinatorial design is a mechanical approach• Does not guarantee that the most common and important combinations

are covered• The operational relation derived from the requirements could get too

little testing• It is important to use domain knowledge and system

knowledge to test relevant combinations.• Oracle problem: You need a reliable way of recognizing a

failure.

45Where can you find relevant relationships between key concepts (operational variables)? -> Domain model

Tool support for combinatorial designhttp://www.pairwise.org

– Articles and resources– over 30 tools listed: http://www.pairwise.org/tools.asp

• Somewhat working combinatorial tool options:– Allpairs

• http://www.satisfice.com/tools.shtml

– PICT• https://github.com/microsoft/pict

• Modelling classification trees, etc., with any mindmapper– Freemind, xmind, etc.

Scenario-Based Testing

47

Testing technique 3

Scenarios in testing• A scenario is a story used to help a person think through a complex

problem or system.• Testing scenarios are complicated and realistic stories of

real usage of the system.• Testing scenarios are carefully designed and documented.• The goal is to focus on business needs and realistic use situations,

instead of technical details and mechanical testing.• Testing scenarios can be executed manually or (partly) automated.

48

Good Scenarios …• Are motivating stories of real usage of the system• Describe who is the user and what are the user’s goals

when using the system– The goal is not to use the system features, but to accomplish

something through using the system– E.g. no one has the goal of “logging in”, it might be an action

that happens in order for the user to accomplish their goal(s)• Are credible and describe complex use of the system • Are easy to evaluate and powerful in revealing potential

problems

49

Scenario testing is good for• Expose failures to deliver desired benefits• Explore expert use of a system• Connect testing to documented requirements

– Bring requirements-related issues to the surface• Learn the product• Make a bug report more motivating

"Scenario testing works best for complex transactions or events, for studying end-to-end delivery of the benefits of the program, for exploring how the program will work in the hands of an experienced user, and for developing more persuasive variations of bugs found using other approaches."Cem Kaner. “An Introduction to Scenario Testing”, STQE, Oct, 2003.http://www.kaner.com/pdfs/ScenarioIntroVer4.pdf

Assignment 4: Software testing• Test strategy

– Describe an overall testing strategy for testing your solution (The ShapeMeal Service).

• Test Design– From the testing viewpoint, select the most relevant use case and

one user story from requirements created in the first assignment– Use case: model the operational variables and relations. – User story: apply scenario based testing– Write sufficient descriptions of all your test designs and give reasoning

for your design decisions. • Reflect how you applied test design techniques and how

they support each other.• See more detailed instructions in MyCourses!

Assigment 4: Test Strategy

SUT

COVERAGE

TECHNIQUES

KNOWLEDGE

THE SYSTEM Under Test

≠

52

QUALITY CRITERIA

Assigment 4: Test Design

SUT

COVERAGE

TECHNIQUES

KNOWLEDGE

THE SYSTEM Under Test

≠

53

QUALITY CRITERIA

1. Operational variable modelling2. Combinatorial testing3. Scenario-based testing