Introduction to
White box and Black Box
Software Testing
Henry MucciniComputer Science Department
University of L'Aquila - [email protected] –
[www.HenryMuccini.com]
Software Testing
Copyright Notice
» The material in these slides may be freely reproduced
and distributed, partially or totally, as far as an explicit
reference or acknowledge to the material author is
2
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
reference or acknowledge to the material author is
preserved.
Henry Muccini
Agenda
» Recap on Software Testing
» White box Testing
- JUnit
»
3
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
» Black box Testing
- Model-based Testing
Recap on Software Testing
4
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
Recap on Software Testing
Software testing consists of:
the dynamic verification of the behavior of a program
An all-inclusive definition
5
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
on a finite set of test cases
suitably selected from the (in practice infinite) input
domain
against the specified expected behavior
[A. Bertolino]
IMP
Main Activities & Terminology
Main Activities:
– Test case Selection
– Test case Execution
– Regression Testing
Terminology:
– Test Coverage
– Oracle
– Adequacy criteria
– Test Objective
6
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
– Test Objective
– …
» Main principles:
- Repeatable
- Measurable
Main Artefacts
7
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
» Any software artifact can be used for testing
purposes
Main Type of Testing
» Unit, Integration, and System Testing
» White Box and Black box Testing
» Stress Testing
8
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
» Performance Testing
» …
White Box Testing
9
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
White Box Testing
Black Box vs. White Box Testing
SELECTED INPUTS
RESULTANT OUTPUTS
DESIRED OUTPUT
10
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
SELECTED
INPUTS
RESULTANT OUTPUTS
INTERNAL BEHAVIOR
DESIRED OUTPUT
SOFTWARE DESIGN
“BLACK BOX” TESTING
“WHITE BOX” TESTING
Main principles
» The code is available
11
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
» Objective: Cover the software structure
- How?
12
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
Example: CFG
» Build a graph representation of the source code
» Cover the graph
13
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
Coverage Criteria
» Based on Control-flow and Data-flow information
» Among the most famous:
- Statement coverage
- Branch coverage
14
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
- Branch coverage
- Path coverage
Category Partition
» In the case of a branch, how to select the inputs?
15
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
JUnit
» Unit Testing di Java classes
» White box Testing
16
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
» Intuizione:
- Data una classe, l’idea consiste nel:
> Creare una instanza di tale classe, con valori specificati (classe di
test)
> Controllare che tale classe di test si comporti come specificato
JUnit
17
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
Black Box Testing
18
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
Black Box Testing
Black Box vs. White Box Testing
SELECTED INPUTS
RESULTANT OUTPUTS
DESIRED OUTPUT
19
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
SELECTED
INPUTS
RESULTANT OUTPUTS
INTERNAL BEHAVIOR
DESIRED OUTPUT
SOFTWARE DESIGN
“BLACK BOX” TESTING
“WHITE BOX” TESTING
Main Principles
» The code is NOT available
- How to select inputs for test cases?
20
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
» Inputs can be selected based on specifications
» Model-based Testing
Modeling: there is not just programming
» A model is a simplification of the reality
- The reality is abstracted, since it is too complex
- Everything is a model ?!?
» Examples
21
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
» Examples
Examples
22
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
Examples
23
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
Examples
24
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
Why to make models? Document and Analyze
25
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
What is MBT?
» It consists in:
- Extracting Test Cases from the Model
- Run the test case over the system implementation
26
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
» Purpose:
- To validate the implementation conformance to the model
> The model itself is the oracle, i.e., it represents the expected
behavior
Test Selection
» A test case, in MBT, is typically a possible “scenario”
- In scenario-based testing:
> A scenario itself is an abstract test case
27
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
- In state-based testing
> Execution flows are extraced from the state machine and represent a
test case
> MB coverage criteria are applied on this model
In practice
28
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
Test Execution
» It consists in running the test cases on the system
implementation
- So to compare the real execution with the expected
behavior
29
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
MBT practical concerns
» Traceability, i.e. relating the abstract values of the specification to the concrete values of the implementation.
30
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
» Execution, i.e. forcing the Implementation Under Test (IUT) to execute the specific sequence of events that has been selected.
Model-based Testing Approaches
TESTO
UMLAU
TOTEM
SOOT
COW-SU
SCENTO
SeDiTe
AGEDI
UMLTe
UCSC-Sy
AsmL
31
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
OR
UT
MTF
UITE
OR
eC IS
est
ystem
L
MBT Tools
» There are some automated tools for MBT:
- Telelogic Tau and Tau Tester
- Leirios Test Designer
- Conformiq Qtronic
32
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
- Conformiq Qtronic
- Rhapsody Test Conductor and Automatic Test Generation
- AGEDIS
MBT Tools
33
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
MBT Tools
34
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
Remarks
» Learn English
» GSEEM
35
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
» GSEEM
- http://informatica.di.univaq.it/gseem
» EURECA Project
- http://www.mrtc.mdh.se/eureca
References
» H. Muccini. “Software Testing: Testing new Software Paradigms and new Artefacts”. In the Wiley Encyclopedia of Computer Science & Engineering –2008
» A. Bertolino. “Software Testing Research: Achievements, Challenges, Dreams”, ICSE 2007
36
SEA Group
© 2009 by H. Muccini
SEA GroupSEA Group
Achievements, Challenges, Dreams”, ICSE 2007 Future of Software Engineering Track
» J. Gao, H. Muccini, and X. Bai, “Software Test Automation from Components to Systems”, John Wiley & Sons, to appear in 2009