Black Box Testing
Adam Hendra Brata
2
Agenda
Testing Strategies
Black Box Testing– Introduction
– Techniques Equivalence Partitioning
Boundary Value Analysis
3
Testing Strategies
4
Testing Strategies
White Box
– White Box Testing is a
software testing method in
which the internal structure/
design/ implementation of
the item being tested is
known to the tester.
Black Box
– Black Box Testing is a
software testing method in
which the internal structure/
design/ implementation of
the item being tested is
NOT known to the tester
5
Test-to-pass and Test-to-fail
Test-to-pass
– assures that the software minimally works
– does not push the capabilities of the software
– applies simple and straightforward test cases
– does not try to “break” the program
Test-to-fail
– designing and running test cases with the sole
purpose of breaking the software
– strategically chosen test cases to probe for common
weaknesses in the software
6
Black Box TestingIntroduction
7
Black Box Testing
Testing software against a specification of its external
behavior without knowledge of internal
implementation (source code) details
– Can be applied to software “units” (e.g., classes) or to entire
programs
– External behavior is defined in API docs, Functional specs,
Requirements specs, etc.
Because black box testing purposely disregards the
program's control structure, attention is focused primarily
on the information domain (i.e., data that goes in, data
that comes out)
The Goal: Derive sets of input conditions (test cases) that
fully exercise the external functionality
8
Black Box Testing
interface
Black Box
(Component or System)outputinput
operation
9
Black Box Testing
Black box testing also known as specification-
based testing
Black box testing refer to test activities using
specification-based testing methods and criteria to
discover program errors based on program
requirements and product specifications
The major testing focuses:
– specification-based function errors
– specification-based component/system behavior errors
– specification-based performance errors
– user-oriented usage errors
– black box interface errors
10
Black Box Testing
Black box testing attempts to find errors in the
external behavior of the code in the following
categories :
1. incorrect or missing functionality
2. interface errors
3. errors in data structures used by interfaces
4. behavior or performance errors
5. initialization and termination errors
Through this testing, we can determine if the
functions appear to work according to specifications
11
Black Box Testing Problems
So exactly, what’s the problem ?
12
Black Box Testing Problems
Domain is all possible inputs to the function
Range is all possible outputs of the function
“there are so many combination of inputs and various outputs,
so what should I do ?”
13
Black Box Testing Problems
“..and there are millions of possible test cases to run, what do I do ?”
14
Black Box Testing TechniquesEquivalence Partitioning
15
Equivalence Partitioning
Typically the universe of all possible test cases is so
large that you cannot try them all
You have to select a relatively small number of test
cases to actually run
Which test cases should you choose?
Equivalence partitioning helps answer this question
Goal: Reduce number of test cases by equivalence
partitioning:
– Divide input conditions into equivalence classes
– Choose test cases for each equivalence class. (Example: If
an object is supposed to accept a negative number, testing
one negative number is enough)
16
Equivalence Partitioning
17
Equivalence Partitioning
Partition the test cases into "equivalence classes"
Each equivalence class contains a set of
"equivalent" test cases
Two test cases are considered to be equivalent if we
expect the program to process them both in the
same way (i.e., follow the same path through the
code)
If you expect the program to process two test cases
in the same way, only test one of them, thus
reducing the number of test cases you have to run
18
Equivalence Partitioning
Steps in equivalence testing:
– identify classes of inputs with same behavior
– test on at least one member of each equivalence class
– assume behavior will be same for all members of
class
Criteria for selecting equivalence classes:
– coverage : every input is in one class
– disjointedness : no input in more than one class
– representation : if error with 1 member of class, will
occur with all
19
Equivalence Partitioning
First-level partitioning : Valid vs. Invalid test cases
Valid Invalid
20
Equivalence Partitioning
Partition valid and invalid test cases into equivalence
classes
21
Equivalence Partitioning
Create a test case for at least one value from each
equivalence class
22
Equivalence Partitioning
When designing test cases, you may use different
definitions of “equivalence”, each of which will
partition the test case space differently
Example: int Add(n1, n2, n3, …)
– Equivalence Definition 1: partition test cases by the number
of inputs (1, 2, 3, etc.)
– Equivalence Definition 2: partition test cases by the number
signs they contain (positive, negative, both)
– Equivalence Definition 3: partition test cases by the
magnitude of operands (large numbers, small numbers,
both)
– Etc.
23
Equivalence Partitioning
When designing test cases, you may use different
definitions of “equivalence”, each of which will
partition the test case space differently
Example: string Fetch(URL)
– Equivalence Definition 1: partition test cases by URL
protocol (“http”, “https”, “ftp”, “file”, etc.)
– Equivalence Definition 2: partition test cases by type of file
being retrieved (HTML, GIF, JPEG, Plain Text, etc.)
– Equivalence Definition 3: partition test cases by length of
URL (very short, short, medium, long, very long, etc.)
– Etc.
24
Equivalence Partitioning : Oracle
If an oracle is available, the test values in each
equivalence class can be randomly generated.
This is more useful than always testing the same
static values.
– Oracle: something that can tell you whether a test passed or
failed
– An oracle is a mechanism for determining whether the
program has passed or failed a test.
Test multiple values in each equivalence class.
Often you’re not sure if you have defined the
equivalence classes correctly or completely, and
testing multiple values in each class is more
thorough than relying on a single value.
25
Equivalence Partitioning Example
26
Equivalence Partitioning Example
Let's consider the behavior of tickets in the Flight
reservation application, while booking a new flight.
Ticket values 1 to 10 are considered valid & ticket is
booked, while value 11 to 99 are considered invalid
for reservation
Here is the test condition
– Any Number greater than 10 entered in the reservation
column (let say 11) is considered invalid.
– Any Number less than 1 that is 0 or below, then it is
considered invalid.
– Numbers 1 to 10 are considered valid
– Any 3 Digit Number say -100 is invalid.
27
Equivalence Partitioning Example
We cannot test all the possible values because if
done, the number of test cases will be more than
100.
– To address this problem, we use equivalence
partitioning hypothesis where we divide the possible
values of tickets into groups or sets as shown below
where the system behavior can be considered the
same
28
Equivalence Partitioning Example
The divided sets are called Equivalence Partitions or
Equivalence Classes. Then we pick only one value
from each partition for testing.
– The hypothesis behind this technique is that if one
condition/value in a partition passes all others will also
pass. Likewise, if one condition in a partition fails, all
other conditions in that partition will fail.
29
Equivalence Partitioning Example
30
Equivalence Partitioning Example
Input Valid Equivalence Classes Invalid Equivalence Classes
A integer N such that:
-99 <= N <= 99 ? ?Phone Number
Area code: [200, 999]
Prefix: (200, 999]
Suffix: Any 4 digits
? ?
Another cases…..
31
Equivalence Partitioning Example
Input Valid Equivalence Classes Invalid Equivalence Classes
A integer N such that:
-99 <= N <= 99
[-99, -10]
[-9, -1]
0
[1, 9]
[10, 99]
< -99
> 99
Malformed numbers
{12-, 1-2-3, …}
Non-numeric strings
{junk, 1E2, $13}
Empty value
Phone Number
Area code: [200, 999]
Prefix: (200, 999]
Suffix: Any 4 digits
? ?
32
Equivalence Partitioning Example
Input Valid Equivalence Classes Invalid Equivalence Classes
A integer N such that:
-99 <= N <= 99
[-99, -10]
[-9, -1]
0
[1, 9]
[10, 99]
< -99
> 99
Malformed numbers
{12-, 1-2-3, …}
Non-numeric strings
{junk, 1E2, $13}
Empty value
Phone Number
Area code: [200, 999]
Prefix: (200, 999]
Suffix: Any 4 digits
555-5555
(555)555-5555
555-555-5555
200 <= Area code <= 999
200 < Prefix <= 999
Invalid format 5555555,
(555)(555)5555, etc.
Area code < 200 or > 999
Area code with non-numeric
characters
Similar for Prefix and Suffix
33
Black Box Testing TechniquesBoundary Value Analysis
34
Boundary Value Analysis
Boundary value analysis
– Testing conditions on bounds between classes of inputs
Why is it useful to test near boundaries?
– likely source of programmer errors (< vs. <=, etc.)
– language has many ways to implement boundary checking
– requirement specs may be fuzzy about behavior on
boundaries
– often uncovers internal hidden limits in code
– example: array list must resize its internal array when it fills
capacity
35
Boundary Value Analysis
When choosing values from an equivalence class to test,
use the values that are most likely to cause the program
to fail
Errors tend to occur at the boundaries of equivalence
classes rather than at the "center"
– If (200 < areaCode && areaCode < 999) {//valid area code
– If (200 <= areaCode && areaCode <= 999) {//valid area code}
– Testing area codes 200 and 999 would catch this error, but a
center value like 770 would not
In addition to testing center values, we should also test
boundary values
– Right on a boundary
– Very close to a boundary on either side
36
Boundary Value Analysis
Testing boundary conditions of eq. classes is more
effective i.e. values directly on, above, and beneath
edges of eq. classes
Choose input boundary values as tests in input eq.
classes instead of, or additional to arbitrary values
Choose also inputs that invoke output boundary values
(values on the boundary of output classes)
Example strategy as extension of equivalence
partitioning:
– choose one (n) arbitrary value in each eq. class
– choose values exactly on lower and upper boundaries of eq.
class
– choose values immediately below and above each
boundary (if applicable)
37
Boundary Value Analysis
Create test cases to test boundaries of equivalence
classes
38
Boundary Value Analysis Example
39
Boundary Value Analysis Example
In our earlier example instead of checking, one value
for each partition you will check the values at the
partitions like 0, 1, 10, 11 and so on. As you may
observe, you test values at both valid and invalid
boundaries.
– Boundary Value Analysis is also called range checking
40
Boundary Value Analysis Example
Input Boundary Cases
A number N such that:
-99 <= N <= 99 ?Phone Number
Area code: [200, 999]
Prefix: (200, 999]
Suffix: Any 4 digits
?
Another cases…..
41
Boundary Value Analysis Example
Input Boundary Cases
A number N such that:
-99 <= N <= 99
-100, -99, -98
-10, -9
-1, 0, 1
9, 10
98, 99, 100
Phone Number
Area code: [200, 999]
Prefix: (200, 999]
Suffix: Any 4 digits
Area code: 199, 200, 201
Area code: 998, 999, 1000
Prefix: 200, 199, 198
Prefix: 998, 999, 1000
Suffix: 3 digits, 5 digits
42
Boundary Value Analysis Example
Imagine we are testing a Date class with a
daysInMonth(month, year)method
– What are some conditions and boundary tests for this
method?
Possible answers
– check for leap years (every 4th yr, no 100s, yes 400s)
– try years such as: even 100s, 101s, 4s, 5s
– try months such as: June, July, Feb, invalid values
43
Boundary Value Analysis Example
Another Example of Equivalence Partitioning and
Boundary Value Analysis– http://www.guru99.com/equivalence-partitioning-boundary-value-
analysis.html
44
Thank You