TODAY’S AGENDABLACK BOX TESTING
What is Black Box Testing
Differences between Black Box Testing and White Box Testing
Different techniques of Black Box Testing
2
WHAT IS BLACK BOX TESTING
Black Box Testing is done without the knowledge of the internals of the system under test
3
LOCK AND KEY EXAMPLE OF BLACK BOX TESTING
4
CHARACTERISTICS OF BLACK BOX TESTINGDone based on requirements
Addresses (Should address) stated as well as implied requirements
Encompasses the end user perspective
Checks for valid and invalid conditions / inputs
May or may not know the technology aspects of the product
5
DIFFERENCES BETWEEN WHITE BOX AND BLACK BOX TESTING
Black Box Testing White Box Testing
No access to program code Has access to program code
Requires external perspective
Requires knowledge of program code
Set of techniques applicable to all other phases of testing
Typically applies only to unit testing, where code is involved
6
PRINCIPLES OF BLACK BOX TESTING
It is not possible to exhaustively test a product
Choose tests so that We can test “as much of the external functionality as possible” Uncover “as many defects as possible” Use “as short time as possible”
7
Have methodologies that choose tests that have a higher likelihood of uncovering new defects
TECHNIQUES / METHODOLOGIES OF BLACK BOX TESTINGRequirements Based TestingPositive and Negative TestingBoundary Value AnalysisDecision TablesEquivalence PartitioningState Based TestingCompatibility TestingUser Documentation TestingDomain Testing (leads to ad hoc testing)
8
GENERAL FORMAT OF DISCUSSION OF TECHNIQUESPresent some reasoning where applicable
List out one or two examples
Walk through the examples
Summarize the process for using the technique
9
REQUIREMENTS BASED TESTING
Done to ensure all requirements in SRS are tested
Difference between implicit and explicit requirements
Review requirements first to ensure they are consistent, correct, complete and testable
Review enables translation of (some of) implied requirements to stated requirements
A reviewed SRS tabulates requirements, along with a requirements id and a priority
This is the genesis of a Requirements Traceability Matrix
10
REQUIREMENTS TRACEABILITY MATRIX –GENERAL FORMAT<REFER TEMPLATE>
11
ZOOMING IN ON TESTING PART OF RTM
Fig 4.5
Test Condition: Different ways of testing a requirement (different types of mapping)
Test Case: Different conditions / scenarios for a given requirement
Phase of testing – helps scheduling
Importance of documenting expected results
12
POSITIVE AND NEGATIVE TESTING
Positive testing to check that the product does what it is supposed to
Behaves correctly when given right inputs Maps to a specific requirement “Coverage” is defined better
Negative Testing to show that the product does not fail when given unexpected inputs
Try to break the system No direct mapping to a specific requirement “Coverage” more challenging
13
BOUNDARY VALUE ANALYSIS
Most defects come up near “boundaries”
Reasons from a White Box perspective Programmers tentativeness in using the right relational
operator (< or <= ?) Multiple ways of implementing loops Confusing array subscripts
Reasons from a Black Box perspective Unclear requirements Ambiguous or “it depends” mindset!
14
BOUNDARY CONDITIONS – EXAMPLE 1:DATABASE BUFFERSDatabase starts with a pre-allocating number of buffers for
caching
Buffers filled up as needed
If full, buffers freed on a FIFO basis
15
16
…
Case i: All buffers free for use
…
Case ii: After inserting two buffers and still having free buffers
…
Case iii: After inserting the last available buffer: No free buffers
…
Case iv: No free buffers and new buffer coming in: First buffer needs freeing
Cases MostLikely to Find defects
EXAMPLE 2:VOLUME DISCOUNT CALCULATIONS
17
Product pricing depends on the quantity procured
Based on pre-defined “slabs”
BOUNDARY VALUE ANALYSIS (BVA):SUMMARY OF PROCESSLook for any kind of gradation or discontinuity in data
values which affect computation – the discontinuities are the boundary values, which require thorough testing.
Look for any internal limits like limits on resources (like the example of buffers given above). The behavior of the product at these limits should also be the subject of boundary value testing.
Also include in the list of boundary values, documented limits on hardware resources. For example, if it is documented that a product will run with minimum 4MB of RAM, make sure you include test cases for the minimum RAM (i.e., 4MB in this case).
The examples given above discuss boundary conditions for input data – the same analysis needs to be done for output variables also.
18
DECISION TABLES
A program’s behavior characterized by several decision variables
Each decision variable specifies a boolean condition
The distinct combinations of these decision variables leads to different scenarios
Each scenario occupies a row in the decision table and the row also has expected results
One representative data point from each scenario needs to be tested
19
EXAMPLE OF DECISION TABLE
20
SUMMARY OF DECISION TABLE
Identify the decision variables.Identify the possible values of each of the decision variables.Enumerate the combinations of the allowed values of each of
the variables.Identify the cases when values assumed by a variable (or by
sets of variables) are immaterial for a given combination of other input variables. Represent such variables by the Don’t Care symbol.
For each combination of values of decision variables (appropriately minimized with the Don’t Care scenarios), list out the action or expected result.
Form a table, listing on each but the last column a decision variable each. On the last column, list the action item for the combination of variables in that row (including Don’t Cares, as appropriate)
21
EQUIVALENCE PARTITIONING (EVP)
Generalization of BVA / Decision Table
Divide the (potentially infinite) set of values into a set of equivalence classes or partitions
One element of the class can act as a representative for the entire class
Results of the test for that one element extrapolated to all the other elements
22
EXAMPLE OF EVP
23
SUMMARY OF PROCESS FOR EVP
Choose criteria for doing the equivalence partitioning (range, list of values, etc.)
Identify the valid equivalence classes, based on the above criteria (number of ranges allowed values, etc.)
Select a sample data from that partitionWrite the expected result based on the requirements
givenIdentify special values, if any, and include it in the tableCheck to have expected results for all the cases preparedIn case, expected result is not clear for any particular
test case, mark appropriately and escalate for corrective actions. If you cannot answer a question, or find an inappropriate answer, consider whether you want to record this issue on your log.
24
STATE OR GRAPH BASED TESTING
Useful for Language Processors or compilers Workflow modeling Dataflow modeling
25
26
1SELECT
2*
3
<col n
ame>
4,
<col name>5
FROM
FROM
6<table name>
7
,
<ta
ble
nam
e>
WHERE
…
Example of Graph Based Testing for a SQL command
PROCESS FOR LANGUAGE PROCESSORS TESTING USING GRAPH METHODIdentify the grammar for the scenario. In the
above example, we have represented the diagram as a State Machine. In some cases, the scenario can be a Context Free Grammar, which may require a more sophisticated representation of a “State Diagram”.
Design test cases corresponding to each valid state-input combination.
Design test cases corresponding to most common invalid combinations of state-input.
27
28
RequestorApprovingAuthority Finance
Vendor
Received Item
AvailableOff the shelf
Buy & Expense(Transaction: Expense Report)
Requisition Approval
Budget Update &
Purchase Order
Delivery
Note
Payment
Item D
elivery
WORKFLOW MODELING
EmployeeDesires Leave
VerifyEligibility
Leave ApplicationForm
ManagerEnsure
Feasibility
Reject
Ineligible
Eligible
Feasible
Not Feasible
Approve
29
GENERAL CHARACTERISTICS OF GRAPH BASED TESTINGThe application can be characterized by a set of states
The data values (screens, mouse clicks, etc) that cause the transition from one state to another is well understood
The methods of processing within each state to process the
input received is also well understood
30
COMPATIBILITY TESTING
To ensure that the product works consistently with infrastructure components
Could be parameters of hardware, OS, network…
Concept of a compatibility matrix
Backward compatibility and forward compatibility
31
USER DOCUMENTATION TESTING
To check if what is stated in the document is available in the product
To check if what is there in the product is available in the document
Documents include user guides, installation guides, setup guides, read me file, software release notes and on-line help
32
BENEFITS OF DOCUMENTATION TESTING
User documentation testing aids in highlighting problems that were overlooked during reviews
High quality user documentation ensures the consistency of documentation and product, thus minimizing possible defects reported by customers.
Results in less difficult support calls. Contributes to better customer satisfaction and better
morale of support staff.New programmers and testers who join a project group
can use the documentation to learn the external functionality of the product.
Customers need less training and can proceed more quickly to advanced training and product usage
Reduced training costs for customers
33
DOMAIN TESTING
34
YES
WHITEBOXTESTING
NO
Knowledge of Program CodeAvailable?
Knowledge of Requirements Specifications Available?
NO
YES
BLACKBOXTESTING
DOMAINTESTING
DOMAIN TESTING
Have “domain people” perform tests by using the software
Business flow determines the test, not “logic” or “steps”
Tests what the users do on a typical business day
Captures the typical problems faced by users (not necessarily captured in SRS)
35
WHEN TO USE WHAT…
When you want to test scenarios that have… The most effective black box testing technique is likely to be…
Output values are dictated by certain conditions depending upon values of input variables
Decision Tables
Input values being in ranges, with each range exhibiting a particular functionality
Boundary Value Analysis
Input values being divided into classes (like ranges, list of values, etc), with each class exhibiting a particular functionality
Equivalence Partitioning
Positive and negative testing
Workflows, process flows or language processors Graph based testing
To ensure that requirements are tested and met properly Requirements Based Testing
To test using the domain expertise rather than the product specification
Domain Testing
To ensure that the documentation is consistent with the product
Documentation Testing
36
TODAY’S AGENDA (RECAP)BLACK BOX TESTING
What is Black Box Testing
Differences between Black Box Testing and White Box Testing
Different techniques of Black Box Testing
37