Test adequacy evaluation
for the user-database
interaction: a specification-
based approach
University of Oviedo
Department of Computing
Software Engineering Research Group
http://giis.uniovi.es/
ICST 2012 – Montreal, April 17-21
Raquel Blanco, Javier Tuya, Rubén V. Seco
{rblanco, tuya, valdesruben}@uniovi.es
Introduction
Software Engineering Research Group (GIIS) 2 Raquel Blanco
Test
Input
User
Input
Test
Database
Database Application Testing
Problem
Bu
sin
es
s L
og
ic
Test
Output
User
Output
Output
Database
Clo
sely
Rela
ted
Database
Warehouse: District:
Customer: Name: Credit:
Order Number: Number of Lines:
Supplying
WarehouseItem Id Item Name Quantity
Execution Status:
Discount:
Warehouse Tax: District Tax:
Date and Time:
Brand-
generic
Item
PriceAmount
Total Amount:
New Order
Complex
Structure
Introduction
3 Raquel Blanco
Test Input
User Input
Test Database
Quality
Goal
Evaluate the test input adequacy
using a test adequacy criterion
Derive situations of interest to be tested
(test requirements)
Software Engineering Research Group (GIIS)
White-Box
Black-Box ?
Problem approach
4 Raquel Blanco
Contributions:
1) Definition of a unique model that represents user input + test
database
2) Modelling the functionality through a set of business rules
3) Elaboration of a MCDC-based criterion over the business rules
to automatically derive the test requirements
4) Automatic evaluation of the test input adequacy (user input +
test database)
Software Engineering Research Group (GIIS)
Specification-based approach
Testi
ng
Pro
cess
M
od
ellin
g
Pro
cess
Raquel Blanco Software Engineering Research Group (GIIS)
System
Specification
Functionality
Problem approach
5
1 2
3
4
5
User
Interface Database
Creation of the
Integrated Data Model (IDM)
Specification of the
Business Rules
Business
Rules IDM
Test
Requirements
All test
requirements
covered?
No
Test Inputs
Yes
Job done
Derivation of the Test
Requirements
Evaluation of the Test
Inputs Adequacy
Generation of new
Test Inputs
6
Integrated Data Model (IDM)
6
Test Assignment
UI_Order
UI_OrderLine
1
*
UI Level
Warehouse
District
Customer
Order
Order_Line
Stock
Item
New_Order
History
1
0..1
1
*
1
*
1
*
1*
1
*
1
1
1*
+
*
Database Level
TestCaseTest Case Level Example: “New-Order Transaction” of
the TPC-C benchmark (test assignment)
Raquel Blanco Software Engineering Research Group (GIIS)
Warehouse
District
Customer
Order
Order_Line
Stock
Item
New_Order
History
1
0..1
1
*
1
*
1
*
1*
1
*
1
1
1*
+
*
Da
tab
ase
Sch
em
a
1
*
1
*
*
1
Business Rules
Type of business rules:
7 Raquel Blanco
Context: PathContext:
Frame
Business Rule
Value
Conditions
Quantification
Conditions
Constraint
Rule
Derivation
Rule
(1) (2) (3)
Value
Conditions
Quantification
Conditions
Derivation
Rule
Value
Conditions
(4) (5)
(3) Derivation rule for conditions in some tuple of a path
(4) Derivation rule for the number of tuples in a path
(5) Derivation rule for conditions in all tuples of a frame
(1) Constraint rule for values
(2) Constraint rule for the number of tuples in a path
Software Engineering Research Group (GIIS)
Warehouse
District
Customer
Order
Order_Line
Stock
Item
New_Order
History
1
0..1
1
*
1
*
1
*
1*
1
*
1
1
1*
+
*
1
1
1
Test Assignment
TestCase
UI_Order
UI_OrderLine
1
*
*
1
*
1
*
Business Rules
Path: the context to evaluate
the conditions of a business rule
8 Raquel Blanco
UI Level
Database Level
Test Case Level
Software Engineering Research Group (GIIS)
Example: the Brand-generic field of
an item is ‘B’ if the brand information
stored for that item in both attributes
Stock.s_data and Item.i_data
includes the string “ORIGINAL”
Logical Path
Test Case Level
tc_id
1
TestCase
(1)
ol_ui_id ol_numbertc_id ….ol_supply_w_id ol_i_id ol_brand ….1 11 ….1 1
1 21 ….1 2
UI_OrderLine
1 31 ….1 3
(1)
(2)
(3)
….….….
w_id ….1
Warehouse
….(1)
d_w_id ….1 1 ….
District
d_id
(1)
c_w_id ….1 1 ….
Customer
c_d_id c_id
1(1)
o_ui_id o_w_idtc_id ….1 11 1 ….
UI_Order
o_d_id o_c_id
1(1)
UI Level
Database Level
s_i_id s_datas_w_id ….1 original1
2 generic1
Stock
3 original1
….….….
(1)
(2)
(3)
….i_datai_id
original1
original2
Item
3
(1)
(2)
(3)
….….….
Test Requirements Masking MCDC-based criterion over each business rule
9 Raquel Blanco
pred1 pred2
Path P1 is UI_OrderLine [ ] Stock [ ] Item
If P1.i_data is like ‘%ORIGINAL%’
and P1.s_data is like %ORIGINAL%’
then P1.ol_brand = ‘B’
cond1
cond2
Test Requirements
ID pred1 pred2 cond1 cond2
1 True True True True
2 True True True False
3 True True i_data is null True
4 True True False True
5 True True True s_data is null
6 False True True True
7 True False True True
Software Engineering Research Group (GIIS)
Implementation
DruidaTest http://in2test.lsi.uniovi.es/sqltools/druidatest/
IDM Relational Database
Test requirements SQL queries
10 Raquel Blanco Software Engineering Research Group (GIIS)
Business
Rules
DB
schema
UI
schema
Test Inputs
Adequacy
IDM Final list of
Coverage
Rules
SQL query &
IDM schema
Coverage
Rules
SQLFpcWSUIBDRules
Execution of Coverage
Rules against IDM
DruidaTest
http://in2test.lsi.
uniovi.es/sqlfpc/
User interface of DruidaTest
Results
Case study: TPC-C benchmark
Implementation: BenchmarkSQL
11
Test
Assignment
Bu
sin
ess R
ule
s
Co
vera
ge R
ule
s
Tu
ple
s ID
M
Test C
ases
Failu
res
No
n D
up
licate
d
Failu
res
New-Order 7 38 113 10 10 5
Payment 6 24 23 9 7 5
Order-Status 5 19 30 7 0 0
Delivery 4 20 27 7 4 2
Stock-Level 4 24 129 8 3 1
Total: 26 125 322 41 24 13
5 caused by input validation
errors
3 caused by null values
3 caused by an incorrect update
of the user interface
2 caused by the incorrect
processing of the test inputs
Raquel Blanco
New-Order:
86 tuples
Payment:
9 tuples
Order-Status:
7 tuples
Delivery:
7 tuples
Stock-Level:
8 tuples
TestCase:
41 tuples
Database Level: 164 tuples
UI Level: 117 tuples
Test Case Level: 41 tuples
Warehouse
District
Customer
Order
Order_Line
Stock
Item
New_Order
History
1
0..1
1
*
1
*
1
*
1*
1
*
1
1
1*
+
*
Software Engineering Research Group (GIIS)
Conclusions and future work
Conclusions:
Specification-based Testing (Black-Box Testing)
System specification = IDM + business rules
Results: test cases obtained are able to detect interesting faults
Future work:
Automatic generation of test inputs
A comprehensive experimentation
Improve the expressiveness of the business rules and model
user’s operations with sequence states
Use the output behaviour of the business rules as a test oracle
12 Raquel Blanco Software Engineering Research Group (GIIS)
Thank you for your attention
Questions?
13 Raquel Blanco Software Engineering Research Group (GIIS)
