Post on 15-Nov-2014
description
transcript
|
Real-life test; Real-life situations
A practical example of the Statistical Usage TestDutch Testing Conference 2014André Verschelling; Sr. test Consultant High Tech
1Real-life test; Real-life situations
|
Customer satisfaction
Number one need: Involvement Customers want to feel like
you care about their needs,and that you’re emotionallyinvested in helping themsolve their problems(Source: http://www.businesspundit.com/top-10-things-that-all-customers-want/)
2Real-life test; Real-life situations
|
Therefore, we test
Because testing … … is a process that provides insight into, and
advice on, quality and the related risks
And quality …… is the totality of features and characteristics
of a product or service that bear on its ability to satisfy stated or implied needs
Thus, …… the customer gets what he really needs, right?
3Real-life test; Real-life situations
|
What is happening
4Real-life test; Real-life situations
Focus shift
|
Does that satisfy the customer?
“On system level we do exploratory testing”
Does that really address the customer needs …
… and does it solves his/her problems?
5Real-life test; Real-life situations
|
What about test design techniques?
6Real-life test; Real-life situations
Test Design Technique
Useful for Test Design Technique
Useful for
Decision Table Verification Evolutionary Algorithms
Verification
Classification Tree Method
Verification State Transition Test
Verification
Elementary Comparison Test
Verification Real-life test: Statistical usage Test
Verification & Validation
Error Guessing Verification Rare Event Test Verification
Exploratory Testing
Verification & Validation
Semantic Test Verification
Data Cycle Test Verification Syntactic Test Verification
Process Cycle test Verification Use Case Test Verification & Validation
|
Exploratory Testing Only explores what the testers has interest in Tester in center stage, instead of user Actually verifying, not validating
Use Case Testing Fixed and pre-defined workflows to represent
user behavior Based upon defined actors/users
Real-life test: Statistical Usage Test Statistically correct user behavior randomly
applied Based upon operational usage of all users
7Real-life test; Real-life situations
Validation Techniques
|
Statistical Usage Test
A truly real-life test Operational Usage is captured in Operational
Profiles Which operation is most likely to be performed
under which circumstances?• Events • ‘States’ (history classes)• Probability of state/event combination
8Real-life test; Real-life situations
Event 1 Event 2 … Event m
History Class 1
P1,1 P1,2 .. P1,m
History Class 2
P2,1 P2,2 .. P2,m
… .. .. .. ..
History Class n
Pn,1 Pn,2 .. Pn,m
| 9Real-life test; Real-life situations
Statistical Usage Test
Example of an operational profile (PVR/VCR)
Events ►History classes ▼
evStop evRewind(X tics)
evPlay(X tics)
evFastForward(X tics)
evRecord(X tics)
Standby 0 0.2 0.5 0.2 0.1
Rewind 0.3 0 0.6 0.1 0
Play(counter < 15000)
0.5 0.3 0 0.2 0
Fast forward 0.5 0.1 0.4 0 0
Record 1 0 0 0 0
Play(counter ≥ 15000)
0.8 0.15 0 0.05 0
| 10Real-life test; Real-life situations
Gather the Operational Profile (1)
Formal analysis / specification Differentiate customers Differentiate users Differentiate system modes Differentiate functions Specify operational profile
for each [customer, user, system mode, function]
Learn by example Watch and record Interviews
| 11Real-life test; Real-life situations
Gather the Operational Profile (2)
Better: Use production data Log files Transaction data Usage data
Most systems are evolutions instead of revolutions Evaluate predecessors Evaluate comparable systems
| 12Real-life test; Real-life situations
Statistical Usage Test
Test case starts with initial state (i.e. History Class
1) Randomly choose next event Example Test Case for the given Operational
Profile(= sequence of transitions):
Test case must continue until execution profile corresponds with Operational Profile
Standby(current count = 15000) | evRewind(10000 Tics) | Rewind(5000) | evPlay(6000) | Play(11000) | evStop | Standby(11000) | evPlay(5000) | Play(16000) | evRewind (8000) | Rewind(8000) | evPlay(3000) | Play(11000) | etc., etc., etc.
|
Everything your users do is done prove it
No fixed sequence, butrandom selection at runtime statistically correct
Thus, caring about the users’needs
But what if the real user finds an anomaly?
13Real-life test; Real-life situations
Run the test
|
OK, there’s more to it
Given a system (mobile X-ray system) with: Intermittent failures Unclear problem descriptions like
• “Sometimes the connection between stand and Mobile Viewing Station is lost”
• “Sometimes images are lost” Problems could not be reproduced by functional
tests Root cause of system failure not yet found Log file contains not enough information on
problem
14Real-life test; Real-life situations
|
Gather the Operational Profile
Log file contains information on usage
Define most important parameters: Switch on / switch
off system Start / stop low dose
fluoroscopy Start / stop high
definition fluoroscopy
Store images
Log file excerpt10-02-10 14:05:34 S INIT > > > > INIT_Main:
Starting application < < < < 10-02-10 14:05:34 S INIT Build by nlybstqvp4-
sys24 on NLYBSTQVP4DT037, 10-02-10 14:05:34 M IPDR DFI Unit Serial
Number: [00-00-10-6B-22-03-01]10-02-10 14:05:34 M IPDR DFI Unit Versions: IN [0x1] PROC [0x1]
CTRL [0x5] DISP [0x1]10-02-10 14:05:34 S IPDR Starting DFI selftest
10-02-10 14:38:55 M SYS USR switch on RIGHT10-02-10 14:38:55 M SYS Xray mode continuous
fluoro10-02-10 14:38:55 M USR START FLUORO RUN RIGHT 10-02-10 14:38:55 M SYS USR switch off at
187683 10-02-10 14:38:56 M USR STOP FLUORO RUN 10-02-10 14:38:56 M FILE Stored [1] new
image(s) on disk10-02-10 14:38:56 M FILE Deleted [0] image(s)
from disk
15Real-life test; Real-life situations
|
Gather the Operational Profile
Data from log file
16Real-life test; Real-life situations
Case#
Start date
dd-mm-yy
Start Time
hh:mm:ss
End Date
dd-mm-yy
End Timehh:mm:s
s
Run Timehh:mm:s
s
Left Fluo#
Right Fluo#
Images Stored
#
Total Fluo time (sec)
Avg Fluo time (sec)
1 10-02-10 14:05:34 10-02-10 14:08:16 0:02:42 0 0 0 0 0
2 10-02-10 14:08:16 10-02-10 14:57:08 0:48:52 2 24 24 15 0.576923077
3 12-02-10 7:12:11 12-02-10 10:27:44 3:15:33 9 74 75 63 0.759036145
38 9-03-10 7:15:20 09-03-10 13:57:47 6:42:27 60 1 18 58 0.950819672
39 10-03-10 7:37:01 10-03-10 7:37:37 0:00:36 0 0 0
Totals 61:27:51 565 480 521 1065
Average 1:34:34 14 12 13 28
|
Gather the Operational Profile
Interpreting the data determine history classes
Basic profile
A ‘case’ is from system start to system stop Sometimes no fluoroscopy is done at all Most of the times a case consist of fluoroscopy
(LDF, HDF or both)17Real-life test; Real-life situations
History class Start system
Stop system
Make fluoroscopy run(s)
System switched off 1 0 0
System started 0 0.08 0.92
Fluoro Runs completed 0 1 0
|
Gather the Operational Profile
Add more details for fluoroscopy history classes
18Real-life test; Real-life situations
History class Start system
Stop system
Make fluoroscopy run(s)
System switched off 1 0 0
System started 0 0.08 0.92
Fluoro Runs completed 0 1 01 – 7 * LDF 8 – 16 * LDF 17 – 45 *
LDF46 – 270 * LDF
Standby 45% 29% 24% 2%
+1 – 7 * HDF 8 – 16 * HDF 17 – 78 *
HDF
Standby 40% 40% 20%
|
Executing the test: Automate
Switch on the systemDecide on the number of fluoroscopy
runs @nr_fluoL = 0 @nr_fluoR = 0 # start throwing the dice for the
left fluoro runs @leftSel = rand() case @leftSel when 0 .. 0.41 @nr_fluoL = 0 when 0.42 .. 0.67 @nr_fluoL = 1 + rand(7) when 0.68 .. 0.84 @nr_fluoL = 8 + rand(9) when 0.85 .. 0.99 @nr_fluoL = 17 + rand(28) else @nr_fluoL = 46 + rand(225) end
# then for the right fluoro runs (= images being stored)
@rightSel = rand() case @rightSel when 0 .. 0.20 @nr_fluoR = 0 when 0.21 .. 0.59 @nr_fluoR = 1 + rand(7) when 0.60 .. 0.78 @nr_fluoR = 8 + rand(9) else @nr_fluoR = 17 + rand(62) end
Perform the selected number of runsSwitch off the system
And start all over again (endless loop)
19Real-life test; Real-life situations
|
Statistical Usage Test results
System starts showing intermittent failures Finally, test results in a Fatal Error (hard disk
broken)
13 days of test versus 1 month of real-life usage Images stored: 80719 versus 521 Total Fluoroscopy time: 135995 sec versus 1065
sec20Real-life test; Real-life situations
Run Timehh:mm:ss
# Left Fluo
# Right Fluo
# Images Stored
Total Fluo time (sec)
Customer system 61:27:51 565 480 521 1065
Test system 198:19:31 71873 89548 80719 135995
|
Statistical Usage Test results
Problem reproduced Test results have proven the data analysis
Possible improvements on the Operational Profile and/or the test Combine more user data (e.g. more log files, more
customers) Define actual contribution on available data (no
assumption of normal distribution)
Automation is mandatory, e.g. using FitNesse
21Real-life test; Real-life situations
|
Conclusion
Real-life testing is doing what your customer wants to do Paying attention to thecustomer needs
Real-life testing helps findingissues Help solving thecustomers’ problems
22Real-life test; Real-life situations
Is your customer satisfied?
Unless you have 100% customer satisfaction…you must improve.Horst Schulze