The Pursuit of Quality: Chasing Tornadoes or Just Hot Air?
Gerrard Consulting LimitedPO Box 347MaidenheadBerkshireSL6 2GUTel: +44 (0) 1628 639173Fax: +44 (0) 1628 630398Web: gerrardconsulting.com
Slide 1Intelligent Testing, Improvement and Assurance
Intelligent Testing, Improvement and Assurance
Agenda
• What is Quality?• Models for quality and testing• Examples of models• Models and stakeholders• Failures of systems, failures of
models• Close
Slide 3
Intelligent Testing, Improvement and Assurance
Weather
• Rain is great for farmers and their crops, but terrible for tourists
• Wind is essential for sailors and windmills but bad for the rest of us
• Quality, like weather, can be good or bad and that depends on who you are.
Slide 4
That’s Fantastic
!
That’s Terrible!
Intelligent Testing, Improvement and Assurance
Quality is a relationship
• Quality is not an attribute of a system
• It is a relationship between systems and stakeholders who take different views
• The model of Quality that prevails has more to do with stakeholders than the system itself
Slide 6
The concepts of quality, risk, comfort, intuitiveness …
• Concepts that most people understand, but few can explain
• But it’s a lot worse than that
• Quality is an all-encompassing, collective term for these and many other difficult concepts
• A term that means all things to all people
• (I try and avoid the Q-word).
Intelligent Testing, Improvement and Assurance
Slide 7
Models for Quality and Testing
Intelligent Testing, Improvement and Assurance
Slide 8
Models
Intelligent Testing, Improvement and Assurance
Slide 9
Models are everywhere
Models and reality
• In our minds we build mental models of everything we experience (and also, many things we don’t experience)
• When we pick up a glass of water, we build models– The 3-dimentional location and relationship between
the glass, the water, the table it sits on and our body– As we reach for the glass, our brain processes the
signals from our eyes, our muscles and the feelings in our fingertips
– It continuously compares experience with the model and adjusts/rebuilds the model many times
• … just to lift a cup of water – incredible!
Intelligent Testing, Improvement and Assurance
Slide 10
Some familiar models
• The project plan is a model– The resources, activities, effort, costs, risks and
future decision making
• System requirements are a model– The “what and how” of the system– What: the features and functionality– How: how the system works (fast, secure,
reliable)
• User personas (16 year old gamer, 30 year old security hacker, 50 year old Man United fan).
Intelligent Testing, Improvement and Assurance
Slide 11
Where quality comes from
• Quality is the outcome of a comparison– Our mental model of perfection– Our experience of reality
• Mental models are internal,personal and unique to us
• We could share them using somekind of Vulcan mind meld
• But usually, we can write them down or we can talk about them
• However we communicate, there is noise and information gets corrupted/lost in translation.
Intelligent Testing, Improvement and Assurance
Slide 12
A quality model?
• The requirements and design describe the behaviour of a system
• Functional– Mapping test cases to requirements is all we need
• Non-Functional– All technical attributes are defined and measured
• Quality and therefore testing assumes a model– Often undocumented, the model may not be shared,
understood, complete, consistent, correct…
Intelligent Testing, Improvement and Assurance
Slide 13
Intelligent Testing, Improvement and Assurance
Test design is based on models• Models describe the environment, system, usage,
users, goals, risks• They simplify the context of the test - irrelevant or
negligible details are ignored in the model• Focus attention on a particular aspect of the
behaviour of the system• Generate a set of unique and diverse tests (within
the context of the model)• Enable the testing to be estimated, planned,
monitored and evaluated for its completeness (coverage).
• Models help us to select tests in a systematic way.
Slide 14
Intelligent Testing, Improvement and Assurance
Examples of test models
• A checklist or sets of criteria– Goals, risks, process paths, interfaces, message
type…
• Diagrams from requirements or design documents
• Analyses of narrative text or tables• Some models are documented, many models
are never committed to paper– Can be mental models constructed specifically to
guide the tester whilst they explore the system under test and guide their next action.
Slide 15
Sources of models
• Test Basis– We analyse the text or diagrams or information that describe
required behaviour (or use past experience and knowledge)
• System architecture:– We identify testable items in its user-interface, structure or
internal design
• Modes of failure (product risks):– We identify potential ways in which the system might fail
that are of concern to stakeholders
• Usage patterns:– We focus on the way the system will be used, operated and
interacted with in a business context using personas
• Everything looks fine – doesn’t it?
Intelligent Testing, Improvement and Assurance
Slide 16
Intelligent Testing, Improvement and Assurance
But all models (over-)simplify• But requirements are never perfect, not all
attributes can be meaningfully measured• Models incorporate implicit assumptions and
are approximate representations• All test models are heuristic, useful in some
situations, always incomplete and fallible• Before we adopt a model, we need to know:
– What aspects of the behaviour, design, modes of failure or usage the model helps us to identify
– What assumptions and simplifications it includes (explicitly or implicitly).
Slide 17
Intelligent Testing, Improvement and Assurance
Formality
• Formal test models– Derived from analyses of requirements or code– Quantitative coverage measure can be obtained from a
formal test mode (mostly)
• Informal test models– E.g. some models are just lists of modes of failure, risks
or vulnerabilities.– Informal models cannot be used to define quantitative
coverage measures
• Ad-hoc models– Some models can be ad-hoc, invented by the tester just
before or even during testing– Can be formal or informal.
Slide 18
Examples of Models
Intelligent Testing, Improvement and Assurance
Slide 19
Basic test design techniques are based on the simplest models• Equivalence partitions and boundary
values:– Presume single input, single output
responses– All values in partitions are equivalent, but
the boundaries are the most important
• These techniques are useful, but they date from the ‘green-screen’ era.
Intelligent Testing, Improvement and Assurance
Slide 20
“Green Screen” equivalence model• Single input, single
output• All input is classified
and partitioned with rules
• One test per rule is enough!
• But we don’t consider:– The state of the
system– Combinations of
values.
Slide 21
If m<1 then“Error”
Else if m>12 then “Error”
Else“OK”
Single Input
Single Output
Intelligent Testing, Improvement and Assurance
State Transition Testing
Slide 22
StartState
RoomRequested
RoomBooked
OnWaiting
List
OvernightStay
BookingCancelled
CheckoutRoom availableDecrement room count
Room request None
Customer arrives None
Customer paysIncrement roomcount
No room availableAdd to waiting list
Customer cancelsRemove from waitinglist
Room availableDecrement room count
Customer cancelsIncrement roomcount
Intelligent Testing, Improvement and Assurance
But the number of states is infinite!• State-Transition considers:
– The states of the system and– The valid/invalid transitions between states
• Some systems have many, many states– A real-time system e.g. telecoms switch may
have 25,000 distinct states– State may depend on many variables that
can have infinite values in combination
• How confident can we be in this model?
Intelligent Testing, Improvement and Assurance
Slide 23
End-to-end/transaction-flow tests• End–to-end tests can follow a path
through a process or a user journey• The mechanics of the experience are
simulated but…
Intelligent Testing, Improvement and Assurance
Slide 24
Bad experience leads to attrition
Slide 25
• Typical form-filling on government sites intended to allow citizens to ‘apply online’ Page 1 Page 2 Page 6Page 5Page 4Page 3 Page 7
45% 72% 48% 21% 85% 80% Conversion by page
45% 32% 16% 3% 3% 2% Cumulative
• Every page ‘works’ but the user-experience is so poor that only 2% finish the journey
• Modelling the journey is good, but not enough…• We need to model the experience too.
Models and Stakeholders
Intelligent Testing, Improvement and Assurance
Slide 26
Intelligent Testing, Improvement and Assurance
Stakeholders and test models• Stakeholders may not tell testers to
use specific test models; you need to explain them to stakeholders so they understand
• The challenge(s):– Stakeholders may be of the opinion that
the models you propose generate too few tests to be meaningful or too many to be economic
– We need to engage stakeholders.Slide 27
‘Measuring quality’ feels good but…• Measurable quality attributes make
techies feel good, but they don’t help stakeholders if they can’t be related to experience
• If statistics don’t inform the stakeholders’ vision or model of quality– We think we do a good job– They think we waste their time and
money.Intelligent Testing, Improvement and
AssuranceSlide 28
Intelligent Testing, Improvement and Assurance
Relevance
• Documented or not, testers need and use models to identify what is important and what to test
• A control flow graph has meaning (and value) to a programmer but not to an end-user
• An equivalence partition may have meaning to users but not the CEO of the company
• Control flow, equivalence partitions are models that have value in some, but never all, contexts.
Slide 29
Intelligent Testing, Improvement and Assurance
Helping stakeholders to make better decisions is the tester’s goal• We need models that
– Do more than identify tests– Take account of the stakeholders’ perspective
and have meaning in the context of their decision-making
• If we ‘measure quality’ using technical models– We delude both our stakeholders and ourselves
into thinking we are in control of Quality– We’re not.
Slide 30
Failures of Systems,
Failures of Models
Intelligent Testing, Improvement and Assurance
Slide 31
F-16 bug (found in flight)
• One of the early problems was that you could flip the plane over and the computer would gladly let you drop a bomb or fuel tank. It would drop, dent the wing, and then roll off.
• http://catless.ncl.ac.uk/Risks/3.44.html#subj1.1
Intelligent Testing, Improvement and Assurance
Slide 32
Poor test model
Slide 33
Poor test model
Intelligent Testing, Improvement and Assurance Slide 34
Poor test model
2. Web Sub-System
Web Server
1. Application (objects)Sub-System
Database Server
Banking System (Credit Card Processor)
Legacy System(s)
3. Order Processing Sub-System
4. Full E-Business System
Scope of testing for E-Commerce
People
Process
Training
Environment
Test strategy
Intelligent Testing, Improvement and Assurance
Slide 36
• Our test strategy must align with our model of quality and our risk-assessmentTest Phase Focus
Requirements, design etc. Relevance, correctness, completeness, ambiguity etc.
Component Input validation, correct behaviour, output validation, statement and branch coverage
Integration Correct, authorised transfer of control, exchange of data, consistency of use and reconciliations
System (-system) End-to-end accuracy, consistency, security, performance and reliability
Acceptance Alignment to business goals, end-to-end ease of use and experience, successful outcomes, user personas
Every focus area requires test
model(s)
Failure of testing is usually a failure in a test model• If the right models are selected, and
commitment is made to cover them– The testing usually gets done
• But often, no model is explicitly selected at all• Where a model fails, it is usually wrong
because:– The model does not represent reality– The scope of the model is too narrow– The model ignores critical aspects (context, people,
process, environment or training/capability).
Intelligent Testing, Improvement and Assurance
Slide 37
Close
• We need to understand what quality is before we can pursue and achieve it
• Testing often fails because test models are not used or understood
• Testers need models to test but the ‘standard’ quality models are too simple
• We need to take stakeholder views into account to create relevant testing models
• Using models sounds techy, but it’s completely natural – it’s part of what makes us human.
Intelligent Testing, Improvement and Assurance
Slide 38
Assurance is NOT Quality Assurance
It’s not about proof or guarantees,
eitherIntelligent Testing,
Improvement and AssuranceSlide 39
The Pursuit of Quality: Chasing Tornadoes or Just Hot Air?
Slide 40Intelligent Testing, Improvement and Assurance
gerrardconsulting.commaelscrum.com
businessstorymanager.com test-axioms.com
uktmf.com
@paul_gerrard