Date post: | 27-Jan-2015 |
Category: |
Technology |
Upload: | paul-gerrard |
View: | 114 times |
Download: | 1 times |
AXIOMS
Paul Gerrard
THE
TESTINGOF
Advancing Testing Using Axioms
Agenda
Axioms – a Brief Introduction Advancing Testing Using Axioms
First Equation of Testing Test Strategy and Approach Testing Improvement A Skills Framework for Testers Quantum Testing
Close
Test Axioms
Formulated as a context-neutral set of rules for testing systems
They represent the critical thinking processes required to test any system
There are clear opportunities to advance the practice of testing using them
Testers Pocketbook: testers-pocketbook.com
Test Axioms Website test-axioms.com
How can we use Test Axioms?• Test Axioms are not beginners guides• They can help you to think critically about
testing• They expose flaws in other people’s thinking
and their arguments about testing• They generate some useful by-products• They help you to separate context from values
• Interesting research areas!• First Equation of Testing, Testing Uncertainty
Principle, Quantum Theory, Relativity, Exclusion Principle...
• You can tell I like physics
16 Proposed Axioms
(in three groups)
Stakeholder, (Test) Design and (Test) Delivery
You have the little book
Stakeholder
Value
Scope Mgt.
Good-Enough
Stakeholderaxioms
Testing needs stakeholders (p64)
Summary:Identify and engage the people or organisations that will use and benefit from the test evidence we are to provide
Consequence if ignored or violated:There will be no mandate or any authority for testing. Reports of passes, fails or enquiries have no audience.
Questions: Who are they? Whose interests do
they represent? What evidence do
they want? What do they need it
for? When do they want
it? In what format? How often?
Test Model
Test Basis
Oracle
Coverage
Prioritis-ation
Fallibility
Design axioms
Test design is based on models (p68)
Summary:Choose test models to derive tests that are meaningful to stakeholders. Recognise the models’ limitations and the assumptions that the models make
Consequence if ignored or violated:Tests design will be meaningless and not credible to stakeholders.
Questions Are design models available to use as
test models? Are they mandatory? What test models could be used to
derive tests from the Test Basis? Which test models will be used? Are test models to be documented or
are they purely mental models? What are the benefits of using these
models? What simplifying assumptions do
these models make? How will these models contribute to
the delivery of evidence useful to the acceptance decision makers?
How will these models combine to provide sufficient evidence without excessive duplication?
How will the number of tests derived from models be bounded?
Confidence
Sequenc-ing
Event
Never-Finishe
d
Repeat-Test
Environment
Delivery axioms
Advancing Testing Using Axioms
First Equatio
n of Testing
Axioms+ Context
+ Values+ Thinking
=Approac
h
Why is the equation useful? Separation of Axioms, context, values
and thinking Tools, methodologies, certification,
maturity models promote approaches without reference to your context or values
No thinking is required!
Without a unifying test theory you have no objective way of assessing these products.
One context, multiple approaches Given context, practitioners can promote
different approaches based on their values Values are preferences or beliefs
Pre-planned v exploratory Predefined v custom process Requirements-driven v goal-based Standard documentation v face-to-face comms.
Some contexts preclude certain practices “No best practices”
Axioms allow (ensure) different approaches and expose positions
Separating axioms, context and values clarifies positions, for example: ‘Structured’ (certified?) test advocates have little
(useful) to say about Agile contexts Exploratory test advocates have little (useful) to
say about contract/requirements-based acceptance
The disputes between these positions is more about values than practices in context
Is a consultant recommendation best for the stakeholders or the consultant?
Test Strategy and Approach
Strategy is a thought process not a document
Contexts of Test Strategy
TestStrategy
Risks
Goals
ConstraintsHuman resourc
e
EnvironmentTimescales
Process(lack of?)
Contract
Culture
Opportunities
User involvement
Automation
De-Duplicatio
n
Early Testing
Skills
Communication
Axioms
Artefacts
IEEE 829 Test Plan Outline1. Test Plan Identifier2. Introduction3. Test Items4. Features to be Tested5. Features not to be
Tested6. Approach7. Item Pass/Fail Criteria8. Suspension Criteria
and Resumption Requirements
9. Test Deliverables10. Testing Tasks11. Environmental
Needs12. Responsibilities13. Staffing and
Training Needs14. Schedule15. Risks and
Contingencies16. ApprovalsBased on IEEE Standard 829-1998
I’m no fan of IEEE 829
Used as a strategy checklist Scarily vague (don’t go there)
Used as a documentation template/standard Flexible, not prescriptive, but
encourages copy and edit mentality (documents that no one reads)
But many many testers seek guidance on What to consider in a test strategy Communicating their strategy to
stakeholders and project participants
IEEE 829 Plan and Axioms Items 1, 2 – Administration Items 3+4+5 – Scope Management, Prioritisation Item 6 – All the Axioms are relevant Items 7+8 – Good-Enough, Value Item 9 – Stakeholder, Value, Confidence Item 10 – All the Axioms are Relevant Item 11 – Environment Item 12 – Stakeholder Item 13 – All the Axioms are Relevant Item 14 – All the Axioms are Relevant Item 15 – Fallibility, Event Item 16 – Stakeholder Axioms
A Better Test Strategy and Plan
1. Stakeholder Objectives Stakeholder management Goal and risk management Decisions to be made and how
(acceptance) How testing will provide
confidence and be assessed How scope will be determined
2. Design approach Sources of knowledge (bases
and oracles) Sources of uncertainty Models to be used for design
and coverage Prioritisation approach
3. Delivery approach Test sequencing policy Repeat test policies Environment requirements Information delivery approach Incident management approach Execution and end-game
approach
4. Plan (high or low-level) Scope Tasks Responsibilities Schedule Approvals Risks and contingencies
Testing Improvement
Test process improvement is a waste
of time
Actuallyits 11
(most were not software related)
The delusion of process models(e.g. CMM)
Google search “CMM” – 22,300,000 “CMM Training” – 48,200 “CMM improves quality” – 74 (BUT really 11 – most of
these have NOTHING to do with software) A Gerrard Consulting client…
CMM level 3 and proud of it (chaotic, hero culture) Hired us to assess their overall s/w process and make
recommendations (quality, time to deliver is slipping) 40+ recommendations, only 7 adopted – they
couldn’t change How on earth did they get through the CMM 3 audit?
“Test Process Improvement is a Waste of Time”
Using process change to fix cultural or organisational problems is never going to work
Improving test in isolation is never going to work either
Need to look at changing context rather than values…
Why you are where you are
Context+ Values
+ Thinking=Approa
ch
<- your values<- your context
<- your thinking<- your approach
Where maturity models come from
Context+ Values
+ Thinking=Approa
ch
<- someone else's
<- someone else's<- someone else's<- someone else's
Making change happen
Axioms+
Context+ Values
+ Thinking=Approa
ch
<- recognise
<- hard to change
<- could change?<- just do some<- your approach
Using the axioms and questions Axioms represent the critical things
to think about Associated questions act as
checklists to: Assess your current approach Identify gaps, inconsistencies in current
approach QA your new approach in the future
Axioms represent the WHAT Your approach specifies HOW
Eight stage change process (after Kotter) Mission Coalition Vision Communication Action Wins Consolidation Anchoring
Changes identified here
If you must use one, this is where your
‘process model’ comes into play
A Skills framework for testers
Axioms indicate WHAT to think about...
...so the Axioms point to SKILLS
Test design is based on models (p68)
Summary:
Choose test models to derive tests that are meaningful to stakeholders. Recognise the models’ limitations and the assumptions that the models make.
Consequence if ignored or violated:
Tests design will be meaningless and not credible to stakeholders.
Questions: Are design models available to use
as test models? Are they mandatory? What test models could be used to
derive tests from the Test Basis? Which test models will be used? Are test models to be documented or
are they purely mental models? What are the benefits of using these
models? What simplifying assumptions do
these models make? How will these models contribute to
the delivery of evidence useful to the acceptance decision makers?
How will these models combine to provide sufficient evidence without excessive duplication?
How will the number of tests derived from models be bounded?
Test design and modelling skills A tester needs to understand:
Test models and how to use them How to select test models from fallible
sources of knowledge How to design test models from fallible
sources of knowledge Significance, authority and precedence of
test models How to use models to communicate The limitations of test models
Familiarity with common modelsIs this all that current certification provides?
Training and certification must change Intellectual skills and capabilities are
more important than the clerical skills Need to re-focus on:
Testing thought processes (Axioms) Testing Stakeholder relationship management Testing as an information provision service Goal and risk-based testing Real-world examples, not theory
Practical, hands-on, real-world training, exercises and coaching.
Quantum Testing
If evidence arrives in discrete quanta...
...can we assign a value to it?
How testing builds confidence As tests are run, every individual test
has some significance Some tests expose failures but
ultimately we want all tests to PASS When all tests pass – the
stakeholders are happy, aren’t they? Can we measure confidence by
counting tests?
Not really...
The value of a test varies by… Coverage model:
A test could cover one or hundreds of functional conditions, ten thousand program statements or ten
Objective: Criticality of the business goal it examples Criticality of the risk it informs
Precedent: The first end-to-end test pass is significant The 100th variation of a similar test is less significant
Functional dependence: A test of shared functionality used thousands of times per hour could
be much more important than a peripheral feature used once/day Stakeholder:
Are customers tests more or less significant than supplier tests? Context:
The same test run at different times in different environments can have different value.
Quantum Testing
Only a stakeholder can assign a value to a test (but that is very hard thing to do) – (but see relativity)
Testers can’t quantify value, but could define significance
A test is significant (to stakeholders) if it: Can be related to a meaningful test objective Increases coverage with respect to a meaningful test
model Is considered in an acceptance decision (at any level)
Significance is a Boolean but could be 0 or 1 The number of insignificant tests should be zero.
Assessing significance
Significance can only be assessed by testers if: Our test objectives, models, coverage goals are
meaningful (to stakeholders) or Testers are authorised to create their own
objectives, measures and coverage goals or Testers are their own stakeholder
Testers need a close, trusting relationship with their stakeholders or authorised autonomy E.g. exploratory testing won’t work if stakeholders
do not allow autonomy Testers should not ‘go it alone’.
The significance of significance Test coverage models and goals that
generate uniform distributions of tests are inefficient and uninformative We need more and better test models Models that are meaningful in context
Significance varies with context and can be used to explain why e.g. some tests aren’t useful as regression
tests How much testing is enough?
Can never be answered by coverage alone.
Using significance (as booleans) A test objective could be deemed as met
when a set of significant tests are passed Test objectives can often be aligned with
business goals (or risks) e.g.: Faultless end to end or straight-through
processing in an ERP system Customer order capture, money-in or complete
financial reconciliations without failure Could the business put a value on these
goals being achieved? Perhaps the business case provides this level of
information?
Our current understanding of test value measurement In terms of a mathematical treatment of
testing we are some way off Compare with:
Understanding of heat and energy in 18th century?
Economics in the early 20th Century? A complete theory of test value may never
be achieved Quantum testing suggests that:
Stakeholders judge value (relatively speaking) Testers judge significance.
Close
Axioms are context-neutral rules for testing The Equation of Testing
Separates axioms, context, values and thinking We can have sensible conversations about
process Axioms and associated questions provide
context neutral checklists for test strategy, assessment/improvement and skills
QT separates significance from value Could it tell us, “how much testing is enough”? Right now, we can ‘feel the heat’, but not
measure temperature.
AXIOMS
Thank-You!
THE
TESTINGOF
testaxioms.comtesters-pocketbook.comgerrardconsulting.comuktmf.com