AmiBug.Com, Inc. August 30, 2015© Robert Sabourin, 2009Slide 1 Testing Under Pressure: Five Key...

AmiBug.Com, Inc.

April 19, 2023 © Robert Sabourin, 2009 Slide 1

Testing Under Pressure: Five Key Principles

Robert Sabourin

President

AmiBug.Com, Inc.

Montreal, [email protected]

AmiBug.Com, Inc.


Under Pressure

• Pain points?– What hurts?– How much?

AmiBug.Com, Inc.

Testing Under Pressure

1. Begin with the end in mind

2. Active context listening

3. Decision making

4. Ruthlessly triage

5. The last best build


AmiBug.Com, Inc.


Under Pressure

(1)Begin with the end in mind

AmiBug.Com, Inc.


Fundamental Question

• How do you know when you are finished?

AmiBug.Com, Inc.


Crosby on Quality

• “Quality is defined as conformance to requirements”

• “Quality is not a measure of GOODNESS”– Phil B. Crosby, Quality is Free

AmiBug.Com, Inc.


Gerald M. Weinberg

“Quality is value to some person”

Exploring Requirements Quality

Before Design Dorset House

AmiBug.Com, Inc.


Edsger W. Dijkstra

• “Program testing can be used to show the presence of bugs, but never to show their absence”

AmiBug.Com, Inc.


Pareto Principle

• Vilfredo Pareto, 1848 - 1923, Economist– 80% of the wealth was in the hands of 20%

of the population

AmiBug.Com, Inc.


Pareto Principle

• Joseph Juran, 1903 - present,Quality Control Engineer– 1950 Quality Control Handbook

– 20% of the study population accounts for 80% of the measure under consideration

AmiBug.Com, Inc.


Under Pressure

(2) Active context listening

AmiBug.Com, Inc.


Context Drivers - BTO

• Business– Value– To whom?– Why?

• Technology– Solutions

• Organization– Corporate Structure– Team Structure– Roles and Responsibilities

AmiBug.Com, Inc.


AmiBug.Com, Inc.


Context Listeners

• Find Sources• Monitor Drivers• Anticipate Change• React

AmiBug.Com, Inc.


Under Pressure

(3) Decision Making

AmiBug.Com, Inc.


First Things First

• Begin with the end in mind– Gain Consensus

• Goals– How do we know we are finished?

• Purpose– Why are we doing this project?

– How will be react to change?

• Meaning– What is a bug?

– What is a test?

– What is quality?

AmiBug.Com, Inc.


"No! Try not, Do. Or do not.

There is no try."

Yoda

AmiBug.Com, Inc.


Bug Flow

• We will be testing … imagine that we actually find a bug!– What are we going to do about it?– How will we decide?– When should we decide how to decide?– When should we change how we decide?– When should we review our past

decisions?

AmiBug.Com, Inc.


Bug Flow

Entered Reviewed

Fixed

AssignedPrioritized

ClosedUnassigned

TRIAGE

REFUSE

CHECK

MANDATE

FAILURE

DESIGNATE

CORRECT

CONFIRM

AmiBug.Com, Inc.


Bug Workflow

Seven Steps1. Identify key stakeholders

2. Learn about decisions

3. Define bug priority & severity

4. List steps when bug is found

5. Build bug flow state model

6. Get stakeholders “buy in”

7. Adapt bug flow as required

AmiBug.Com, Inc.


Bug Workflow

Identify key stakeholders

1

AmiBug.Com, Inc.


Bug Workflow

How is a decision made?

2

AmiBug.Com, Inc.


Bug Workflow

UrgentSevere

UrgentNot Severe

Not UrgentSevere

Not UrgentNot Severe

Define Priority and Severity Scheme

3

AmiBug.Com, Inc.


Bug Workflow

Define steps to follow– bug is discovered in testing or reported from the field– a bug report form is completed– the bug report form is reviewed– the bug report is added to the bug list– a decision is made, at a bug review meeting, about whether

the bug should be fixed– if the bug is fixed then the software is re-tested to reconfirm

that the bug has indeed been fixed– if the bug is not fixed (on purpose!) then a description of the

work around is published or made available to help desk staff4

AmiBug.Com, Inc.


Bug Workflow

Entered Reviewed

Fixed

AssignedPrioritized

ClosedUnassigned

TRIAGE

REFUSE

CHECK

MANDATE

FAILURE

DESIGNATE

CORRECT

CONFIRM

Build State Model

5

AmiBug.Com, Inc.


Bug Workflow

Get stakeholder “buy-in”

6

AmiBug.Com, Inc.


R2-D2 and Chewbacca are playing the holographic game aboard the Millennium Falcon Chewbacca: Aaaaaaaaaaaaaaaarrrgh C-3PO: He made a fair move. Screaming about it can't help you.Han Solo: Let him have it. It's not wise to upset a Wookiee.C-3PO: But sir, nobody worries about upsetting a droid.Han Solo: That's 'cause droids don't pull people's arms out of their sockets when they lose. Wookiees are known to do that.Chewbacca: Grrf.C-3PO: I see your point, sir. I suggest a new strategy, R2: let the Wookiee win.

AmiBug.Com, Inc.


Bug Workflow

Adapt as required

7

AmiBug.Com, Inc.


Under Pressure

(4) Ruthlessly triage

AmiBug.Com, Inc.


Test Idea Sources

• Capabilities• Failure Modes• Quality Factors• Usage Scenarios• Creative Ideas• States• Data• Environments• White Box• Taxonomies

Cap

ture

tes

ting

idea

s

AmiBug.Com, Inc.


Triage

• Criticality• Resources• Trade offs

– Benefit– Consequence– Credibility

AmiBug.Com, Inc.


Which test?

•Impact estimation–For each test idea guesstimate:

• benefit of implementation• consequence of implementation• benefit for not implementing• consequence of not implementing

–How credible is the information?

Tria

ge t

estin

g id

eas

AmiBug.Com, Inc.


How to Decide?Understanding Complex Technology Quantitatively By Tom Gilb

Rank Credibility0.0 Wild guess, no credibility

0.1 We know it has been done somewhere

0.2 We have one measurement somewhere

0.3 There are several measurements in the estimated range

0.4 The measurements are relevant to our case

0.5 The method of measurement is considered reliable

0.6 We have used the method in-house

0.7 We have reliable measurements in-house

0.8 Reliable in-house measurements correlate to independent external measurements

0.9 We have used the idea on this project and measured it

1.0 Perfect credibility, we have rock solid, contract- guaranteed, long-term, credible experience with this idea on this project and, the results are unlikely to disappear

Tria

ge t

estin

g id

eas

AmiBug.Com, Inc.


Which test?

Test Idea Rejection – What If?–If the cost/benefit does not make business sense then consider implementing:• part of the test, could that lead to part of the

benefit at a more reasonable cost?• more than the stated test, would that generate

more benefit?• a different test than the stated idea, could that

generate more benefit for less cost?

Tria

ge t

estin

g id

eas

AmiBug.Com, Inc.


Test Triage

• Test Triage– JIT Projects– High Frequency– Daily Test Triage Session– Experience dictates

• Early AM (Rob Preference)• Late PM (several clients)

Tria

ge t

estin

g id

eas

AmiBug.Com, Inc.


Test Triage

• Test Triage Meeting– Review Context

• Business• Technical

– Information since last triage

• Test results• Bug results• New testing ideas

Tria

ge t

estin

g id

eas

AmiBug.Com, Inc.


Test Triage

• Allocate Testing Assignments to Testers– Make sure testers know context– Best thing to test– Best person to test it– Best people to explore it– Best lead– Are subject matter experts required

Tria

ge t

estin

g id

eas

AmiBug.Com, Inc.


Test Triage

• Requirement Triage• Change Control• Test Triage• Bug Flow

– Combined– Equivalent to CCB– Few people– Fluid

Tria

ge t

estin

g id

eas

AmiBug.Com, Inc.


Test Triage

Life of a test ideaa. Comes into existence

b. Clarified

c. Prioritizeda. Test Now (before further testing)

b. Test before shipping

c. Nice to have

d. May be of interest in some future release

e. Not of interest in current form

f. Will never be of interest

d. Integrate into a testing objective

Tria

ge t

estin

g id

eas

AmiBug.Com, Inc.


Which test is next?

• Questions– Given state of project, state of business, state of

technology, our abilities, our experience and our history, what we know and what we do not know, what should we test next?

– How much effort are we willing to spend continuing to test this project?

– Can we ship yet?

Tria

ge t

estin

g id

eas

AmiBug.Com, Inc.


Which test is next?

• Magic crystal ball– If it existed how would you use it?– What question would you ask?– What question would it ask?

Tria

ge t

estin

g id

eas

AmiBug.Com, Inc.


Deciding what not to test?

• Time pressure

– Should we skip a test?– If test failed could system still be of

value to some stakeholder?– If test was skipped could important

bugs have been otherwise found?

Tria

ge t

estin

g id

eas

AmiBug.Com, Inc.


Guidelines and Decisions

• To each stakeholder– risk of failure– consequence of failure– value of success– how much certainty do we have – is it a wild guess or an absolute

truth?

Get

Sta

rted

Rig

ht

AmiBug.Com, Inc.


Under Pressure

(5) The last best build

AmiBug.Com, Inc.

Under Pressure

• Always know the last best build.

• Rank recent builds.• If forced to ship at a

fixed date then I often let project stakeholders trade off between recent builds.


AmiBug.Com, Inc.


Under Pressure

The right stuff

AmiBug.Com, Inc.


Testing Under Pressure

So what exactly did they throw over the wall?

AmiBug.Com, Inc.


Getting Things Done

Adapt to change

Triage Testing

Prioritize Bugs

Track Progress

Revised risks?New test objectives?

What to test?What not to test?

What to fix?What not to fix?

What do we know so far?What don’t we know yet?

Per

iod

ic

Smoke Test Should the new build be tested at all?On failure continue with previous build in test.

Confirmation Test Have bugs really been fixed? Double check!

Stress Testing How well does the application behave in harsh conditions?Experiment.

BU

ILD

FAST Test Each testable object has a simple test.Is the testable object stable enough to test

Regression Test Does application still work as before?Did we accidentally break something?

AmiBug.Com, Inc.


Smoke Testing

• Smoke test is run on a new build of software to make sure all functions operate well enough to continue testing

• “Turn on a new appliance at the store”

AmiBug.Com, Inc.


FAST Testing

• Functional Acceptance Simple Tests– Wide in breadth, low

in depth– Exercise every

function of the application at least once

AmiBug.Com, Inc.


Regression Testing

• Previously executed tests are re-executed against a new version of the application– Have code changes broken something that

used to work– Have we introduced new defects

AmiBug.Com, Inc.


Confirmation Testing

• Typically:– Tester confirms that the

fixed bug is really fixed in the appropriate software build

AmiBug.Com, Inc.


Stress Testing

• Testing operational characteristics of application within a harshly constrained environment– Limit processor– Limit memory– Limit disk space– Diminish access to

shared resources

AmiBug.Com, Inc.


Under Pressure

Chartered Session Based Testing

AmiBug.Com, Inc.


C. Northcote Parkinson

Parkinson’s Law:“…work expands so as to fill the

time available for its completion…”

AmiBug.Com, Inc.


Just In Time Testing

Exploratory Testing

AmiBug.Com, Inc.


Exploratory Testing

• James Bach (www.satisfice.com)

• General Functionality and Stability Test Procedure for Windows 2000 Application Certification

AmiBug.Com, Inc.


Mandate to explore

The object of your mission is to explore the Missouri river, & such principle streams of it, as, by its course and communicationwith the waters of the Pacific ocean...may offer the most direct & practicable water communication across this continent for thepurposes of commerce.

- Thomas Jefferson's letter to Meriwether Lewis, June 1803

William Clark Meriwether Lewis

AmiBug.Com, Inc.


Make intelligent decisions

Take notes about your decisions

Map out where you have been

Others can use the result

AmiBug.Com, Inc.


Chart as you explore

Further exploration yields a good idea of the state of the world!

One bit at a time

AmiBug.Com, Inc.


Exploration Notes

- Tabular- Chronological- Schematic- Point form- Concise

AmiBug.Com, Inc.


Exploratory Testing

• Test Cases – Not known in advance– Defined & executed “on the fly” while you learn

about the product

• Map Making Skills– Consistent note taking style– Practice

AmiBug.Com, Inc.


Exploratory Testing

• During test we must capture– Function, options or sub-functions being explored– Test cases attempted– Comments, notes, images or attachments – Hints, reminders and observations which may be useful to

future testers– Date, Platform, Build or Configuration under test– Name of person running test– Oracles, “strategy to assess correctness”– Other relevant details

AmiBug.Com, Inc.


An Exploratory Test Process

Kick Off

Prepare

Run

Complete

Review

Follow Up

Confirm Test ObjectiveEnsure context known

Ensure HW and SW OKAll tools available

Chunk of 90 to 120 minTest, Plan, Discover

Wrap upCollect all notes data

Review results withTest Lead

Reassess goalsPiece together map

AmiBug.Com, Inc.


Finished?

How do you know you are finished?

AmiBug.Com, Inc.


You know you are finished when

…

… the only bugs left are the ones are acceptable (based on your objective test team input) ...

AmiBug.Com, Inc.


… the only bugs left are the ones are acceptable (based on your objective test team input) ...

At least for now!

You know you are finished when

…

AmiBug.Com, Inc.


Bottom Line

• My experience is that it is better to omit a test on purpose than to skip it because you ran out of time or forgot about it!

Get

Sta

rted

Rig

ht • Systematically collecting, evaluating and triaging testing ideas helps me decide what not to test - at least for now?

AmiBug.Com, Inc.


Thank You

• Questions?

Date post:	25-Dec-2015
Category:	Documents
Upload:	charlene-bates
View:	217 times
Download:	0 times

AmiBug.Com, Inc. August 30, 2015© Robert Sabourin, 2009Slide 1 Testing Under Pressure: Five Key...

Documents