'Best Practices' & 'Context-Driven' - Building a bridge (2003)

“Best Practices” and “Context-Driven: Building a Bridge

Neil Thompsonwww.TiSCL.com+44 (0)7000 NeilTh (634584)

STAREast Neil Thompson slide 0 of 29T15

Test Management

Thompson informationSystemsConsulting Limited

© 2003

track presentation

Presentation contents

• Theme: what to bridge, and how.................• Learning objectives• Instead of Best Practice: “always-good” practices• How deep is the schism, how strong the bridge supports?• Goldratt’s Theory of Constraints• The thinking tools and how to apply them• Relationship to process improvement• Conclusions & key references • Lessons learned, and way forward

Thompson informationSystemsConsulting Limited2003

©

STAREast Neil Thompson 1 of 29T15

Theme: from conflict to integrated framework

BestPractice

Context-Driven

Constraints,Requirements,Objectives etc

“Always-Good”Practices

“fossilisedthinking”

“formalisedsloppiness”

Unifying principles

Goldratt’s“thinking tools”

Expert pragmatismwith structure


©

What How


Learning objectives• Understand the Best Practice v. Context-Driven debate• If your allegiance is already fixed, please un-fix at least for

now: consider “always-good” principles & practices• Gain knowledge on Goldratt’s Theory of Constraints

(like it or not, it’s widely used)• Appreciate the subtleties of extending its use beyond

manufacturing etc• Learn enough about the thinking tools to use them• Understand enough of my examples to determine if, and

how, you can use some or all of this framework


©


Instead of Best Practice: “always-good” practices (the top-down view)

• “doing the right things”

Always-goodEffectiveness

Efficiency

Risk management

Quality management

Insurance Assurance

• “doing things right” (eg optimising speed & minimising cost)

• detecting errors,faults & failures

• giving confidence in fitness-for-purpose Thompson

informationSystemsConsulting Limited2003

©


Principles ElementsWhat testing against,QA context

Detecting errors: the V-model Giving confidence: the W-model Good enough quality Quality & risk management

Risk use & testprioritisation

Risk management & testing Risks by layer of V-model Tests’ priorities based on risks

Appropriate stepwiserefinement

Strategy, plan, design, cases, scripts, data, execution &management procedures

Structured &controlled execution

Test, check results, debug, fix, retest & regression-test(problems & changes, by urgency & importance)

Informed decision-making

Test coverage & handover criteria Metrics (progress & problems) Role of metrics in risk management Summary: quantify residual risks & confidence

Appropriate skills Business, technical & testing; roles & responsibilities;independent testing

Appropriatetechniques

Glass-box, black-box etc Rehearsing acceptance

Appropriate tools Capture-replay, management etc

Testing processoverall

Process review & improvement, eg symptom-based,Capability Maturity Model

Decide targets, and improve as appropriate

Effective-ness

Efficiency

Always-good practices (bottom-up)


©


How deep is the schism?

• CD views of BP: “fossilised thinking, disenfranchising, a disservice to testing”

• BP views of CD: “formalised sloppiness, posturing, “I agree”, stating the obvious”

Each side may be merely competitive, or ethically outraged


©


• CD more outraged? • BP more willing to appease / compromise?

Not trying to be controversial: is this schism good for testing?

How strong are the bridge supports?

• survived some scrutiny so far• not guaranteed; I modify them

slightly over time• but you could use your own• other authors say fixed “principles”:

this is just adding a level below• deliberately omit trivial-obvious, eg

use good env’ts (?)• omit if unclear message, eg “test

before code” (?)• the ? marks illustrate value of the

thinking tools: I may add these!

• began in manufacturing• since applied to other areas, especially

project management• limited attempts on software dev’t,

but...• principles used by Agile methods• new here is applying all the thinking

tools to process context (not just improvement)

This bridge is supported by “always-good” practices and Goldratt


©


How always-good are these practices really?

How applicable is Goldrattto this situation?

Goldratt’s Theory of Constraints


©


diagram based on those in “The Race”, E.M. Goldratt & R. Fox 1986

Drum Rope

Buffer

Goal: to win the warObjective: to maximise throughput (right soldiers doing right things)Constraint on throughput: slowest marcher

Critical chain: weakest link is all we need fix, by means of...Five focussing steps: identify constraint, exploit it, subordinate all else, elevate it (i.e. strengthen so not now weakest), then... identify next constraintBut now it’s no longer simple: so need iterative tools for: what to change, what to change to, howFive thinking tools (based on sufficient causes & necessary conditions)

Five logical tools for process improvement (Dettmer*, after Goldratt)

Core problem+(other) Root causes

Intermediateeffects

Undesirableeffects

Prerequisites+Conflicts

Requirements +INJECTIONS

Objective

CURRENT REALITY+ Injections

Intermediateeffects

Desired effects

Intermediateobjectives

Obstacles

Objective

Needs+Specificactions

Intermediateeffects

Objective

CURRENTREALITY

........... What to change to .......(2) (3)

CONFLICTRESOLUTION

.... How to change ....(4) (5)

PRE-REQUISITES

TRANSITION

FUTUREREALITY

What tochange (1)

* very slightly paraphrased here



©

The five logical tools applied to testing context, practices etc.

Rootcauses

Intermediateeffects

Effects

REALITY + Injections

Intermediateeffects

Desired effects

Intermediatesub-prerequisites

Obstacles

Sub-prerequisite

Specificactions

Intermediateeffects

Sub-objectives

ContextAlways-good practices

CONFLICTRESOLUTION

Questions toconsider

PRE-REQUISITES

TRANSITION

FUTUREREALITY

CURRENTREALITY

What “appropriate” meansin this context

Choice categories& actions

“Prerequisites”

Requirements

Objectives

POSITIONING+ Justifications

Extremes

Sub-requirements(valid & invalidassumptions)+ INJECTIONS

Choicecategories +NEEDS +

INTERACTIONS

Objectives

Actions & sub-objectives

• Methodology unhappy with ( actions)

• Unsure how best to test ( conditions)

Good practicesin thiscontext


©


12a

2b

3

4

5a

5b

Context (CURRENT REALITY)

• Unsure how best

to testMethodologyhappy with

• Methodologyunhappy with

Business/org. sector Nation

(eg USA)

Corporate culture

Technology

Legal constraints:• regulation• standards

Moralconstraints, eg:• human safety• money, property• convenience Process

constraints, eg:• quality management• configuration mgmt

Job type & size:• project/programme• bespoke/product• new/maintenance

Resources:• money ( skills, environments)• time

SCO

PE, CO

ST, TIME,

QU

ALITY

/ RISK

FAC

TOR

S

App type

1


©


Always-good practices (CONFLICT RESOLUTION upper level)

Always-goodEffectiveness

EfficiencyRisk management Quality management

Insurance Assurance

V-model: what testing against W-model: quality management

Risks: list & evaluate

Define & detect errors (UT,IT,ST)Give confidence (AT)

Prioritise tests based on risks

Tailor risks & priorities etc to factors

Refine test specifications progressively: Plan based on priorities & constraints Design flexible tests to fit Allow appropriate script format(s) Use synthetic + lifelike data

Allow & assess for coverage changes Document execution & management procedures

Distinguish problems from change requests Prioritise urgency & importance

Distinguish retesting from regression testing

Use handover & acceptance criteria

Define & measure test coverage

Measure progress & problem significance

Be pragmatic over quality targets

Quantify residual risks & confidence

Decide process targets & improve over time

Define & use metrics

Assess where errors originally made

Define & agree roles & responsibilities

Use appropriate skills mix

Use independent system & acceptance testers

Use appropriate techniques & patterns

Plan early, thenrehearse-run,acceptance tests

Use appropriate tools

Optimise efficiency

2a



©

What are the dimensions of formality?

• adherence to standards and/or proprietary methods

• detail• amount of

documentation• scientific-ness• degree of control• repeatability

• consistency• contracted-ness• trained-ness and

certification of staff• “ceremony”, eg

degree to which tests need to be witnessed, results audited, progress reported

• any others?


©


Next diagram will take each box from the previous diagram and assess it on a formal-informal continuum, so...

In preparation for this: what do we mean by “formality”?

Good practices in this context (CONFLICT RESOLUTION lower level)

APPROPRIATE USE OFV-model: what testing against

Use a waterfallV-model

Don’tuse a V-model

• NEED NOT BE 4 LEVELS

V-model isdiscredited

We want to be trendy, anyway

We’re too lazy to think

We want baselines to test against

We want to test viewpoints of:• users• someone expert & independent• designers• programmers

We’re doing adaptive development (no specs)

MANY PEOPLESTAND BY V-MODEL

We’re doing iterative development

We’re object-oriented • V-MODEL IS IMPLICIT IN BINDER’S BOOKTesting OO systems: models,

patterns & tools

Documentation must be minimised

“Conflict”

We have little time

• SOME SPECS AREOUT OF DATE / IMPERFECT,BUT WE COPE

Different levelsmitigate different risks

Two heads are better than one

• CAN USE EXPLORATORY TESTING AGAINST CONSENSUS BASIS

• NEED NOT BE 1-1 CORRESP SPECS-LEVELS

• MULTIPLE PARTIAL PASSES

• THEY ARE LEVELS, NOT STAGES

All systems are integrated from parts

2b


©


Summary of conflict resolution

Questioning some"positive" statements

Questioning"negatively"

Neutral questions

we must always... oh, really? so what? it's absolutely

impossible to... who says? why is that?

is that still true? are there no

exceptions at all?

The positioning of our always-good practice on the formal-informal scale has been prepared by:• splitting it into assumed reasons eg “documentation must be minimised” • challenging the validity of those assumption and inserting “injections”

of things not originally thought of• questioning generally, which also helps understand causes & effects...

By doing the above, we have also distilled out some basic justifications of why this practice is “always-good” Thompson

informationSystemsConsulting Limited2003

©


What “appropriate” means in this context (FUTURE REALITY)

V-model with only 3 levels: acceptance (v. consensus) system (v. spec) unit (informal)

We don’t need aseparateintegration test level

The system has:• users• (potentially) expert & independent testers• designers (where significant)• programmers

• NEED NOT BE 4 LEVELS

Our system isvery simple We do need separate

development & acceptancetest levels

Our user requirementsare out of date andwere vague when written

Our programmers hatedocumentation

We do have a good functional specand independent testers available

3


©


Where are we up to?Remember, can apply this partially

ContextAlways-good practices

CONFLICTRESOLUTION


PRE-REQUISITES

TRANSITION

FUTURE REALITY

CURRENTREALITY

What “appropriate” meansin this context


Good practicesin thiscontext


©


12a

2b

3

4

5a

5b

Canstophere

Canstophere

Canstophere

Overall

Effectiveness

Efficiency

Positioning

Specifics

Questions to consider (PREREQUISITES)

USING ON THIS JOBa V-model with only 3 levels: acceptance (v. consensus) system (v. spec) unit (informal)

Tried before?What happened& why?

Under what circumstanceswould this work?

What are besteffects on:• you?• other stakeholders?

Howwillyouknowif itworked?(considera pilot)

Who is most likelyto benefit?

Under what circumstanceswould this not work?

What’s differenthere, and wouldit matter?

Who is mostlikely to bedisadvantaged?

Whatwillyoulearn?

What are worsteffects on:• you?• other stakeholders?

What if akey persondisagrees?

Howovercomeobjections& sell this?

Effectiveness

Efficiency

4


©


Based on Kaner, Bach & Pettichord (2002)Lessons learned in software testing:a Context-Driven approach

Choices (TRANSITION lower level)USING ON THIS JOBa V-model with only 3 levels: acceptance (v. consensus) system (v. spec) unit (informal)

Test against one basisor multiple?

For each testing level:

Each basis formally CM-baselined,or loose CM?

Each basis documented, partially verbal, orwholly verbal?

If verbal, how communicate andhow get agreement?

If mixed, how manage?

If documented, what format(s)?• natural language• pseudo-code• UML• ELHs, STDs, FDs etc• formal / mathematicalFunctional /

non-functionalconstituents?

Where non-functional, relationship toquality standards?

In what “language” andlevel of detail arerisks expressed?

5a


©


Intranetsite

Choice categories & actions (TRANSITION upper level)

Overall objectives, eg determineprocess improvement targets

and improve over time

Effectiveness objectives,rg quantify residual risks& confidence for each job

Efficiency objectives,eg optimise (within context)

Processes

Independence

Techniques

Tools

Coverage

Ceremony

Investment

CMM/TMMlevel targets

TPI®

targetsSymptom-basedtargets eg TOM™

NEEDS

V-model: what testing against

Define & use metrics Assess where errors originally made

Provide standard progress reports why?

Use independent system & acceptance testers

Communicationsplan:• training courses• workshops• documented

in-house guidelines

Use appropriatetools

Use appropriatetechniques

Allow & assess for coverage changes

INTER

AC

TION

S

5b


©


So why not just start with a process improvement method?

• Very good question! Possible answers:– “we’re not ready for process improvement, just want a

structure to apply Context-Driven”– “ we want to choose a process improvement method”– “the methods are too restrictive, we want to build our own”

• Theory of Constraints answer:– need to do some analysis before we know where best

to focus (or to improve processes)• Thinking tools answer:

– want to look at interactions between choices... Thompson informationSystemsConsulting Limited2003

©


Interactions between choice categoriesProcesses Independence

Techniques

Tools

Coverage

Ceremony

Investment

INFLUENCES

INFLUENCESINFLUENCE

INFLUENCE CONSTRAINGREATLY

HELP

SOMEHELPMEASURE

SOMEHELPSMEASURE DOCUMENTATION

HELPS

HINDERS

IMPROVES

PROVIDE DATA

FOR


©


Can use tables before / instead of / after diagrams

ContextGood practices


PREREQUISITES TRANSITION

FUTUREREALITY

CURRENTREALITY

What “appropriate”means


Canstophere

Canstophere

Principles &elements

Extremes &variables

CONFLICTRESOLUTION

Needs eg TMM

Canstophere

Canstophere

Canstophere


©


Conclusions• Yes, there are differences between Context-Driven

& Best Practice, but:– not as great as may first appear– each side attempts to be inclusive (but sees other as not!)– partly due to different backgrounds & industry sectors– this public-domain method can bridge the gap– need not be interpreted rigidly, or executed in full– the basic thinking is causes & effects: anyone can do it

• To summarise:– the framework, in effect, deconstructs then

reconstructs in your context... Thompson informationSystemsConsulting Limited2003

©


How many diagrams, where and whenCONFLICT RESOLUTION upper

PRE-REQUISITES

TRANSITIONlower

FUTUREREALITY

CURRENTREALITY

TRANSITIONupper

wherewhen how many

Your context

“Universal”


What “appropriate”means in your context

Choicecategories& actions

Each requirementto examine

Choices

2a-b

CONFLICTRESOLUTION

lowereg a V-model

eg testwarelifecycle

eg patterns+techniques

... 2-6(out of 25)

2b-3

... 2-6 or more(out of 25)

... 2-6(out of 25)

2-6(out of 25)

...

one

one

one

stoppointsSTAREast Neil Thompson 25 of 29T15


©

3-4

4-5a

5a-b

Deconstruct then reconstruct: the framework is a “meta-V-model”

Your context

All possiblecontexts


What “appropriate”means in your context

Choicecategories& actions

Each practiceto examine

Choices

CONFLICT RESOLUTION upper

PRE-REQUISITES

TRANSITIONlower

CURRENTREALITY

TRANSITIONupper

CONFLICTRESOLUTION

lower

FUTUREREALITY

FUTUREREALITY


©


Key references• Context-Driven:– Kaner, Bach & Pettichord (2002) Lessons learned in software testing, Wiley

• Best Practice:– Erik van Veenendaal et al. (2002) The Testing Practitioner, U.T. Nolthenius

• My inspiration:– Jens Pas (EuroSTAR 1998) Software testing metrics– Gregory Daich (STAREast 2002) Software documentation superstitions

• Theory of Constraints understanding:– Eliyahu M. Goldratt (1984 then 1992 with Jeff Cox) The Goal; (1997) Critical Chain, N. River Press

• TOC overview and the thinking tools:– H. William Dettmer (1997) Goldratt’s TOC: a systems approach to cont. improv’t, ASQ

• Related (but differently-specialised) thinking from Agile community:– Alistair Cockburn: A methodology per project, www.crystalmethodologies.org– Mary Poppendieck: Lean development: an agile toolkit,

www.poppendieck.com Thompson informationSystemsConsulting Limited2003

©


Lessons learned, and way forward• Reactions so far:

– “but this looks very like our “Best Practice” training course syllabus”– “this may be too controversial”– “I don’t understand what you’re on about”– “why bother to analyse statements of the obvious?” – “we should be able to come up with something more specific / detailed /

quantitative / prescriptive than this”

• So main lessons so far are: – it may be obvious (and excitingly new) to me, but others differ, and...– the reactions are mutually inconsistent!

• Ways forward:– for me: can the other parts of Goldratt’s thinking also usefully apply to testing?– for you (and therefore also for me!): can you use this framework?


©


• Questions please?

Neil [email protected]+44 (0)7000 NeilTh (634584)23 Oast House CrescentFarnham, Surrey, EnglandGU9 0NP, United Kingdom

STAREast Neil Thompson slide 29 of 29T15

Test Management

Thompson informationSystemsConsulting Limited

© 2003

track presentation

• Contact details...

• Thank you!

www.TiSCL.com

Date post:	16-Apr-2017
Category:	Technology
Upload:	neil-thompson
View:	468 times
Download:	2 times