Better Testing

transcript

better testingbetter testing

who are we?who are we?

We ship software.

We have operational authority over the process, the “how.” And we are held accountable for the consequences of our choices.

We ship software.

We have operational authority over the process, the “how.” And we are held accountable for the consequences of our choices.

matthew 8:9matthew 8:9

“For I also am a man under authority, having under myself soldiers: and I say to this one, Go, and he goeth; and to another, Come, and he cometh; and to my servant, Do this, and he doeth it.”

what we won’t saywhat we won’t say

We aren’t going to explain why we should test. We aren’t going to talk about how to become a QA manager. We’re looking for the extra edge.

This is about how to improve the testing we already do and already trust.

We aren’t going to explain why we should test. We aren’t going to talk about how to become a QA manager. We’re looking for the extra edge.

This is about how to improve the testing we already do and already trust.

psyche: hardly any metrics

In the South Seas there is a cargo cult of people…

Richard Feynman

In the South Seas there is a cargo cult of people…

Richard Feynman

“The function of QA is to know (and articulate) the quality of the product at all times in the development cycle.”

Jim McCarthy

“The function of QA is to know (and articulate) the quality of the product at all times in the development cycle.”

Jim McCarthy

“A management report is a report that is used to make a management decision. All other reports are window dressing.”

Peter Holden

“A management report is a report that is used to make a management decision. All other reports are window dressing.”

Peter Holden

holden defines ‘better testing’

Any test that provides information used to make a management decision is better than a test that only provides window dressing.

the management decisionthe management decision

ship, continue, or quit?ship, continue, or quit?

This is the fundamental management decision in software development. Even if you feel you have no choice about quitting, you must be able to articulate whether you can ship.

We don’t go on because it’s ready;we go on because it’s 11:30.”

Lorne Michaels

We don’t go on because it’s ready;we go on because it’s 11:30.”

Lorne Michaels

testing readinesstesting readiness

How many bugs?

How much functionality?

How many bugs?

How much functionality?

the end?the end?

You have a plan. Execute on it.

If you don’t ship, you must not be trying hard enough. Right?

You have a plan. Execute on it.

If you don’t ship, you must not be trying hard enough. Right?

the planthe plan

Measure whether we can ship at regular intervals. When the answer is ‘ship,’ ship.

What’s the problem?

Measure whether we can ship at regular intervals. When the answer is ‘ship,’ ship.

What’s the problem?

ass, you, meass, you, me

The plan assumes that the software becomes ‘more ready’ over time.

Must defects fall over time?

Must functionality increase towards readiness over time?

The plan assumes that the software becomes ‘more ready’ over time.

Must defects fall over time?

Must functionality increase towards readiness over time?

sotto vocesotto voce

Some teams do not report how much the software does. They report the discrepancy between what the software does and what they want it to do.

If your team works this way, your management is more complex than teams that report what the software does. But this doesn’t affect the choice of tests and test processes.

Some teams do not report how much the software does. They report the discrepancy between what the software does and what they want it to do.

If your team works this way, your management is more complex than teams that report what the software does. But this doesn’t affect the choice of tests and test processes.

are we converging?are we converging?

Are defects falling over time? Are we confident they will continuing to fall? Do we know which activities present the greatest risk of introducing defects?

Is functionality increasing? Are we climbing ladders or slipping on snakes?

Are defects falling over time? Are we confident they will continuing to fall? Do we know which activities present the greatest risk of introducing defects?

Is functionality increasing? Are we climbing ladders or slipping on snakes?

ass, you, me tooass, you, me too

The plan assumes that we can tell how many defects are present with an acceptable certainty.

The plan assumes that we can tell how much functionality is present with acceptable certainty.

The plan assumes that we can tell how many defects are present with an acceptable certainty.

The plan assumes that we can tell how much functionality is present with acceptable certainty.

add uncertainty to readiness

How many bugs with what level of certainty?

How much functionality with what level of certainty?

How many bugs with what level of certainty?

How much functionality with what level of certainty?

testing, revisitedtesting, revisited

Measure how many defects and how much functionality we have.

Articulate convergence towards readiness and uncertainty about readiness.

Measure how many defects and how much functionality we have.

Articulate convergence towards readiness and uncertainty about readiness.

a better plana better plan

Measure whether we can ship at regular intervals. Ship if we’re ready with acceptable certainty.

Change the production process if we aren’t converging. Change the testing process if we aren’t certain about our state.

Measure whether we can ship at regular intervals. Ship if we’re ready with acceptable certainty.

Change the production process if we aren’t converging. Change the testing process if we aren’t certain about our state.

testing readinesstesting readiness

Everyone’s an expert on whether the software is ready.

My sole tip today is to get everyone on the same page.

Nothing smarts more than shipping the wrong application.

Everyone’s an expert on whether the software is ready.

My sole tip today is to get everyone on the same page.

Nothing smarts more than shipping the wrong application.

getting on the same pagegetting on the same page

Shared and/or executable requirements. See FITnesse.

Screen shots are good. Movies are better. The key is to have development and QA testing the same thing.

That means that developers need to be able to reproduce every bug. If they can’t, investigate: there’s a process or environment problem.

Shared and/or executable requirements. See FITnesse.

Screen shots are good. Movies are better. The key is to have development and QA testing the same thing.

That means that developers need to be able to reproduce every bug. If they can’t, investigate: there’s a process or environment problem.

testing convergencetesting convergence

You must test consistently over time.

You must test against the source repository with precision. This allows development to manage their risks.

You must test consistently over time.

You must test against the source repository with precision. This allows development to manage their risks.

snakes or ladders?snakes or ladders?

You must regression test: you must know whether you are on a snake or a ladder.

Report regressions/persists/come-backs separately from new bugs. Use this information to manage production activities.

You must regression test: you must know whether you are on a snake or a ladder.

Report regressions/persists/come-backs separately from new bugs. Use this information to manage production activities.

managing uncertaintymanaging uncertainty

We manage uncertainty around readiness by choosing how much to test, what to test, and when to test it.

And we build uncertainty into our management report.

We manage uncertainty around readiness by choosing how much to test, what to test, and when to test it.

And we build uncertainty into our management report.

what to test?what to test?

Start with the acceptance tests for promised functionality.

Add regression tests for every bug as you go along.

It’s really that simple.

Start with the acceptance tests for promised functionality.

Add regression tests for every bug as you go along.

It’s really that simple.

sumsum

Tie every test to a management decision. If it isn’t tied to a management decision, either start managing or drop the test.

Tie every management decision to tests. If it isn’t ties to tests, either stop deciding or start testing.

Tie every test to a management decision. If it isn’t tied to a management decision, either start managing or drop the test.

Tie every management decision to tests. If it isn’t ties to tests, either stop deciding or start testing.

overtimeovertime

what tests support these decisions?

Adding a function to the software without slipping the date?

Trying to work harder/smarter so we have higher quality?

“Easter? No way. We need this March 15th or you’ll be writing your résumé!”

Adding a function to the software without slipping the date?

Trying to work harder/smarter so we have higher quality?

“Easter? No way. We need this March 15th or you’ll be writing your résumé!”

what decisions do we make on these tests?what decisions do we make on these tests?

Time between bug injection and detection?

Unit test code coverage and acceptance test code coverage?

Average number of come-backs per bug? Ratio of come-backs to bugs?

Average bug age?

Time between bug injection and detection?

Unit test code coverage and acceptance test code coverage?

Average number of come-backs per bug? Ratio of come-backs to bugs?

Average bug age?

thanks!thanks!

Better Testing

Documents