KEY IDEA

It Boils Down To…

YOU: Skills, equipment, experience, attitude

THE BALL: The product, testing tasks, bugs

YOUR TEAM: Coordination, roles, support

THE GAME: Risks, rewards, project environment, corporate environment, your mission as a tester

YOUR MOVES: How you spend your attention and energy to help your team win the game.

Rapid Testing

Develop your scientific mind. Know your coverage and evaluation strategy. Run crisp test cycles that focus first on areas of risk. Use a diversified test strategy that serves the mission. Assure that your testing fits the logistics of the project. Let your tests evolve as new information comes to light.

Copyright © 1996-2001, Satisfice, Inc.

v1.0

Rapid Test Management

Establish a strong and supportive testing role. Re-visit your strategy and mission every day. Advocate for bugs and testability. Maintain three lists: risks, issues, coverage. Test with your team. Continuously report the status and dynamics of

testing.

KEY IDEA

This is our role.We make informed decisions about quality possible.

We can do this under a variety of conditions.

Testers light the way.

This is the formula…

You must demand… complete specs and quantifiable criteria. protected schedule and early involvement. zero defect philosophy and control over release. resources to achieve complete test coverage.

…for hypochondria.

Look, testing is a difficult job.

“The perfect cure for hypochondria, 100 percent effective, is to contract a potentially fatal disease. It cures you instantly. It happened to me.”

Gene WeingartenThe Hypochondriac's Guide to Life and Death

That’s why they hire smart people, like us, to do it for them.

Too many textbooks seem to thinktesters are wind-up toys.

“This model identifies a standards-based life cycle testing process that concentrates on developing formal test documentation to implement repeatable structured testing on a software or hardware/software system. The general intent is that the test documentation be developed based on a formal requirements specification document...Once the documentation is developed, the test is executed.”

-- from a real article about testing. (I added the boldfacing to emphasize instructions)

Where’s the tester in this picture?

Testing skills compensate for a difficult project environment.

complete specs quantifiable criteria protected schedule early involvement zero defect philosophy control over release complete test coverage

– implicit specs & inference– meaningful criteria– risk-driven schedule– good working relationship– good enough quality– don’t be the gatekeeper– enough information

Instead of this… consider this.

Goal setting Risk identification

Task planning Risk planning

Task completion Risk reduction

failure

failure

success

success

We think about failure,and that can be “negative”.

Eight Commitmentsworth making to developers

We’ll test your code as soon as we can after it’s built. We’ll test important things first, and focus on important problems. We’ll write clear, thoughtful, and respectful problem reports. We’ll try not to be a bottleneck for development. We’ll tell you how we’re testing, and consider your suggestions. We’ll look for ways to test better and faster. We’ll try to accommodate how you like to work. We will not waste your time.

Our job is information.If they don’t want information…

Stop fretting and just be as helpful as you can. Let events play out. Carefully record what happens. Detach yourself from the outcome and refocus

on improving the next project, instead. Help the team use the experience as a

reference point for improvement.

use the Reality Steamroller technique:

KEY IDEA

Public Relations Problem

“What’s the status of testing?”

“What are you doing today?”

“When will you be finished?”

“Why is it taking so long?”

“Have you tested _______, yet?”

The Problem

Management has little patience for detailed test status reports.

Management doesn’t understand testing. Testing is confused with improving. Testing is considered a linear, independent task. Testing is assumed to be exhaustive. Testing is assumed to be continuous. Test results are assumed to stay valid over time. Impact of regression testing is not appreciated. Test metrics are hard to interpret.

A Solution

Report test cycle progress in a simple, structured way...

...that shows progress toward a goal... ...manages expectations... ...and inspires support... ...for an effective test process.

No process?

No Problem!Easy setup

The Dashboard Concept

Project conference roomLarge dedicated whiteboard

“Do Not Erase”

Project status meeting

Test Cycle Report

Test Effort

Test Coverage

Quality Assessment

Time

Product Areas

vs.

vs.

Areafile/editviewinsertformattoolsslideshowonline helpclipartconvertersinstallcompatibilitygeneral GUI

Efforthighlowlowlowblockedlowblockednonenonestart 3/17start 3/17low

C.11+22+12011003

Q. Comments

1345, 1363, 1401

automation broken

crashes: 1406, 1407

animation memory leak

new files not delivered

need help to test...

need help to test...

lab time is scheduled

Testing Dashboard Updated: Build:

2/21 38

Product Area

15-30 areas (keep it simple) Avoid sub-areas: they’re confusing. Areas should have roughly equal value. Areas together should be inclusive of

everything reasonably testable. “Product areas” can include tasks or risks-

but put them at the end. Minimize overlap between areas. Areas must "make sense" to your clients,

or they won’t use the board.

Areafile/editviewinsertformattoolsslideshowonline helpclipartconvertersinstallcompatibilitygeneral GUI

Test Effort

None

Start

Low

High

Pause

Blocked

Ship

Not testing; not planning to test.

Regression or spot testing only; maintaining coverage.

Focused testing effort; increasing coverage.

No testing yet, but expect to start soon.

Temporarily ceased testing, though area is testable.

Can’t effectively test, due to blocking problem.

Going through final tests and signoff procedure.

Test Effort

Use red to denote significant problems or stoppages, as in blocked, none, or pause.

Color ship green once the final tests are complete and everything else on that row is green.

Use neutral color (such as black or blue, but pick only one) for others, as in start, low, or high.

Test Coverage

0

1

1+

2

2+

3

We don’t have good information about this area.

More than sanity, but many functions not tested.

Common & Critical:

Sanity Check:

Some data, state, or error coverage beyond level 2.

Complex Cases:

all functions touched; common & critical tests executed.

strong data, state, error, or stress testing.

major functions & simple data.

Test Coverage

Color green if coverage level is acceptable for ship, otherwise color black.

Level 1 and 2 focus on functional requirements and capabilities: can this product work at all?

Level 2 may span 50%-90% code coverage. Level 2+ and 3 focus on information to judge

performance, reliability, compatibility, and other “ilities”: will this product work under realistic usage?

Level 3 or 3+ implies “if there were a bad bug in this area, we would probably know about it.”

Quality Assessment

“We know of no problems in this area that threaten to stop ship or interrupt testing, nor do we have any definite suspicions about any.”

“We know of problems that are possible showstoppers, or we suspect that there are important problems not yet discovered.”

“We know of problems in this area that definitely stop ship or interrupt testing.”

Use the comment field to explainanything colored red, or any non-green

quality indicator.

Comments

Problem ID numbers. Reasons for pausing, or delayed start. Nature of blocking problems. Why area is unstaffed.

Using the Dashboard

Updates: 2-5/week, or at each build, or prior to each project meeting.

Progress: Set expectation about the duration of the “Testing Clock” and how new builds reset it.

Justification: Be ready to justify the contents of any cell in the dashboard. The authority of the board depends upon meaningful, actionable content.

Going High Tech: Sure, you can put this on the web, but will anyone actually look at it???