BIO PRESENTATION
International Conference on Software Testing Analysis and Review
October 16-20, 2006 Anaheim, CA USA
F1
10/20/2006 10:00:00 AM
KEEPING IT BETWEEN THE DITCHES: A DASHBOARD TO
GUIDE YOUR TESTING Randy Rice
Rice Consulting Services Inc
Randall W. Rice Randy Rice is a leading author, speaker and consultant in the field of software testing and software quality. Rice, an ASTQB Certified Tester – Foundation Level (CTFL), a Certified Software Quality Analyst and a Certified Software Test Engineer, has worked with organizations worldwide to improve the quality of their information systems and optimize their testing processes. Randy has over 28 years experience building and testing mission-critical projects in a variety of environments and has authored over 20 training courses in software testing and software engineering. He is a popular speaker at international conferences on software testing and is also publisher of The Software Quality Advisor newsletter. He is co-author with William E. Perry of the book, Surviving the Top Ten Challenges of Software Testing and Testing Dirty Systems, published by Dorset House Publishing Co. Randy also serves on the board of directors of the American Software Testing Qualifications Board (ASTQB). In 1990, Randy founded Rice Consulting Services, of which he is Principal Consultant and Trainer.
Keeping It Between the Ditches
A Dashboard to Guide Your Testing
© 2006, Rice Consulting Services, Inc.
2
The Roadmap for This Session
Driver’s Ed. RevisitedThe TechniquesUnderstanding the DashboardOther Things Needed to Keep Testing on Track
3
Driver’s Ed Revisited
Remember what they told us?Keep your eyes on the roadPay full attention to your drivingFasten your seatbeltBe courteousDon’t speedObey all laws
Some of these apply to us in “driving” software testing projects.
4
A Testing Project is Somewhat Like Driving
You need to know your:DestinationLocationOrientation (Direction)ProgressSpeedResource levels (gas, oil, etc.)Engine operation (temp, charge, etc.)
5
The Goal
Arrive at the desired destination safelyStay on the roadMake good progressDon’t get lostDon’t run out of fuelOnly one driver at a time
6
The TechniquesAn effective testing strategy
Defines the test objectives, scope and approach early in the project.
A workable test planDefines scope, resources, schedules, risks, contingencies, etc.
A dashboardMonitors defect levels, test progress, resource levels.
7
The Test StrategyThis is a concise, high-level document that communicates for a specific project:
The purpose of testingThe major test objectivesThe scope of testingWhat is unique about this project?Critical success factorsThe test approach
The test strategy helps ensure you are headed in the right direction.
9
The Test Plan
The test plan is a project plan for the test that describes:
The test objectivesThe scope of testing
What should and should not be testedThe resources neededThe schedule and timelinesRisks and contingenciesPass/fail criteriaGeneral test procedures
The test plan is a roadmap for testing that directs you to your destination.
11
The Test DashboardDashboards are nothing new
This has been a common topic in articles and at conferences for several years.
At the same time, testers often struggle with how to convey accurate and timely information to management.So…let’s explore dashboards and look at some examples.Then, we’ll look at the issues behind test measurement and reporting.
12
The Main Objective
To provide simple, meaningful and reliable information in one place to help guide the testing effort and convey that information to our clients.
13
What is a Testing Dashboard?
A testing dashboard, just like a car’s dashboard, is a set of indicators that show the current status of testing.Dashboards can be seen from various perspectives:
ProjectTestingOngoing system maintenance
14
Why Have a Testing Dashboard?
For fast and easy reporting test results to managementTo have all of your testing information in one placeTo help guide the testing effortTo help make good decisionsTo build project learning
Better estimates in the futureTo build the credibility and visibility of testing
15
What is Required for a Dashboard?
Accurate and meaningful measurements and metricsA culture of trust and opennessNon-intrusive ways to measure
Ideally, the measures should come from activities already being tracked.
Defect tracking systems
16
Types of Dashboards
Low-techWhite boards
SpreadsheetsElaborate
Allows input from distributed teamsExamples
DART (http://public.kitware.com/Dart/HTML/Index.shtml)SPAWAR (U.S. Navy)
17
What is Shown on a Typical Testing Dashboard
CoverageRequirementsFunctionsTest case
StatusOf testingDefect resolutionReadiness for deployment
ProgressBased on test goals and objectivesBlockages
RiskTechnicalBusinessProject
18
What Makes a Good Metric?Simple
Easily measured and understood
Can be automatedSo we don’t have to take time to measure and record manuallyAlso, people don’t get the chance to manipulate the measures
MeaningfulWe can gain useful information to make decisions
Examples of Dashboards
20
Low-Tech
Source: James Bach, “A Low Tech Testing Dashboard”, http://www.satisfice.com/presentations/dashboard.pdf
21
Low-Tech Legend
AreaThis can be any function, feature or attribute to be tested.
EffortRanges from “none” to “ship”
CoverageRanges from “0” (no info) to “3” (most rigorous)
Quality AssessmentSummary of status
CommentsAny additional helpful information
For more details on each dashboard item, see “A Low Tech Testing Dashboard”, by James Bach - http://www.satisfice.com/presentations/dashboard.pdf
22
Low-Tech Dashboard Considerations
Keep the number of areas under 20 or so.Also, try to be consistent in the value of areasArea names should be easily understood
Color coding helpsRed: Bad stuff happeningYellow: WarningGreen: Looking good
Frequencyat each build2 – 5 per week
23
Low Tech Dashboard
AdvantagesEasy to implementEasy to understandEasy to update
ChallengesDistribution of informationMaking sure it doesn’t get erased
24
Spreadsheets
Adapted from, “By the Dashboard Light – Providing Information, Not Data by Johanna Rothman, www.stickyminds.com
25
Spreadsheets (2)
Easily maintainedEasily distributedNot limited by board space
New columns can be easily added
26
Randy’s Graphical Dashboard
80% 70%75% 65%
% Test casesexecuted
% Test casespassed
% Requirementstested
% Requirementstested & passed
Correctness
% Open defect reports
50%
Usability
Security
InteroperabilityCompatibility
Efficiency
Maintainability
% Testerutilization
67%
Portability
27
Kiviat ChartsCorrectness
Reliability
UsabilityPortability
Integration
100% 80% 60% 40% 20%
Each of these areas are desired attributes of the application or system.
Each ring shows the relative score for each attribute.
Example: Six of ten usability tests have passed.
28
Kiviat Charts (2)
Easily understoodShow at a glance:
CoverageStrengths and weaknesses
Can be a part of other dashboardsAre great to show when you don’t have much time in a presentation.
29
Expanding the View
Project DashboardsHave the same characteristics, but more points of measurement.Contain testing measures.Guide the entire project, not just testing.
30
Sample Project Dashboard
Source: U.S. Navy - sepo.spawar.navy.mil/Metrics.ppt
31
Sometimes You Need A Navigator
I use a GarminStreetPilot 2820 GPS navigator.You might need a mentor, a consultant, etc.
Don’t be afraid to ask for help.However, the guide could be wrong!
32
Words of WarningToo many items on a dashboard can be distracting and confusing.
Unless you are flying a plane!Metrics can be abused.
If people don’t understand human behavior, more harm than good can result.
Stuff happens.Things not shown on your dashboard can derail your test.
33
Other ConcernsIs the testing process working as desired?Are we keeping our eyes on the objectives?Are there multiple people trying to drive at one time?Are we avoiding the potholes?
34
Keeping the Process Working
The dashboard tells you about vehicle (process) malfunctions.
In testing, the process is the engine.
The process might not be documented.How you perform the process determines whether or not you reach the intended destination.
35
Keeping Your Eyes on the Road
Test objectives and plans keep us on track…
If we pay attention to them!
36
Dealing With Multiple DriversThere can be team input, but ultimately only one leader.When things start to go wrong, people tend to lose confidence in the leader.
This means the leader must:Make the right correctionsReassure the teamSeek input from the teamBe decisive
37
Common Potholes
Excessive defect levels“Out of scope” distractionsUnexpected changes
ReorganizationsApplication/System changesProject changes
Team strife“Be courteous”
38
Final Thought
A key purpose of testing is to provide meaningful information to management to make informed decisions.
39
Other Resources
By the Dashboard Light - Providing Information, Not Data by Johanna Rothman
www.stickyminds.com
A Low-Tech Testing Dashboard by James Bach
http://www.satisfice.com/presentations/dashboard.pdf
F1.pdfKeeping It Between the DitchesThe Roadmap for This SessionDriver’s Ed RevisitedA Testing Project is Somewhat Like DrivingThe GoalThe TechniquesThe Test StrategyThe test strategy helps ensure you are headed in the right direction.The Test PlanThe test plan is a roadmap for testing that directs you to your destination.The Test DashboardThe Main ObjectiveWhat is a Testing Dashboard?Why Have a Testing Dashboard?What is Required for a Dashboard?Types of DashboardsWhat is Shown on a Typical Testing DashboardWhat Makes a Good Metric?Examples of DashboardsLow-TechLow-Tech LegendLow-Tech Dashboard ConsiderationsLow Tech DashboardSpreadsheetsSpreadsheets (2)Randy’s Graphical DashboardKiviat ChartsKiviat Charts (2)Expanding the ViewSample Project DashboardSometimes You Need A NavigatorWords of WarningOther ConcernsKeeping the Process WorkingKeeping Your Eyes on the RoadDealing With Multiple DriversCommon PotholesFinal ThoughtOther Resources
CoverBioPresentation
