27
2
BUILDING A CUSTOMER QUALITY DASHBOARD John Ruberto VP of Quality Engineering – Clover, a First Data Company
3
4
First, A story
5
9.86
6
7
What is 9.86? Metric First Negative Link in Google Source Lines of Code 3 Cyclomatic Complexity 19 Function Points 11 Code Coverage 6 Defect Removal Efficiency 6 Defect Density 9 Bug Count 15
8
Principles for metrics • Related to our goals • Leading vs lagging indicators • Process metrics vs outcome metrics • Use the right technology to display
9
Related to our goals
10
Use the right technology to collect & display
11
Provides actionable insights
12
Goal-Questions-Metric • GQM • Victor Basili
• Align on a set of goals • Ask questions about those goals • Design & collect metrics to answer the questions
13
Why • Setting goals, in alignment with the wider organization, gains acceptance
• Focus on what’s most important to your stakeholders
• Provide “line of sight” from your metrics to your goals
• Build comprehensive view of your goals.
14
Example - Context • Software as a service application • > 500K active users • Paying monthly subscription
15
Example • Goal: Deliver better quality to our customers • Questions:
• How many defects do our customers report? • How are we trending on customer reported defects? • How quickly to we fix the defects? • What are the top causes of these defects? • Why aren’t we catching these bugs before release?
16
Example –Delivered Quality
0% 10% 20% 30% 40% 50% 60% 70%
RCA Pareto
0% 5%
10% 15% 20% 25% 30% 35% 40% 45%
Reasons for Escape
17
Characteristics to think about • Process Metrics vs Outcomes
• LOC / Review hour vs Defects found per review
• Leading Indicators vs Lagging Indicators • Code coverage vs delivered quality
• Median vs Average (Better yet: percentile) • Median page load vs Average page load • % fixes within SLA vs Average Age • 2012 average income in San Mateo County
18
Principles in using metrics • Direct measures instead of derived
• “quality score” • Apdex
• Actionable • Total crashes vs crash code pareto
• Live data is best data • No powerpoint…
19
Fallacies of Metrics - Gamification • Goal: Improve Testing Efficiency • Metric: Testing Efficiency: (fixed bugs / total submitted)
20
Fallacies of Metrics – Confirmation Bias • Incoming bug rate improved dramatically – our quality must be outstanding!
Image Credit: By User:KAMiKAZOW (Transferred from en.wikipedia to Commons.) [Public domain], via Wikimedia Commons
21
Fallacies of Metrics – Survivor Bias
Image Credit: WyrdLight.com [CC BY-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons
22
Fallacies of Metrics – Survivor Bias
0
20
40
60
80
100
120
140
160
180
Critical Major Minor
Open bugs By Severity
23
Measurement Bias
2% 3%
5%
8% 10%
3% 3%
8% 9%
12%
5%
13%
20%
30%
35%
29% 28%
24% 24% 22%
33% 33%
28%
21%
18%
28%
20%
15%
8%
3%
0%
5%
10%
15%
20%
25%
30%
35%
40%
Release 1 Release 2 Release 3 Release 4 Release 5
Phase Detection By Release
Rqmts Design Code Int Sys Customer
24
Vanity Metrics • Don’t measure things that matter • Easily manipulated • But, make us feel good
25
Keeping the gains • Process Wrapper
• Monitor & regulate • Automatic trigger • Wide distribution
27
Photo Credits
Nadia Comm: Ben Sutherland https://www.flickr.com/photos/bensutherland/ Bull: By Hollingsworth John and Karen, U.S. Fish and Wildlife Service [Public domain], via Wikimedia Commons Cow: By Keith Weller/USDA (www.ars.usda.gov: Image Number K5176-3) [Public domain], via Wikimedia Commons
