Date post: | 07-Jul-2015 |
Category: |
Technology |
Upload: | techwellpresentations |
View: | 49 times |
Download: | 0 times |
4/16/2013
1
1
Lee CopelandSoftware Quality Engineering
The Mismeasure of Software
The Last Talk on Measurement You’ll Ever Need to Hear
22
Topics
• What is measurement?
• The “Four Don’ts” of measurement
• My favorite indicator—isn’t a measurement
• Lee’s Zeroth Law of Measurement
4/16/2013
2
33
What Is Measurement?
• “Measurement is the empirical, objective assignment of numbers, according to a rule derived from a model or theory, to attributes of objects or events with the intent of describing them”
–Cem Kaner
44
A Classic Example
• Phrenology
– The brain is the organ of the mind
– The brain is not a homogenous entity, but an aggregate of “mental organs” with specific functions
– These organs are topographically localized
4/16/2013
3
55
A Classic Example
• Phrenology
– The relative size of any particular mental organ indicates the power or strength of that organ
– External measurement can be used to determine the internal states of our mental powers
66
A Volunteer From The Audience?
4/16/2013
4
77
A Classic Example
• Even automated devices for phrenological analysis were created. The psychographnetted its inventors about $200,000 at the 1934 Century of Progress Exposition in Chicago
• Some employers even required phrenological certificationsbefore hiring employees
88
What Is Measurement?
• Did you notice that Kaner’s definition only requires intent. It says nothing about the measurement actually being useful?
4/16/2013
5
99
Two Causes of Measurement Failure
• Map-territory error
• Reification error
1010
What Is Measurement?
• “Measurement is the empirical, objective assignment of numbers, according to a rule
derived from a model or theory, to attributes
of objects or events with the intent of describing them”
–Cem Kaner
4/16/2013
6
1111
Map-Territory Error
• A common error is the confusion of a model with reality
• “The map is not the territory”
– Alfred Korzybski, 1931
• Mathematical or simulation models may helpus understand a system or situation but real life always differs from the model
1212
Map-Territory Error
• The model / theory / map may just be WRONG!
4/16/2013
7
1313
What Is Measurement?
• “Measurement is the empirical, objective assignment of numbers, according to a rule derived from a model or theory, to attributes of
objects or events with the intent of
describing them”
–Cem Kaner
1414
Reification Error
• A Reification error occurs when an abstraction (abstract belief or hypothetical construct) is treated as if it were a physical object or a real event
• Counting them is dangerous! Comparing the counts is even more dangerous!!!
4/16/2013
8
1515
Reification Error
• Examples include:
– Intelligence
– Performance
– Commitment
– Motivation
– Leadership
– Attitude
1616
Reification Error
• Now do you understand why you feel so violated after your Annual Performance Review?
4/16/2013
9
1717
But Wait, There’s More !
• Consider these reification errors:
– Requirements
– Test cases
– Defects
1818
Topics
• What is measurement?
• The “Four Don’ts” of measurement
• My favorite indicator—isn’t a measurement
• Lee’s Zeroth Law of Measurement
4/16/2013
10
1919
The Four Don’ts
Don’t #1
Don’t #3
Don’t #2
Don’t #4
2020
Don’t #1
• Don’t measure it if you don’t know what it means
4/16/2013
11
2121
Don’t #1
• Does the number actually represent the attribute we are trying to measure?
• Examples:
• Lines of code – program size
• Number of test cases – thoroughness
• Number of defects found – product quality
• Superclass:Subclass ratio – good systemdesign
2222
Don’t #2
Thanks for all your
hard work producing
this measurement
report
• Don’t measure it if you’re not going to do something because of the measurement
4/16/2013
12
2323
Don’t #2
• What does this tell your staff?
– You’re a clown
– You’re an idiot
– You’re an H
2424
Don’t #3
• Don’t turn your measurement into a goal
• Goodhart’s Law — Proposed in 1975 by Charles Goodhart—a former advisor to the Bank of England and Emeritus Professor at the London School of Economics
4/16/2013
13
2525
Don’t #3
• The law states that “once an indicator is made a target for the purpose of guiding policy, then it will lose the information content that originally made it useful”
• Goodhart drones on, “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes”
2626
Don’t #3
• Professor Marilyn Strathern has restated Goodhart's Law more succinctly and more generally:
“When a measure becomes a target, it ceases to be a good measure”
4/16/2013
14
2727
Don’t #3
• A classic metric of this type is “lines of code written per day.” When used as a measure, this metric can be valuable for estimating, planning, and tracking
• However, when LOC/day becomes a goal, developers may be enticed to write more lines of (less efficient) code
2828
Don’t #3
For example, a classic Java “for” loop has the generic form:
for (initialization;
termination; increment) {
statement(s)
}
An example of this loop is:
for (int i=1; i<11; i++) {System.out.println("Count is: " + i);}
4/16/2013
15
2929
Don’t #3
If writing more lines of code is rewarded, I can write the equivalent code as:
int i;i=1; while (i<11)
{System.out.println("Count is: " + i);i++;}
thus increasing the LOC count while providing the identical functionality
3030
Don’t #3
• Goodhart’s Law reminds us that when rewards and punishments are connected to the achievement of specific goals, unintended consequences are often the result
4/16/2013
16
3131
Don’t #3
• If LOC is too “old-school”, consider Velocity, a premier agile metric
• What have you seen happening with Velocity?
Goal
3232
Don’t #4
• Don’t focus on measuring effort– Number of test cases created– Number of test cases executed– Number of test cases passed– H
4/16/2013
17
3333
Don’t #4
• Instead, focus on measuring accomplishments– Number of defects discovered before
installation– Costs saved through testing– H
3434
Topics
• What is measurement?
• The “Four Don’ts” of measurement
• My favorite indicator—isn’t a measurement
• Lee’s Zeroth Law of Measurement
4/16/2013
18
3535
My Favorite Indicator
• Lines of code
• Percent test cases passed
• Defect density
• Defect detection percentage
3636
My Favorite Indicator
• My favorite indicator is a
4/16/2013
19
3737
My Favorite Indicator
is
3838
My Favorite Indicator Is
• Surprise
– “To discover suddenly and unexpectedly, without warning”
– “To become aware of something not previously perceived”
– “A spontaneous awareness of a discontinuity in our understanding”
4/16/2013
20
3939
My Favorite Indicator Is
• Surprise is an interesting indicator for use in software development
• We could examine the surprise itself
• We could examine the reason(s) for our surprise
4040
Satir Interaction Model
Virginia Satir
4/16/2013
21
4141
Intake
• Intake is the process of observation, of bringing part of the “world out there” into our consciousness
4242
Intake Surprises
• As humans, we fail to observe huge amounts of information; there is simply too much
• Some individuals and organizations maintain an “oblivious culture” – they chose not to systematically observe anything
4/16/2013
22
4343
Intake Surprises
• A second type observes, but quickly filters out data that does not match their world-view
• A third, to prevent having to deal with realities, actually prohibits observing
4444
Meaning
• Meaning is the process of mapping our observations to our context, experience, and feelings
• Biases, agendas, pressures, expectations, and even past unresolved personal issuescan cloud our assignment of meaning
4/16/2013
23
4545
Meaning Surprises
• If not careful, we may assume that the first meaning we assign is the correct meaning is the only meaning
• Remember Weinberg’s Rule of Three – “if you can’t think of at least three different meanings, you haven’t thought enough about it”
4646
Significance
• Significance is the process of deciding whether the observation and meaning are important enough to affect our actions
4/16/2013
24
4747
Significance Surprises
• We may fail to assign the proper significance because:
– the observation does not fit into our previous experience
– we just can’t evaluate its importance
– we may be operating under rules that do not serve us well
– we may have been conditioned to minimize the significance
4848
Significance Surprises
4/16/2013
25
4949
Surprise
• Focus on your processes of Intake, Meaning, and Significance
• To learn something, examine the surprise
• To learn even more, examine the reasons why you were surprised
5050
Topics
• What is measurement?
• The “Four Don’ts” of measurement
• My favorite indicator—isn’t a measurement
• Lee’s Zeroth Law of Measurement
4/16/2013
26
5151
Lee’s Zeroth Law of Measurement
5252
Lee’s Zeroth Law of Measurement
• “If you measure the wrong thing, and reward the wrong thing, don’t be surprised if you get the wrong thing”
FamousGuru
&Grandpa
4/16/2013
27
5353
Recap
• What is measurement?
• The “Four Don’ts” of measurement
• My favorite indicator—isn’t a measurement
• Lee’s Zeroth Law of Measurement
5454
Thanks
Shamelesscommercial
message