Copyright HDR 2007 [email protected] 1 How To Measure Anything: Finding the Value of...

Copyright HDR 2007 Copyright HDR 2007 [email protected]@hubbardresearch.com

11

How To Measure Anything:

Finding the Value of ‘Intangibles’ in Business


22

How to Measure Anything• It started 12 years ago…• I conducted over 60 major risk/return

analysis projects so far that included a variety of “impossible” measurements

• I found such a high need for measuring difficult things that I decided I had to write a book

• The book was released in July 2007 with the publisher John Wiley & Sons

• This is a “sneak preview” of many of the methods in the book


33

How To Measure Anything• “I love this book. Douglas Hubbard helps us create a path to know the answer to almost any question, in business, in science or in life.” Peter Tippett, Ph.D., M.D. Chief Technology Officer at CyberTrust and inventor of the first antivirus software

• “Doug Hubbard has provided an easy-to-read, demystifying explanation of how managers can inform themselves to make less risky, more profitable business decisions.” Peter Schay, EVP and COO of The Advisory Council

• “As a reader you soon realize that actually everything can be measured while learning how to measure only what matters. This book cuts through conventional clichés and business rhetoric and it offers practical steps to using measurements as a tool for better decision making.” Ray Gilbert, EVP Lucent

• “This book is remarkable in it's range of measurement applications and it's clarity of style. A must read for every professional who has ever exclaimed ‘Sure, that concept is important but can we measure it?’” Dr. Jack Stenner, CEO and co-founder of MetaMetrics, Inc.

CFO Measurement Problem1. The Risk Paradox: The largest and riskiest decisions

often get the least quantitative risk analysis

2. The Measurement Inversion: According to an economic valuation of the benefits of a measurement, most measurement priorities are the opposite of the optimal solution.

3. Better alternatives than: – Traditional business cases that don’t quantify

uncertainty, risks, and intangibles – “Scores” that quantify nothing


44

Style vs. Substance• If you are adding and multiplying subjective “scores” on a scale of 1-

5 for things like risk, alignment, etc. chances are your method doesn’t improve on your intuition

• Also don’t be fooled by the terms “structured” or “formal” (Astrology is both structured and formal, it just doesn’t work)

• The Following Charts Mean Nothing:


55

0

2

4

6

8Strategy

Efficiency

EffectivenessCustomer Value

Alignment

Relationship Process

Innovation

66

Assessing Assessment Methods

• “Proven” should mean more than some previous users feel good about it (the “testimonial proof”)

• Only empirical evidence that forecasts and decisions are actually improved can separate real benefits from a “placebo effect”

• Effective methods for evaluating IT investments should have a lot in common with well-known methods in other fields (actuarial science, portfolio optimization, etc.)



77

My Three Measurement “Heroes”

• Eratosthenes – measured the Earth’s circumference to within 1% accuracy

• Enrico Fermi – the physicist who used “Fermi Questions” to break down any uncertain quantity (and was the first to estimate the yield of the first atom bomb)

• Emily Rosa – the 11 yr old who was published in JAMA (youngest author ever) for her experiment that debunked “therapeutic touch”


88

Three Illusions of Intangibles(The “howtomeasureanything.com” approach)

• The perceived impossibility of measurement is an illusion caused by not understanding:– the Concept of measurement– the Object of measurement– the Methods of measurement

• See my “Everything is Measurable” article in CIO Magazine (go to “articles” link on www.hubbardresearch.com


99

An Approach• Model what you know now• Compute the value of additional

information• Where economically justified, conduct

observations that reduce uncertainty• Update the model and optimize the

decision


1010

Uncertainty, Risk & Measurement• Measuring Uncertainty, Risk and the Value of Information are closely

related concepts, important measurements themselves, and precursors to most other measurements

• The “Measurement Theory” definition of measurement: “A measurement is an observation that results in information (reduction of uncertainty) about a quantity.”

• We model uncertainty statistically – with Monte Carlo simulations

A Few of My Examples• Risk of IT• The value of better information• The value of better security• Forecasting the demand for space tourism• Forecasting fuel for Marines in the battlefield• Measuring the effectiveness of combat training to reduce

roadside bomb/IED casualties• The Risk of obsolescence• The value of a human life• The value of saving an endangered species• The value of public health• The value of IQ points lost by children exposed to Methyl-

Mercury


1111

1212

Calibrated Estimates• Decades of studies show that most managers are

statistically “overconfident” when assessing their own uncertainty

• Studies also show that measuring your own uncertainty about a quantity is a general skill that can be taught with a measurable improvement

• Training can “calibrate” people so that of all the times they say they are 90% confident, they will be right 90% of the time


1313

Giga Analysts

Giga Clients

Statistical Error

“Ideal” Confidence

30%

40%

50%

60%

70%

80%

90%

100%

50% 60% 80% 90% 100%

25

75 71 65 58

21

17

68152

65

4521

70%

Assessed Chance Of Being Correct

Per

cent

Cor

rect

99 # of Responses

• 16 IT Industry Analysts and 16 CIO’s , the analysts were calibrated• In January 1997, they were asked To Predict 20 IT Industry events• Example: Steve Jobs will be CEO of Apple again, by Aug 8, 1997 - True or False?

1997 Calibration Experiment

Source: Hubbard Decision Research


The Value of Information


1414

EVI p r V p r V p r V p r EVi j j ij

z

j j i l j j ij

z

j

z

i

k

( ) max ( | ), ( | ),... ( | ), *, , ,1

12

111

What it means:1.Information reduces uncertainty2.Reduced uncertainty improves decisions3.Improved decisions have observable consequences with measurable value

-0.5-0.4-0.3-0.2-0.1

00.10.20.30.40.5

10

2

46

8

11

0.20.4

0.60.8

.010.1

0.040.06

0.08

100

20

40

6080

10

100

20

4060

80

102468

1

10

0.20.40.60.81

0.1 0.05

I use macro in Excel for this formula. In the book, I made a simple table that can estimate it with some simple multiplication


1515

Next Step: Observations• Now that we know what to measure, we

can think of observations that would reduce uncertainty

• The value of the information limits what methods we should use, but we have a variety of methods available

• Take the “Nike Method”: Just Do It – don’t let imagined difficulties get in the way of starting observations


1616

Some Useful Suggestions

• It has been done before• You have more data than you think• You need less data than you think• It is more economical than you think

The “Math-less” Statistics Table• Measurement is based on

observation and most observations are just samples

• Reducing your uncertainty with random samples is not made intuitive in most statistics texts

• This table makes computing a 90% confidence interval easy


1717


1818

Measuring to the Threshold• Measurements have

value usually because there is some point where the quantity makes a difference

• Its often much harder to ask “How much is X” than “Is X enough” Samples Below Threshold

20%

30%

40%

50%

0.1%

1%

10%

4 5 6 7 8 9 10

2 4 6 8 10 12 16 20Number Sampled

Cha

nce

the

Med

ian

is B

elow

the

Thr

esh

old

1 2 3

1814

2%

5%

0.2%

0.5%

0


1919

Statistics Goes to War

• Several clever sampling methods exist that can measure more with less data than you might think

• Examples: estimating the population of fish in the ocean, estimating the number of tanks created by the Germans in WWII, extremely small samples, etc.


2020

Reducing Inconsistency• The “Lens Model” is another method used to improve on expert intuition• The chart shows the reduction in error from this method on intuitive estimates• In every case, this method equaled or bettered the judgment of experts

0% 10% 20% 30% 40%

Cancer patient life-expectancy

Life-insurance salesrep performance

Graduate students grades

Changes in stock prices

Mental illness using personality tests

Student ratings of teaching effectiveness

IQ scores using Rorschach tests

Psychology course grades

Business failures using financial ratios

Reduction in Errors

Battlefield Fuel Forecasts

IT Portfolio PrioritiesMy StudiesMy Studies

Source: Hubbard Decision Research


2121

The Simplest Method

• Bayesian methods in statistics use new information to update prior knowledge

• Bayesian methods can be even more elaborate that other statistical methods BUT…

• It turns out that calibrated people are already mostly “instinctively Bayesian”


2222

Comparison of Methods

Calibrated Estimator

Bayesian

Typical Un-calibrated

Estimator

Non-BayesianStatistics

Ignores Prior Knowledge; Emphasizes new data

Ignores New data; Emphasizes Prior

Knowledge

StubbornGullible

Under-confident (Stated uncertainty is higher than

rational)

Overconfident (Stated uncertainty is lower than

rational)

CautiousVacillating,Indecisive

CalibratedSkeptic


2323

Risk/ROI w/ “Monte Carlo”

Administrative Cost Reduction

Total Project Cost

Customer Retention Increase

5% 10% 15%

10% 20% 30%

$2 million $4 million $6 million

ROI-50% 50% 100%0%

• A Monte Carlo simulation generates thousands of random scenarios using the defined probabilities and ranges

• The result is a range ROI not a point ROI

Quantifying Risk Aversion


2424

Acceptable Risk/Return Boundary

Investment Region

The simplest element of the Nobel Prize-winning method “Modern Portfolio Theory” is documenting how much risk an investor accepts for a given return

Investment

Define Decision Model

Define Decision Model

Calibrate EstimatorsCalibrate

Estimators

Conduct Value of Information Analysis (VIA)

Conduct Value of Information Analysis (VIA)

Measure according to VIA

results and update model

Measure according to VIA

results and update model

Populate Model with Calibrated

Estimates & Measurements

Populate Model with Calibrated

Estimates & Measurements

Analyze Remaining Risk

Analyze Remaining Risk

Optimize DecisionOptimize Decision

Approach Summary

2525Copyright HDR 2007 Copyright HDR 2007 [email protected]@hubbardresearch.com


2626

Final Tips• Learn how to think about uncertainty, risk and

information value in a quantitative way• Assume its been measured before• You have more data than you think and you

need less data than you think• Methods that reduce your uncertainty are more

economical than many managers assume• Don’t let “exception anxiety” cause you to avoid

any observations at all• Just do it

2727

Questions?

Doug Hubbard

Hubbard Decision Research

[email protected]

www.hubbardresearch.com

630 858 2788


Date post:	26-Mar-2015
Category:	Documents
Upload:	ella-duffy
View:	219 times
Download:	0 times

Copyright HDR 2007 [email protected] 1 How To Measure Anything: Finding the Value of...

Documents