Analyzing+Opportunities+ +AO 27July

7/28/2019 Analyzing+Opportunities+ +AO 27July

1/30

Analyzing Opportunities

In 2004, two Serbian men and a Hungarian woman won $2 million from the Ritz Casino in

London. Luck? Robbery? Or simply smart betting? How about a combination of all three. The

team had a small camera that scanned the roulette wheel and the position of the ball when it was

released by the croupier and sent this information to a remote location. Based on historical data

of that croupier and a simulation model of the mechanics underlying the movement of the ball, a

workstation recommended bets. Was the system able to perfectly forecast the outcome of the

roulette? No, but it was allegedly able to improve the gambling odds from the traditional 37 to 1

to 6 to 1. In other words, by making use of data analysis and a clever process, a losers gamble

was turned around into a profitable businessat least until the casino caught on.

Financial evaluations of innovation opportunities at times can feel like betting in a casino. After

all, many innovations fail, and most will not pay back their investment. But by following the

methods that we outline below, you can emulate that trio of London gamblers and begin to turn

the odds in your favor. By incorporating into your forecasts information about past innovations

and the innovation process, you can increase your likelihood of success.

version 080907

Horizon 1 Horizon 2 Horizon 3

Nature of

Uncertainty

Type of

Uncertainty

Example of a

Market

Uncertainty

Example of a

Technology

Uncertainty

Uncertainty about a

number, such as a sales

forecast

Uncertainty about a

scenario, such as an

approval or a cancellation

Many types; not even clear

which type matters

We think that sales volume

for the new product will be

between 120k and 150kper year.

We think there exists a 40%

probability that the results

of our pilot market warranta full launch of the new

product. Otherwise, the

product will be cancelled.

The market for transportation

devices smaller than cars seems to

have big potential. However, weneither understand the exact user

needs to peoples willingness to pay

We might be able to improve the

efficacy of drug x by customizing

the treatment to the patient.However, we dont know how to go

about this customization.

We think that the drug will

have a 20%-30% higher

efficacy than the drugcurrently on the market

We think that the drugs

toxicity might require the

termination of all clinicaltrials with a probability

of 20%.

Horizon 1

Horizon 2

Exhibit ESTIMATION FRAMEWORK: The three different types of opportunitiesdeal with different types of uncertainty requiring different analytical tools in theirevaluation.


2/30

AO-2

As we discussed in the previous chapter, innovations can be classified into three groups, based

on their level of uncertainty and time horizon. For Horizon 1 opportunities, your main concern is

estimating the likely sales. For Horizon 2 opportunities, sales are also uncertain, but youre less

concerned with that than with a bigger riskwhether opportunity will reach the market. Early

cancellation is a real risk, and you have to factor that into your analysis. For Horizon 3

opportunities, the uncertainty is so great that a formal financial analysis is almost impossible.

You should instead seek basic insights about the quality of the opportunity.

Horizon 1 and Horizon 2 opportunities lend themselves to sophisticated analyses, such as

stochastic calculations that may seem intimidating at first sight. We call this rocket science

innovation management. Just as the trio at the Ritz benefited from modeling and analyzing

thousands of previous spins of the roulette wheel, so too can you benefit by applying rigorousanalysis to data from past innovations.

Evaluating Horizon 1 Opportunities

When evaluating a Horizon 1 opportunity you want to create an expected discounted cash flow

(DCF) model. You will also want to estimate the uncertainty of your DCF and can view the

variance of the DCF as a measure of risk. You create a DCF model in five steps, which Exhibit

HORIZON1 EVALUATION summarizes:

(1) forecast the expected sales of the opportunity;

(2) adjust the forecast based on the experience from prior opportunities;

(3) compute the variance of the forecast;

(4) use a typical product lifecycle pattern to estimate monthly (yearly) sales;

(5) build an economic model of costs and revenues.


3/30

AO-3

version 081407

To Portfolio

Planning / Final

Selection

Define

opportunity

including target

segment and price

Concept test

with input from

potentialcustomers

Aggregate

opinions of

multiple experts

Computeexpected sales

and adjust for past

forecast errors

Spread sales

over the product

lifecycle. Build

a DCF modelComputevariance based

on disagreement

among experts.

Compute

variance

based on old

AF ratios

From Screening

and Scoring

Exhibit HORIZON1 EVALUATION: For each Horizon 1 opportunity coming fromscreening, five steps need to be taken to create an expected discounted cash flow.

Forming a sales forecast

Entire books have been written about forecasting sales for of new products, so we will focus on

the process more than the technical details. You can obtain a forecast using either of two basic

methods: surveying customers or relying on experts.

When doing a survey, identify customers in the market you are targeting with the opportunity.

Then try to discover their purchase intent with questions such as, If a web service reminding

you of important dates like your anniversary were available, how likely would you be to use it:

Very, Somewhat, Maybe/maybe not, somewhat unlikely, very unlikely. From this, you can

compute an average purchase probability and expected sales as shown in Exhibit CONCEPT

TEST. Note that in the exhibit, the customers expressed purchase intent is discounted (a 0.4

weight is applied to definite buy, and a 0.2 weight is applied to probably buy). Research has

shown that customers are much less likely to adopt an innovation compared with what they say

in surveys. These sorts of parameters as well as adjustments for product awarenessyou can

only purchase products you are aware ofdiffer widely by industry and should be obtained via

market research.


4/30

AO-4

SpoilMy Spouse is an internet based service that will send an email or SMS to subscribers on a

weekly basis offering ideas on how to make their spouse happy (e.g. bring her flowers, come

home early, etc). These ideas are user-rated and recommendations are made based on the

experience of like-minded subscribers.

How likely are you to purchase subscription to this service?

Definitely

would buy

Probably

would buy

Might or might

not buyProbably

would not buy

Definitely

would not buySurv

eytoCustomers

InternalAnalysis

ofResults

Results

Proportion

Purchase

Probability

10 22 48 59 61

10/200 22/200 48/200 59/200 61/200

0.4 x + 0.2 x = 0.042

Expected sales = Population size x Awareness x Purchase Probability = 4M x 0.2 x 0.042 = 33,600version 080907

22200

10200

Exhibit CONCEPT TEST: To form a sales forecast, you estimate the averagepurchase probability for the population by learning the expressed purchase intent andthen adjusting based on historical data. You then multiply with the number ofpotential adopters aware of the product

1.

Alternatively, you can rely on experts to forecast sales. This method works best if your

innovation resembles products or services that have been launched in the past. The method is

similar to what you saw in the previous chapter on scoring and screening. The experts make

estimates independently. You then convene them a consensus-building discussion or simply

average their forecasts.

Dealing with forecast errors

Sales forecasts rarely are exactly right. If you forecast that 13 million people would see a movie

and 14 million did, you would pop open the champagne. But if you forecast 20 million people

would see it and only 5 million do, you would weep. The problem, of course, is that you only

know the accuracy of your forecast after the fact.


5/30

AO-5

Project Forecast Actual AF ratio

7 5600 1000 0.178571

12 2500 500 0.2

5 6700 4000 0.597015

14 2100 1500 0.714286

1 11100 8000 0.720721

4 6700 5000 0.746269

2 9800 7500 0.765306

8 3879 3000 0.773395

18 1146 900 0.78534

10 3300 3300 1

9 3800 4000 1.052632

11 2500 3000 1.2

6 6300 8000 1.269841

20 1100 1400 1.272727

3 9000 11500 1.277778

16 1400 2000 1.428571

19 1100 2000 1.818182

17 1200 2400 2

Average 0.988924

St-Dev 0.483351

Product Forecast Actual

AF

Ratio Wine Year Color Forecast Actual

AF

Ratio

ZEN-ZIP 2MM FULL 470 116 0.246809MONTAGNE ST-EMILION 02 rouge 4000 468 0.117

ZEN 3/2 3190 1195 0.374608MONTAGNE ST-EMILION 02 rouge 6000 1236 0.206

ZEN 4/3 430 239 0.555814 CARTON PANACHE 1 n.a. mixed 1800 372 0.206667

WMS ELITE 3/2 650 364 0.56 BORDEAUX SUP 01 Rouge 1200 252 0.21

WMS HAMMER 3/2 FULL 6490 3673 0.565948 IROULEGUY 01 Rouge 3000 726 0.242JR HAMMER 3/2 1220 721 0.590984 CARTON PANACHE 2 n.a. mixed 900 297 0.33

HEATWAVE 4/3 430 274 0.637209 BORDEAUX 01 Rouge 1800 612 0.34

ZEN 2MM S/S FULL 680 453 0.666176 APREMONT 02 Blanc 1200 567 0.4725

EVO 4/3 440 623 1.415909 ROSE DE LOIRE 03 Ros 2300 2934 1.275652

WMS EPIC 4/3 1060 1552 1.464151 CTES DU RHNE (6) 01 Rouge 4000 5370 1.3425

JR EPIC 4/3 380 571 1.502632 CTES DU RHNE (6) 01 Rouge 3000 4146 1.382

EVO 3/2 380 587 1.544737 BORDEAUX 02 Rouge 2500 4057 1.6228

JR ZEN FL 3/2 90 140 1.555556CHTEAUNEUF DUPAPE 00 Rouge 300 703 2.343333

EPIC 3/2 2190 3504 1.6 CARTON PANACHE 6 na mixed 2160 5064 2.344444

Average 0.9966 Average 0.858337

St-Dev 0.369666 St-Dev 0.472292

Exhibit AF-RATIOS: Three cases of comparing forecasted numbers with the actualoutcomes in diverse settings including power and automation, surf apparel, andwines.


6/30

AO-6

Even though forecasts are faulty, you should take time to examine your past forecasts and

contrast them with your actual performance. For each opportunity, compute the ratio between the

actual number and the forecasted number, providing anAF ratio for each opportunity. This

works for sales and financial forecasts. AF ratios can vary from figures that are significantly less

than onea ratio of 0.1 indicates that you over-forecasted by a factor of 10to figures of 3 or

highera ratio of 3 indicates that the real number was three times as high as the forecast.

Exhibit AF ratios shows examples from very different industries, including power and

automation, surf apparel, and French wines. At first sight, the analysis does not say much.

Sometimes, the forecasts are too high; sometimes, too low. But with a little more analysis and

graphical support similar to what is shown in Exhibit AF-GRAPH, you can obtain a couple of

powerful insights2.

By looking at the average AF ratio, you can see if a firm is forecasting correctly on average.

Most companies consistently forecast more than they actually sell; humans are optimistic

animals. As you can see in Exhibit AF-RATIO, the power and automation and surf apparel

forecasts are very accurate on average, with an average ratio close to 1. At least for the company

that provided the data underlying the exhibit, wines suffer from over-forecasting; on average,

sales are only 85 percent of the forecasts. (Maybe the forecasters spent too much time sampling

the product.) In the future, if you need to forecast the sales for a new wine, you should adjust the

forecast by multiplying it by 0.85 (and keep the forecasters away from the bottles).

The AF ratios also help you estimate your risk. A company that consistently produces forecasts

with AF ratios between 0.9 and 1.1 faces less risk (a lower forecast variance) than an outfit with

AF ratios ranging between 0.5 and 2. (Exhibit AF-RATIO (right) shows how you can use the

distribution of past AF ratios to estimate the standard deviation for the new product sales. For

every dollar forecasted, you really face a distribution of potential outcomes as is shown in the

distribution in Exhibit AF-GRAPH.)

Risk follows a common statistical pattern. To see this, compare the three distributions of AF

ratios in AF-Graph. The left and the right histograms resemble the normal distribution while the


7/30

AO-7

middle one seems closer to a uniform distribution. Once you understand the statistical behavior

of your old forecasts, you can build a forecast model that incorporates your tendency to err. Risk

can be modeled, in other words, whether the opportunity is a robot, surf shorts, or a Saint

Emilion wine.

version 082007

Frequencywithwhich

AFRatiowasObserved

0.0 0 1 .0 0 2 .000

1

2

3

4

5

6

Financial Returns for

Power and Automation

Products

AF Ratio

0.2 0 1 .0 0 1 .6 00

1

2

3

4

5

Product Sales of

Surf Apparel and

Wet-suits

AF Ratio

0 .00 1.0 0 2 .500

2

4

6

8

10

12

Product Sales of

French Wines

AF Ratio

Exhibit AF-GRAPH: Each of the three graphs shows the empirical distribution ofthe AF ratios shown in Exhibit AF-RATIOS.

Building a financial model

Once you have a sales forecast, you can create your overall financial model. For this, you first

distribute the total sales over the lifecycle of the product using a lifecycle similar to that of other

innovations of this type. Considering your development and launch costs as well as the major

fixed and variable costs, you can compute the discounted cash flow (DCF) for this innovation.

Exhibit ECONOMICS-EXAMPLE provides an example of this by plotting the cumulative cash-

flows (inflows and outflows) with the innovation. This graphic can easily be created in Excel and

provides a basis for the most common financial evaluations of Horizon 1 opportunities. (In

Chapter FI, we discuss several aspects related to how to appropriately choose discount factors

a topic that can lead to heated debates when evaluating innovation opportunities.)

Given your focus on value creation, your most important measure is the financial returns relative

to your investment. Yet with the discounted cash flow analysis, you can also compute other

commonly used metrics, including break-even time, time to your first commercial sale, and the

return on investment from the innovation.


8/30

AO-8

Cumu

lative

Inflowor

Ou

tflow

($)

version 081507

Ramp-up Costs

Marketing and Support Costs

Production Costs

+$

-$

Sales Revenues

TimeDevelopment Costs

Exhibit ECONOMICS EXAMPLE: Most opportunities begin with a cash outflow,typically covering the costs of development and launch. Successful opportunities arethose in which the cumulative cash inflows (revenues net of production costs) justifythe initial investment3.

Now that you have computed the expected sales and the corresponding standard deviation, you

can also obtain the standard deviation of your financial measures. The easiest way to do this is to

perform a Monte-Carlo simulation in the Excel model, varying the sales number (or other

numbers that experience suggests are uncertain such as development time) according to the

histogram computed in Exhibit AF RATIOS. This can be simply done by randomly creating an

AF ratio between zero and two and then multiplying that AF ratio with the sales forecast.

As the opportunity moves through development, all forecasted numbers should be compared

with actual performance and updated. Because Horizon 1 opportunities are seldom terminated,

the development process should be executed with a focus on efficiency and timeliness.


9/30

AO-9

Evaluating Horizon 2 opportunities

Earlier in this chapter we compared financial evaluations of innovations with bets at a casino.

Before you take your money to the roulette table, you want to make sure that you understand the

rules of the games and the odds. Imagine that you place a $100 bet on #14 when playing roulette.

You know that (unless you have secret camera) the odds of winning are 1 in 37. If you win, you

will receive $3,600. More likely, though, you will lose, and the casino will take your $100. From

simple statistics, you can value this bet using the expected payoff, which is computed as:

Expected payoff of putting $100 on #14 = 36/37 (-$100) + 1/37 ($3600-$100) = -$2.70

Thus your expected payoff is negative, and the casino makes more money than you do.

(Surprise, surprise.) Exhibit CASINO-EXAMPLE explains this bet. The concept of the expected

payoff works in cases where you make a lot of bets and what matters is the average payoff. If

you bet only once, you might also consider other measures, such as the probability of losing

money or the payoff in the worst (best) case scenario. On the left of the exhibit, you can see how

the uncertainty of the innovation can play out (also called an event tree). On the right, you can

see the various possible outcomes and associated probabilities.

version 081507

Probability

36/37

-100

1/37

+3500

Outcome is 0 (Probability 1/37) Pay-off is 0





Exhibit CASINO-EXAMPLE: An event tree describes the odds of winning at the

roulette table (left) and a histogram shows the potential payoffs (right).

Expected payoff

Enough roulette. Now lets consider a real innovation project. In Exhibit PHARMA-EXAMPLE,

you have the event tree for a new chemical compound considered for investment. Based on


10/30

AO-10

Phase 1 clinical trials, you will learn how toxic the drug is for humans. If people cannot tolerate

the drug at all (10 percent probability), all trials stop. Otherwise, the drug moves into Phase 2

trials. A home run would be a drug with medium toxicity but high efficacy, in which case you

would make $500 million. If the toxicity is medium, you have a 20 percent chance of obtaining a

high efficacy score. For a more toxic drug (strong toxic activity, 40 percent probability), you

have a better chance of obtaining a high efficacy (80 percent). But because of the higher toxicity,

fewer patients could use the drug, and the payoff would be only $100 million.

version 081507

Probability

0.82

-30

0.1

+370

0.08

-130Pay-Off

Phase I shows mediumtoxic activity (p=0.5)

Phase Ishows strongtoxic activity(p=0.4)

Phase I shows substancecannot be tolerated byhumans (p=0.1)

Phase II showsmedium efficacy (p=0.8)

Pay-off: 100

Phase II showshigh efficacy

Pay-off: 500

Phase II showsmedium efficacy (p=0.2)

Pay-off: 0

Phase II showshigh efficacy (p=0.8)

Pay-off: 100

Investment Phase I: $30M Investment Phase II: $100M

Exhibit PHARMA-EXAMPLE: An event tree and histogram for a pharmaceuticalcompound.

To calculate the expected payoff for this drug, apply the same logic as before. The expected

payoff (not yet including the necessary investment) can be calculated as:

Expected payoff

= 0.5 Payoff if medium toxicity + 0.4 Payoff if strong toxicity + 0.1 Payoff if too toxic

= 0.5 (0.8 $100M + 0.2 $500M) + 0.4 (0.2 $0M + 0.8 $100M) + 0.1 $0M

= 0.5 $180M + 0.4 $80M = $90M + $32M = $122M


11/30

AO-11

If you factor in a development cost of $30 million for the Phase 1 trials and $100 million for the

Phase 2 trials, you end up with a net gain (loss) of: $122 million - $130 million = -$8 million.

And the drug seems as risky as a bet in roulette. But this analysis misses something critical.

Understanding option value

Despite the similarities with gambling, evaluating an innovation requires a more detailed

analysis of how the innovation will be created and how it will proceed through selection. In other

words, you can make an intermediate decision after obtaining information from Phase 1 but

before making the additional (large) investment in Phase 2. To see why a more detailed analysis

is needed, consider the three possible outcomes of the Phase 1 trials:

Case 1 (Medium toxicity): When you learn about the outcome of Phase 1, you have already

spent the associated $30 million. These costs are what economists call sunkthe money is gone

and you cant recoup it, so its no longer relevant to your analysis. Knowing a medium level of

toxicity, you can expect to earn a payoff of 0.8 $100 million + 0.2 $500 million = $180

million. For this, you would have to spend another $100 million. Would you do this? The answer

is an enthusiasticyes! This investment would create an expected $80 million in value.

Case 2 (Strong toxicity): When you learn about strong toxicity, you again should look forward,

not backward. What matters is that you have an expected payoff of $80 million (0.2 $0 + 0.8

$100M), not that you have spent $30 million. This is especially important when deciding on the

next $100 million investment. Investing $100 million to obtain an expected $80 million destroys

value. So, as painful as it might be having spent $30 million for nothing, you kill the project

(more on the organizational aspects of this later).

Case 3 (Too toxic to tolerate): Again, the $30 million is sunk and, because you have no hope of

making any money going forward, you should terminate the project after Phase 1.

Now, lets return to the initial decision concerning the initiation of Phase 1 trials. Is this project a

good investment or not? You have a 50 percent shot at $80 million and a 50 percent (40 percent

+10 percent) shot at nothing. For this, you have to spend $30 million for the Phase 1 trials.

Suddenly, what looked like a bad investment a moment ago looks much more attractivea $40


12/30

AO-12

million payoff for a $30 million investment. Note also that this investment is now less risky, as

you have eliminated the scenario in which you suffer a $130 million loss.

What can you learn from this example? First, most investments in innovation happen in phases,

and the early phases are usually less expensive than the later ones. While the first phases by

themselves typically do not lead to financial rewards, they provide something almost as valuable:

information. At the beginning of Phase 1, you had a compound about which you knew little.

Once you have spent the initial $30 million, you knew a lot more. The additional information

had the same effect as the techniques the gamblers used at Ritz: it improved your odds. You

could invest in promising compounds (those with medium toxicity) and abandon unpromising

ones.

Second, when investing into a new innovation, keep in mind that you have the right, even the

responsibility, to walk away if the information you gather points to failure. Failing is part of

innovation, and failing early makes more sense than failing late. Walking away from an

innovation is often referred to as an exit option, as it resembles the financial option of buying (or

not buying) a stock at a predefined price.

Estimating the probabilities of success

Once you have outlined all the ingredients for the calculations in Exhibit PHARMA-EXAMPLE,

valuing the compound (and making the necessary decisions) is a matter of calculus. The

calculation turns out to be simple, once you write them down. In contrast, the much harder

problem is to get the numbers required for the inputs. So how do you estimate success

probabilities? What does it mean to have a 50 percent probability of success? To see the

importance of these questions, consider again the payoffs described in Exhibit PHARMA-

EXAMPLE. By re-doing the above calculations with slightly lower (or higher) estimates for theprobability of obtaining a medium toxicity, you observe the following:

A 1 percentage point increase in probability (for example, a move the probabilityfrom 50 percent to 51 percent) is worth $0.8 million in value, which corresponds to

roughly 10 percent of the overall value of the innovation.


13/30

AO-13

Moving the probability from 45 percent to 55 percent increases project value by 100percent.

Thus not only are the odds of innovation success important, their exact magnitude can have a

dramatic effect on your decision-making. For this reason, we recommend that you do the

computations outlined above with a range of plausible success probabilities, not just one. If the

opportunity is profitable only for a small range of probabilities, you probably should not pursue

it.

In general, you can choose among three estimation methods that yield probabilities of success

(POS). Given that even small changes in POS have large impact on the NPV of an opportunity,

consider using two or ideally all three of them.

Historical outcome data. You can use historical data about the proportion of similaropportunities that succeeded to estimate the POS for opportunities currently under

evaluation. Exhibit PHARMA-POS shows a set of probabilities for product approval

in the drug development process. Given that the POS data shows significant

differences across illnesses, you would want to group the data points of prior

opportunities according to illness. The main strength of forecasting POS using

historical data is that it provides a rigorous statistical approach. But it also assumes

that the future will resemble the past.


14/30


15/30

AO-15

the opportunity under consideration. But they also can become mired in office or

professional politics.

Criteria. Many firms have clear criteria that define what POS level any givenopportunity deserves. Proctor & Gamble has found that by using explicit criteria of

what corresponds to, say, a 50 percent POS, expert discussion can focus more on

facts and less on opinions5. Experts, for example, are likely to agree on whether

theres a 90 percent probability that a new product can be made on existing machines

at high volume. This approach is objective, but defining the criteria consumes a lot of

time. And even more time gets spent translating the criteria to numeric probabilities.

Again, there is no one right way to forecast POS. You should use multiple approaches and

understand that you face an extra level of risk if these approaches arrive at very different

estimates. Any POS forecast should always be pressure tested: if a 5 percent reduction in POS

turns your return into a loss, the opportunity is probably not worth funding. As we discussed in

the previous chapter, accidentally killing a good opportunity might be painful, but is typically far

less expensive than further developing a poor one.

All three approaches benefit from feeding back the actual outcome into your forecasting for

future opportunities. We will elaborate on this point after discussing Horizon 3 opportunities.

Exhibit HORIZON2-EVALUATION summarizes the overall process.

version 081507

Forecast the pay-

off collected upon

launch (expected

value is enough)

Estimate the

probabilities that

the milestone is

completed (POS)

based on:

Historical data

Opinion

aggregation

Criteria

From Screening

and Scoring

Define all costs

required to get the

opportunity

launched.

Identify milestone(s)

at which the

opportunity could be

killed (and no furthercost incurred)

To Portfolio

Planning / Final

Selection

Build an integrated

model:

- Exit option

- Probability for

each pay-off

Exhibit HORIZON2-EVALUATION: Summary of steps that need to be takenwhen analyzing a Horizon 2 opportunity.


16/30

AO-16

Evaluating Horizon 3 opportunities

Horizon 3 opportunities are all about uncertainty. But, in contrast to the tidy risks of Horizon 1

opportunities, the uncertainty for Horizon 3 opportunities is, simply put, one big mess. You dont

know many critical facts and you dont even know that you dont know even more things. So

how do you evaluate an opportunity like this?

Exhibit SEGWAY: The Segway scooter is a two-wheeled mobility device.

Consider the example of the Segway scooter, a personal-transportation device launched in 2001.

Given the radical nature of the innovation, the Segway team faced a number of uncertainties,

including:

Can you produce a self-balancing scooterthe Segway has two wheels on the sameaxleand sell it for less than $5,000?

Will local governments allow Segways on sidewalks? Are you able to produce more 100,000 units a year? Can you create a sufficient product awareness to sell more than 100,000 units a year? How will consumers respond to the Segway?


17/30

AO-17

Focus on hypothesis testing

With the Segway, youre not dealing with a Horizon 1 or 2 uncertainty. At inception, the Segway

represented a new radically technology is search of a market. The innovators did not know who

might adopt it. Commuters? People with disabilities and the elderly? Cops? Tour operators?When facing this much uncertainty, you focus on learning. You have only hypotheses about how

it might meet peoples needs but no solid evidence. For this reason, Horizon 3 evaluations focus

on the creation of knowledge, not potential sales.

Issue Current Understanding Type of Uncertainty

Technical Feasibility Technology has been used in a previous product. Detailed engineering not clearyet, but uncertainty primarily relates to component costs. Costs will be

between$3,000 and $5,000.

Horizon 1: Parameter uncertainty

Regulatory Approval Not clear if Segways will be allowed on sidewalks or if they will require driverslicense.

Horizon 2: Scenario uncertainty

Ability to Mass Produce No prior experience with mass production of a product of this type. Theproduct appears to be similar in its production process to golf carts and othersimple electric vehicles.


Marketing Campaign Are you able to create sufficient product awareness? Currently, severalnewspapers and online chat rooms are hotly debating what product you willlaunch.


Consumer Interest You are hoping to fundamentally change the way consumers think abouttransportation. You have no interest in niche markets such as golf carts. Giventhe risk that somebody could steal your idea, youve done no market research.

Horizon 3: Unknown unknowns

Exhibit HYPOTHESES1: To analyze a Horizon 3 opportunity, the various sourcesof uncertainty need to be identified and categorized into three horizons defined inChapter SO (Screening Opportunities).

When evaluating a Horizon 3 opportunity, you start by creating a table like the one shown in

Exhibit HYPOTHESES1. This table lists all aspects of this opportunity, such as its technical

feasibility, its fit with current regulations, and the customer response. For each one, you thenassess your current level of understanding:

Level 1 understanding (parameter uncertainty). You exactly understand the problembut have estimates for only some of its parameters (for example, the unit cost of the

Segway will between $3,000 and $5,000).


18/30

AO-18

Level 2 understanding (scenario uncertainty). You can envision all possibleoutcomes (and have some probability estimates) for your hypotheses (for example,

whether each state or major city will either approve the Segway for sidewalk use).

Level 3 understanding (unknown unknowns): You dont even understand all therelevant variables related to the hypothesis (for example, to support your sales

hypothesis, you need to first set a price point, identify market segments, learn the

purchase probabilities and appropriately adjust these probabilities)6.

version 081607

H1

: We can produce a self-balancingscooter for a unit cost below $5k

H2: Governments will allow people touse our scooters on side-walk

H3: We can produce at volumes>100k per year

H4: We can create product awarenesswithout having an established brand

H5: More than 100k consumers per yearwill want to purchase a scooter

Uncertainty

(phrased as hypothesis)

Full Understanding

(no uncertainty)

Build a protype ($2M)

Lobby for changes in regulation (>$5M)

Purchase intent test ($10M)

Public relations ($1M)

Exhibit HYPOTHESES2: Each uncertainty can be resolved at a cost. Your goal is

to find the uncertainty for which you can create a maximum of learning per dollar ofinvestment.

Once you have mapped out the uncertainties, you can answer two questions. What are the

biggest (and scariest) uncertainties? And where do you get the biggest amount of learning per

dollar of investment? As is illustrated by the leftward-pointing arrows in Exhibit

HYPOTHESES2, it is relatively expensive to reduce the uncertainty about regulatory approval.

True, regulatory approval is crucial, but before spending millions lobbying for Segways to beallowed on sidewalks, you might want to find out if consumers want to ride the machine at all.

Maybe, they want to use them only on golf courses or at airports. Creating a table such as

Exhibit HYPOTHESES is very helpful in pinpointing where small investments might create new

knowledge. In this case, a simple purchase-intent survey such as outlined in the section on

Horizon 1 evaluations would have revealed that mainstream consumers were not interested in the


19/30

AO-19

Segway as means of transportation at a price of $5,000. With this knowledge, the management

team probably would have waited to build a mass-production factory.

Learning vs. opportunity costs of time

Learning and testing hypotheses seem sensible steps. But they suffer from the motherhood and

apple-pie syndromethat is, who could disagree with them? Theyre too easy to pursue

mindlessly. Thus, before you plunge in, you need to consider what youre sacrificing when you

spend time on them. Your willingness to move ahead should be driven by two considerations.

The first is the opportunity cost of time, or, put differently, the urgency of a launch. This variable

is typically determined by the competitive landscape. The second is the chance of inexpensive

uncertainty resolution. This depends on how Exhibit HYPOTHESES looks for the opportunity at

hand, specifically if you have a key hypothesis that you can test for a small investment7.

Based on these two variables, you can create a matrix as described in Exhibit UNCERTAINTY

RESOLUTION. In some situations, you simply must place a big bet: if the opportunity cost of

time is high and you cant reduce uncertainty, there are no intermediate steps. You either

launch the rocket now or never. Granted, this scenario is rare. In many situations, you can simply

wait (and potentially become the second mover) or follow a staged investment approach as

discussed above.


20/30

AO-20

version 080907

Rate at which Uncertainty Can be Reduced with

Incremental Investment.

OpportunityCostofTime

(e.g.,

immedia

cyofcompetitivethreat)

Uncertainty is high for

some time regardless

of our investment.

Uncertainty can be

reduced substantially

with modest investment.

HighOpportunity

CostofDelay

LowO

pportun

ity

CostofDelay

Go for it and translate

into Horizon 2

development

Adopt a dynamic,

contingent-action

approachstages

with decision points.

Wait and see

possibly

as second mover.

Take slow, deliberate,

and careful steps to

reduce uncertainty.

Exhibit UNCERTAINTY RESOLUTION: Depending on the possibility ofuncertainty resolution (that is, of learning via small investments) and on theopportunity cost of time, different strategies become optimal.

So how to you make a final decision? Again, recall that Horizon 3 opportunities need to be

evaluated based on the learning that they create. The investment in the opportunity needs to be

staged with the milestones corresponding to the hypotheses that you want to test. As you saw

with Segway, you generally want to start with the biggest uncertainty reduction per dollar

invested. Whether you make the initial investment, kill the opportunity or postpone requires a

group evaluation similar to what you have seen elsewhere in this chapter. If you cannot resolve

any of the uncertainty resolution quickly and you face time pressures (upper left box in Exhibit

UNCERTAINTY RESOLUTION), the potential payoff better be big and likely to happen under

a wide range of possible scenarios.

One venture capital firm that we studied uses the following approach to make the initial funding

decision for Horizon 3 opportunities. After hearing the creators pitch, each partner takes a turn


21/30

AO-21

and discusses (a) the key hypotheses that she would like to see tested and then (b) gives a

recommendation with respect to funding or not funding the opportunity. These discussions start

with the most junior partner and are wrapped up by the most senior partner. That way, opinions

of less experienced partners arent influenced by those of their more senior colleagues, who

should be more comfortable expressing opinions that dont line up with the groups consensus.

The final vote is typically pro forma. In most cases, the partners have already reached an

agreement.

Which Opportunities to Move Forward?

Our discussion so far has been on analyzing the three types of opportunities yet, we still have

not addressed the question of which opportunities to now move forward. Recall from our earlier

discussion of opportunity classification (see Chapter SO (Screening Opportunities)) that we

should not compare one type with another. The amount of resources that should be allocated to

each type of opportunity will be discussed at length in the following chapter (see Chapter SP

(Innovation Strategy and Opportunity Portfolio)).

So, how do we compare within each type? For Horizon 1 and Horizon 2, you select the setof

opportunities that creates the biggest value. In other words, you maximize the value of the

innovation portfolio. Too often, companies just rank opportunities according to expected returns

and then pursue the ones at the top of the list. But this approach ignores interdependencies

among the opportunities. The value of the portfolio is not just the sum of the individual values.

(See Chapter SP (Innovation Strategy and Opportunity Portfolio) for more details.)

For Horizon 3 opportunities, you should compare the required investments and the associated

learnings across opportunities and determine which ones would create knowledge most relevant

to your company as a whole. (Chapter SP (Innovation Strategy and Opportunity Portfolio) also

helps more with this question.)


22/30

AO-22

Human and Organizational Aspects of Opportunity Evaluation

All of the techniques we have discussed for evaluating opportunities are probabilistic in nature.

This reflects the uncertain reality of innovation. But in trying to implement these techniques, you

have to keep in mind that probabilistic thinking does not come naturally to many people.

version 081607

Forecast Probability of Rain

938

82159

147

203

172

257

589

575

2820

0.2

0.4

0.6

0.8

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

ProportionofDayswithRain

Exhibit POS-FORECASTING1: A comparison of the forecasted probability of rain(x-axis) plotted against with the proportion of days on which rain came (y-axis). Eachnumber along the plotted line corresponds to the number of times a specific forecast

was made. For example, a 0.3 probability of rain was forecasted 257 times8.

To get more comfortable with forecasting the probability of success of an innovation, consider

the example shown in Exhibit POS-FORECASTS-1. It shows data from another area in which

people often speak in terms of probabilities, namely weather forecasting. You might have asked

yourself in the past what it really means if you have a 70 percent probability of rain. After all, it

will either rain tomorrow or it will not. The graphic will help you ponder that question.

On the x-axis, you have different levels of rain probability that the National Weather Service has

announced for a particular region and period. Pick the point of 0.7, that is, a 70 percent

probability. By moving upwards in the graphic, you hit the point labeled 159, which means

that, for this region and period, the National Weather Service had announced a 0.7 rain

probability for 159 days. But what you want to know is how many times it actually rained the


23/30

AO-23

following day. To get to that number, move from this point to the left, towards the y-axis. If the

forecast had been perfect, you would expect that rain would have fallen in 70 percent of the

cases in which the National Weather Service forecast a 70 percent probability. You see that the

actual number was only slightly higher (about 72 percent). Overall, thats an impressive

performance.

version 081007

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Forecast Probability of Pneumonia

ProportionwithPne

umonia

,

asDeterminedby

X-ray

Exhibit POS-FORECASTING2: The graphic compares the forecasted probabilityof pneumonia as expressed by the doctors participating in the study (x-axis) with theproportion of times the patient indeed suffered from pneumonia. For example, only 5percent of the patients for whom doctors estimated a 25 percent probability ofpneumonia suffered from it

9.

Now consider a second example, created in the exact same way. Instead of weather forecasters,

our subjects are doctors who need to estimate the probability that patients have pneumonia.

Exhibit POS-FORECASTS2 shows the proportion of patients who turned out to have pneumonia

as a function of the probability forecast of the treating physician. For cases in which the doctor

forecast a 50 percent probability of pneumonia, x-rays indicated that only 10 percent of the

patients had the disease.


24/30

AO-24

Unlike the weather forecasts, in which all data points tidily matched the predictions, you see that

their data points stray far from the good forecasting line Of course, just because doctors fail at

forecasting does not mean they mistreat their patients. From a forecasting perspective, you might

complain about their performance. From a medical perspective, the picture changes. What the

doctor really might have been saying when predicting 50 percent was, This patient is at risk for

pneumonia, and we should take a chest x-ray. Put differently, the doctor did not distinguish

clearly between the forecast (50 percent) and ordering an action (the chest x-ray).

Why does this distinction matter? Because the same problem exists in most innovation

processes. Both those proposing innovations to be fundedinventors, scientists, entrepreneurs

and those who control the requested resourcesmanagers and venture capitalistsoften confuse

the forecast (this innovation will or will not work) with the action (should we fund thisinnovation?). Of course, advocates will argue that the odds are good and will attempt to shift the

45 percent odds to 55 percent odds, while funders will argue the opposite. But to make a prudent

financial valuation, you must draw a clear line between your risk assessment and the associated

payoffs.

To improve your forecasting capability, the following three steps are useful:

(a) ensure that the evaluation of opportunities is done by independent reviewers (as opposed to

people who have a stake in the success of the opportunities);

(b) increase the independence of the review and improve its predictive power by involving a

larger population inside or outside your company;

(c) give regular feedback to all units involved in forecasting.

The need for an independent review board

The adage that You dont let the turkeys vote on Thanksgiving! has a direct relevance to the

evaluation of opportunities. You must separate those who select opportunities from those who

create and develop them, and this is where independent review boards come in. To understand

their usefulness, consider Exhibit INLICENSE-POS. The data summarizes a McKinsey study on

the innovation odds of compounds that drug companies created internally versus those that they

licensed from outsiders.


25/30

AO-25

version 081407

Clinical Success Probabilities Among the Leading Pharmaceutical Companies

54

48

39

7068

80

14

27

Phase I Phase II Phase III

Cumulative SuccessClinical Trials

Internally developed

compounds

Licensed compounds

Exhibit INLICENSE-POS: Success probabilities for internally developed chemicalcompounds compared with licensed compounds. Internally developed compounds are

more likely to pass the Phase 1 review, yet their overall success rate is only half ofthat of licensed compounds

10.

Looking at the overall success ratesthat is, the probability of the innovation surviving all three

phasesyou notice that licensed compounds are twice as likely to succeed. Why is this? When

licensing a compound, an organization does exactly what we described above. It has a set of

criteria and applies them to the licensing candidates. In this setting, the people involved in the

valuation have no personal stake in the innovation and can be dispassionate.

But when you are evaluating an internally developed innovation, people involved with its

creation probably have some influence on the review. This often leads to a company being

reluctant to terminate a project early on. Just as parents would not harm their own children, many

innovators do not want to kill their projects. For this reason, as you can see in INLICENSE-POS,

internally developed projects are more likely pass through the first phase but also more likely to

fail later. This tendency can cost you real money. As you have seen in the context of valuing

individual projects, the value of an innovation depends crucially on your ability to walk away

from it. A review board can be truly independent in making these tough decisions.

An effective board is not just independent. Its also diverse, including the perspectives of

disinterested innovators as well as from such corporate divisions as finance, marketing, and

operations. The link between innovation and finance is often weak in big corporations. As a


26/30

AO-26

consequence, finance people will attempt to develop models similar to the one described in

Exhibit PHARMA-EXAMPLE but fail to incorporate insights from staffers most familiar with

the innovation.

Using large populations to support the evaluation process

In addition to this traditional approach towards forecasting success probabilities, one

additional tool deserves discussion:prediction markets. Prediction markets aggregate peoples

opinions using a basic principle of economics. If I believe that an event will occur with a 70

percent probability, I should be willing to pay up to 70 cents for a lottery ticket that pays $1 if

the innovation succeeds and nothing otherwise. If you feel more confident about the success of

the innovation, you should be willing to pay more. Prediction markets let you (and many others)

trade your bets in ways similar to how stocks are traded at Wall Street11. Just as stock prices

mirror the markets aggregate opinion on a companys future cash flow, a prediction market

reveals the participants aggregate opinion on the odds of an innovations success.

Several successful implementations of prediction markets for innovation opportunities have been

reported (links available at www.MasteringInnovation.com). You can trade prediction shares in

nearly anything, ranging from the next pope to the outcomes of elections. But there is little

empirical evidence that these markets really work when it comes to forecasting innovation

success.

Prediction markets certainly represent a step forward for most firms, relative to the typical status

quo of having no system of forecasting. Whether they can replace three-step probability

forecasting discussed above will depend on the specific application. Chapter OI (Open

Innovation) will elaborate more on how one can benefit from the wisdom of the crowds.

Feedback and calibration

One of the reasons that weather men do so well in forecasting the probability of rain (see Exhibit

POS FORECAST1) is that they have many opportunities to practice. But there is more to this

than practice alone. Practice only leads to better forecasting if there exists somefeed-back.


27/30

AO-27

Doing many weather forecasts alone will not help, doing the forecasts and then contrasting them

with the actual weather, however, can be very powerful.

Why cant we do the same exercise that every new weather-person needs to go through for those

in charge of forecasting the odds that an innovation turns out successful? The idea ofcalibrating

the innovation forecasting process is compelling. However, it requires that the organization

overcomes two obstacles:

Forecasts will never be perfect, and this reality has to be accepted by forecasters andthe managers who evaluate them. Nobody likes to be reminded of yesterdays errors,

so most innovation forecasters prefer to not go on record when making a prediction.

Many companies fail to keep track of their old forecasts. Old forecasts get over-written by new forecasts if they are stored electronically, or they are filed way, never

to be retrieved.

Overcome these obstacles, and your firm can create a graph like the one shown in Exhibit POS-

FORECASTING3, which tracks the forecasting performance of Ely Lilly. Notice that the

probability forecasts by the companys review board are remarkably close to the actual

innovation success rates. With enough discipline and the right inputs, forecasting innovation

odds is possible.


28/30

AO-28

version 080907

0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

IRB Probability Assessment

ActualSuccessRate

76

73

21

296722

15

12

9

815

6

IRB Performance, All Raw Data

Most of the high sample size

observations lie within a 0.10 band

about the target line

Exhibit POS-FORECASTING3: The graph compares the forecasted probability ofsuccess as expressed by the independent review board (x-axis) with the proportion ofactual innovation successes at Eli Lilly. Each number along the plotted linecorresponds to the number of times a specific forecast was made. For example, a 0.3probability of success was forecasted 8 times

12.

Chapter Summary

The objective of opportunity evaluation is to assign financial values to opportunities and to

identify the types and sources of risk. Where possible, you should aim to create rocket-science

evaluation models.

Horizon 1 opportunities can be evaluated based on discounted cash flows. The biggest threats to

a good evaluation are overly optimistic forecasts of sales and cost overruns. Their development

process should be highly structured and data driven.

Horizon 2 opportunities require an options-based evaluation, as they typically allow the

organization to abandon the opportunity after some bad news has surfaced. The biggest threats to

a good evaluation are overly optimistic POS forecasts and the unwillingness to kill an

opportunity.


29/30

AO-29

Horizon 3 opportunities should not be valued financially. Their focus should be learning, and the

company should try to maximize the knowledge per dollar of investment. If possible,

investments should be staged. Staging investments enables you to cash in on the full upside of a

good opportunity, yet limits the downside of a bad one.

Evaluation of opportunities should be done by an independent review board. Explicit data-based

feedback to a board helps it overcome decision biases.

Diagnostics

What measures do you use to evaluate the financial attractiveness of opportunities?Do you distinguish between different horizons?

Do you use measures for the probability of success? How are these probabilitiescomputed?

Does your company keep data on old forecasts with respect to financial returns, salesvolume, and probability of success?

When analyzing the financials of a new opportunity, do you factor in how previousforecasts fared?

Are you able to spot weak opportunities early on or do your opportunities tend tolinger, regardless of their prospects, once they have started development?

Do you evaluate Horizon 3 opportunities with a focus on financial returns orlearning?

Are your review boards independent or might members have conflicts of interestswhen selecting opportunities?

Chapter Notes

1 See the chapter of concept testing in Ulrich and Eppinger. K.T. Ulrich and S. Eppinger, Product Design and

Development,2nd ed. (New York: McGraw-Hill, 2000).


30/30

2 See Cachon and Terwiesch for a detailed discussion of how to interpret AF ratios and how to use this information

to make decisions under uncertainty. G. Cachon and C. Terwiesch, Matching Supply with Demand: An Introduction

to Operations Management(New York: McGraw-Hill, 2006).

3 See Ulrich and Eppinger concerning details on how to build a discounted cash flow model for Horizon 1

opportunities. K.T. Ulrich and S. Eppinger, Product Design and Development,2nd ed. (New York: McGraw-Hill,

2000).

4 Source: ADIS International, see Girotra et al for further details. Karan Girotra, Karl Ulrich, and Christian

Terwiesch, Risk Management in New Product Portfolios: A Study of Late Stage Drug Failures, forthcoming in

Management Science.

5 Panel discussion of the POMS college on Product Innovation and Technology Management, Boston 2006.

6 See Loch et al. for more details on managing unknown unknowns as well as on how to create a table similar to

Figure HYPOTHESES 1. Christian Terwiesch and Christoph H. Loch, Measuring the Effectiveness of

Overlapping Development Activities,Management Science Vol. 45, Number 4 (1999): 455-465.

7 Terwiesch and Loch discuss the concept of uncertainty resolution. To measure uncertainty resolution, we have to

map out the amount of residual uncertainty over time. A steep initial decline in the resulting graph corresponds to a

fast uncertainty resolution. Christian Terwiesch and Christoph H. Loch, Measuring the Effectiveness of

Overlapping Development Activities,Management Science Vol. 45, Number 4 (1999): 455-465.

8 Example and data is taken from Murphy and Winkler 1977. A.H. Murphy and R.L. Winklder, Can weather

forecasters formulate reliable probability forecasts of precipitation and temperature?,National Weather Digest2

(1977): 2-9.

9 Example and data is taken from Christensen-Szalanski and Busyhead 1981. Christensen-Szalanski and

Bushyhead, Human Perception and Performance, Journal of Experimental Psychology Vol. 4 (August 7, 1981):928-35.

10 Example and data is taken from Booth et al 2004. B.L. Booth, D.J. Lennon, and E. J. McCafferty, Improving the

pharma research pipeline,McKinsey Quarterly (Web Exclusive 2004).

11 Wolfers and Zitzowitz discuss prediction markets including applications to new product sales forecasting. J.

Wolfers and E. Zitzewitz, Prediction Markets,Journal of Economic Perspectives Vol. 18 Issue 2 (Spring 2004).

12 John S. Andersen, Assessing Technical Feasibility of R&D Projects in Portfolio Management, presented at

INFORMS, Seattle, 1998, TA01.3.

Date post:	03-Apr-2018
Category:	Documents
Upload:	idoscribd
View:	213 times
Download:	0 times

Analyzing+Opportunities+ +AO 27July

Documents