How predictable is technological progress? ∗

J. Doyne Farmer1,2,3 and François Lafond1,4,5

1Institute for New Economic Thinking at the Oxford Martin School, University of Oxford, Oxford OX26ED, U.K.

2Mathematical Institute, University of Oxford, Oxford OX1 3LP, U. K.3Santa-Fe Institute, Santa Fe, NM 87501, U.S.A

4London Institute for Mathematical Sciences, London W1K 2XF, U.K.5United Nations University - MERIT, 6211TC Maastricht, The Netherlands

September 30, 2015

Abstract

Recently it has become clear that many technologiesfollow a generalized version of Moore’s law, i.e. coststend to drop exponentially, at different rates that de-pend on the technology. Here we formulate Moore’slaw as a correlated geometric random walk with drift,and apply it to historical data on 53 technologies.We derive a closed form expression approximating thedistribution of forecast errors as a function of time.Based on hind-casting experiments we show that thisworks well, making it possible to collapse the forecasterrors for many different technologies at different timehorizons onto the same universal distribution. This isvaluable because it allows us to make forecasts for anygiven technology with a clear understanding of thequality of the forecasts. As a practical demonstrationwe make distributional forecasts at different time hori-zons for solar photovoltaic modules, and show how ourmethod can be used to estimate the probability that agiven technology will outperform another technologyat a given point in the future.

∗Acknowledgements: We would like to acknowledge DianaGreenwald and Aimee Bailey for their help in gathering andselecting data, as well as Glen Otero for help acquiring dataon genomics, Chris Goodall for bringing us up to date on de-velopments in solar PV, and Christopher Llewellyn-Smith, JeffAlstott and three referees for comments. This project was sup-ported by the European Commission project FP7-ICT-2013-611272 (GROWTHCOM) and by the U.S. Dept. of Solar En-ergy Technologies Office under grant DE-EE0006133. Contacts:doyne.farmer@inet.ox.ac.uk; francois.lafond@inet.ox.ac.uk

Keywords: forecasting, technological progress,Moore’s law, solar energy.

JEL codes: C53, O30, Q47.

1 Introduction

Technological progress is widely acknowledged as themain driver of economic growth, and thus any methodfor improved technological forecasting is potentiallyvery useful. Given that technological progress de-pends on innovation, which is generally thought ofas something new and unanticipated, forecasting itmight seem to be an oxymoron. In fact there are sev-eral postulated laws for technological improvement,such as Moore’s law and Wright’s law, that have beenused to make predictions about technology cost andperformance. But how well do these methods work?

Predictions are useful because they allow us to plan,but to form good plans it is necessary to know prob-abilities of possible outcomes. Point forecasts are oflimited value unless they are very accurate, and whenuncertainties are large they can even be dangerous ifthey are taken too seriously. At the very least oneneeds error bars, or better yet, a distributional fore-cast, estimating the likelihood of different future out-comes. Although there are now a few papers testingtechnological forecasts1 there is as yet no method that

1See e.g. Alchian (1963), Alberth (2008). Nagy et al. (2013)test the relative accuracy of different methods of forecastingstatistically but do not produce and test a distributional esti-

1

gives distributional forecasts based on an empiricallyvalidated stochastic process. In this paper we remedythis situation by deriving the distributional errors fora simple forecasting method and testing our predic-tions on empirical data on technology costs.

To motivate the problem that we address, con-sider three technologies related to electricity gener-ation: coal mining, nuclear power and photovoltaicmodules. Fig. 1 compares their long-term historicalprices. Over the last 150 years the inflation-adjustedprice of coal has fluctuated by a factor of three or so,but shows no long term trend, and indeed from thehistorical time series one cannot reject the null hy-pothesis of a random walk with no drift2 (McNerneyet al. 2011). The first commercial nuclear power plantwas opened in 1956. The cost of electricity generatedby nuclear power is highly variable, but has generallyincreased by a factor of two or three during the pe-riod shown here. In contrast, since the time of theirfirst practical use as a power supply for the VanguardI satellite in 1958, solar photovoltaic modules havedramatically dropped in price, by a factor of about2,330 between 1956 and 2013, and since 1980 havedecreased in cost at an average rate of about 10% peryear3.

In giving this example we are not trying to make ahead-to-head comparison of the full system costs forgenerating electricity. Rather, we are comparing threedifferent technologies, coal mining, nuclear powerand photovoltaic manufacture. Generating electric-ity with coal requires plant construction (whose his-torical cost has dropped considerably since the firstplants came online at the beginning of the 20th cen-tury). Generating electricity via solar photovoltaicshas balance of system costs that have not dropped asfast as that of modules in recent years. Our point

mate of forecast reliability for any particular method. McCrory,cited in Jantsch (1967), assumes a Gaussian distribution anduses this to calculate the probability that a targeted level ofprogress be met at a given horizon. Here we assume and test aGaussian distribution for the natural log.

2To drive home the point that fossil fuels show no long termtrend of dropping in cost, after adjusting for inflation coal nowcosts about what it did in 1890, and a similar statement appliesto oil and gas.

3The factor of 2,330 is based on the fact that a one watt PVcell cost $286 in 1956 (Perlin 1999), which using the US GDPdeflator corresponds to $1,910 in 2013 dollars, vs. $0.82 for awatt of capacity in 2013.

here is that different technologies can decrease in costat very different rates.

Predicting the rate of technological improvementis obviously very useful for planning and investment.But how consistent are such trends? In response to aforecast that the trends above will continue, a skepticwould rightfully respond, “How do we know that thehistorical trend will continue? Isn’t it possible thatthings will reverse, and over the next 20 years coalwill drop in price dramatically and solar will go backup?".

Our paper provides a quantitative answer to thisquestion. We put ourselves in the past, pretend wedon’t know the future, and use a simple method toforecast the costs of 53 different technologies. Ac-tually going through the exercise of making out-of-sample forecasts rather than simply doing in-sampleregressions has the essential advantage that it fullymimics the process of making forecasts, and allows usto say precisely how well forecasts would have per-formed. Out-of-sample testing such as we do here isparticularly important when models are mis-specified,which one expects for a complicated phenomenon suchas technological improvement.

We show how one can combine the experience fromforecasting many technologies to make reliable distri-butional forecasts for a given technology. For solarPV modules, for example, we can say, “Based on ex-perience with many other technologies, the probabil-ity is roughly 5% that in 2030 the price of solar PVmodules will be greater than or equal to their currentprice". We can assign a probability to different pricelevels at different points in the future, as is done laterin Fig. 10 (where we show that very likely the pricewill drop significantly). We can also compare differ-ent technologies to assess the likelihood of differentfuture scenarios for their relative prices, as is done inFig. 11.

Technological costs occasionally experience struc-tural breaks where trends change. Indeed, there areseveral clear examples in our historical data, and al-though we have not explicitly modeled this, their ef-fect on forecast errors is included in the empiricalanalysis we have done here. The point is that, whilesuch structural breaks happen, they are not so largeand so common as to over-ride our ability to forecast.Every technology has its own story, its own specificset of causes and effects, that explain why costs went

2

up or down in any given year. Nonetheless, as wedemonstrate here, the long term trends tend to beconsistent, and can be captured via historical timeseries methods with no direct information about theunderlying technology-specific stories.

1900 1950 2000 2050

1e−

021e

+00

1e+

02

pric

e in

$/k

Wh

0.17$/kWh= 2013 price in $/Wp

DOE SunShot Target = coal elec. price

Coal fuel price for electricity generation

nuclear

solar energy

0.1

0.5

5.0

50.0

pric

e in

$/W

p

photovoltaic module pricesU.S. nuclear electricity pricesU.K. Hinkley Point price in 2023

All prices in 2011 U.S. dollars

Figure 1: A comparison of long-term price trends for coal,nuclear power and solar photovoltaic modules. Prices forcoal and nuclear power are levelized costs in the US indollars per kilowatt hour (scale on the left) whereas so-lar modules are in dollars per watt-peak, i.e. the cost forthe capacity to generate a watt of electricity in full sunlight(scale on the right). For coal we use units of the cost of thecoal that would need to be burned in a modern US plant ifit were necessary to buy the coal at its inflation-adjustedprice at different points in the past. Nuclear prices arelevelized electricity costs for US nuclear plants in the yearin which they became operational (from Cooper (2009)).The alignment of the left and right vertical axes is purelysuggestive; based on recent estimates of levelized costs, wetook $0.177/kWh = $0.82/Wp in 2013 (2013$). The num-ber $0.177/kWh is a global value produced as a projectionfor 2013 by the International Energy Agency (Table 4 inInternational Energy Agency (2014)). We note that it iscompatible with estimated values (Table 1 in Baker et al.(2013), Fig. 4 in International Energy Agency (2014)).The red cross is the agreed price for the planned UK Nu-clear power plant at Hinkley Point which is scheduled tocome online in 2023 (£0.0925 ≈ $0.14). The dashed linecorresponds to an earlier target of $0.05/kWh set by thethe U.S. Department of Energy.

In this paper we use a very simple approach to fore-casting, which was originally motivated by Moore’sLaw. As everyone knows, Intel’s ex-CEO, Gordon

Moore, famously predicted that the number of tran-sistors on integrated circuits would double every twoyears, i.e. at an annual rate of about 40%. Makingtransistors smaller also brings along a variety of otherbenefits, such as increased speed, decreased powerconsumption, and less expensive manufacture costsper unit of computation. As a result it quickly be-came clear that Moore’s law applies more broadly, forexample, implying a doubling of computational speedevery 18 months.

Moore’s law stimulated others to look at relateddata more carefully, and they discovered that expo-nential improvement is a reasonable approximationfor other types of computer hardware as well, suchas hard drives. Since the performance of hard drivesdepends on physical factors that are unrelated to tran-sistor density this is an independent fact (though ofcourse the fact that mass storage is essential for com-putation causes a tight coupling between the two tech-nologies). Lienhard, Koh and Magee, and others4 ex-amined data for other products, including many thathave nothing to do with computation or informationprocessing, and postulated that exponential improve-ment is a much more general phenomenon that appliesto many different technologies, even if in most casesthe exponential rates are much slower.

Although Moore’s law is traditionally applied as aregression of the log of the cost on a deterministictime trend, we reformulate it here as a geometric ran-dom walk with drift. This has several advantages.On average it results in more accurate forecasts, es-pecially at short horizons, indicating that it is indeeda better model. In addition, this allows us to usestandard results from the time series forecasting liter-ature5. The technology time series in our sample are

4Examples include Lienhard (2006), Koh & Magee (2006,2008), Bailey et al. (2012), Benson & Magee (2014a,b), Nagyet al. (2013). Studies of improvement in computers over longspans of time indicate super-exponential improvement (Nord-haus 2007, Nagy et al. 2011), suggesting that Moore’s law mayonly be an approximation reasonably valid over spans of timeof 50 years or less. See also e.g. Funk (2013) for an explanationof Moore’s law based on geometric scaling, and Funk & Magee(2014) for empirical evidence regarding fast improvement priorto large production increase.

5Several methods have been defined to obtain prediction in-tervals, i.e. error bars for the forecasts (Chatfield 1993). Theclassical Box-Jenkins methodology for ARIMA processes uses atheoretical formula for the variance of the process, but does notaccount for uncertainty due to parameter estimates. Another

3

typically rather short, often only 15 or 20 points long,so to test hypotheses it is essential to pool the data.Because the geometric random walk is so simple it ispossible to derive formulas for the forecast errors inclosed form. This makes it possible to estimate theforecast errors as a function of both sample size andforecasting horizon, and to combine data from manydifferent technologies into a single analysis. This al-lows us to get highly statistically significant results.And most importantly, because this allows us to sys-tematically test the method on data for many differ-ent technologies, this allows us to make distributionalforecasts for a single technology and have confidencein the results.

Motivated by structure we find in the data, we fur-ther extend Moore’s law to allow for the possibilitythat changes in price are positively autocorrelatedin time. We assume that the logarithm of the costfollows a random walk with drift and autocorrelatednoise, more specifically an Integrated Moving Aver-age process of order (1,1), i.e. an IMA(1,1) model.Under the assumption of sufficiently large autocorre-lation this method produces a good fit to the empiri-cally observed forecasting errors. We derive a formulafor the errors of this more general model, assumingthat all technologies have the same autocorrelationparameter and the forecasts are made using the sim-ple random walk model. We use this to forecast thelikely distribution of the price of photovoltaic solarmodules, and to estimate the probability that solarmodules will undercut a competing technology at agiven date in the future.

We want to stress that we do not mean to claimthat the generalizations of Moore’s law explored hereprovide the most accurate possible forecasts for tech-nological progress. There is a large literature on ex-perience curves6, studying the relationship betweencost and cumulative production originally suggestedby Wright (1936), and many authors have proposed

approach is to use the empirical forecast errors to estimate thedistribution of forecast errors. In this case, one can use ei-ther the in-sample errors (the residuals, as in e.g. Taylor &Bunn (1999)), or the out-of-sample forecast errors (Williams& Goodman 1971, Lee & Scholtes 2014). Several studies havefound that using residuals leads to prediction intervals whichare too tight (Makridakis & Winkler 1989).

6Arrow (1962), Alchian (1963), Argote & Epple (1990), Dut-ton & Thomas (1984), Thompson (2012).

alternatives and generalizations7. Nagy et al. (2013)tested these alternatives using a data set that is veryclose to ours and found that Moore’s and Wright’slaws were roughly tied for first place in terms of theirforecasting performance. An important caveat is thatNagy et al.’s study was based on a trend stationarymodel, and as we argue here, the difference station-ary model is superior, both for forecasting and forstatistical testing. It seems likely that methods usingauxiliary data such as production, patent activity, orR&D can be used to make forecasts for technologicalprogress that incorporate more factors, and that suchmethods should yield improvements over the simplemethod we use here8.

The key assumption made here is that all technolo-gies follow the same random process, even if the driftand volatility parameters of the random process aretechnology specific. This allows us to develop distri-butional forecasts in a highly parsimonious mannerand efficiently test them out of sample. We restrictourselves to forecasting unit cost in this paper, forthe simple reason that we have data for it and it iscomparable across different technologies. The workpresented here provides a simple benchmark againstwhich to compare forecasts of future technologicalperformance based on other methods.

The approach of basing technological forecasts onhistorical data that we pursue here stands in sharpcontrast to the most widely used method, which isbased on expert opinions. The use of expert opin-ions is clearly valuable, and we do not suggest thatit should be supplanted, but it has several seriousdrawbacks. Expert opinions are subjective and canbe biased for a variety of reasons (Albright 2002), in-cluding common information, herding, or vested in-terest. Forecasts for the costs of nuclear power in theUS, for example, were for several decades consistentlylow by roughly a factor of three (Cooper 2009). Asecond problem is that it is very hard to assess theaccuracy of expert forecasts. In contrast the methodwe develop here is objective and the quality of theforecasts is known. Nonetheless we believe that bothmethods are valuable and that they should be used

7See Goddard (1982), Sinclair et al. (2000), Jamasb (2007),Nordhaus (2009).

8See for example Benson & Magee (2014b) for an exampleof how patent data can be used to explain variation in rates ofimprovement among different technologies.

4

side-by-side.9

The remainder of the paper develops as follows: InSection 2 we derive the error distribution for forecastsbased on the geometric random walk as a functionof time horizon and other parameters and show howthe data for different technologies and time horizonsshould be collapsed. We also show how this can begeneralized to allow for autocorrelations in the dataand derive similar (approximate) formulas. In Sec-tion 3 we describe our data set and present an em-pirical relationship between the variance of the noiseand the improvement rate for different technologies.In Section 4 we describe our method of testing themodels against the data, and present the results inSection 5. We then apply our method to give a distri-butional forecast for solar module prices in Section 6and show how this can be used to forecast the likeli-hood that one technology will overtake another. Fi-nally we give some concluding remarks in Section 7.A variety of technical results are given in the appen-dices.

2 Models

2.1 Geometric random walk

In this section we discuss how to formulate Moore’slaw in the presence of noise, and argue that the bestmethod is the geometric random walk with drift. Wethen present a formula for the distribution of expectederrors as a function of the time horizon and the otherparameters of the model, and generalize the formulato allow for autocorrelation in the data generatingprocess. This allows us to pool the errors for manydifferent technologies, which is extremely useful be-cause it makes it possible to test the validity of theseresults using many short time series (such as the datawe have here).

The generalized version of Moore’s law we studyhere is a postulated relationship which in its deter-ministic form is

pt = p0eµt,

where pt is either the unit cost or the unit price of atechnology at time t; we will hereafter refer to it as

9For additional discussion of the advantages and drawbacksof different methods of technology forecasting, see Ayres (1969),Martino (1993) and National Research Council (2009)

the cost. p0 is the initial cost and µ is the exponentialrate of change. (If the technology is improving thenµ < 0.) In order to fit this to data one has to allowfor the possibility of errors and make an assumptionabout the structure of the errors. Typically the liter-ature has treated Moore’s law using linear regression,minimizing squared errors to fit a model of the form

yt = y0 + µt+ et, (1)

where yt = log(pt). From the point of view of theregression, y0 is the intercept, µ is the slope and et isindependent and identically distributed (IID) noise.

But records of technological performance such asthose we study here are time series, giving the costs pjtfor technology j at successive times t = 1, 2, . . . , Tj .It is therefore more natural to use a time series model.The simplest possible choice that yields Moore’s lawin the deterministic limit is the geometric randomwalk with drift,

yt = yt−1 + µ+ nt. (2)

As before µ is the drift and nt is an IID noise process.Letting the noise go to zero recovers the deterministicversion of Moore’s law in either case. When the noiseis nonzero, however, the models behave quite differ-ently. For the regression model the shocks are purelytransitory, i.e. they do not accumulate. In contrast,if y0 is the cost at time t = 0, Eq. 2 can be iteratedand written in the form

yt = y0 + µt+t∑i=1

ni. (3)

This is equivalent to Eq. (1) except for the last term.While in the regression model of Eq. (1) the valueof yt depends only on the current noise and the slopeµ, in the random walk model (Eq. 2) it depends onthe sum of previous shocks. Hence, shocks in the ran-dom walk model accumulate, and the forecasting er-rors grow with time horizon as one would expect, evenif the parameters of the model are perfectly estimated.

An important difference between the two methodsis that for the geometric random walk the residual as-sociated with the most recently observed data point isby definition zero. For the regression model, in con-trast, the most recent point often has a large error(this depends on the way the points happen to devi-ate from the data), often leading to unnaturally large

5

one step ahead forecast errors. This is an importantreason why the geometric random walk is more legit-imate10.

For time series models a key question is whetherthe process has a unit root. Most of our time seriesare much too short for unit root tests to be effective(Blough 1992). Nonetheless, we found that our timeseries forecasts are consistent with the hypothesis ofa unit root and that they perform better than severalalternatives.

2.2 Prediction of forecast errors

We now derive a formula for the forecast errors of thegeometric random walk as a function of time horizon.We assume that all technologies follow the geometricrandom walk, i.e. our noisy version of Moore’s law,but with technology-specific parameters. RewritingEq. (2) slightly, it becomes

yjt = yj,(t−1) + µj + njt,

where the index j indicates technology j. For con-venience we assume that noise njt is IID normal, i.e.njt ∼ N (0,K2

j ). This means that technology j ischaracterized by a drift µj and the standard devia-tion of the noise increments Kj . We will typically notinclude the indices for the technology unless we wantto emphasize the dependence on the technology.

We now derive the expected error distribution forEq. (2) as a function of the time horizon τ . Eq. (2)implies that

yt+τ = yt + µτ +

t+τ∑i=t+1

ni (4)

The point forecast τ steps ahead is11

yt+τ = yt + µτ, (5)10Nagy et al. (2013) used regression models to study a simi-

lar dataset. Their short term forecasts were inaccurate due tothe problem described here. In addition, because the regressionmodel is an unnatural choice for time series forecasting, Nagyet al. had to make ad hoc assumptions to pool data from differ-ent horizons. These problems are avoided using the geometricrandom walk.

11The point forecast is the expected logarithm of the cost forthe random walk with drift model, E[yt+τ ]. We assume yt+τis normally distributed. This means the cost is log-normallydistributed, and the forecast of the median cost is eE[yt+τ ]. Be-cause the mean of a log-normal distribution also depends on thevariance of the underlying normal distribution, the expected

where µ is the estimated µ. The forecast error is de-fined as

E = yt+τ − yt+τ . (6)

Putting Eqs. (4) and (5) into Eq. (6) gives

E = τ(µ− µ) +t+τ∑i=t+1

ni, (7)

which separates the error into two parts. The firstterm is the error due to the fact that the mean is anestimated parameter, and the second term representsthe error due to the fact that unpredictable randomshocks accumulate (Sampson 1991). Assuming thatthe noise increments are i.i.d normal, and that theestimation of the parameters is based on a trailingsample of m data points, in Appendix B.1 we derivethe scaling of the errors with m, τ and K, where K2

is the estimated variance.Because we want to aggregate forecast errors for

technologies with different volatilities, to study howthe errors grow as a function of τ we use the nor-malized mean squared forecast error Ξ(τ). Assumingm > 3 it is

Ξ(τ) ≡ E

[(EK

)2]

=m− 1

m− 3

(τ +

τ2

m

), (8)

where E represents the expectation.This formula makes intuitive sense. The diffusion

term τ is due to the accumulation of noisy fluctuationsthrough time. This term is present even in the limitm → ∞, where the estimation is perfect. The τ2/mterm is due to estimation error in the mean. Theneed to estimate the variance causes the prefactor12

(m−1)/(m−3), and also means that the distributionis Student t rather than normal, i.e.

ε =1√A

(EK

)∼ t(m− 1), (9)

cost diverges when τ →∞ due to parameter uncertainty. Ourforecasts here are for the median cost, which has the importantadvantage that (unlike the mean or the mode) it does not re-quire an estimate of the variance, and is therefore simpler andmore robust.

12The prefactor is significantly different from one only whenm is small. Sampson (1991) derived the same formula butwithout the prefactor since he worked with the true variance.Sampson (1991) also showed that the square term due to errorin the estimation of the drift exists for the regression on a timetrend model, and for more general noise processes. See alsoClements & Hendry (2001).

6

withA = τ + τ2/m. (10)

Eq. (9) is universal, in the sense that the right handside is independent of µj , Kj , and τ . It depends nei-ther on the properties of the technology nor on thetime horizon. As a result we can pool forecast errorsfor different technologies at different time horizons.This property is extremely useful for statistical test-ing, and can also be used to construct distributionalforecasts for a given technology.

2.3 Generalization for autocorrelation

We now generalize the formula above to allow for au-tocorrelations in the error terms. Although the uncor-related random walk model above does surprisinglywell, there is good evidence that there are positiveautocorrelations in the data. In order to incorpo-rate this structure we extend the results above foran ARIMA(0,1,1) (autoregressive integrated movingaverage) model. The zero indicates that we do notuse the autoregressive part, so we will abbreviate thisas an IMA(1,1) model in what follows. The IMA(1,1)model is of the form

yt − yt−1 = µ+ vt + θvt−1, (11)

with the noise vt ∼ N (0, σ2). This model is also a ge-ometric random walk, but with correlated incrementswhen θ 6= 0 (the autocorrelations of the time seriesare positive when θ > 0).

We chose this model rather than other alternativesmainly for its simplicity13. Moreover, our data areoften time-aggregated, that is, our yearly observa-tions are averages of the observed costs over the year.It has been shown that if the true process is a ran-dom walk with drift then aggregation can lead to sub-stantial autocorrelation (Working 1960). In any case,while every technology certainly follows an idiosyn-cratic pattern and may have a complex autocorrela-tion structure and specific measurement errors, usingthe IMA(1,1) as a universal model allows us to par-simoniously understand the empirical forecast errorsand generate robust prediction intervals.

13Our individual time series are very short, which makes itvery difficult to find the proper order of differencing and todistinguish between different ARMA models. For instance,slightly different ARIMA models such as (1,1,0) are far fromimplausible for many technologies.

A key quantity for pooling the data is the variance,which by analogy with the previous model we call Kfor this model as well. It is easy to show that

K2 ≡ var(yt − yt−1) = var(vt + θvt−1) = (1 + θ2)σ2,

see e.g. Box & Jenkins (1970). The relevant formulasfor this case are derived in Appendix B.2. We makethe same point forecasts as before given by Eq. (5).If the variance is known the distribution of forecasterrors is

E ∼ N (0, σ2A∗), (12)

with

A∗ = −2θ+

(1 +

2(m− 1)θ

m+ θ2

)(τ +

τ2

m

). (13)

Note that we recover Eq. (10) when θ = 0. In theusual case where the variance has to be estimated,we derive an approximate formula for the growth anddistribution of the forecast errors by assuming that Kand E are independent. The expected mean squarednormalized error is

Ξ(τ) ≡ E

[(EK

)2]

=m− 1

m− 3

A∗

1 + θ2, (14)

and the distribution of rescaled normalized forecasterrors is

ε∗ =1√

A∗/(1 + θ2)

(EK

)∼ t(m− 1). (15)

These formulas are only approximations so we com-pare them to more exact results obtained through sim-ulations in Appendix B.2 – see in particular Fig. 12.For m > 30 the approximation is excellent, but thereare discrepancies for small values of m.

As before the right hand side is independent of allthe parameters of the technology as well as the timehorizon. Eq. (15) can be viewed as the distribution oferrors around a point forecast, which makes it possibleto collapse many technologies onto a single distribu-tion. This property is extremely useful for statisticaltesting. Alternatively, it can be used to make a distri-butional forecast for the future costs of a given tech-nology. Whenm is sufficiently large the Student t dis-tribution is well-approximated by a standard normal.Using the mean given by Eq. (5) and the variance de-termined by Eqs. (12-13), the distributional forecast

7

for the future logarithm of the cost yt+τ conditionedon (yt, . . . , yt−m+1) is14

yt+τ ∼ N (yt + µτ, K2A∗/(1 + θ2)). (16)

We will return later to the estimation of θ.

2.4 Alternative hypotheses

In addition to autocorrelation, we investigated otherways to generalize the model, such as heavy tails andlong-memory. As discussed in Appendix C.4, based onforecast errors we found little evidence for heavy tails.Long-memory is in a sense an extreme version of theautocorrelation hypothesis15, which produces errorsthat grow faster as a function of the forecasting hori-zon τ than a random walk. Given that long-memoryis a natural result of nonstationarity, which is com-monly associated with technological change, our priorwas that it was a highly plausible alternative. How-ever, as we will see, the geometric random walk withnormal noise increments and autocorrelations seemsto give good agreement for the time scaling of fore-casting errors, so we did not investigate long-memoryfurther.

3 Data

3.1 Data collection

The bulk of our data on technology costs comesfrom the Santa Fe Institute’s Performance CurveDataBase16, which was originally developed by BelaNagy and collaborators; we augment it with a fewother datasets. These data were collected via litera-ture search, with the principal criterion for selectionbeing availability. Fig. 2 plots the time series for each

14Note that although we make the estimate of the varianceθ-dependent, we always use the estimate of the mean corre-sponding to θ = 0. We do this because this is simpler and morerobust.

15A process has long-memory if the autocorrelation functionof its increments is not integrable. Under the long-memoryhypothesis one expects the diffusion term of the normalizedsquared errors to scale as Ξ(τ) ∼ τ2H , where H is the Hurstexponent. In the absence of long-memory H = 1/2, but forlong-memory 1/2 < H < 1. Long-memory can arise from manycauses, including nonstationarity. It is easy to construct plau-sible processes with the µ parameter varying where the meansquared errors grow faster than τ2.

16pcdb.santafe.edu

data set. The motley character of our dataset is clear:The time series for different technologies are of differ-ent lengths and they start and stop at different times.The sharp cutoff for the chemical data, for example,reflects the fact that it comes from a book publishedby the Boston Consulting Group in 1972. Table 1gives a summary of the properties of the data andmore description of the sources can be found in Ap-pendix A. This plot also makes it clear that technolo-gies improve at very different rates.

1940 1960 1980 2000

1e−

081e

−05

1e−

021e

+01

1e+

04

cost

ChemicalHardwareConsumer GoodsEnergyFoodGenomics

Figure 2: Cost vs. time for each technology in our dataset.This shows the 53 technologies out of the original set of66 that have a significant rate of cost improvement (DNAsequencing is divided by 1000 to fit on the plot; the y-axisis in log scale). More details can be found in Table 1 andAppendix A.

A ubiquitous problem in forecasting technologicalprogress is finding invariant units. A favorable exam-ple is electricity. The cost of generating electricity canbe measured in dollars per kWh, making it possibleto sensibly compare competing technologies and mea-sure their progress through time. Even in this favor-able example, however, making electricity cleaner andsafer has a cost, which has affected historical pricesfor technologies such as coal and nuclear power in re-cent years, and means that their costs are difficult tocompare to clean and safe but intermittent sources ofpower such as solar energy. To take an unfavorableexample, our dataset contains appliances such as tele-vision sets, that have dramatically increased in quality

8

through time17. Yet another problem is that some ofthem are potentially subject to scarcity constraints,which might potentially introduce additional trendsand fluctuations.

One should therefore regard our results here as alower bound on what is possible, in the sense thatperforming the analysis with better data in which alltechnologies had invariant units would very likely im-prove the quality of the forecasts. We would love tobe able to make appropriate normalizations, but thework involved is prohibitive; if we dropped all ques-tionable examples we would end with little remainingdata. Most of the data are costs, but in a few casesthey are prices; again, this adds noise but if we wereable to be consistent that should only improve our re-sults. We have done various tests removing data andthe basic results are not sensitive to what is includedand what is omitted (see Fig. 14 in the appendix).

We have removed some technologies that are toosimilar to each other from the Performance CurveDatabase. For instance, when we have two datasetsfor the same technology, we keep only one of them.Our choice was based on data quality and length ofthe time series. This selection left us with 66 tech-nologies, belonging to different sectors that we labelas chemistry, genomics, energy, hardware, consumerdurables and food.

3.2 Data selection and descriptive statis-tics

In this paper we are interested in technologies thatare improving, so we restrict our analysis to thosetechnologies whose rate of improvement is statisticallysignificant based on the available sample. We useda simple one-sided t-test on the first-difference (log)series and removed all technologies for which the p-value indicates that we can’t reject the null that µj =0 at a 10% confidence level18.

Table 1 reports the p-values for the one sided t-testand the bottom of the table shows the technologiesthat are excluded as a result. Table 1 also shows the

17Gordon (1990) provides quality change adjustments for anumber of durable goods. These methods (typically, hedonicregressions) require additional data.

18If the true data generating process is ARIMA(0,1,1), thevariance of the sampling distribution of µ is different, but usingit would require the knowledge of θ.

Technology Industry T µ p value K θ

Transistor Hardware 38 -0.50 0.00 0.24 0.19Geothermal.Electricity Energy 26 -0.05 0.00 0.02 0.15

Milk..US. Food 79 -0.02 0.00 0.02 0.04DRAM Hardware 37 -0.45 0.00 0.38 0.14

Hard.Disk.Drive Hardware 20 -0.58 0.00 0.32 -0.15Automotive..US. Cons. Goods 21 -0.08 0.00 0.05 1.00

Low.Density.Polyethylene Chemical 17 -0.10 0.00 0.06 0.46Polyvinylchloride Chemical 23 -0.07 0.00 0.06 0.32Ethanolamine Chemical 18 -0.06 0.00 0.04 0.36

Concentrating.Solar Energy 26 -0.07 0.00 0.07 0.91AcrylicFiber Chemical 13 -0.10 0.00 0.06 0.02

Styrene Chemical 15 -0.07 0.00 0.05 0.74Titanium.Sponge Chemical 19 -0.10 0.00 0.10 0.61VinylChloride Chemical 11 -0.08 0.00 0.05 -0.22Photovoltaics Energy 34 -0.10 0.00 0.15 0.05

PolyethyleneHD Chemical 15 -0.09 0.00 0.08 0.12VinylAcetate Chemical 13 -0.08 0.00 0.06 0.33Cyclohexane Chemical 17 -0.05 0.00 0.05 0.38BisphenolA Chemical 14 -0.06 0.00 0.05 -0.03

Monochrome.Television Cons. Goods 22 -0.07 0.00 0.08 0.02PolyethyleneLD Chemical 15 -0.08 0.00 0.08 0.88Laser.Diode Hardware 13 -0.36 0.00 0.29 0.37

PolyesterFiber Chemical 13 -0.12 0.00 0.10 -0.16Caprolactam Chemical 11 -0.10 0.00 0.08 0.40

IsopropylAlcohol Chemical 9 -0.04 0.00 0.02 -0.24Polystyrene Chemical 26 -0.06 0.00 0.09 -0.04

Polypropylene Chemical 10 -0.10 0.00 0.07 0.26Pentaerythritol Chemical 21 -0.05 0.00 0.07 0.30

Ethylene Chemical 13 -0.06 0.00 0.06 -0.26Wind.Turbine..Denmark. Energy 20 -0.04 0.00 0.05 0.75

Paraxylene Chemical 12 -0.10 0.00 0.09 -1.00DNA.Sequencing Genomics 13 -0.84 0.00 0.83 0.26NeopreneRubber Chemical 13 -0.02 0.00 0.02 0.83Formaldehyde Chemical 11 -0.07 0.00 0.06 0.36SodiumChlorate Chemical 15 -0.03 0.00 0.04 0.85

Phenol Chemical 14 -0.08 0.00 0.09 -1.00Acrylonitrile Chemical 14 -0.08 0.01 0.11 1.00Beer..Japan. Food 18 -0.03 0.01 0.05 -1.00

Primary.Magnesium Chemical 40 -0.04 0.01 0.09 0.24Ammonia Chemical 13 -0.07 0.02 0.10 1.00Aniline Chemical 12 -0.07 0.02 0.10 0.75Benzene Chemical 17 -0.05 0.02 0.09 -0.10Sodium Chemical 16 -0.01 0.02 0.02 0.42Methanol Chemical 16 -0.08 0.02 0.14 0.29

MaleicAnhydride Chemical 14 -0.07 0.03 0.11 0.73Urea Chemical 12 -0.06 0.03 0.09 0.04

Electric.Range Cons. Goods 22 -0.02 0.03 0.04 -0.14PhthalicAnhydride Chemical 18 -0.08 0.03 0.15 0.31

CarbonBlack Chemical 9 -0.01 0.03 0.02 -1.00Titanium.Dioxide Chemical 9 -0.04 0.04 0.05 -0.41Primary.Aluminum Chemical 40 -0.02 0.06 0.08 0.39

Sorbitol Chemical 8 -0.03 0.06 0.05 -1.00Aluminum Chemical 17 -0.02 0.09 0.04 0.73

Free.Standing.Gas.Range Cons. Goods 22 -0.01 0.10 0.04 -0.30CarbonDisulfide Chemical 10 -0.03 0.12 0.06 -0.04Ethanol..Brazil. Energy 25 -0.05 0.13 0.22 -0.62

Refined.Cane.Sugar Food 34 -0.01 0.23 0.06 -1.00CCGT.Power Energy 10 -0.04 0.25 0.15 -1.00

HydrofluoricAcid Chemical 11 -0.01 0.25 0.04 0.13SodiumHydrosulfite Chemical 9 -0.01 0.29 0.07 -1.00

Corn..US. Food 34 -0.02 0.30 0.17 -1.00Onshore.Gas.Pipeline Energy 14 -0.02 0.31 0.14 0.62

Motor.Gasoline Energy 23 -0.00 0.47 0.05 0.43Magnesium Chemical 19 -0.00 0.47 0.04 0.58Crude.Oil Energy 23 0.01 0.66 0.07 0.63

Nuclear.Electricity Energy 20 0.13 0.99 0.22 -0.13

Table 1: Descriptive statistics and parameter estimates(using the full sample) for all available technologies. Theyare ordered by the p-value of a one-sided t-test for µ, i.e.based on how strong the evidence is that they are improv-ing. The improvement of the last 13 technologies is notstatistically significant and so they are dropped from fur-ther analysis – see the discussion in the text.

9

estimated drift µj and the estimated standard devia-tion Kj based on the full sample for each technologyj. (Throughout the paper we use a hat to denote esti-mates performed within an estimation window of sizem and a tilde to denote the estimates made using thefull sample). Histograms of µj , Kj , sample size Tjand θj are given19 in Fig. 3.

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8

02

46

810

12

− µ~

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8

05

1015

2025

K~

Fre

quen

cy

0 20 40 60 80

02

46

810

12

T

Fre

quen

cy

−1.0 −0.5 0.0 0.5 1.0

02

46

8

θ~

Figure 3: Histogram for the estimated parameters for eachtechnology i based on the full sample (see also Table 1).µj is the annual logarithmic rate of decrease in cost, Kj

is the standard deviation of the noise, Tj is the number ofavailable years of data and θj is the autocorrelation.

3.3 Relation between drift and volatility

Fig. 4 shows a scatter plot of the estimated stan-dard deviation Kj for technology j vs. the esti-mated improvement rate −µj . A linear fit gives K =0.02 − 0.76µ with R2 = 0.87 and standard errors of0.008 for the intercept and 0.04 for the slope, as shownin the figure. A log-log fit gives K = e−0.68(−µ)0.72

with R2 = 0.73 and standard errors for the scalingconstant of 0.18 and for the exponent of 0.06. Thisindicates that on average the uncertainty Kj gets big-ger as the improvement rate −µj increases. There isno reason that we are aware of to expect this a priori.One possible interpretation is that for technologicalinvestment there is a trade-off between risk and re-turns. Another possibility is that faster improvementamplifies fluctuations.

19The θj are estimated by maximum likelihood letting µMLE

be different from µ.

0.02 0.05 0.10 0.20 0.50

0.02

0.05

0.10

0.20

0.50

− µ~

K~

log−log fitlinear fit

ChemicalHardwareConsumer GoodsEnergyFoodGenomics

Figure 4: Scatter plot of the estimated standard deviationKj for technology j against its estimated improvement rate−µj . The dashed line shows a linear fit (which is curvedwhen represented in log scale); the solid line is a log-logfit. Technologies with a faster rate of improvement havehigher uncertainty in their improvement.

4 Estimation procedures

4.1 Statistical validation

We use hindcasting for statistical validation, i.e. foreach technology we pretend to be at a given date inthe past and make forecasts for dates in the future rel-ative to the chosen date20. We have chosen this pro-cedure for several reasons. First, it directly tests thepredictive power of the model rather than its good-ness of fit to the data, and so is resistant to overfitting.Second, it mimics the same procedure that one wouldfollow in making real predictions, and third, it makesefficient use of the data available for testing.

We fit the model at each time step to the m mostrecent changes in cost (i.e. the most recent m + 1years of data). We use the same value of m for alltechnologies and for all forecasts. Because most ofthe time series in our dataset are quite short, andbecause we are more concerned here with testing theprocedure we have developed rather than with makingoptimal forecasts, unless otherwise noted we choosem = 5. This is admittedly very small, but it has theadvantage that it allows us to make a large number of

20This method is also sometimes called backtesting and is aform of cross-validation.

10

forecasts. We will return later to discuss the questionof which value of m makes the best forecasts.

We perform hindcasting exhaustively in the sensethat we make as many forecasts as possible given thechoice of m. For technology j, the cost data yt =log pt exists in years t = 1, 2, . . . , Tj . We then makeforecasts for each feasible year and each feasible timehorizon, i.e. we make forecasts yt0+τ (t0) rooted inyears t0 = (m + 1, . . . , Tj − 1) with forecast horizonτ = (1, . . . , Tj − t0).

0 10 20 30 40 50

010

2030

4050

τ

# of

tech

nolo

gies

0 10 20 30 40 50

010

020

030

040

050

060

070

0

# of

fore

cast

s

# technologies# of forecasts

Figure 5: Data available for testing as a function of theforecast time horizon. Here # of technologies refers to thenumber of technology time series that are long enough tomake at least one forecast at a given time horizon τ . Sim-ilarly, # of forecasts refers to the total number of forecaststhat can be made at time horizon τ . The horizontal lineat τ = 20 indicates our (somewhat arbitrary) choice of amaximum time horizon.

Since our dataset includes technologies of differentlength (see Table 1 and Fig. 2), the number of pos-sible forecasts that can be made with a given histor-ical window m is highest for τ = 1 and decreases forlonger horizons21. Fig. 5 shows the total number ofpossible forecasts that can be made with our datasetat a given horizon τ , and the number of technology

21The number of possible forecasts that can be made using atechnology time series of length Tj is [Tj − (m+ 1)][Tj −m]/2which is O(T 2

j ). Hence the total number of forecast errors con-tributed by a given technology time series is disproportionatelydependent on its length. However, we have checked that aggre-gating the forecast errors so that each technology has an equalweight does not qualitatively change the results.

time series that are long enough to make at least oneforecast at horizon τ . This shows that the amount ofavailable data decreases dramatically for large fore-cast horizons. We somewhat arbitrarily impose an up-per bound of τmax = 20, but find this makes very lit-tle difference in the results (see Appendix C.3). Thereare a total of 8212 possible forecasts that can be madewith an historical window of m = 5, and 6391 fore-casts that can be made with τ ≤ 20.

To test for statistical significance we use a surrogatedata procedure (explained below). There are threereasons for doing this: The first is that, although wederived approximate formulas for the forecast errorsin Eq. (14) and (15), when θ 6= 0 the approximationis not very good for m = 5. The second is that therolling window approach we use for hindcasting im-plies overlaps in both the historical sample used toestimate parameters at each time t0 and overlappingintervals in the future for horizons with τ > 1. Thisimplies substantial correlation in the empirical fore-cast errors, which complicates statistical testing. Thethird reason is that, even if the formulas were exact,we expect finite sample fluctuations. That is, with alimited number of technologies and short time series,we do not expect to find the predicted result exactly;the question is then whether the deviation that weobserve is consistent with the expected deviation.

The surrogate data procedure estimates a null dis-tribution for the normalized mean squared forecasterror under the hypothesized model. This is done bysimulating both the model and the forecasting proce-dure to create a replica of the dataset and the fore-casts. This is repeated for many different realizationsof the noise process in order to generate the null dis-tribution. More specifically, for each technology wegenerate Tj pseudo cost data points using Eq. (11)with µ = µj , K = Kj and a given value of θ, therebymimicking the structure of the data set. We then esti-mate the parameters and perform hindcasting just aswe did for the real data, generating the same numberof forecasts and computing the mean squared forecasterror. This process is then repeated many times withdifferent random number seeds to estimate the distri-bution. This same method can be used to estimateexpected deviations for any quantity, e.g. we also usethis to estimate the expected deviation of the finitesample distribution from the predicted distribution offorecast errors.

11

4.2 Parameter estimation

We estimate the mean and the variance for each tech-nology dynamically, using a rolling window approachto fit the parameters based on the m+ 1 most recentdata points. In each year t0 for which forecasts aremade the drift µt0 is estimated as the sample mean ofthe first differences.

µt0 =1

m

t0−1∑i=t0−m

(yi+1 − yi) =yt0 − yt0−m

m, (17)

where the last equality follows from the fact that thesum is telescopic, and implies that only two pointsare needed to estimate the drift. The volatility isestimated using the unbiased estimator22

K2t0 =

1

m− 1

t0−1∑i=t0−m

[(yi+1 − yi)− µt0 ]2. (18)

This procedure gives us a variable number of forecastsfor each technology j and time horizon τ , rooted atall feasible times t0. We record the forecasting errorsEt0,τ = yt+τ (t0)−yt+τ (t0) and the associated values ofKt0 for all t0 and all τ where we can make forecasts.

The autocorrelation parameter θ for the generalizedmodel has to be treated differently. Our time seriesare simply too short to make reasonable rolling win-dow, technology-specific estimates for θ. With suchsmall values of m the estimated autocorrelations arehighly unreliable.

Our solution is to use a global value of θ, i.e. weuse the same value for all technologies and all pointsin time. It may well be that θ is technology specific,but given the short amount of data it is necessaryto make a choice that performs well under forecast-ing. This is a classic bias-variance trade-off, wherethe variance introduced by statistical estimation of aparameter is so large that the forecasts produced by abiased model with this parameter fixed are superior.With very long time series this could potentially beavoided. As we will see, this procedure seems to workwell. It leaves us with a parameter that has to be esti-mated in-sample, but since this is only one parameter

22This is different from the maximum likelihood estimator,which does not make use of Bessel’s correction (i.e. dividingby (m − 1) instead of m). Our choice is driven by the factthat in practice we use a very small m, making the bias of themaximum likelihood estimator rather large.

estimated from a sample of more than 6, 000 forecaststhe resulting estimate should be reasonably reliable.

Evidence concerning autocorrelations is given inFig. 3, where we present a histogram for the valuesof θj , estimated for each technology j using the fullsample. The results are highly variable. Excludingeight likely outliers where θj = ±1, the mean acrossthe sample is 0.27, and 35 out of the remaining 45 im-proving technologies have positive values of θj . Thisseems to suggest that θ tends to be positive.

We use two different methods for estimating aglobal value of θ. The first method takes advantageof the fact that the magnitude of the forecast errorsis an increasing function of θ (we assume θ > 0) andchooses θm (m as in “matched”) to match the empir-ically observed forecast errors, leading to θm = 0.63as described in the next section. The second methodtakes a weighted average θw (w as in “weighted”) cal-culated as follows. We exclude all technologies forwhich the estimate of θ reveals specification or esti-mation issues (θ ≈ 1 or θ ≈ −1). Then at each hori-zon we compute a weighted average, with the weightsproportional to the number of forecasts made withthat technology. Finally we take the average of thefirst 20 horizon-specific estimated values of θ, leadingto θw = 0.25. See Appendix D.

5 Comparison of models to data

In comparing the model to data we address the fol-lowing five questions:

1. Is the scaling law for the increase in forecastingerrors as a function of time derived in Eqs. (8)and (14) consistent with the data?

2. Does there exist a value of θ such that the nullhypothesis of the model is not rejected? If so,what is this value, and how strong is the evidencethat it is positive?

3. When the normalized errors for different tech-nologies at different time horizons are col-lapsed onto a single distribution, does this agreewith the Student distribution as predicted byEq. (15)?

4. Do the errors scale with the trailing sample sizem as predicted under the assumption that the

12

random process is stationary (i.e. that parame-ters are not changing in time)?

5. Is the model well-specified?

We will see that we get clear affirmative answers tothe first four questions, but we are unable to answerquestion (5).

5.1 Normalized forecast errors as a func-tion of τ

To answer the first question we compute the sampleestimate of the mean squared normalized forecast er-ror Ξ(τ), averaging over all available forecasts for alltechnologies at each time horizon with τ ≤ 20 (seeEq. (14)). Fig. 6 compares the empirical results tothe model with three different values of the autocor-relation parameter θ. Because the approximate errorestimates derived in Eq. (14) break down for small val-ues of m, for each value of θ we estimate the expectedmean squared errors under the null hypothesis of themodel via the surrogate data procedure described inSection 4.123.

The model does a good job of predicting the scal-ing of the forecast errors as a function of the timehorizon τ . The errors are predicted to grow approxi-mately proportional to (τ + τ2/m); at long horizonsthe error growth at each value of θ closely parallelsthat for the empirical forecasts. This suggests thatthis scaling is correct, and that there is no strongsupport for modifications such as long-memory thatwould predict alternative rates of error growth.

Using θm = 0.63 gives a good match to the em-pirical data across the entire range of time horizons.Note that even though we chose θm in order to get thebest possible match, given that we are rescaling datafor different technologies by the empirically measuredsample standard deviations over very short samplesof length m = 5, and that we are predicting across 20different time horizons simultaneously, the ability tofind a value of the parameter θ that matches this wellwas far from guaranteed. (It is completely possible,for example, that there would simply not exist a valueof θ < 1 yielding errors that were sufficiently large).

To test the statistical significance of the results fordifferent values of θ and τ we use the surrogate data

23When θ = 0 the simulated and analytical results are visu-ally indistiguishable. Fig. 6 used the analytical formula, Eq. (8)

1 2 5 10 20

510

2050

100

200

500

Forecast horizon τ

Ξ(τ)

real datatheory(θ = 0, mean)theory(θ = 0.25, mean)theory(θ = 0.63, mean)theory(θ = 0.63, 95 % CI)

Figure 6: Growth of the mean squared normalized forecasterror Ξ(τ) for the empirical forecasts compared to predic-tions using different values of θ. The empirical value ofthe normalised error Ξ(τ) is shown by black dots. Thegrey area corresponds to the 95% confidence intervals forthe case θ = θm. The dashed line represents the predictedsquared normalized error with θ = 0, the dot-dash line isfor θw = 0.25, and the solid line is for θm = 0.63.

procedure described at the end of Section 4.1. Forθm = 0.63 we indicate error bars by showing in greythe region containing the 95% of the simulated real-izations with errors closest to the mean. For τ = 1and τ = 2 the predicted errors are visibly below theempirical observations, but the difference is within theerror bars (though on the edge of the error bars forτ = 1); the agreement is very good at all other val-ues of τ . The autocorrelation parameter θw = 0.25is weakly rejected for τ between 1 and 6 and weaklyaccepted elsewhere, indicating that it is very roughlythe lowest value of θ that is consistent with the dataat the two standard deviations level. In contrast thecase θ = 0, which gives normalized error predictionsthat are lower by about a factor of two, is clearlywell outside of the error bars (note the logarithmicscale). This strongly indicates that a positive value ofθ is required to match the observed errors, satisfyingθ > θw = 0.25.

5.2 Distribution of forecast errors

We now address question (3) by testing whether wecorrectly predict the distribution of forecast errors.

13

Fig. 7 shows the distribution of rescaled forecast er-rors using θm = 0.63 with Eq. (15) to rescale theerrors. Different values of τ are plotted separately,and each is compared to the predicted Student dis-tribution. Overall, the fit is good but at longer hori-zons forecast errors tend to be positive, that is, re-alized technological progress is slightly slower thanpredicted. We have tested to see if this forecast biasis significant, and for τ ≤ 11 we cannot reject thenull that there is no bias even at the 10% level. Athigher horizons there is evidence of forecast bias, butwe have to remember that at these horizons we havemuch less data (and fewer technologies) available fortesting.

−10 −5 0 5 10

0.00

20.

005

0.02

00.

050

0.20

00.

500

cum

ulat

ive

dist

ribut

ion

τ1…20

Figure 7: Cumulative distribution of empirical rescalednormalized forecast errors at different forecast horizons τ .The forecast errors for each technology j are collapsed us-ing Eq. (15) with θ = θm = 0.63. This is done for eachforecast horizon τ = 1, 2, . . . , 20 as indicated in the leg-end. The green thick curve is the theoretical prediction.The positive and negative errors are plotted separately.For the positive errors we compute the number of errorsgreater than a given value X and divide by the total num-ber of errors to estimate the cumulative probability andplot in semi-log scale. For the negative errors we do thesame except that we take the absolute value of the errorand plot against −X.

Fig. 8 shows the empirical distribution with all val-ues of τ pooled together, using rescalings correspond-ing to θ = 0, θw, and θm. The predicted distributionis fairly close to the theoretical prediction, and as ex-pected the fit with θm = 0.63 is better than with

θw = 0.25 or θ = 0.

−15 −10 −5 0 5 10 15

2e−

041e

−03

5e−

032e

−02

1e−

015e

−01

cum

ulat

ive

dist

ribut

ion

Student t(m−1)θ = 0.63θ = 0.25θ = 0

Figure 8: Cumulative distribution of empirical rescalednormalized forecast errors with all τ pooled together forthree different values of the autocorrelation parameter,θ = 0 (dashed line), θ = 0.25 (dot-dash line), and θ = 0.63(solid line). See the caption of Fig. 7 for a description ofhow the cumulative distributions are computed and plot-ted.

To test whether the observed deviations of the em-pirical error distribution from the predicted distribu-tion are significant, we once again use the surrogatedata approach described at the end of Section 4.1.As before we generate many replicas of the datasetand forecasts. For each replica of the dataset andforecasts we compute a set of renormalized errors ε∗

and construct their distribution. We then measurethe average distance between the surrogate distribu-tion and the Student distribution as described in Ap-pendix E. Repeating this process 10,000 times resultsin the sampling distribution of the deviations from theStudent distribution under the null hypothesis thatthe model is correct. We then compare this to the cor-responding value of the average distance between thereal data and the Student distribution, which givesus a p-value under the null hypothesis. We find thatthe model with θm = 0.63 is accepted. In contrastθw = 0.25 is rejected with p-values ranging from 1%to 0.1%, depending on the way in which the averagedistance is computed. The case with θ = 0 is verystrongly rejected.

These results make it clear that the positive auto-

14

correlations are both statistically significant and im-portant. The statistical testing shows that θ = 0.63provides a good estimate for the observed forecastingerrors across a large range of time horizons, with nor-malized forecasting errors that are well-described bythe Student distribution.

5.3 Dependence on sample size m

So far we have used only a small fraction of the datato make each forecast. The choice for the trailingsample of m = 5 was for testing purposes, allowing usto generate a large number of forecasts and test ourmethod for estimating their accuracy.

We now address the question of the optimal valueof m. If the process is stationary in the sense that theparameters (µ,K, θ) are constant, one should alwaysuse the largest possible value of m. If the processis nonstationary, however, it can be advantageous touse a smaller value of m, or alternatively a weightedaverage that decays as it goes into the past. Howstationary is the process generating technology costs,and what is the best choice of m?

We experimented with increasing m as shown inFig. 9, and compared this to the model with θm =0.63. We find that the errors drop as m increasesroughly as one would expect if the process were sta-tionary24, and that the model does a reasonably goodjob of forecasting the errors (see also Appendix C.1).This indicates that the best choice is the largest possi-ble value of m, which in this case is m = 16. Howeverwe should emphasize that it is entirely possible thattesting on a sample with longer time series might yieldan optimal value25 of m > 16.

24Note that to check forecast errors for high m we have usedonly technologies for which at least m+ 2 years were available.For large values of m the statistical variation increases due tolack of data.

25We present the results up to m = 16 because less than athird of the technologies can be used with larger sample sizes.We have performed the same analysis up tom = 35, where only5 technologies are left, and the results remain qualitatively thesame.

1 2 5 10 20

25

1020

5010

050

0

Forecast horizon τ

Ξ(τ)

m

481216

Figure 9: Mean squared normalized forecast error Ξ as afunction of the forecast horizon τ for different sizes of thetrailing sample size m. This is done for m = (4, 8, 12, 16),as shown in the legend. The corresponding theoreticalpredictions are made using θm = 0.63, and are shown assolid curves ordered in the obvious way from top (m = 4)to bottom (m = 16).

5.4 Is the model well-specified?

Because most of our time series are so short it is diffi-cult to say whether or not the model is well-specified.As already noted, for such short series it is impossibleto usefully estimate technology-specific values of theparameter θ, which has forced us to use a global valuefor all technologies. Averaging over the raw samplessuggests a relatively low value θw = 0.25, but a muchhigher value θm = 0.63 is needed to match the empir-ically observed errors. However we should emphasizethat with such short series θ is poorly estimated, andit is not clear that averaging across different technolo-gies is sufficient to fix this problem.

In our view it would be surprising if there are nottechnology-specific variations in θ; after all µj and Kj

vary significantly across technologies. So from thispoint of view it seems likely that the model with aglobal θ is mis-specified. It is not clear whether thiswould be true if we were able to measure technology-specific values of θj . It is remarkable that such a sim-ple model can represent a complicated process such astechnological improvement as well as it does, and inany case, as we have shown, using θ = θm does a goodjob of matching the empirically observed forecasting

15

errors. Nonetheless, testing with more data is clearlydesirable.

6 Application to solar PV modules

In this section we provide a distributional forecast forthe price of solar photovoltaic modules. We then showhow this can be used to make a comparison to a hypo-thetical competing technology in order to estimate theprobability that one technology will be less expensivethan another at a given time horizon.

6.1 A distributional forecast for solar en-ergy

We have shown that the autocorrelated geometricrandom walk can be used to forecast technologicalcost improvement, and that the formula we have de-rived for the distribution of forecast errors works wellwhen applied to many different technologies. We nowdemonstrate how this can be used to make a distri-butional forecast for the cost improvement of a giventechnology. The fact that the method has been ex-tensively tested on many technologies in the previoussection gives us some confidence that this forecast isreliable.

PV

mod

ule

pric

e in

201

3 $/

Wp

1980 1986 1992 1998 2004 2010 2016 2022 2028

0.02

0.14

17.

3954

.6

± 1 std. dev.± 1.5 std. dev.± 2 std. dev.

Figure 10: Forecast for the cost of photovoltaic modules in2013 $/Wp. The point forecasts and the error bars areproduced using Eq. (19) and the parameters as discussedin the text. Shading indicates the quantiles of the distri-bution corresponding to 1, 1.5 and 2 standard deviations.

We make the forecast using Eq. (16). We use allavailable years of past data (m = 33) to fit the pa-rameters µS = µS = −0.10 and KS = KS = 0.15,and we used θ = θm = 0.63. The forecast is givenby Eq. (16) with appropriate substitutions of param-eters, i.e.

yS(t+ τ) ∼ N (yS(t) + µSτ, K2SA∗/(1 + θm

2)), (19)

where A∗(θm) is defined in Eq. (13). Fig. 10 shows thepredicted distribution of likely prices for solar photo-voltaic modules for time horizons up to 2030. Theintervals corresponding to plus or minus two standarddeviations in Fig. 10 are 95% prediction intervals.

The prediction says that it is likely that solar PVmodules will continue to drop in cost at the roughly10% rate that they have in the past. Nonethelessthere is a small probability (about 5%) that the pricein 2030 will be higher than it is now26. While it mightseem remarkable to forecast 15 years ahead with only33 years of past data, note that throughout most ofthe paper we were forecasting 20 years ahead withonly five years of data. As one uses more past data,the width of the distributional forecast decreases. Inaddition, there are considerable variations in the stan-dard deviations Kj of the technologies in Table 1;these variations are reflected in the width of the dis-tribution at any given forecasting horizon. The largedeviation from the trend line that solar module costsmade in the late 90’s and early part of the millenniumcause the estimated future variance to be fairly large.

Except for the estimation of θ, no data from othertechnologies was used in this forecast. Nonetheless,data from other technologies was key in giving us con-fidence that this forecast is reliable.

26This forecast is consistent with the one made several yearsago by Nagy et al. (2013) using data only until 2009. It is dif-ficult to compare this forecast with expert’s elicitation studies,which are often more precise in terms of which PV technologyand which market is predicted, and are often concerned withlevelized costs. Individual experts’ distributional predictionsfor LCOE (see Fig. 6 in Bosetti et al. (2012)) seem tight ascompared to ours (for modules only). However, the predic-tions for the probability that PV will cost less than $0.30/Wpin 2030 reported in Fig.3 of Curtright et al. (2008) are overallcomparable with ours.

16

6.2 Estimating the probability that onetechnology will be less expensive thananother

Suppose we want to compute the probability that agiven technology will be less expensive than anothercompeting technology at a given point in the future.We illustrate how this can be done by comparing thelog cost of photovoltaic modules yS with the log costof a hypothetical alternative technology yC . Both thecost of photovoltaic modules and technology C are as-sumed to follow Eq. (19), but for the sake of argumentwe assume that, like coal, technology C has histori-cally on average had a constant cost, i.e. µC = 0. Wealso assume that the estimation period is the same,and that θC = θS = θm. We want to compute theprobability that τ steps ahead yS < yC . The proba-bility that yS < yC is the probability that the randomvariable Z = yC − yS is positive. Since yS and yC arenormal, assuming they are independent their differ-ence is normal

Z ∼ N(µZ , σ

2Z

),

where µZ = (yC(t) − yS(t)) + τ(µC − µS) and σ2Z =

(A∗/(1 + θ2m))(K2

S + K2C). The probability that yS <

yC is the integral for the positive part, which is ex-pressed in terms of the error function

Pr(yS < yC) =

∫ ∞0

fZ(z)dz

=1

2

[1 + Erf

(µZ√2 σZ

)].

(20)

In Fig. 11 we plot this function using the param-eters estimated for photovoltaics, assuming that thecost of the competing technology is a third that ofsolar at the starting date in 2013, and that it is onaverage not dropping in cost, i.e. µC = 0. We con-sider three different levels of the noise parameter KC

for technology C. Note that changing the noise pa-rameter does not change the expected time when thecurves cross.

The main point of this discussion is that with ourmethod we can reliably forecast the probability thata given technology will surpass a competitor.

0.0

0.2

0.4

0.6

0.8

Forecast horizon

Pro

b(y S

<y C

)

2013 2017 2021 2025 2029 2033 2037 2041

KC = 0KC = 0.1KC = 0.2

Figure 11: Probability that solar photovoltaic modules be-come less expensive than a hypothetical competing technol-ogy C whose initial cost is one third that of solar but ison average not improving, i.e. µC = 0. The curves showEq. (20) using µS = −0.10, KS = 0.15, m = 33 for solarPV and three different values of the noise parameter KC

for technology C. The crossing point is at τ ≈ 11 (2024)in all three cases.

6.3 Discussion of PV relative to coal-firedelectricity and nuclear power

In the above discussion we have carefully avoided dis-cussing a particular competing technology. A forecastfor the full cost of solar PV electricity requires pre-dicting the balance of system costs, for which we lackconsistent historical data, and unlike module costs,the full cost depends on factors such as insolation, in-terest rates and local installation costs. As solar PVgrows to be a significant portion of the energy sup-ply the cost of storage will become very important.Nonetheless, it is useful to discuss it in relation to thetwo competitors mentioned in the introduction.

An analysis of coal-fired electricity, breaking downcosts into their components and examining each of thetrends separately, has been made by McNerney et al.(2011). They show that while coal plant costs (whichare currently roughly 40% of total cost) dropped his-torically, this trend reversed circa 1980. Even if the re-cent trend reverses and plant construction cost dropsdramatically in the future, the cost of coal is likely toeventually dominate the total cost of coal-fired elec-tricity. As mentioned before, this is because the his-

17

torical cost of coal is consistent with a random walkwithout drift, and currently fuel is about 40% of to-tal costs. If coal remains constant in cost (except forrandom fluctuations up or down) then this places ahard bound on how much the total cost of coal-firedelectricity can decrease. Since typical plants have ef-ficiencies the order of 1/3 there is not much room formaking the burning of coal more efficient – even aspectacular efficiency improvement to 2/3 of the the-oretical limit is only an improvement of a factor oftwo, corresponding to the average progress PV mod-ules make in about 7.5 years. Similar arguments applyto oil and natural gas27.

Because historical nuclear power costs have tendedto increase, not just in the US but worldwide, evena forecast that they will remain constant seems op-timistic. Levelized costs for solar PV powerplantsin 2013 were as low as 0.078-0.142 Euro/kWh (0.09-0.16$) in Germany (Kost et al. 2013)28, and in 2014solar PV reached a new record low with an acceptedbid of $0.06/kWh for a plant in Dubai29. When theseare compared to the projected cost of $0.14/kWh in2023 for the Hinkley Point nuclear reactor, it appearsthat the two technologies already have roughly equalcosts, though of course a direct comparison is difficultdue to factors such as intermittency, waste disposal,insurance costs, etc.

As a final note, skeptics have claimed that solar PVcannot be ramped up quickly enough to play a signif-icant role in combatting global warming. A simpletrend extrapolation of the growth of solar energy (PVand solar thermal) suggests that it could represent

27Though much has been made of the recent drop in the priceof natural gas due to fracking, which has had a large effect,one should bear in mind that the drop is tiny in comparisonto the factor of about 2,330 by which solar PV modules havedropped in price. The small change induced by fracking is onlyimportant because it is competing in a narrow price range withother fossil fuel technologies. In work with other collaboratorswe have examined not just oil, coal and gas, but more than ahundred minerals; all of them show remarkably flat historicalprices, i.e. they all change by less than an order of magnitudeover the course of a century.

28Levelized costs decrease more slowly than module costs,but do decrease (Nemet 2006). For instance, installation costsper watt have fallen in Germany and are now about half whatthey are in the U.S. (Barbose et al. 2014).

29See http://www.renewableenergyworld.com/rea/news/article/2015/01/dubai-utility-dewa- procures-the-worlds-cheapest-solar-energy-ever

20% of the energy consumption by 2027. In contrastthe "hi-Ren" (high renewable) scenario of the Interna-tional Energy Agency, which is presumably based onexpert analysis, assumes that PV will generate 16%of total electricity in 2050. Thus even in their op-timistic forecast they assume PV will take 25 yearslonger than the historical trend suggests to generatea lower target. We hope in the future to formulatesimilar methods for forecasting production so that wecan better assess the reliability of such forecasts. SeeAppendix F.

7 Conclusion

Many technologies follow a similar pattern of progressbut with very different rates. In this paper we haveproposed a simple method based on the autocorre-lated geometric random walk to provide robust pre-dictions for technological progress that are stated asdistributions of outcomes rather than point forecasts.We assume that all technologies follow a similar pro-cess except for their rates of improvement and volatil-ity. Under this assumption we can pool forecast er-rors of different technologies to obtain an empiricalestimation of the distribution of forecast errors.

One of the essential points of this paper is that theuse of many technologies allows us to make a betterforecast for a given technology, such as solar PV mod-ules. Although using many technologies does not af-fect our point forecast, it is the essential element thatallows us to make a distributional forecast. The pointis that by treating all technologies as essentially thesame except for their parameters, and collapsing allthe data onto a single distribution, we can pool datafrom many technologies to gain confidence in and cal-ibrate our method for a given technology. It is ofcourse a bold assumption to say that all technologiesfollow a random process with the same form, but theempirical results indicate that this a good hypothesis.

We do not want to suggest in this paper that wethink that Moore’s law provides an optimal forecast-ing method. Quite the contrary, we believe that bygathering more historical data, and by adding otherauxiliary variables, such as production, R&D, patentactivity, there should be considerable room for im-proving forecasting power. In the future we antici-pate that theories will eventually provide causal ex-

18

planations for why technologies improve at such dif-ferent rates, and this will result in better forecasts.Nonetheless, in the meantime the method we haveintroduced here provides a benchmark against whichother approaches can be measured. It provides a proofof principle that technologies can be successfully fore-cast, and that the errors in the forecasts can be reli-ably predicted.

From a policy perspective we believe that ourmethod can be used to provide an objective point ofcomparison to expert forecasts, which are often biasedby vested interests and other factors. The fact that wecan associate uncertainties with our predictions makesthem far more useful than simple point forecasts. Theexample of solar PV modules illustrates that differ-ences in the improvement rate of competing technolo-gies can be dramatic, and that an underdog can beginfar behind the pack and quickly emerge as a front-runner. Given the urgency of limiting greenhouse gasemissions, it is fortuitous that a green technology alsohappens to have such a rapid improvement rate, andis likely to eventually surpass its competition within10 − 20 years. In a context where limited resourcesfor technology investment constrain policy makers tofocus on a few technologies that have a real chanceto eventually achieve and even exceed grid parity, theability to have improved forecasts and know how ac-curate they are should prove particularly useful.

Appendix

A Data

The data are mostly taken from the Santa-Fe Performance Curve DataBase, accessible atpcdb.santafe.edu. The database has been constructedfrom personal communications and from Colpier &Cornland (2002), Goldemberg et al. (2004), Lieber-man (1984), Lipman & Sperling (1999), Zhao (1999),McDonald & Schrattenholzer (2001), Neij et al.(2003), Moore (2006), Nemet (2006), Schilling & Es-mundo (2009). The data on photovoltaic prices hasbeen collected from public releases of Strategies Un-limited, Navigant and SPV Market Research. Thedata on nuclear reactors in Table 1 is from Koomey& Hultman (2007). The data for nuclear power inFig. 1 is from Cooper (2009). The DNA sequencing

data is fromWetterstrand (2015) (cost per human-sizegenome), and for each year we took the last availablemonth (September for 2001-2002 and October after-wards) and corrected for inflation using the US GDPdeflator.

B Distribution of forecast errors

B.1 Random walk with drift

This section derives the distribution of forecast errors.Note that by definition yt+1 − yt = ∆y ∼ N (µ,K2).To obtain µ we assume m sequential independent ob-servations of ∆y, and compute the average. The sam-pling distribution of the mean of a normal variable is

µ ∼ N (µ,K2/m). (21)

Moreover, nt ∼ N (0,K2) implies

t+τ∑i=t+1

ni ∼ N (0, τK2). (22)

Using Eqs. (21) and (22) in Eq. (7) we see that thedistribution of forecast errors is Gaussian

E = τ(µ− µ) +t+τ∑i=t+1

ni ∼ N (0,K2A), (23)

where A = τ + τ2/m (10). Eq. 23 implies

1√A

EK∼ N (0, 1). (24)

Eq. (23) leads to E[E2] = K2(τ + τ2/m), which ap-pears in more general form in Sampson (1991). How-ever we also have to account for the fact that we haveto estimate the variance. Since K2 is the sample vari-ance of a normally distributed random variable, thefollowing standard result holds

(m− 1)K2

K2∼ χ2(m− 1). (25)

If Z ∼ N (0, 1), U ∼ χ2(r), and Z and U are inde-pendent, then Z/

√U/r ∼ t(r). Taking Z from Eq.

(24), U from Eq. (25) and assuming independence,we find that the rescaled normalized forecast errorshave a Student t distribution

1√A

EK∼ t(m− 1). (26)

19

Note that the t distribution has mean 0 but vari-ance df/(df−2), where df are the degrees of freedom.Hence the expected squared rescaled normalized fore-cast error is

E

[(1√A

EK

)2]

= 0 + V ar

[1√A

EK

]=m− 1

m− 3,

leading to Eq. (8) in the main text.

B.2 Integrated Moving Average

Here we derive the distribution of forecast errors giventhat the true process is an IMA(1,1) with known θ,µ and K are estimated assuming that the process isa random walk with drift, and the forecasts are madeas if the process was a random walk with drift. Firstnote that, from Eq. (11),

yt+τ = yt + µτ +t+τ∑i=t+1

[vi + θvi−1].

Using Eq. (5) to make the prediction implies that

E = yt+τ − yt+τ = τ(µ− µ) +t+τ∑i=t+1

[vi + θvi−1].

Now we can substitute

µ =1

m

t−1∑i=t−m

(yi+1 − yi) = µ+1

m

t−1∑i=t−m

[vi+1 + θvi]

to obtain

E =τ

m

(−

t−1∑i=t−m

[vi+1 + θvi]

)+

t+τ∑i=t+1

[vi + θvi−1].

Expanding the two sums, this can be rewritten

E =− τθ

mvt−m −

τ(1 + θ)

m

t−1∑i=t−m+1

vi

+(θ − τ

m

)vt + (1 + θ)

t+τ−1∑i=t+1

vi + vt+τ .

Note that the term vt enters in the forecast errorboth because it has an effect on parameter estimationand because of its effect on future noise. Now thatwe have separated the terms we are left with a sum of

independent normal random variables. Hence we canobtain E ∼ N (0, σ2A∗), where

A∗ ≡(τθ

m

)2

+ (m− 1)

(τ(1 + θ)

m

)2

+(θ − τ

m

)2+ (τ − 1)(1 + θ)2 + 1.

can be simplified as (13) in the main text.

1 2 5 10 20 50

110

100

1000

1000

0

Forecast horizon τ

m=5

m=15

m=40

m → ∞

simulationanalyticalasymptotic

Figure 12: Error growth for large simulations of aIMA(1,1) process, to check Eq. (14) and (15). Simulationsare done using 5000 time series of 100 periods, all withwith µ = 0.04, K = 0.05, θ = 0.6. The insets show thedistribution of forecast errors, as in Fig. 8, for m = 5, 40

To obtain the results with estimated (instead oftrue) variance (Eq. (14) and (15)), we follow the sameprocedure as in Appendix B.1, which assumes inde-pendence between the error and the estimated vari-ance. Fig. 12 shows that the result is not exact butworks reasonably well if m > 15.

C Robustness checks

C.1 Size of the learning window

As a check on the results presented in Section 5.3,we test the dependence of the forecast errors on thesample window m for several different forecast hori-zons. The results are robust to a change of the sizeof learning window m. It is not possible to go belowm = 4, because when m = 3 the Student distributionhas m − 1 = 2 degrees of freedom, hence an infinite

20

variance. Note that to make forecasts using a largem, only the datasets which are long enough can beincluded. The results for a few values of m are shownin Fig. 9. Fig. 13 shows that the normalized meansquared forecast error consistently decreases as thelearning window increases.

4 6 8 10 12 14 16

23

45

m

Ξ(τ)

τ = 1

4 6 8 10 12 14 16

510

1520

m

Ξ(τ)

τ = 2

4 6 8 10 12 14 16

2040

6080

100

m

Ξ(τ)

τ = 5

4 6 8 10 12 14 16

100

300

500

m

Ξ(τ)

τ = 15

Figure 13: Empirical mean squared normalized forecast er-rors as a function of the size of learning window for dif-ferent forecast horizons. The dots are the empirical errors,and the plain lines are those expected if the true modelwas an IMA(1,1) with θm = 0.63.

C.2 Data selection

We have checked how the results change when abouthalf of the technologies are randomly selected and re-moved from the dataset. The shape of the normalizedmean squared forecast error growth does not changeand is shown in Fig. 14. The procedure is based on10000 random trials selecting half the technologies.

1 2 5 10 20

510

2050

100

200

500

Forecast horizon τ

Ξ(τ)

All technologiesHalf technologies: MeanHalf technologies: 95% C.I.

Figure 14: Robustness to dataset selection. Mean squarednormalized forecast errors as a function of τ , when usingonly half of the technologies (26 out 53), chosen at random.The 95% confidence intervals, shown as dashed lines, arefor the mean squared normalized forecast errors when werandomly select 26 technologies.

C.3 Increasing τmax

1 2 5 10 20 50

510

2050

200

500

2000

Forecast horizon τ

Ξ(τ)

Figure 15: Robustness to increasing τmax. Main results(i.e as in Fig. 6 and 8) using τmax = 73. We use θ = 0 andθ = 0.63.

In the main text we have shown the results for a fore-cast horizon up to τmax = 20. Moreover, we haveused only the forecast errors up to τmax to constructthe empirical distribution of forecast errors in Fig. 8

21

and to estimate θ in Appendix D. Fig. 15 shows thatif we use all the forecast errors up to the maximumwith τ = 73 the results do not change significantly.

C.4 Heavy tail innovations

To check the effect of non-normal noise increments onΞ(τ), we simulated random walks with drift with noiseincrements drawn from a Student distribution with 3or 7 degrees of freedom. Fig. 16 shows that fat tailnoise increments do not change the long horizon errorsvery much. While the IMA(1,1) model produces aparallel shift of the errors at medium to long horizons,the Student noise increments generate larger errorsmostly at short horizons. Thus fat-tail innovations arenot the most important source of discrepancy betweenthe geometric random walk model and the empiricaldata.

1 2 5 10 20 50 100

510

5050

050

00

Forecast horizon τ

Ξ(τ)

IMA(1,1)RWD, NormalRWD, t(df=3)RWD, t(df=7)

Figure 16: Effect of fat tail innovations on error growth.The figure shows the growth of the mean squared normal-ized forecast errors for four models, showing that introduc-ing fat tail innovations in a random walk with drift (RWD)mostly increases errors only at short horizons.

D Procedure for selecting the auto-correlation parameter θ

We select θ in several ways. The first method is tocompute a variety of weighted means for the θj esti-mated on individual series. The main problem withthis approach is that for some technology series the

0.0 0.4 0.8

1.0

1.4

1.8

2.2

θ

Z θm*

5 10 15 20

0.24

50.

255

τ

θ w(τ

)

θw* = E(θw(τ))

Figure 17: Estimation of θ as a global parameter

0.0 0.2 0.4 0.6

−2

01

2

θ

ln(Ξ

IMA^

Ξ RW

D^

)x10

0 τ1210

5 10 15 20

01

23

4

τ

−ln

(ΞIM

A^Ξ R

WD

^) x

100

EmpiricalSimulation: meanSimulation: 95% C.I.

Figure 18: Using the IMA model to make better forecasts.The right panel uses θ = 0.25

estimated θ was very close to 1 or -1, indicating mis-specification or estimation problems. After removingthese 8 technologies the mean with equal weights foreach technology is 0.27 with standard deviation 0.35.We can also compute the weighted mean at each fore-cast horizon, with the weights being equal to the shareof each technology in the number of forecast errorsavailable at a given forecast horizon. In this case, theweighted mean θw(τ) will not necessarily be constantover time. Fig. 17 (right) shows that θw(τ) oscillatesbetween 0.24 and 0.26. Taking the average over thefirst 20 periods, we have θw = 1

20

∑20τ=1 θw(τ) = 0.25.

When doing this, we do not mean to imply that ourformulas are valid for a system with heterogenous θj ;we simply propose a best guess for a universal θ.

The second approach is to select θ in order to matchthe errors. As before we generate many artificial datasets using the IMA(1,1) model. Larger values of θimply that using the simple random walk model tomake the forecasts will result in higher forecast er-rors. Denote by Ξ(τ)empi the empirical mean squarednormalized forecast error as depicted in Fig. 6, and byΞ(τ)sim,θ the expected mean squared normalized fore-cast error obtained by simulating IMA(1,1) datasets3,000 times with a particular global value of θ andtaking the average. We study the ratio of the these

22

two, averaged over all 1 . . . τmax = 20 periods, i.e.Z(θ) = 1

20

∑20τ=1

Ξ(τ)empiΞ(τ)sim,θ

. The values are shown inFig. 17 (left). The value at which |Z − 1| is minimumis at θm = 0.63.

We also tried to make forecasts using the IMAmodel to check that forecasts are improved: whichvalue of θ allows the IMA model to produce betterforecasts? We apply the IMA(1,1) model with differ-ent values of θ to make forecasts (with the usual es-timate of the drift term µ) and study the normalizederror as a function of θ. We record the mean squarednormalized error and repeat this exercise for a rangeof values of θ. The results for horizons 1,2, and 10 arereported in Fig. 18 (left). This shows that the bestvalue of θ depends on the time horizon τ . The curveshows the mean squared normalized forecast error ata given forecast horizon as a function of the value ofθ assumed to make the forecasts. The vertical linesshow the minima, at 0.26, 0.40, and 0.66. To makethe curves fit on the plot, given that the mean squarednormalized forecast error increases with τ , the valuesare normalized by the mean squared normalized fore-cast error for θ = 0, that is, obtained when assumingthat the true process is a random walk with drift.We also see that as the forecast horizon increases theimprovement from taking the autocorrelation into ac-count decreases (Fig. 18, right), as expected theoret-ically from an IMA process. Note that the improve-ment in forecasting error is only a few percent, evenfor τ = 1, indicating that this makes little difference.

E Comparison of the empirical dis-tribution of rescaled errors to thepredicted Student distribution

In this section we check whether the deviations of theempirical forecast errors from the predicted theoret-ical distribution shown in Fig. 8 are consistent withstatistical sampling error. For a given value of θ wegenerate a surrogate data set and surrogate forecastsmimicking our empirical data as described at the endof Section 4.1. We then construct a sample surrogate(cumulative) distribution Pk for the pooled rescalederrors ε∗ of Eq. (15). We measure the distributionPk over 1,000 equally spaced values xk on the inter-val [−15; 15]. Pk is estimated by simply counting the

number of observations less than xk. This is thencompared to the predicted Student distribution tk bycomputing the difference ∆k = Pk − tk between thesurrogate distribution and the Student distribution ineach interval. We measure the overall deviation be-tween the surrogate and the Student using three dif-ferent measures of deviation:

∑k |∆k|,

∑k(∆k)

2, andmax ∆k. We then repeat this process 10,000 times togenerate a histogram for each of the measures above,and compare this to the measured value of the devia-tion for the real data.

Results for doing this for θw = 0.25 and θm = 0.63are reported in Fig. 19. For θw the resulting p-values(the shares of random datasets with a deviation higherthan the empirical deviation) are (0.001, 0.002, 0.011)respectively using (

∑k |∆k|,

∑k(∆k)

2, max ∆k) tomeasure the deviation. In contrast for θm = 0.63the p-values are (0.21, 0.16, 0.20). Thus θm = 0.63 isaccepted and θw = 0.25 is rejected. The uncorrelatedcase θ = 0 is rejected even more strongly.

Fre

quen

cy

2 4 6 8 10

020

040

060

0

∑ ∆

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8

050

015

00

∑(∆)2

Fre

quen

cy

0.00 0.04 0.08

020

040

060

0

max ∆

Fre

quen

cy

0 5 10 15

020

060

010

00

∑ ∆

Fre

quen

cy

0.0 0.4 0.8 1.2

020

060

0

∑(∆)2

Fre

quen

cy

0.02 0.06 0.10

020

040

0

max ∆

Figure 19: Expected deviations of the distribution of therescaled variable ε∗ of Eq. (15) from the Student distri-bution for hindcasting experiments as we do here using adataset with the same properties as ours. The histogramsshow the sampling distribution of a given statistic, and thethick black line shows the empirical value on real data. Thesimulations use θ = 0.25 (3 upper panels) and θ = 0.63 (3lower panels).

F A trend extrapolation of solarenergy capacity

In this paper we have been concerned with forecast-ing costs. For some applications it is also useful toforecast production. Our exploratory work so far sug-

23

gests that, while the same basic methods can be ap-plied, production seems more likely to deviate system-atically from increasing exponentially. Nonetheless,Nagy et al. (2013) found that as a rough approxi-mation most of the technologies in our data set canbe crudely (but usefully) approximated as having ex-ponentially increasing production for a long span oftheir development cycle, and solar PV is no excep-tion. Trend extrapolation can add perspective, evenif it comes without good error estimates, and the ex-ample we present below motivates the need for morework to formulate better methods for assessing the re-liability of production forecasts (for an example, seeShlyakhter et al. (1994)).

Many analysts have expressed concerns about thetime required to build the needed capacity for solarenergy to play a role in reducing greenhouse gas emis-sions. The "hi-Ren" (high renewable) scenario of theInternational Energy Agency assumes that PV willgenerate 16% of total electricity30 in 2050; this wasrecently increased from the previous estimate of only11%. As a point of comparison, what do past trendssuggest?

Though estimates vary, over the last ten years cu-mulative installed capacity of PV has grown at animpressive rate. According to BP’s Statistical Reviewof World Energy 2014, during the period from 1983-2013 solar energy as a whole grew at an annual rate of42.5% and in 2014 represented about 0.22% of totalprimary energy consumption, as shown in Fig. 20. Bycomparison total primary energy consumption grewat an annual rate of 2.6% over the period 1965-2013.Given that solar energy is an intermittent source, it ismuch easier for it to contribute when it supplies only aminority of energy: new supporting technologies willbe required once it becomes a major player. If wesomewhat arbitrarily pick 20% as a target, assumingboth these trends continue unaltered, a simple calcu-lation shows that this would be achieved in about 13.7years31. That is, under these assumptions in 2027 so-lar would represent 20% of energy consumption. Ofcourse this is only an extrapolation, but it puts intoperspective claims that solar energy cannot play anessential role in mitigating global warming on a rela-

30Electricity generation uses about 40% of the world’s pri-mary energy but is expected to grow significantly.

31In this deterministic setting, the time to meet this goal isthe solution for t of 0.0022(1.425)t = 0.2(1.026)t.

tively short timescale.

1980 2000 2020 2040 2060

1e−

031e

−01

1e+

011e

+03

mill

ion

tons

oil

equi

vale

nt

Primary Energy

Oil

Gas

Coal

Nuclear

Hydro

Solar

WindGeothermal, Biomass, and other renewables

202720% primary energy

Figure 20: Global energy consumption due to each of themajor sources from BP Statistical Review of World Energy(BP 2014). Under a projection for solar energy obtainedby fitting to the historical data the target of 20% of globalprimary energy is achieved in 2027.

Of course the usual caveats apply, and the limita-tions of such forecasting is evident in the historicalseries of Fig. 20. The increase of solar is far fromsmooth, wind has a rather dramatic break in its slopein roughly 1988, and a forecast for nuclear power madein 1980 based on production alone would have beenfar more optimistic than one today. It would be in-teresting to use a richer economic model to forecastcost and production simultaneously, but this is be-yond the scope of this paper. The point here wassimply to show that if growth trends continue as theyhave in the past significant contributions by solar areachievable.

References

Alberth, S. (2008), ‘Forecasting technology costs viathe experience curve: Myth or magic?’, Technolog-ical Forecasting and Social Change 75(7), 952–983.

Albright, R. E. (2002), ‘What can past technologyforecasts tell us about the future?’, TechnologicalForecasting and Social Change 69(5), 443–464.

Alchian, A. (1963), ‘Reliability of progress curves

24

in airframe production’, Econometrica 31(4), 679–693.

Argote, L. & Epple, D. (1990), ‘Learning curves inmanufacturing’, Science 247(4945), 920–924.

Arrow, K. J. (1962), ‘The economic implications oflearning by doing’, The Review of Economic Studies29(3), 155–173.

Ayres, R. U. (1969), ‘Technological forecasting andlong-range planning’.

Bailey, A., Bui, Q. M., Farmer, J., Margolis, R. &Ramesh, R. (2012), Forecasting technological inno-vation, in ‘ARCS Workshops (ARCS), 2012’, pp. 1–6.

Baker, E., Fowlie, M., Lemoine, D. & Reynolds, S. S.(2013), ‘The economics of solar electricity’, AnnualReview of Resource Economics 5, 387–426.

Barbose, G., Darghouth, N., Weaver, S. & Wiser, R.(2014), Tracking the Sun VI: An Historical Sum-mary of the Installed Price of Photovoltaics in theUnited States from 1998 to 2012, NREL.

Benson, C. L. & Magee, C. L. (2014a), ‘On improve-ment rates for renewable energy technologies: SolarPV, wind turbines, capacitors, and batteries’, Re-newable Energy 68, 745–751.

Benson, C. L. & Magee, C. L. (2014b), ‘Quantitativedetermination of technological improvement frompatent data’, ESD Working Paper Series 2014-29.

Blough, S. R. (1992), ‘The relationship between powerand level for generic unit root tests in finite sam-ples’, Journal of Applied Econometrics 7(3), 295–308.

Bosetti, V., Catenacci, M., Fiorese, G. & Verdolini,E. (2012), ‘The future prospect of pv and csp solartechnologies: An expert elicitation survey’, EnergyPolicy 49, 308–317.

Boston Consulting Group (1972), Perspectives on Ex-perience, 3 edn, The Boston Consulting Group,Inc., One Boston Place, Boston, Massachusetts02106.

Box, G. E. & Jenkins, G. M. (1970), Time SeriesAnalysis: Forecasting and Control, Holden-Day,San Francisco.

BP (2014), ‘BP statistical review of world energy’.URL: http://www.bp.com/en/global/corporate/about-bp/energy-economics/statistical-review-of-world-energy.html

Chatfield, C. (1993), ‘Calculating interval fore-casts’, Journal of Business & Economic Statistics11(2), 121–135.

Clements, M. P. & Hendry, D. F. (2001), ‘Forecast-ing with difference-stationary and trend-stationarymodels’, Econometrics Journal 4(1), 1–19.

Colpier, U. C. & Cornland, D. (2002), ‘The economicsof the combined cycle gas turbine, an experiencecurve analysis’, Energy Policy 30(4), 309–316.

Cooper, M. (2009), The economics of nuclear reactors:Renaissance or relapse, Technical report, VermontLaw School.

Curtright, A. E., Morgan, M. G. & Keith, D. W.(2008), ‘Expert assessments of future photovoltaictechnologies’, Environmental Science & Technology42(24), 9031–9038.

Dutton, J. M. & Thomas, A. (1984), ‘Treatingprogress functions as a managerial opportunity’,Academy of Management Review 9(2), 235–247.

Funk, J. L. (2013), ‘What drives exponentialimprovements?’, California Management Review55(3), 134–155.

Funk, J. L. & Magee, C. L. (2014), ‘Rapid improve-ments with no commercial production: How do theimprovements occur?’, Research Policy .

Goddard, C. (1982), ‘Debunking the learning curve’,Components, Hybrids, and Manufacturing Technol-ogy, IEEE Transactions on 5(4), 328–335.

Goldemberg, J., Coelho, S. T., Nastari, P. M. & Lu-con, O. (2004), ‘Ethanol learning curve, the Brazil-ian experience’, Biomass and Bioenergy 26(3), 301–304.

Gordon, R. J. (1990), The Measurement of DurableGoods Prices, University of Chicago Press.

25

International Energy Agency (2014), Technologyroadmap: Solar photovoltaic energy (2014 ed.),Technical report, OECD/IEA, Paris.

Jamasb, T. (2007), ‘Technical change theory andlearning curves: patterns of progress in electric-ity generation technologies’, The Energy Journal28(3), 51–71.

Jantsch, E. (1967), Technological forecasting in per-spective, OCDE.

Koh, H. & Magee, C. L. (2006), ‘A functional ap-proach for studying technological progress: Ap-plication to information technology’, TechnologicalForecasting and Social Change 73(9), 1061–1083.

Koh, H. & Magee, C. L. (2008), ‘A functional ap-proach for studying technological progress: Exten-sion to energy technology’, Technological Forecast-ing and Social Change 75(6), 735–758.

Koomey, J. & Hultman, N. E. (2007), ‘A reactor-level analysis of busbar costs for US nuclear plants,1970–2005’, Energy Policy 35(11), 5630 – 5642.

Kost, C., Mayer, J., Thomsen, J., Hartmann, N.,Senkpiel, C., Philipps, S., Nold, S., Lude, S., Saad,N. & Schleg, T. L. (2013), Levelized cost of elec-tricity: Renewable energy technologies., Technicalreport, Fraunhofer Institute for Solar Energy Sys-tems.

Lee, Y. S. & Scholtes, S. (2014), ‘Empirical predictionintervals revisited’, International Journal of Fore-casting 30(2), 217–234.

Lieberman, M. B. (1984), ‘The learning curve andpricing in the chemical processing industries’, TheRAND Journal of Economics 15(2), 213–228.

Lienhard, J. H. (2006), How invention begins: Echoesof old voices in the rise of new machines, OxfordUniversity Press.

Lipman, T. E. & Sperling, D. (1999), ‘Experiencecurves for policy making: the case of energy tech-nologies’.

Makridakis, S. & Winkler, R. L. (1989), ‘Samplingdistributions of post-sample forecasting errors’, Ap-plied Statistics 38(2), 331–342.

Martino, J. P. (1993), Technological forecasting fordecision making, McGraw-Hill, Inc.

McDonald, A. & Schrattenholzer, L. (2001), ‘Learn-ing rates for energy technologies’, Energy Policy29(4), 255–261.

McNerney, J., Farmer, J. D. & Trancik, J. E. (2011),‘Historical costs of coal-fired electricity and impli-cations for the future’, Energy Policy 39(6), 3042–3054.

Moore, G. E. (2006), ‘Behind the ubiquitous mi-crochip’.

Nagy, B., Farmer, J. D., Bui, Q. M. & Trancik, J. E.(2013), ‘Statistical basis for predicting technologi-cal progress’, PloS one 8(2), e52669.

Nagy, B., Farmer, J. D., Trancik, J. E. & Gonzales,J. P. (2011), ‘Superexponential long-term trends ininformation technology’, Technological Forecastingand Social Change 78(8), 1356–1364.

National Research Council (2009), Persistent Fore-casting of Disruptive Technologies., The NationalAcademies Press.

Neij, L., Andersen, P. D., Durstewitz, M., Helby, P.,Hoppe-Kilpper, M. & Morthorst, P. (2003), Expe-rience curves: A tool for energy policy assessment,Environmental and Energy Systems Studies, Univ.

Nemet, G. F. (2006), ‘Beyond the learning curve: fac-tors influencing cost reductions in photovoltaics’,Energy Policy 34(17), 3218–3232.

Nordhaus, W. D. (2007), ‘Two centuries of productiv-ity growth in computing’, The Journal of EconomicHistory 67(01), 128–159.

Nordhaus, W. D. (2009), The perils of the learn-ing model for modeling endogenous technologicalchange, Technical report, National Bureau of Eco-nomic Research.

Perlin, J. (1999), From space to earth: the story ofsolar electricity, Earthscan.

Sampson, M. (1991), ‘The effect of parameter uncer-tainty on forecast variances and confidence intervalsfor unit root and trend stationary time-series mod-els’, Journal of Applied Econometrics 6(1), 67–76.

26

Schilling, M. A. & Esmundo, M. (2009), ‘Technologys-curves in renewable energy alternatives: Analy-sis and implications for industry and government’,Energy Policy 37(5), 1767 – 1781.

Shlyakhter, A. I., Kammen, D. M., Broido, C. L. &Wilson, R. (1994), ‘Quantifying the credibility ofenergy projections from trends in past data: Theus energy sector’, Energy Policy 22(2), 119–130.

Sinclair, G., Klepper, S. & Cohen, W. (2000), ‘What’sexperience got to do with it? sources of cost reduc-tion in a large specialty chemicals producer’, Man-agement Science 46(1), 28–45.

Taylor, J. W. & Bunn, D. W. (1999), ‘A quantileregression approach to generating prediction inter-vals’, Management Science 45(2), 225–237.

Thompson, P. (2012), ‘The relationship between unitcost and cumulative quantity and the evidence fororganizational learning-by-doing’, The Journal ofEconomic Perspectives 26(3), 203–224.

Wetterstrand, K. (2015), ‘DNA sequencing costs:Data from the NHGRI genome sequencing program(GSP)’.URL: www.genome.gov/sequencingcosts

Williams, W. & Goodman, M. (1971), ‘A simplemethod for the construction of empirical confidencelimits for economic forecasts’, Journal of the Amer-ican Statistical Association 66(336), 752–754.

Working, H. (1960), ‘Note on the correlation of firstdifferences of averages in a random chain’, Econo-metrica 28(4), pp. 916–918.

Wright, T. P. (1936), ‘Factors affecting the cost ofairplanes’, Journal of the Aeronautical Sciences3(4), 122—128.

Zhao, J. (1999), ‘The diffusion and costs of naturalgas infrastructures’.

27