Automatic Construction and Natural-Language Description of Nonparametric Regression Models James Robert Lloyd Department of Engineering University of Cambridge David Duvenaud Department of Engineering University of Cambridge Roger Grosse Brain and Cognitive Sciences Massachusetts Institute of Technology Joshua B. Tenenbaum Brain and Cognitive Sciences Massachusetts Institute of Technology Zoubin Ghahramani Department of Engineering University of Cambridge Abstract This paper presents the beginnings of an automatic statisti- cian, focusing on regression problems. Our system explores an open-ended space of possible statistical models to discover a good explanation of the data, and then produces a detailed report with figures and natural-language text. Our approach treats unknown functions nonparametrically using Gaussian processes, which has two important conse- quences. First, Gaussian processes model functions in terms of high-level properties (e.g. smoothness, trends, periodicity, changepoints). Taken together with the compositional struc- ture of our language of models, this allows us to automati- cally describe functions through a decomposition into addi- tive parts. Second, the use of flexible nonparametric models and a rich language for composing them in an open-ended manner also results in state-of-the-art extrapolation perfor- mance evaluated over 13 real time series data sets from vari- ous domains. 1 Introduction Automating the process of statistical modeling would have a tremendous impact on fields that currently rely on expert statisticians, machine learning researchers, and data scien- tists. While fitting simple models (such as linear regression) is largely automated by standard software packages, there has been little work on the automatic construction of flexible but interpretable models. What are the ingredients required for an artificial intelligence system to be able to perform sta- tistical modeling automatically? In this paper we conjecture that the following ingredients may be useful for building an AI system for statistics, and we develop a working system which incorporates them: • An open-ended language of models expressive enough to capture many of the modeling assumptions and model composition techniques applied by human statisticians to capture real-world phenomena • A search procedure to efficiently explore the space of models spanned by the language • A principled method for evaluating models in terms of their complexity and their degree of fit to the data • A procedure for automatically generating reports which explain and visualize different factors underlying the data, make the chosen modeling assumptions explicit, This component is approximately periodic with a period of 10.8 years. Across periods the shape of this function varies smoothly with a typical lengthscale of 36.9 years. The shape of this function within each period is very smooth and resembles a sinusoid. This component applies until 1643 and from 1716 onwards. This component explains 71.5% of the residual variance; this increases the total variance explained from 72.8% to 92.3%. The addition of this component reduces the cross validated MAE by 16.82% from 0.18 to 0.15. Posterior of component 4 1650 1700 1750 1800 1850 1900 1950 2000 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 Sum of components up to component 4 1650 1700 1750 1800 1850 1900 1950 2000 1360 1360.5 1361 1361.5 1362 Figure 8: Pointwise posterior of component 4 (left) and the posterior of the cumulative sum of components with data (right) Figure 1: Extract from an automatically-generated report describing the model components discovered by automatic model search. This part of the report isolates and describes the approximately 11-year sunspot cycle, also noting its dis- appearance during the 16th century, a time known as the Maunder minimum (Lean, Beer, and Bradley, 1995). and quantify how each component improves the predic- tive power of the model In this paper, we introduce a system for modeling time- series data containing the above ingredients which we call the Automatic Bayesian Covariance Discovery (ABCD) sys- tem. The system defines an open-ended language of Gaus- sian process models via a compositional grammar. The space is searched greedily, using marginal likelihood and the Bayesian Information Criterion (BIC) to evaluate mod- els. The compositional structure of the language allows us to develop a method for automatically translating components of the model into natural-language descriptions of patterns in the data. We show examples of automatically generated reports which highlight interpretable features discovered in a vari- ety of data sets (e.g. figure 1). The supplementary material to this paper includes 13 complete reports automatically gen- erated by ABCD 1 . Good statistical modeling requires not only interpretabil- ity but predictive accuracy. We compare ABCD against ex- 1 A link to these reports will be maintained at http://mlg. eng.cam.ac.uk/lloyd/ arXiv:1402.4304v2 [stat.ML] 24 Feb 2014
Transcript
Automatic Construction and Natural-Language Descriptionof Nonparametric Regression Models
James Robert LloydDepartment of EngineeringUniversity of Cambridge
David DuvenaudDepartment of EngineeringUniversity of Cambridge
Roger GrosseBrain and Cognitive Sciences
Massachusetts Institute of Technology
Joshua B. TenenbaumBrain and Cognitive Sciences
Massachusetts Institute of Technology
Zoubin GhahramaniDepartment of EngineeringUniversity of Cambridge
Abstract
This paper presents the beginnings of an automatic statisti-cian, focusing on regression problems. Our system exploresan open-ended space of possible statistical models to discovera good explanation of the data, and then produces a detailedreport with figures and natural-language text.Our approach treats unknown functions nonparametricallyusing Gaussian processes, which has two important conse-quences. First, Gaussian processes model functions in termsof high-level properties (e.g. smoothness, trends, periodicity,changepoints). Taken together with the compositional struc-ture of our language of models, this allows us to automati-cally describe functions through a decomposition into addi-tive parts. Second, the use of flexible nonparametric modelsand a rich language for composing them in an open-endedmanner also results in state-of-the-art extrapolation perfor-mance evaluated over 13 real time series data sets from vari-ous domains.
1 IntroductionAutomating the process of statistical modeling would havea tremendous impact on fields that currently rely on expertstatisticians, machine learning researchers, and data scien-tists. While fitting simple models (such as linear regression)is largely automated by standard software packages, therehas been little work on the automatic construction of flexiblebut interpretable models. What are the ingredients requiredfor an artificial intelligence system to be able to perform sta-tistical modeling automatically? In this paper we conjecturethat the following ingredients may be useful for building anAI system for statistics, and we develop a working systemwhich incorporates them:• An open-ended language of models expressive enough
to capture many of the modeling assumptions and modelcomposition techniques applied by human statisticians tocapture real-world phenomena
• A search procedure to efficiently explore the space ofmodels spanned by the language
• A principled method for evaluating models in terms oftheir complexity and their degree of fit to the data
• A procedure for automatically generating reportswhich explain and visualize different factors underlyingthe data, make the chosen modeling assumptions explicit,
2.4 Component 4 : An approximately periodic function with a period of 10.8 years. Thisfunction applies until 1643 and from 1716 onwards
This component is approximately periodic with a period of 10.8 years. Across periods the shape ofthis function varies smoothly with a typical lengthscale of 36.9 years. The shape of this functionwithin each period is very smooth and resembles a sinusoid. This component applies until 1643 andfrom 1716 onwards.
This component explains 71.5% of the residual variance; this increases the total variance explainedfrom 72.8% to 92.3%. The addition of this component reduces the cross validated MAE by 16.82%from 0.18 to 0.15.
Posterior of component 4
1650 1700 1750 1800 1850 1900 1950 2000−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6Sum of components up to component 4
1650 1700 1750 1800 1850 1900 1950 20001360
1360.5
1361
1361.5
1362
Figure 8: Pointwise posterior of component 4 (left) and the posterior of the cumulative sum ofcomponents with data (right)
Residuals after component 4
1650 1700 1750 1800 1850 1900 1950 2000−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
Figure 9: Pointwise posterior of residuals after adding component 4
Figure 1: Extract from an automatically-generated reportdescribing the model components discovered by automaticmodel search. This part of the report isolates and describesthe approximately 11-year sunspot cycle, also noting its dis-appearance during the 16th century, a time known as theMaunder minimum (Lean, Beer, and Bradley, 1995).
and quantify how each component improves the predic-tive power of the model
In this paper, we introduce a system for modeling time-series data containing the above ingredients which we callthe Automatic Bayesian Covariance Discovery (ABCD) sys-tem. The system defines an open-ended language of Gaus-sian process models via a compositional grammar. Thespace is searched greedily, using marginal likelihood andthe Bayesian Information Criterion (BIC) to evaluate mod-els. The compositional structure of the language allows us todevelop a method for automatically translating componentsof the model into natural-language descriptions of patternsin the data.
We show examples of automatically generated reportswhich highlight interpretable features discovered in a vari-ety of data sets (e.g. figure 1). The supplementary material tothis paper includes 13 complete reports automatically gen-erated by ABCD1.
Good statistical modeling requires not only interpretabil-ity but predictive accuracy. We compare ABCD against ex-
1A link to these reports will be maintained at http://mlg.eng.cam.ac.uk/lloyd/
isting model construction techniques in terms of predictiveperformance at extrapolation, and we find state-of-the-artperformance on 13 time series. In the remainder of this paperwe describe the components of ABCD in detail.
2 A language of regression modelsThe general problem of regression consists of learning afunction f mapping from some input space X to some out-put space Y . We would like an expressive language whichcan represent both simple parametric forms of f such as lin-ear, polynomial, etc. and also complex nonparametric func-tions specified in terms of properties such as smoothness,periodicity, etc. Fortunately, Gaussian processes (GPs) pro-vide a very general and analytically tractable way of captur-ing both simple and complex functions.
Gaussian processes are distributions over functionssuch that any finite subset of function evaluations,(f(x1), f(x2), . . . f(xN )), have a joint Gaussian distribu-tion (Rasmussen and Williams, 2006). A GP is completelyspecified by its mean function, µ(x) = E(f(x)) and kernel(or covariance) function k(x, x′) = Cov(f(x), f(x′)). It iscommon practice to assume zero mean, since marginaliz-ing over an unknown mean function can be equivalently ex-pressed as a zero-mean GP with a new kernel. The structureof the kernel captures high-level properties of the unknownfunction, f , which in turn determines how the model gener-alizes or extrapolates to new data. We can therefore define alanguage of regression models by specifying a language ofkernels.
The elements of this language are a set of base kernelscapturing different function properties, and a set of compo-sition rules which combine kernels to yield other valid ker-nels. Our base kernels are white noise (WN), constant (C),linear (LIN), squared exponential (SE) and periodic (PER),which on their own encode for uncorrelated noise, constantfunctions, linear functions, smooth functions and periodicfunctions respectively2. The composition rules are additionand multiplication:
k1 + k2 = k1(x, x′) + k2(x, x
′) (2.1)
k1 × k2 = k1(x, x′)× k2(x, x′) (2.2)
Combining kernels using these operations can yield ker-nels encoding for richer structures such as approximate pe-riodicity (SE× PER) or smooth functions with linear trends(SE + LIN).
This kernel composition framework (with different basekernels) was described by Duvenaud et al. (2013). We ex-tend and adapt this framework in several ways. In particular,we have found that incorporating changepoints into the lan-guage is essential for realistic models of time series (e.g.figure 1). Changepoints can be defined through addition andmultiplication with sigmoidal functions:
CP(k1, k2) = k1 × σ + k2 × σ (2.3)
where σ = σ(x)σ(x′) and σ = (1 − σ(x))(1 − σ(x′)).Changewindows CW(·, ·) can be defined similarly by re-placing σ(x) with a product of two sigmoids.
2Definitions of kernels are in the supplementary material.
We also expanded and reparametrised the set of base ker-nels so that they were more amenable to automatic descrip-tion and to extend the number of common regression modelsincluded in the language. Table 1 lists common regressionmodels that can be expressed by our language.
Regression model Kernel
GP smoothing SE + WNLinear regression C + LIN + WNMultiple kernel learning
∑SE + WN
Trend, cyclical, irregular∑
SE +∑
PER + WNFourier decomposition C +
∑cos + WN
Sparse spectrum GPs∑
cos + WNSpectral mixture
∑SE × cos + WN
Changepoints e.g. CP(SE, SE) + WNHeteroscedasticity e.g. SE + LIN ×WN
Table 1: Common regression models expressible in our lan-guage. cos is a special case of our reparametrised PER.
3 Model Search and EvaluationAs in Duvenaud et al. (2013) we explore the space of regres-sion models using a greedy search. We use the same searchoperators, but also include additional operators to incorpo-rate changepoints; a complete list is contained in the supple-mentary material.
After each model is proposed its kernel parameters areoptimised by conjugate gradient descent. We evaluate eachoptimized model, M , using the Bayesian Information Crite-rion (BIC) (Schwarz, 1978):
BIC(M) = −2 log p(D |M) + p log n (3.1)
where p is the number of kernel parameters, log p(D|M) isthe marginal likelihood of the data, D, and n is the numberof data points. BIC trades off model fit and complexity andimplements what is known as “Bayesian Occam’s Razor”(e.g. Rasmussen and Ghahramani, 2001; MacKay, 2003).
4 Automatic description of regression modelsOverview In this section, we describe how ABCD gen-erates natural-language descriptions of the models found bythe search procedure. There are two main features of our lan-guage of GP models that allow description to be performedautomatically.
First, the sometimes complicated kernel expressionsfound can be simplified into a sum of products. A sum ofkernels corresponds to a sum of functions so each productcan be described separately. Second, each kernel in a prod-uct modifies the resulting model in a consistent way. There-fore, we can choose one kernel to be described as a noun,with all others described using adjectives.
Sum of products normal form We convert each kernelexpression into a standard, simplified form. We do this byfirst distributing all products of sums into a sum of products.
Next, we apply several simplifications to the kernel expres-sion: The product of two SE kernels is another SE with dif-ferent parameters. Multiplying WN by any stationary kernel(C, WN, SE, or PER) gives another WN kernel. Multiplyingany kernel by C only changes the parameters of the originalkernel.
After applying these rules, the kernel can as be written asa sum of terms of the form:
K∏m
LIN(m)∏n
σ(n), (4.1)
whereK is one of WN, C, SE,∏
k PER(k) or SE∏
k PER(k)
and∏
i k(i) denotes a product of kernels, each with different
parameters.
Sums of kernels are sums of functions Formally, iff1(x) ∼ GP(0, k1) and independently f2(x) ∼ GP(0, k2)then f1(x) + f2(x) ∼ GP(0, k1 + k2). This lets us de-scribe each product of kernels separately.
Each kernel in a product modifies a model in a consis-tent way This allows us to describe the contribution ofeach kernel in a product as an adjective, or more generallyas a modifier of a noun. We now describe how each kernelmodifies a model and how this can be described in naturallanguage:• Multiplication by SE removes long range correlations
from a model since SE(x, x′) decreases monotonically to0 as |x−x′| increases. This can be described as making anexisting model’s correlation structure ‘local’ or ‘approxi-mate’.
• Multiplication by LIN is equivalent to multiplying thefunction being modeled by a linear function. If f(x) ∼GP(0, k), then xf(x) ∼ GP (0, k × LIN). This causes thestandard deviation of the model to vary linearly withoutaffecting the correlation and can be described as e.g. ‘withlinearly increasing standard deviation’.
• Multiplication by σ is equivalent to multiplying thefunction being modeled by a sigmoid which means thatthe function goes to zero before or after some point. Thiscan be described as e.g. ‘from [time]’ or ‘until [time]’.
• Multiplication by PER modifies the correlation struc-ture in the same way as multiplying the functionby an independent periodic function. Formally, iff1(x) ∼ GP(0, k1) and f2(x) ∼ GP(0, k2) then
Cov [f1(x)f2(x), f1(x′)f2(x′)] = k1(x, x′)k2(x, x
′).
This can be loosely described as e.g. ‘modulated by a pe-riodic function with a period of [period] [units]’.
Constructing a complete description of a product of ker-nels We choose one kernel to act as a noun which is thendescribed by the functions it encodes for when unmodifiede.g. ‘smooth function’ for SE. Modifiers corresponding tothe other kernels in the product are then appended to thisdescription, forming a noun phrase of the form:
Determiner + Premodifiers + Noun + Postmodifiers
As an example, a kernel of the form SE×PER×LIN×σcould be described as an
SE︸︷︷︸approximately
× PER︸︷︷︸periodic function
× LIN︸︷︷︸with linearly growing amplitude
× σ︸︷︷︸until 1700.
where PER has been selected as the head noun.In principle, any assignment of kernels in a product to
these different phrasal roles is possible, but in practice wefound certain assignments to produce more interpretablephrases than others. The head noun is chosen according tothe following ordering:
PER > WN, SE,C >∏m
LIN(m) >∏n
σ(n)
i.e. PER is always chosen as the head noun when present.
Ordering additive components The reports generated byABCD attempt to present the most interesting or importantfeatures of a data set first. As a heuristic, we order com-ponents by always adding next the component which mostreduces the 10-fold cross-validated mean absolute error.
4.1 Worked exampleSuppose we start with a kernel of the form
SE × (WN × LIN + CP(C, PER)).
This is converted to a sum of products:
SE ×WN × LIN + SE × C × σ + SE × PER × σ.
which is simplified to
WN × LIN + SE × σ + SE × PER × σ.
To describe the first component, the head noun descriptionfor WN, ‘uncorrelated noise’, is concatenated with a mod-ifier for LIN, ‘with linearly increasing standard deviation’.The second component is described as ‘A smooth functionwith a lengthscale of [lengthscale] [units]’, correspondingto the SE, ‘which applies until [changepoint]’, which corre-sponds to the σ. Finally, the third component is describedas ‘An approximately periodic function with a period of [pe-riod] [units] which applies from [changepoint]’.
5 Example descriptions of time seriesWe demonstrate the ability of our procedure to discoverand describe a variety of patterns on two time series. Fullautomatically-generated reports for 13 data sets are providedas supplementary material.
5.1 Summarizing 400 Years of Solar ActivityWe show excerpts from the report automatically generatedon annual solar irradiation data from 1610 to 2011 (figure 2).This time series has two pertinent features: a roughly 11-year cycle of solar activity, and a period lasting from 1645 to1715 with much smaller variance than the rest of the dataset.This flat region corresponds to the Maunder minimum, a pe-riod in which sunspots were extremely rare (Lean, Beer, and
1 Executive summary
The raw data and full model posterior with extrapolations are shown in figure 1.
Figure 1: Raw data (left) and model posterior with extrapolation (right)
The structure search algorithm has identified eight additive components in the data. The first 4additive components explain 92.3% of the variation in the data as shown by the coefficient of de-termination (R2) values in table 1. The first 6 additive components explain 99.7% of the variationin the data. After the first 5 components the cross validated mean absolute error (MAE) does notdecrease by more than 0.1%. This suggests that subsequent terms are modelling very short termtrends, uncorrelated noise or are artefacts of the model or search procedure. Short summaries of theadditive components are as follows:
• A constant.
• A constant. This function applies from 1643 until 1716.
• A smooth function. This function applies until 1643 and from 1716 onwards.
• An approximately periodic function with a period of 10.8 years. This function applies until1643 and from 1716 onwards.
• A rapidly varying smooth function. This function applies until 1643 and from 1716 on-wards.
• Uncorrelated noise with standard deviation increasing linearly away from 1837. This func-tion applies until 1643 and from 1716 onwards.
• Uncorrelated noise with standard deviation increasing linearly away from 1952. This func-tion applies until 1643 and from 1716 onwards.
• Uncorrelated noise. This function applies from 1643 until 1716.
Table 1: Summary statistics for cumulative additive fits to the data. The residual coefficient ofdetermination (R2) values are computed using the residuals from the previous fit as the target values;this measures how much of the residual variance is explained by each new component. The meanabsolute error (MAE) is calculated using 10 fold cross validation with a contiguous block design;this measures the ability of the model to interpolate and extrapolate over moderate distances. Themodel is fit using the full data and the MAE values are calculated using this model; this double use ofdata means that the MAE values cannot be used reliably as an estimate of out-of-sample predictiveperformance.
Figure 2: Solar irradiance data.
1 Executive summary
The raw data and full model posterior with extrapolations are shown in figure 1.
Figure 1: Raw data (left) and model posterior with extrapolation (right)
The structure search algorithm has identified eight additive components in the data. The first 4additive components explain 92.3% of the variation in the data as shown by the coefficient of de-termination (R2) values in table 1. The first 6 additive components explain 99.7% of the variationin the data. After the first 5 components the cross validated mean absolute error (MAE) does notdecrease by more than 0.1%. This suggests that subsequent terms are modelling very short termtrends, uncorrelated noise or are artefacts of the model or search procedure. Short summaries of theadditive components are as follows:
• A constant.
• A constant. This function applies from 1643 until 1716.
• A smooth function. This function applies until 1643 and from 1716 onwards.
• An approximately periodic function with a period of 10.8 years. This function applies until1643 and from 1716 onwards.
• A rapidly varying smooth function. This function applies until 1643 and from 1716 on-wards.
• Uncorrelated noise with standard deviation increasing linearly away from 1837. This func-tion applies until 1643 and from 1716 onwards.
• Uncorrelated noise with standard deviation increasing linearly away from 1952. This func-tion applies until 1643 and from 1716 onwards.
• Uncorrelated noise. This function applies from 1643 until 1716.
Table 1: Summary statistics for cumulative additive fits to the data. The residual coefficient ofdetermination (R2) values are computed using the residuals from the previous fit as the target values;this measures how much of the residual variance is explained by each new component. The meanabsolute error (MAE) is calculated using 10 fold cross validation with a contiguous block design;this measures the ability of the model to interpolate and extrapolate over moderate distances. Themodel is fit using the full data and the MAE values are calculated using this model; this double use ofdata means that the MAE values cannot be used reliably as an estimate of out-of-sample predictiveperformance.
Figure 3: Automatically generated descriptions of the com-ponents discovered by ABCD on the solar irradiance dataset. The dataset has been decomposed into diverse structureswith simple descriptions.
Bradley, 1995). ABCD clearly identifies these two features,as discussed below.
Figure 3 shows the natural-language summaries of the topfour components chosen by ABCD. From these short sum-maries, we can see that our system has identified the Maun-der minimum (second component) and 11-year solar cycle(fourth component). These components are visualized in fig-ures 4 and 1, respectively. The third component correspondsto long-term trends, as visualized in figure 5.
2.2 Component 2 : A constant. This function applies from 1643 until 1716
This component is constant. This component applies from 1643 until 1716.
This component explains 37.4% of the residual variance; this increases the total variance explainedfrom 0.0% to 37.4%. The addition of this component reduces the cross validated MAE by 31.97%from 0.33 to 0.23.
Posterior of component 2
1650 1700 1750 1800 1850 1900 1950 2000−0.8
−0.7
−0.6
−0.5
−0.4
−0.3
−0.2
−0.1
0Sum of components up to component 2
1650 1700 1750 1800 1850 1900 1950 20001360
1360.5
1361
1361.5
1362
Figure 4: Pointwise posterior of component 2 (left) and the posterior of the cumulative sum ofcomponents with data (right)
Residuals after component 2
1650 1700 1750 1800 1850 1900 1950 2000−1
−0.5
0
0.5
1
1.5
Figure 5: Pointwise posterior of residuals after adding component 2
Figure 4: One of the learned components corresponds to theMaunder minimum.
5.2 Finding heteroscedasticity in air traffic dataNext, we present the analysis generated by our procedureon international airline passenger data (figure 6). The modelconstructed by ABCD has four components: LIN + SE ×PER × LIN + SE + WN × LIN, with descriptions given in
2.3 Component 3 : A smooth function. This function applies until 1643 and from 1716onwards
This component is a smooth function with a typical lengthscale of 23.1 years. This componentapplies until 1643 and from 1716 onwards.
This component explains 56.6% of the residual variance; this increases the total variance explainedfrom 37.4% to 72.8%. The addition of this component reduces the cross validated MAE by 21.08%from 0.23 to 0.18.
Posterior of component 3
1650 1700 1750 1800 1850 1900 1950 2000−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8Sum of components up to component 3
1650 1700 1750 1800 1850 1900 1950 20001360
1360.5
1361
1361.5
1362
Figure 6: Pointwise posterior of component 3 (left) and the posterior of the cumulative sum ofcomponents with data (right)
Residuals after component 3
1650 1700 1750 1800 1850 1900 1950 2000−1
−0.5
0
0.5
1
Figure 7: Pointwise posterior of residuals after adding component 3
Figure 5: Characterizing the medium-term smoothness ofsolar activity levels. By allowing other components to ex-plain the periodicity, noise, and the Maunder minimum,ABCD can isolate the part of the signal best explained bya slowly-varying trend.
1 Executive summary
The raw data and full model posterior with extrapolations are shown in figure 1.
Raw data
1950 1952 1954 1956 1958 1960 1962100
200
300
400
500
600
700Full model posterior with extrapolations
1950 1952 1954 1956 1958 1960 19620
100
200
300
400
500
600
700
Figure 1: Raw data (left) and model posterior with extrapolation (right)
The structure search algorithm has identified four additive components in the data. The first 2additive components explain 98.5% of the variation in the data as shown by the coefficient of de-termination (R2) values in table 1. The first 3 additive components explain 99.8% of the variationin the data. After the first 3 components the cross validated mean absolute error (MAE) does notdecrease by more than 0.1%. This suggests that subsequent terms are modelling very short termtrends, uncorrelated noise or are artefacts of the model or search procedure. Short summaries of theadditive components are as follows:
• A linearly increasing function.• An approximately periodic function with a period of 1.0 years and with linearly increasing
amplitude.• A smooth function.• Uncorrelated noise with linearly increasing standard deviation.
Table 1: Summary statistics for cumulative additive fits to the data. The residual coefficient ofdetermination (R2) values are computed using the residuals from the previous fit as the target values;this measures how much of the residual variance is explained by each new component. The meanabsolute error (MAE) is calculated using 10 fold cross validation with a contiguous block design;this measures the ability of the model to interpolate and extrapolate over moderate distances. Themodel is fit using the full data and the MAE values are calculated using this model; this double use ofdata means that the MAE values cannot be used reliably as an estimate of out-of-sample predictiveperformance.
Model checking statistics are summarised in table 2 in section 4. These statistics have not revealedany inconsistencies between the model and observed data.
The rest of the document is structured as follows. In section 2 the forms of the additive componentsare described and their posterior distributions are displayed. In section 3 the modelling assumptionsof each component are discussed with reference to how this affects the extrapolations made by themodel. Section 4 discusses model checking statistics, with plots showing the form of any detecteddiscrepancies between the model and observed data.
Figure 6: International airline passenger monthly volume(e.g. Box, Jenkins, and Reinsel, 2013).
figure 7.The second component (figure 8) is accurately described
as approximately (SE) periodic (PER) with linearly growingamplitude (LIN). By multiplying a white noise kernel by alinear kernel, the model is able to express heteroscedasticity(figure 9).
5.3 Comparison to equation learningWe now compare the descriptions generated by ABCD toparametric functions produced by an equation learning sys-tem. We show equations produced by Eureqa (Nutonian,2011) for the data sets shown above, using the default meanabsolute error performance metric.
The learned function for the solar irradiance data is
where t is time and constants are replaced with symbolsfor brevity. This equation captures the constant offset of thedata, and models the long-term trend with a product of si-nusoids, but fails to capture the solar cycle or the Maunderminimum.
The learned function for the airline passenger data is
Passengers(t) = αt+ β cos(γ − δt)logistic(εt− ζ)− η
which captures the approximately linear trend, and the pe-riodic component with approximately linearly (logistic) in-
1 Executive summary
The raw data and full model posterior with extrapolations are shown in figure 1.
Raw data
1950 1952 1954 1956 1958 1960 1962100
200
300
400
500
600
700Full model posterior with extrapolations
1950 1952 1954 1956 1958 1960 19620
100
200
300
400
500
600
700
Figure 1: Raw data (left) and model posterior with extrapolation (right)
The structure search algorithm has identified four additive components in the data. The first 2additive components explain 98.5% of the variation in the data as shown by the coefficient of de-termination (R2) values in table 1. The first 3 additive components explain 99.8% of the variationin the data. After the first 3 components the cross validated mean absolute error (MAE) does notdecrease by more than 0.1%. This suggests that subsequent terms are modelling very short termtrends, uncorrelated noise or are artefacts of the model or search procedure. Short summaries of theadditive components are as follows:
• A linearly increasing function.• An approximately periodic function with a period of 1.0 years and with linearly increasing
amplitude.• A smooth function.• Uncorrelated noise with linearly increasing standard deviation.
Table 1: Summary statistics for cumulative additive fits to the data. The residual coefficient ofdetermination (R2) values are computed using the residuals from the previous fit as the target values;this measures how much of the residual variance is explained by each new component. The meanabsolute error (MAE) is calculated using 10 fold cross validation with a contiguous block design;this measures the ability of the model to interpolate and extrapolate over moderate distances. Themodel is fit using the full data and the MAE values are calculated using this model; this double use ofdata means that the MAE values cannot be used reliably as an estimate of out-of-sample predictiveperformance.
Model checking statistics are summarised in table 2 in section 4. These statistics have not revealedany inconsistencies between the model and observed data.
The rest of the document is structured as follows. In section 2 the forms of the additive componentsare described and their posterior distributions are displayed. In section 3 the modelling assumptionsof each component are discussed with reference to how this affects the extrapolations made by themodel. Section 4 discusses model checking statistics, with plots showing the form of any detecteddiscrepancies between the model and observed data.
Figure 7: Short descriptions and summary statistics for thefour components of the airline model.
2.2 Component 2 : An approximately periodic function with a period of 1.0 years and withlinearly increasing amplitude
This component is approximately periodic with a period of 1.0 years and varying amplitude. Acrossperiods the shape of this function varies very smoothly. The amplitude of the function increaseslinearly. The shape of this function within each period has a typical lengthscale of 6.0 weeks.
This component explains 89.9% of the residual variance; this increases the total variance explainedfrom 85.4% to 98.5%. The addition of this component reduces the cross validated MAE by 63.45%from 34.03 to 12.44.
Figure 5: Pointwise posterior of residuals after adding component 2
Figure 8: Capturing non-stationary periodicity in the airlinedata
creasing amplitude. However, the annual cycle is heavily ap-proximated by a sinusoid and the model does not captureheteroscedasticity.
6 Related workBuilding Kernel Functions Rasmussen and Williams(2006) devote 4 pages to manually constructing a compos-ite kernel to model a time series of carbon dioxode concen-trations. In the supplementary material, we include a reportautomatically generated by ABCD for this dataset; our pro-cedure chose a model similar to the one they constructedby hand. Other examples of papers whose main contributionis to manually construct and fit a composite GP kernel areKlenske (2012) and Lloyd (2013).
Diosan, Rogozan, and Pecuchet (2007); Bing et al. (2010)and Kronberger and Kommenda (2013) search over a simi-lar space of models as ABCD using genetic algorithms butdo not interpret the resulting models. Our procedure is basedon the model construction method of Duvenaud et al. (2013)which automatically decomposed models but componentswere interpreted manually and the space of models searchedover was smaller than that in this work.
2.4 Component 4 : Uncorrelated noise with linearly increasing standard deviation
This component models uncorrelated noise. The standard deviation of the noise increases linearly.
This component explains 100.0% of the residual variance; this increases the total variance explainedfrom 99.8% to 100.0%. The addition of this component reduces the cross validated MAE by 0.00%from 9.10 to 9.10. This component explains residual variance but does not improve MAE whichsuggests that this component describes very short term patterns, uncorrelated noise or is an artefactof the model or search procedure.
Figure 8: Pointwise posterior of component 4 (left) and the posterior of the cumulative sum ofcomponents with data (right)
Figure 9: Modeling heteroscedasticity
Kernel Learning Sparse spectrum GPs (Lazaro-Gredillaet al., 2010) approximate the spectral density of a station-ary kernel function using delta functions which correspondsto kernels of the form
∑cos. Similarly, Wilson and Adams
(2013) introduce spectral mixture kernels which approxi-mate the spectral density using a scale-location mixture ofGaussian distributions corresponding to kernels of the form∑
SE × cos. Both demonstrate, using Bochner’s theorem(Bochner, 1959), that these kernels can approximate anystationary covariance function. Our language of kernels in-cludes both of these kernel classes (see table 1).
There is a large body of work attempting to construct richkernels through a weighted sum of base kernels called multi-ple kernel learning (MKL) (e.g. Bach, Lanckriet, and Jordan,2004). These approaches find the optimal solution in poly-nomial time but only if the component kernels and parame-ters are pre-specified. We compare to a Bayesian variant ofMKL in section 7 which is expressed as a restriction of ourlanguage of kernels.
Equation learning Todorovski and Dzeroski (1997),Washio et al. (1999) and Schmidt and Lipson (2009) learnparametric forms of functions specifying time series, or re-lations between quantities. In contrast, ABCD learns a para-metric form for the covariance, allowing it to model func-tions without a simple parametric form.
Searching over open-ended model spaces This work wasinspired by previous successes at searching over open-endedmodel spaces: matrix decompositions (Grosse, Salakhutdi-nov, and Tenenbaum, 2012) and graph structures (Kemp andTenenbaum, 2008). In both cases, the model spaces were de-fined compositionally through a handful of components andoperators, and models were selected using criteria whichtrade off model complexity and goodness of fit. Our workdiffers in that our procedure automatically interprets the cho-sen model, making the results accessible to non-experts.
Natural-language output To the best of our knowledge,our procedure is the first example of automatic descriptionof nonparametric statistical models. However, systems with
natural language output have been built in the areas of videointerpretation (Barbu et al., 2012) and automated theoremproving (Ganesalingam and Gowers, 2013).
7 Predictive AccuracyIn addition to our demonstration of the interpretability ofABCD, we compared the predictive accuracy of variousmodel-building algorithms at interpolating and extrapolat-ing time-series. ABCD outperforms the other methods onaverage.
Data sets We evaluate the performance of the algorithmslisted below on 13 real time-series from various domainsfrom the time series data library (Hyndman, Accessed sum-mer 2013); plots of the data can be found at the beginning ofthe reports in the supplementary material.
Algorithms We compare ABCD to equation learning us-ing Eureqa (Nutonian, 2011) and six other regression algo-rithms: linear regression, GP regression with a single SEkernel (squared exponential), a Bayesian variant of multi-ple kernel learning (MKL) (e.g. Bach, Lanckriet, and Jor-dan, 2004), change point modeling (e.g. Garnett et al., 2010;Saatci, Turner, and Rasmussen, 2010; Fox and Dunson,2013), spectral mixture kernels (Wilson and Adams, 2013)(spectral kernels) and trend-cyclical-irregular models (e.g.Lind et al., 2006).
We use the default mean absolute error criterion whenusing Eureqa. All other algorithms can be expressed as re-strictions of our modeling language (see table 1) so we per-form inference using the same search methodology and se-lection criterion3 with appropriate restrictions to the lan-guage. For MKL, trend-cyclical-irregular and spectral ker-nels, the greedy search procedure of ABCD corresponds toa forward-selection algorithm. For squared exponential andlinear regression the procedure corresponds to marginal like-lihood optimisation. More advanced inference methods aretypically used for changepoint modeling but we use the sameinference method for all algorithms for comparability.
We restricted to regression algorithms for comparability;this excludes models which regress on previous values oftimes series, such as autoregressive or moving-average mod-els (e.g. Box, Jenkins, and Reinsel, 2013). Constructing alanguage for this class of time-series model would be an in-teresting area for future research.
Interpretability versus accuracy BIC trades off model fitand complexity by penalizing the number of parameters ina kernel expression. This can result in ABCD favoring ker-nel expressions with nested products of sums, producing de-scriptions involving many additive components. While thesemodels have good predictive performance the large numberof components can make them less interpretable. We exper-imented with distributing all products over addition duringthe search, causing models with many additive components
3We experimented with using unpenalised marginal likelihoodas the search criterion but observed overfitting, as is to be expected.
to be more heavily penalized by BIC. We call this proce-dure ABCD-interpretability, in contrast to the unrestrictedversion of the search, ABCD-accuracy.
Extrapolation To test extrapolation we trained all algo-rithms on the first 90% of the data, predicted the remain-ing 10% and then computed the root mean squared error(RMSE). The RMSEs are then standardised by dividing bythe smallest RMSE for each data set so that the best perfor-mance on each data set will have a value of 1.
Figure 10 shows the standardised RMSEs acrossalgorithms. ABCD-accuracy outperforms ABCD-interpretability but both versions have lower quartilesthan all other methods.
Overall, the model construction methods with greater ca-pacity perform better: ABCD outperforms trend-cyclical-irregular, which outperforms Bayesian MKL, which outper-forms squared exponential. Despite searching over a richmodel class, Eureqa performs relatively poorly, since veryfew datasets are parsimoniously explained by a parametricequation.
Not shown on the plot are large outliers for spectral ker-nels, Eureqa, squared exponential and linear regression withvalues of 11, 493, 22 and 29 respectively. All of these out-liers occurred on a data set with a large discontinuity (seethe call centre data in the supplementary material).
Interpolation To test the ability of the methods to interpo-late, we randomly divided each data set into equal amountsof training data and testing data. The results are similar tothose for extrapolation and are included in the supplemen-tary material.
8 ConclusionTowards the goal of automating statistical modeling we havepresented a system which constructs an appropriate modelfrom an open-ended language and automatically generatesdetailed reports that describe patterns in the data capturedby the model. We have demonstrated that our procedure candiscover and describe a variety of patterns on several timeseries. Our procedure’s extrapolation and interpolation per-formance on time-series are state-of-the-art compared to ex-isting model construction techniques. We believe this pro-cedure has the potential to make powerful statistical model-building techniques accessible to non-experts.
Source Code Source code to perform all experiments isavailable on github4.
ReferencesBach, F. R.; Lanckriet, G. R.; and Jordan, M. I. 2004. Mul-
tiple kernel learning, conic duality, and the SMO algo-
4http://www.github.com/jamesrobertlloyd/gpss-research. All GP parameter optimisation was per-formed by automated calls to the GPML toolbox available athttp://www.gaussianprocess.org/gpml/code/.
Figure 10: Raw data, and box plot (showing median and quartiles) of standardised extrapolation RMSE (best performance = 1)on 13 time-series. The methods are ordered by median.
rithm. In Proceedings of the twenty-first internationalconference on Machine learning, 6. ACM.
Barbu, A.; Bridge, A.; Burchill, Z.; Coroian, D.; Dick-inson, S.; Fidler, S.; Michaux, A.; Mussman, S.;Narayanaswamy, S.; Salvi, D.; Schmidt, L.; Shangguan,J.; Siskind, J.; Waggoner, J.; Wang, S.; Wei, J.; Yin, Y.;and Zhang, Z. 2012. Video in sentences out. In Confer-ence on Uncertainty in Artificial Intelligence.
Bing, W.; Wen-qiong, Z.; Ling, C.; and Jia-hong, L. 2010.A GP-based kernel construction and optimization methodfor RVM. In International Conference on Computer andAutomation Engineering (ICCAE), volume 4, 419–423.
Bochner, S. 1959. Lectures on Fourier integrals, volume 42.Princeton University Press.
Box, G. E.; Jenkins, G. M.; and Reinsel, G. C. 2013. Timeseries analysis: forecasting and control. Wiley. com.
Diosan, L.; Rogozan, A.; and Pecuchet, J. 2007. Evolvingkernel functions for SVMs by genetic programming. InMachine Learning and Applications, 2007, 19–24. IEEE.
Duvenaud, D.; Lloyd, J. R.; Grosse, R.; Tenenbaum, J. B.;and Ghahramani, Z. 2013. Structure discovery innonparametric regression through compositional kernelsearch. In Proceedings of the 30th International Confer-ence on Machine Learning.
Fox, E., and Dunson, D. 2013. Multiresolution GaussianProcesses. In Neural Information Processing Systems 25.MIT Press.
Ganesalingam, M., and Gowers, W. T. 2013. A fully au-tomatic problem solver with human-style output. CoRRabs/1309.4501.
Garnett, R.; Osborne, M. A.; Reece, S.; Rogers, A.; andRoberts, S. J. 2010. Sequential bayesian prediction inthe presence of changepoints and faults. The ComputerJournal 53(9):1430–1446.
Grosse, R.; Salakhutdinov, R.; and Tenenbaum, J. 2012.Exploiting compositionality to explore a large space ofmodel structures. In Uncertainty in Artificial Intelligence.
Hyndman, R. J. Accessed summer 2013. Time series datalibrary.
Kemp, C., and Tenenbaum, J. 2008. The discovery of struc-tural form. Proceedings of the National Academy of Sci-ences 105(31):10687–10692.
Klenske, E. 2012. Nonparametric System Identificationand Control for Periodic Error Correction in Telescopes.Ph.D. Dissertation, University of Stuttgart.
Kronberger, G., and Kommenda, M. 2013. Evolution of co-variance functions for gaussian process regression usinggenetic programming. arXiv preprint arXiv:1305.3794.
Lazaro-Gredilla, M.; Quinonero-Candela, J.; Rasmussen,C. E.; and Figueiras-Vidal, A. R. 2010. Sparse spec-trum gaussian process regression. The Journal of MachineLearning Research 99:1865–1881.
Lean, J.; Beer, J.; and Bradley, R. 1995. Reconstruc-tion of solar irradiance since 1610: Implications for cli-mate change. Geophysical Research Letters 22(23):3195–3198.
Lind, D. A.; Marchal, W. G.; Wathen, S. A.; and Magazine,B. W. 2006. Basic statistics for business and economics.McGraw-Hill/Irwin Boston.
Lloyd, J. R. 2013. GEFCom2012 hierarchical load forecast-ing: Gradient boosting machines and gaussian processes.International Journal of Forecasting.
MacKay, D. J. 2003. Information theory, inference andlearning algorithms. Cambridge university press.
Nutonian. 2011. Eureqa.
Rasmussen, C., and Ghahramani, Z. 2001. Occam’s razor.In Advances in Neural Information Processing Systems.
Rasmussen, C., and Williams, C. 2006. Gaussian Processesfor Machine Learning. The MIT Press, Cambridge, MA,USA.
Saatci, Y.; Turner, R. D.; and Rasmussen, C. E. 2010. Gaus-sian process change point models. In Proceedings ofthe 27th International Conference on Machine Learning(ICML-10), 927–934.
Schmidt, M., and Lipson, H. 2009. Distilling free-form nat-ural laws from experimental data. Science 324(5923):81–85.
Schwarz, G. 1978. Estimating the dimension of a model.The Annals of Statistics 6(2):461–464.
Todorovski, L., and Dzeroski, S. 1997. Declarative bias inequation discovery. In International Conference on Ma-chine Learning, 376–384.
Washio, T.; Motoda, H.; Niwa, Y.; et al. 1999. Discover-ing admissible model equations from observed data basedon scale-types and identity constraints. In InternationalJoint Conference On Artifical Intelligence, volume 16,772–779.
Wilson, A. G., and Adams, R. P. 2013. Gaussian process co-variance kernels for pattern discovery and extrapolation.In Proceedings of the 30th International Conference onMachine Learning.
AppendicesA Kernels
A.1 Base kernelsFor scalar-valued inputs, the white noise (WN), constant(C), linear (LIN), squared exponential (SE), and periodickernels (PER) are defined as follows:
WN(x, x′) = σ2δx,x′ (A.1)C(x, x′) = σ2 (A.2)
LIN(x, x′) = σ2(x− `)(x′ − `) (A.3)
SE(x, x′) = σ2 exp(− (x−x′)2
2`2
)(A.4)
PER(x, x′) = σ2
exp
cos2π(x−x′)
p
`2
−I0( 1`2)
exp( 1`2)−I0( 1
`2)
(A.5)
where δx,x′ is the Kronecker delta function, I0 is the modi-fied Bessel function of the first kind of order zero and othersymbols are parameters of the kernel functions.
A.2 Changepoints and changewindowsThe changepoint, CP(·, ·) operator is defined as follows:
CP(k1, k2)(x, x′) = σ(x)k1(x, x′)σ(x′)
+(1− σ(x))k2(x, x′)(1− σ(x′))(A.6)
where σ(x) = 0.5 × (1 + tanh( `−xs )). This can also bewritten as
CP(k1, k2) = σk1 + σk2 (A.7)
where σ(x, x′) = σ(x)σ(x′) and σ(x, x′) = (1−σ(x))(1−σ(x′)).
Changewindow, CW(·, ·), operators are defined similarlyby replacing the sigmoid, σ(x), with a product of two sig-moids.
A.3 Properties of the periodic kernelA simple application of l’Hopital’s rule shows that
PER(x, x′)→ σ2 cos
(2π(x− x′)
p
)as `→∞.
(A.8)This limiting form is written as the cosine kernel (cos).
B Model construction / searchB.1 OverviewThe model construction phase of ABCD starts with the ker-nel equal to the noise kernel, WN. New kernel expressionsare generated by applying search operators to the currentkernel. When new base kernels are proposed by the searchoperators, their parameters are randomly initialised with sev-eral restarts. Parameters are then optimized by conjugategradients to maximise the likelihood of the data conditionedon the kernel parameters. The kernels are then scored by theBayesian information criterion and the top scoring kernel is
selected as the new kernel. The search then proceeds by ap-plying the search operators to the new kernel i.e. this is agreedy search algorithm.
In all experiments, 10 random restarts were used for pa-rameter initialisation and the search was run to a depth of10.
B.2 Search operatorsABCD is based on a search algorithm which used the fol-lowing search operators
S → S + B (B.1)S → S × B (B.2)B → B′ (B.3)
where S represents any kernel subexpression and B is anybase kernel within a kernel expression i.e. the search opera-tors represent addition, multiplication and replacement.
To accommodate changepoint/window operators we in-troduce the following additional operators
where C is the constant kernel. The last two operators resultin a kernel only applying outside or within a certain region.
Based on experience with typical paths followed by thesearch algorithm we introduced the following operators
S → S × (B + C) (B.8)S → B (B.9)
S + S ′ → S (B.10)S × S ′ → S (B.11)
where S ′ represents any other kernel expression. Their in-troduction is currently not rigorously justified.
C Predictive accuracyInterpolation To test the ability of the methods to interpo-late, we randomly divided each data set into equal amountsof training data and testing data. We trained each algorithmon the training half of the data, produced predictions for theremaining half and then computed the root mean squared er-ror (RMSE). The values of the RMSEs are then standardisedby dividing by the smallest RMSE for each data set i.e. thebest performance on each data set will have a value of 1.
Figure 11 shows the standardised RMSEs for the differentalgorithms. The box plots show that all quartiles of the dis-tribution of standardised RMSEs are lower for both versionsof ABCD. The median for ABCD-accuracy is 1; it is the bestperforming algorithm on 7 datasets. The largest outliers ofABCD and spectral kernels are similar in value.
Changepoints performs slightly worse than MKL despitebeing strictly more general than Changepoints. The intro-duction of changepoints allows for more structured models,but it introduces parametric forms into the regression mod-els (i.e. the sigmoids expressing the changepoints). This re-sults in worse interpolations at the locations of the change
points, suggesting that a more robust modeling languagewould require a more flexible class of changepoint shapesor improved inference (e.g. fully Bayesian inference overthe location and shape of the changepoint).
Eureqa is not suited to this task and performs poorly. Themodels learned by Eureqa tend to capture only broad trendsof the data since the fine details are not well explained byparametric forms.
C.1 Tabels of standardised RMSEsSee table 2 for raw interpolation results and table 3for raw extrapolation results. The rows follow the orderof the datasets in the rest of the supplementary mate-rial. The following abbreviations are used: ABCD-accuracy(ABCD-acc), ABCD-interpretability ((ABCD-int), Spectralkernels (SP), Trend-cyclical-irregular (TCI), Bayesian MKL(MKL), Eureqa (EL), Changepoints (CP), Squared exponen-tial (SE) and Linear regression (Lin).
D Guide to the automatically generatedreports
Additional supplementary material to this paper is 13 reportsautomatically generated by ABCD. A link to these reportswill be maintained at http://mlg.eng.cam.ac.uk/lloyd/. We recommend that you read the report for ‘01-airline’ first and review the reports that follow afterwardsmore briefly. ‘02-solar’ is discussed in the main text. ‘03-mauna’ analyses a dataset mentioned in the related work.‘04-wheat’ demonstrates changepoints being used to captureheteroscedasticity. ‘05-temperature’ extracts an exactly pe-riodic pattern from noisy data. ‘07-call-centre’ demonstratesa large discontinuity being modeled by a changepoint. ‘10-sulphuric’ combines many changepoints to create a highlystructured model of the data. ‘12-births’ discovers multipleperiodic components.
