MNRAS 000, 1–15 (2020) Preprint 13 January 2020 Compiled using MNRAS LATEX style file v3.0
Trend Filtering – I. A Modern Statistical Tool forTime-Domain Astronomy and Astronomical Spectroscopy
Collin A. Politsch,1,2,3? Jessi Cisewski-Kehe,4 Rupert A. C. Croft,3,5,6
and Larry Wasserman1,2,3
1 Department of Statistics & Data Science, Carnegie Mellon University, Pittsburgh, PA 152132 Machine Learning Department, Carnegie Mellon University, Pittsburgh, PA 152133 McWilliams Center for Cosmology, Carnegie Mellon University, Pittsburgh, PA 152134 Department of Statistics and Data Science, Yale University, New Haven, CT 065205 Department of Physics, Carnegie Mellon University, Pittsburgh, PA 152136 School of Physics, University of Melbourne, VIC 3010, Australia
Accepted XXX. Received YYY; in original form 2019 August 20
ABSTRACT
The problem of denoising a one-dimensional signal possessing varying degrees ofsmoothness is ubiquitous in time-domain astronomy and astronomical spectroscopy.For example, in the time domain, an astronomical object may exhibit a smoothlyvarying intensity that is occasionally interrupted by abrupt dips or spikes. Likewise,in the spectroscopic setting, a noiseless spectrum typically contains intervals of relativesmoothness mixed with localized higher frequency components such as emission peaksand absorption lines. In this work, we present trend filtering, a modern nonparamet-ric statistical tool that yields significant improvements in this broad problem spaceof denoising spatially heterogeneous signals. When the underlying signal is spatiallyheterogeneous, trend filtering is superior to any statistical estimator that is a linearcombination of the observed data—including kernel smoothers, LOESS, smoothingsplines, Gaussian process regression, and many other popular methods. Furthermore,the trend filtering estimate can be computed with practical and scalable efficiency viaa specialized convex optimization algorithm, e.g. handling sample sizes of n & 107
within a few minutes. In a companion paper, we explicitly demonstrate the broadutility of trend filtering to observational astronomy by carrying out a diverse set ofspectroscopic and time-domain analyses.
Many astronomical observations produce one-dimensionaldata with varying (or unknown) degrees of smoothness.These include data from time-domain astronomy, wheretransient events such as supernovae can show light-curvevariations on timescales ranging from seconds to years (e.g.,Dimitriadis et al. 2017; Tolstov et al. 2019). Similarly, inastronomical spectroscopy, with wavelength (or frequency)as the input variable, sharp absorption or emission-linefeatures can be present alongside smoothly varying black-body or other continuum radiation (see, e.g., Tennyson2019). In each of these general settings, we observe a sig-nal plus noise and would like to denoise the signal as
accurately as possible. Indeed the set of statistical toolsavailable for addressing this general problem is quite vast.Commonly used nonparametric regression methods includekernel smoothers (e.g., Hall et al. 2002; Croft et al. 2002),local polynomial regression (LOESS; e.g., Maron & Howes2003; Persson et al. 2004), splines (e.g., Peiris & Verde 2010;Contreras et al. 2010; Dhawan et al. 2015), Gaussian processregression (e.g., Gibson et al. 2012; Aigrain et al. 2016;Gomez-Valent & Amendola 2018), and wavelet decomposi-tions (e.g., Fligge & Solanki 1997; Theuns & Zaroubi 2000;Golkhou & Butler 2014). A rich and elegant statistical lit-erature exists on the theoretical and practical achievementsof these methods (see, e.g., Gyorfi et al. 2002; Wasserman2006; Hastie et al. 2009 for general references). However,when the underlying signal is spatially heterogeneous, i.e.exhibits varying degrees of smoothness, the power of clas-
sical statistical literature is quite limited. Kernels, LOESS,smoothing splines, and Gaussian process regression belongto a broad family of nonparametric methods called linearsmoothers, which has been shown to be uniformly subop-timal for estimating spatially heterogeneous signals (Ne-mirovskii et al. 1985; Nemirovskii 1985; Donoho & John-stone 1998). The common limitation of these methods isthat they are not locally adaptive; i.e., by construction, theydo not adapt to local degrees of smoothness in a signal. Inparticular, continuing with the example of a smoothly vary-ing signal with occasional sharp features, a linear smootherwill tend to oversmooth the sharp features and/or overfitthe smooth regions in its effort to optimally balance statis-tical bias and variance. Considerable effort has been madeto address this problem by locally varying the hyperparam-eter(s) of a linear smoother. For example, locally varyingthe kernel bandwidth (e.g., Muller & Stadtmuller 1987; Fan& Gijbels 1992, 1995; Lepski et al. 1997; Gijbels & Mam-men 1998) irregularly varying spline knot locations (e.g.,De Boor 1974; Jupp 1978; Dimatteo et al. 2001), and con-structing non-stationary covariance functions for Gaussianprocess regression (e.g., Schmidt & O’Hagan 2003; Paciorek& Schervish 2004, 2006). However, since hyperparameterstypically need to be estimated from the data, such expo-nential increases in the hyperparameter complexity severelylimit the practicality of choosing the hyperparameters ina fully data-driven, generalizable, and computationally ef-ficient fashion. Wavelet decompositions offer an elegant so-lution to the problem of estimating spatially heterogeneoussignals, providing both statistical optimality (e.g., Donoho& Johnstone 1994, 1998) and only requiring data-driven tun-ing of a single (scalar) hyperparameter. Wavelets, however,possess the practical limitation of requiring a stringent anal-ysis setting, e.g. equally-spaced inputs and sample size equalto a power of two, among other provisions; and when theseconditions are violated, the optimality guarantees are void.So, seemingly at an impasse, the motivating question for thiswork is can we have the best of both worlds? More precisely,is there a statistical tool that simultaneously possesses thefollowing properties:
P1. Statistical optimality for estimating spatially het-erogeneous signals
P2. Practical analysis assumptions; for example, notlimited to equally-spaced inputs
P3. Practical and scalable computational speedP4. A one-dimensional hyperparameter space, with au-
tomatic data-driven methods for selection
In this paper we introduce trend filtering (Tibshirani 2014),a statistical method that is new to the astronomical litera-ture and provides a strong affirmative answer to this ques-tion.
The layout of this paper is as follows. In Section 2 weprovide both theoretical and empirical evidence of the su-periority of trend filtering for estimating spatially hetero-geneous signals compared to classical statistical methods.In Section 3 we introduce trend filtering, including a gen-eral overview of the estimator’s machinery, its connection tospline methods, automatic methods for choosing the hyper-parameter, uncertainty quantification, generalizations, andrecommended software implementations in various program-ming languages. In Politsch et al. (2020)—hereafter referred
to as Paper II—we directly illustrate the broad utility oftrend filtering to astronomy by conducting various analysesof spectra and light curves.
We begin this section by providing background and motiva-tion for the nonparametric approach to estimating (or de-noising) signals. We then discuss statistical optimality forestimating spatially heterogeneous signals, with an empha-sis on providing evidence for the claim that trend filtering issuperior to classical statistical methods in this highly gen-eral setting. Finally, we end this section by illustrating thissuperiority with a direct empirical comparison of trend fil-tering and several popular classical methods on simulatedobservations of a spatially heterogeneous signal.
2.1 Nonparametric regression
Suppose we observe noisy measurements of a response vari-able of interest (e.g., flux, magnitude, photon counts) ac-cording to the data generating process (DGP)
f (ti) = f0(ti) + εi, i = 1, . . . , n (1)
where f0(ti) is the signal at input ti (e.g., a time or wave-length) and εi is the noise at ti that contaminates the signal,giving rise to the observation f (ti). Let t1, . . . , tn ∈ (a, b) de-note the observed input interval and E[εi] = 0 (where we useE[·] to denote mathematical expectation). Here, the generalstatistical problem is to estimate (or denoise) the underlyingsignal f0 from the observations as accurately as possible. Inthe nonparametric setting, we refrain from making stronga priori assumptions about f0 that could lead to signifi-cant modeling bias, e.g. assuming a power law or a light-curve/spectral template fit. Mathematically, a nonparamet-ric approach is defined through the deliberately weak as-sumption f0 ∈ F (i.e. the signal belongs to the function classF ) where F is infinite-dimensional. In other words, the as-sumed class of all possible signals F cannot be spanned by afinite number of parameters. Contrast this to the assumptionthat the signal follows a pth degree power law, i.e. f0 ∈ FPL
where
FPL =
{f0 : f0(t) = β0 +
p∑j=1
βj t j}, (2)
a class that is spanned by p+ 1 parameters. Similarly, givena set of p spectral/light-curve templates b1(t), . . . , bp(t), theusual template-fitting assumption is that f0 ∈ FTEMP where
FTEMP =
{f0 : f0(t) = β0 +
p∑j=1
βjbj ((t − s)/v)}
(3)
and s and v are horizontal shift and scale hyperparame-ters, respectively. Both (2) and (3) represent very stringentassumptions about the underlying signal f0. If the signalis anything other than exactly a power law in t—a highlyunlikely occurrence—nontrivial statistical bias will arise bymodeling it as such. Likewise, if a class of signals has arich physical diversity (e.g., Type Ia supernova light curves;Woosley et al. 2007) that is not sufficiently spanned by the
MNRAS 000, 1–15 (2020)
Trend Filtering in Astronomy 3
library of templates used in modeling, then statistical biaseswill arise. Depending on the size of the imbalance betweenclass diversity and the completeness of the template basis,the biases could be significant. Moreover, these biases arerarely tracked by uncertainty quantification. To be clear, thisis not a uniform criticism of template-fitting. For example,templates are exceptionally powerful tools for object clas-sification and redshift estimation (e.g., Howell et al. 2005,Bolton et al. 2012). Furthermore, much of our discussionin Paper II centers around utilizing the flexible nonpara-metric nature of trend filtering to construct more completespectral/light-curve template libraries for various observa-tional objects and transient events.
Let f0 be any statistical estimator for the signal f0, de-rived from the noisy observations in (1). Further, let pt (t)denote the probability density function (pdf) that specifiesthe sampling distribution of the inputs on the interval (a, b),and let σ2(t) = Var(ε(t)) denote the noise level at input t. Inorder to assess the accuracy of the estimator it is commonto consider the mean-squared prediction error (MSPE):
R( f0) = E[( f0 − f )2
](4)
= E[( f0 − f0)2
]+ σ2 (5)
=
∫ b
a
(Bias2( f0(t)) +Var( f0(t))
)· pt (t)dt + σ2, (6)
where
Bias( f0(t)) = E[ f0(t)] − f0(t) (7)
Var( f0(t)) = E(
f0(t) − E[ f0(t)])2
(8)
σ2 =
∫ b
aσ2(t) · pt (t)dt . (9)
The equality in (6) is commonly referred to as the bias-variance decomposition. The first term is the squared biasof the estimator f0 (integrated over the input interval) andintuitively measures how appropriate the chosen statisticalestimator is for modeling the observed phenomenon. Thesecond term is the variance of the estimator that measureshow stable or sensitive the estimator is to the observed data.And the third term is the irreducible error—the minimumprediction error we cannot hope to improve upon. The bias-variance decomposition therefore illustrates that an opti-mal estimator is one that combines appropriate modelingassumptions (low bias) with high stability (low variance).
2.1.1 Statistical optimality (minimax theory)
In this section, we briefly discuss a mathematical frameworkfor evaluating the performance of statistical methods overnonparametric signal classes in order to demonstrate thatthe superiority of trend filtering is a highly general result.Ignoring the irreducible error, the problem of minimizing theMSPE of a statistical estimator can be equivalently statedas a minimization of the first term in (5)—the mean-squaredestimation error (MSEE). In practice, low bias is attained byonly making very weak assumptions about what the underly-ing signal may look like, e.g. f0 has k continuous derivatives.An ideal statistical estimator for estimating signals in sucha class (call it F ) may then be defined as
inff0
(supf0∈FE[( f0 − f0)2
] ). (10)
That is, we would like our statistical estimator to be the min-imizer (infimum) of the worst-case (supremum) MSEE overthe signal class F . This is rarely a mathematically tractableproblem for any practical signal class F . A more tractableapproach is to consider how the worst-case MSEE behaves asa function of the sample size n. A reasonable baseline metricfor a statistical estimator is to require that it satisfies
supf0∈FE[( f0 − f0)2
]→ 0 (11)
as n → ∞. That is, for any signal f0 ∈ F , when a largeamount of data is available, f0 gets arbitrarily close tothe true signal. In any practical situation, this is not truefor parametric models because the bias component of thedecomposition never vanishes. This, however, is a widely-held—perhaps, defining—property of nonparametric meth-ods. Therefore, in order to distinguish optimality amongnonparametric estimators, we require a stronger metric. Inparticular, we study how quickly the worst-case error goesto zero as more data are observed. This is the core ideaof a rich area of statistical literature called minimax theory(see, e.g., Van der Vaart 1998; Wasserman 2006; Tsybakov2008). For many infinite-dimensional classes of signals, the-oretical lower-bounds exist on the rate at which the MSEEof any statistical estimator can approach zero. Therefore,if a statistical estimator is shown to achieve that rate, itcan be considered optimal for estimating that class of sig-nals. Formally, letting g(n) be the rate at which the MSEEof the theoretically optimal estimator (10) goes to zero (amonotonically decreasing function in n), we would like our
estimator f0 to satisfy
supf0∈FE[( f0 − f0)2
]= O(g(n)), (12)
where we use O(·) to denote big O notation. If this is shownto be true, we say the estimator achieves the minimax rateover the signal class F . Loosely speaking, we are stating thata minimax optimal estimator is an estimator that learns thesignal from the data just as quickly as the theoretical goldstandard estimator (10).
2.1.2 Spatially heterogeneous signals
Thus far we have only specified that the signal underlyingmost one-dimensional astronomical observations should beassumed to belong to a class F that is infinite-dimensional(i.e. nonparametric). Further, in Section 2.1.1 we introducedthe standard metric used to measure the performance ofa statistical estimator over an infinite-dimensional class ofsignals. Recalling the discussion in the abstract and Section1, trend filtering provides significant advances for estimatingsignals that exhibit varying degrees of smoothness across theinput domain. We restate this definition below.
Definition. A spatially heterogeneous signal is asignal that exhibits varying degrees of smoothnessin different regions of its input domain.
Example. A smooth light curve with abrupt tran-sient events.
MNRAS 000, 1–15 (2020)
4 C. A. Politsch et al.
Example. An electromagnetic spectrumwith smooth continuum radiation and sharpabsorption/emission-line features.
To complement the above definition we may also loosely de-fine a spatially homogeneous signal as a signal that is eithersmooth or wiggly1 across its input domain, but not both.As“smoothness” can be quantified in various ways these def-initions are intentionally mathematically imprecise. A classthat is commonly considered in the statistical literature isthe L2 Sobolev class:
F2,k (C1) :=
{f0 :
∫ b
af (k)0 (t)
2dt < C1
}, C1 > 0, k ∈ N. (13)
That is, an L2 Sobolev class is a class of all signals suchthat the integral of the square (the “L2 norm”) of the kthderivative of each signal is less than some constant C1. Sta-tistical optimality in the sense of Section 2.1.1 for estimat-ing signals in these classes (and some other closely relatedones) is widely held among statistical methods in the clas-sical toolkit; for example, kernel smoothers (Ibragimov &Hasminiskii 1980; Stone 1982), LOESS (Fan 1993; Fan et al.1997), and smoothing splines (Nussbaum 1985). However, aseminal result by Nemirovskii et al. (1985) and Nemirovskii(1985) showed that a statistical estimator can be minimaxoptimal over signal classes of the form (13) and still per-form quite poorly on other signals. In particular, the authorsshowed that, when considering the broader L1 Sobolev class
F1,k (C2) :=
{f0 :
∫ b
a
�� f (k)0 (t)��dt < C2
}, C2 > 0, k ∈ N, (14)
all linear smoothers2—including kernels, LOESS, smooth-ing splines, Gaussian process regression, and many othermethods—are strictly suboptimal. The key difference be-tween these two types of classes is that L2 Sobolev classesare rich in spatially homogeneous signals but not spatiallyheterogeneous signals, while L1 Sobolev classes3 are rich inboth (see, e.g., Donoho & Johnstone 1998).
The intuition of this result is that linear smoothers can-not optimally recover signals that exhibit varying degrees ofsmoothness across their input domain because they operateas if the signal possesses a fixed degree of smoothness. Forexample, this intuition is perhaps most clear when consid-ering a kernel smoother with a fixed bandwidth. The resultof Nemirovskii et al. (1985) and Nemirovskii (1985) there-fore implies that, in order to achieve statistical optimalityfor estimating spatially heterogeneous signals, a statisticalestimator must be nonlinear (more specifically, it must belocally adaptive). Tibshirani (2014) showed that trend filter-ing is minimax optimal for estimating signals in L1 Sobolev
1 This is, in fact, a technical term used in the statistical literature.2 A linear smoother is a statistical estimator that is a linear com-
bination of the observed data. Many popular statistical estima-tors, although often motivated from seemingly disparate premises,can be shown to fall under this definition. See, e.g., Wasserman
(2006) for more details.3 The L1 Sobolev class is often generalized to a nearly equiv-
alent but slightly larger class—namely, signals with derivativesof bounded variation. See Tibshirani (2014) for the generalized
definition.
classes. Since L2 Sobolev classes are contained within L1Sobolev classes, this result also guarantees that trend fil-tering is also minimax optimal for estimating signals in L2Sobolev classes. Wavelets share this property, but require re-strictive assumptions on the sampling of the data (Donoho& Johnstone 1994).
How large is this performance gap? The collective re-sults of Nemirovskii et al. (1985), Nemirovskii (1985), andTibshirani (2014) reveal that the performance gap betweentrend filtering and linear smoothers when estimating spa-tially heterogeneous signals is significant. For example, whenk = 0, the minimax rate over L1 Sobolev classes (whichtrend filtering achieves) is n−2/3, but linear smoothers can-not achieve better than n−1/2. To put this in perspective,this result says that the trend filtering estimator, trainingon n data points, learns these signals with varying smooth-ness as quickly as a linear smoother training on n4/3 datapoints. As we demonstrate in the next section, this gap intheoretical optimality has clear practical consequences.
In order to minimize the pervasion of technical statis-tical jargon throughout the paper, henceforth we simply re-fer to a statistical estimator that achieves the minimax rateover L2 Sobolev classes as statistically optimal for estimatingspatially homogeneous signals, and we refer to a statisticalestimator that achieves the minimax rate over L1 Sobolevclasses as statistically optimal for estimating spatially hetero-geneous signals. As previously mentioned, the latter impliesthe former, but not vice versa.
2.2 Empirical comparison
In this section we analyze noisy observations of a simu-lated spatially heterogeneous signal in order to compare theempirical performance of trend filtering and several classi-cal statistical methods—namely, LOESS, smoothing splines,and Gaussian process regression. The mock observations aresimulated on an unequally-spaced grid t1, . . . , tn ∼ Unif(0, 1)according to the data generating process
f (ti) = f0(ti) + εi (15)
with
f0(ti) = 63∑
k=1(ti − 0.5)k + 2.5
4∑j=1(−1)jφ j (ti), (16)
where φ j (t), j = 1, . . . , 4 are compactly-supported radial ba-sis functions distributed throughout the input space andεi ∼ N(0, 0.1252). We therefore construct the signal f0 tohave a smoothly varying global trend with four sharp local-ized features—two dips and two spikes. The signal and noisyobservations are shown in the top panel of Figure 1.
In order to facilitate the comparison of methods we uti-lize a metric for the total statistical complexity (i.e. totalwiggliness) of an estimator known as the effective degrees offreedom (see, e.g., Tibshirani 2015). Formally, the effective
degrees of freedom of an estimator f0 is defined as
df( f0) = σ −2n∑i=1
Cov( f0(ti), f (ti)) (17)
where σ2 is defined in (9). In Figure 1 we fix all estimatorsto have 55 effective degrees of freedom. This exercise pro-vides insight into how each estimator relatively distributes
MNRAS 000, 1–15 (2020)
Trend Filtering in Astronomy 5
Figure 1. Comparison of statistical methods on data simulated from a spatially heterogeneous signal. Each statistical estimator is fixedto have 55 effective degrees of freedom in order to facilitate a direct comparison. The trend filtering estimator is able to sufficientlydistribute its effective degrees of freedom such that it simultaneously recovers the smoothness of the global trend, as well as the abrupt
localized features. The LOESS, smoothing spline, and Gaussian process regression each estimates the smooth global trend reasonablywell here, but significantly oversmooths the sharp peaks and dips. Here, we utilize quadratic trend filtering (see Section 3.2).
MNRAS 000, 1–15 (2020)
6 C. A. Politsch et al.
Figure 2. (Continued): Comparison of statistical methods on data simulated from a spatially heterogeneous signal. Here, each of thelinear smoothers (i.e. the LOESS, smoothing spline, and Gaussian process regression) is fixed at 192 effective degrees of freedom—thecomplexity necessary for each estimator to recover the sharp localized features approximately as well as the trend filtering estimator with
55 effective degrees of freedom. While the linear smoothers now estimate the four abrupt features well, each severely overfits the data inthe other regions of the input domain.
MNRAS 000, 1–15 (2020)
Trend Filtering in Astronomy 7
its complexity across the input domain. In the second panelwe see that the trend filtering estimate has sufficiently recov-ered the underlying signal, including both the smoothnessof the global trend and the abruptness of the localized fea-tures. All three of the linear smoothers, on the other hand,severely oversmooth the localized peaks and dips. Gaus-sian process regression also exhibits some undesirable os-cillatory features that do not correspond to any real trendin the signal. In order to better recover the localized fea-tures the linear smoothers require a more complex fit, i.e.smaller LOESS kernel bandwidth, smaller smoothing splinepenalization, and smaller Gaussian process noise-signal vari-ance. In Figure 2 we show the same comparison, but wegrant the linear smoothers more complexity. Specifically, inorder to recover the sharp features comparably with thetrend filtering estimator with 55 effective degrees of free-dom, the linear smoothers require 192 effective degrees offreedom—approximately 3.5 times the complexity. As a re-sult, although they now adequately recover the peaks anddips, each linear smoother severely overfits the data in theother regions of the input domain, resulting in many spuri-ous fluctuations.
As discussed in Section 2.1.2, the suboptimality ofLOESS, smoothing splines, and Gaussian process regressionillustrated in this example is an inherent limitation of thebroad linear smoother family of statistical estimators. Linearsmoothers are adequate tools for estimating signals that ex-hibit approximately the same degree of smoothness through-out their input domain. However, when a signal is expectedto exhibit varying degrees of smoothness across its domain,a locally-adaptive statistical estimator is needed.
3 TREND FILTERING
Trend filtering, in its original form, was independently pro-posed in the computer vision literature (Steidl et al. 2006)and the applied mathematics literature (Kim et al. 2009),and has recently been further developed in the statisticaland machine learning literature, most notably with Tibshi-rani & Taylor (2011), Tibshirani (2014), Wang et al. (2016),and Ramdas & Tibshirani (2016). This work is in no wayrelated to the work of Kovacs et al. (2005), which goes by asimilar name. At a high level, trend filtering is closely relatedto two familiar nonparametric regression methods: variable-knot regression splines and smoothing splines. We elaborateon these relationships below.
3.1 Closely-related methods
Splines have long played a central role in estimating com-plex signals (see, e.g., De Boor 1978 and Wahba 1990 forgeneral references). Formally, a kth order spline is a piece-wise polynomial (i.e. piecewise power law) of degree k that iscontinuous and has k−1 continuous derivatives at the knots.As their names suggest, variable-knot regression splines andsmoothing splines center around fitting splines to observa-tional data. Recall from (1) the observational data generat-ing process (DGP)
where f (ti) is a noisy measurement of the signal f0(ti), andE[εi] = 0. Given a set of knots κ1, . . . , κp ∈ (a, b), the spaceof all kth order splines on the interval (a, b) with knots atκ1, . . . , κp can be parametrized via a basis representation
m(t) =∑j
βjηj (t), (19)
where {ηj } is typically the truncated power basis or B-splinebasis. A suitable estimator for the signal f0 may then be
f0(t) =∑j
βjηj (t), (20)
where the βj are the ordinary least-squares (OLS) estimatesof the basis coefficients. This is called a regression spline.The question of course remains where to place the knots.
3.1.1 Variable-knot regression splines
The variable-knot (or free-knot) regression spline approachis to consider all regression spline estimators with knots ata subset of the observed inputs, i.e. {κ1, . . . , κp} ⊂ {t1, . . . , tn}for all possible p. Formally, the variable-knot regressionspline estimator is the solution to the following constrainedleast-squares minimization problem:
min{β j }
n∑i=1
(f (ti) −
∑j
βjηj (ti))2
s.t.∑
j≥k+21{βj , 0} = p
p ≥ 0
(21)
where p ≥ 0 is the number of knots in the spline and 1(·) isthe indicator function satisfying
1{βj , 0} ={
1 βj , 0,0 βj = 0.
(22)
Furthermore, note that the equality constraint on the basiscoefficients excludes those of the “first” k + 1 basis functionsthat span the space of global polynomials and only countsthe number of active basis functions that produce knots. Thevariable-knot regression spline optimization is therefore aproblem of finding the best subset of knots for the regressionspline estimator. Due to the sparsity of the coefficient con-straint, the variable-knot regression spline estimator allowsfor highly locally-adaptive behavior for estimating signalsthat exhibit varying degrees of smoothness. However, theproblem itself cannot be solved in polynomial time, requir-ing an exhaustive combinatorial search over all ∼2n feasiblemodels. It is common to utilize stepwise procedures basedon iterative addition and deletion of knots in the active set,but these partial searches over the feasible set inherentlyprovide no guarantee of finding the optimal global solutionto (21).
In order to make the connection to trend filtering moreexplicit it is helpful to reformulate the constrained minimiza-tion (21) into the following penalized unconstrained mini-mization problem:
min{β j }
n∑i=1
(f (ti) −
∑j
βjηj (ti))2
+ γ∑
j≥k+21{βj , 0}, (23)
MNRAS 000, 1–15 (2020)
8 C. A. Politsch et al.
where γ > 0 is a hyperparameter that determines the num-ber of knots in the spline and the sum of indicator func-tions serves as a smoothness “penalty” on the ordinary least-squares minimization. Penalized regression is a popular areaof statistical methodology (see, e.g., Hastie et al. 2009), inwhich the cost functional (i.e. the quantity to be minimized)quantifies a tradeoff between the training error of the esti-mator (here, the sum of squared residuals) and the statisticalcomplexity of the estimator (here, the number of knots inthe spline). In particular, (23) is known as an `0-penalizedleast-squares regression because of the penalty’s connectionto the mathematical `0 vector quasi-norm.
3.1.2 Smoothing splines
Smoothing splines counteract the computational issue facedby variable-knot regression splines by simply placing knotsat all of the observed inputs t1, . . . , tn and regularizing thesmoothness of the fitted spline. For example, letting G bethe space of all cubic natural splines with knots at t1, . . . , tn,the cubic smoothing spline estimator is the solution to theoptimization problem
minm∈G
n∑i=1
(f (ti) − m(ti)
)2+ γ
∫ b
a
(m′′(t)
)2dt, (24)
where m′′ is the second derivative of m and γ > 0 tunesthe amount of regularization. Letting η1, . . . , ηn be a basisfor cubic natural splines with knots at the observed inputs,(24) can be equivalently stated as a minimization over thebasis coefficients:
min{β j }
n∑i=1
(f (ti) −
∑j
βjηj (ti))2
+ γ
n∑j,k=1
βj βkωjk (25)
where
ωjk =
∫ b
aη′′j (t)η
′′k (t)dt. (26)
The cost functional (25) is differentiable and leads to a lin-ear system with a special sparse structure (i.e. bandedness),which yields a solution that can both be found in closed-form and computed very quickly—in O(n) elementary op-erations. This particular choice of cost functional, however,produces an estimator that is a linear combination of theobservations—a linear smoother. Therefore, as discussed anddemonstrated in Section 2, smoothing splines are suboptimalfor estimating spatially heterogeneous signals. Equation (25)is known as an `2-penalized least-squares regression becauseof the penalty’s connection to the mathematical `2 vector-norm.
3.2 Definition
Trend filtering can be viewed as a blending of the strengthsof variable-knot regression splines (local adaptivity and in-terpretability) and the strengths of smoothing splines (sim-plicity and speed). Mathematically, this is achieved bychoosing an appropriate set of basis functions and penalizingthe least-squares problem with an `1 norm on the basis co-efficients (sum of absolute values), instead of the `0 norm ofvariable-knot regression splines (sum of indicator functions)or the `2 norm of smoothing splines (sum of squares).
This section is primarily summarized from Tibshirani(2014) and Wang et al. (2014). Let the inputs be orderedwith respect to the index, i.e. t1 < · · · < tn. For thesake of simplicity, we consider the case when the inputst1, . . . , tn ∈ (a, b) are equally spaced with ∆t = ti+1 − ti . Seethe aforementioned papers for the generalized definition oftrend filtering to unequally spaced inputs.
For any given integer k ≥ 0, the kth order trend fil-tering estimate is a piecewise polynomial of degree k withknots automatically selected at a sparse subset of the ob-served inputs t1, . . . , tn. In Figure 3, we provide an exampleof a trend-filtered data set for orders k = 0, 1, 2, and 3. Specif-ically, the panels of the figure respectively display piecewiseconstant, piecewise linear, piecewise quadratic, and piece-wise cubic fits to the data with the automatically-selectedknots indicated by the tick marks on the horizontal axes.Constant trend filtering is equivalent to total variation de-noising (Rudin et al. 1992), as well as special forms of thefused lasso of Tibshirani et al. (2005) and the variable fusionestimator of Land & Friedman (1996). Linear trend filter-ing was independently proposed by Steidl et al. (2006) andKim et al. (2009). Higher-order polynomial trend filtering(k ≥ 2) was developed by Tibshirani & Taylor (2011) andTibshirani (2014). In the Figure 3 example, the quadraticand cubic trend filtering estimates are nearly visually indis-tinguishable, and this is true in general. Although, as we seehere, trend filtering estimates of different orders typicallyselect different sets of knots.
Like the spline methods discussed in Section 3.1, forany order k ≥ 0, the trend filtering estimator has a basisrepresentation
m(t) =n∑j=1
βjhj (t), (27)
but, here, the trend filtering basis {h1, . . . , hn} is the fallingfactorial basis, which is defined as
Like the truncated power basis, the first k+1 basis functionsspan the space of global kth order polynomials and the restof the basis adds the piecewise polynomial structure. How-ever, the knot-producing basis functions of the falling fac-torial basis hj , j ≥ k + 2 have small discontinuities in theirjth order derivatives at the knots for all j = 1, . . . , k − 1,and therefore for orders k ≥ 2 the trend filtering estimate isclose to, but not quite a spline. The discontinuities are smallenough, however, that the trend filtering estimate definedthrough the falling factorial basis representation is visuallyindistinguishable from the analogous spline produced by thetruncated power basis (see Tibshirani 2014 and Wang et al.2014). The advantage of utilizing the falling factorial basis inthis context instead of the truncated power basis (or the B-spline basis) comes in the form of significant computationalspeedups, as we detail below.
Analogous to the continuous smoothing spline prob-lem (24), we let Hk be the space of all functions spannedby the kth order falling factorial basis, and pose the trendfiltering problem as a least-squares minimization with aderivative-based penalty on the fitted function. In partic-ular, the kth order trend filtering estimator is the solution
MNRAS 000, 1–15 (2020)
Trend Filtering in Astronomy 9
Figure 3. Piecewise polynomials with adaptively-chosen knots produced by trend filtering. From top to bottom, we show trend filteringestimates of orders k = 0, 1, 2 and 3, which take the form of piecewise constant, piecewise linear, piecewise quadratic, and piecewise cubic
polynomials, respectively. The adaptively-chosen knots of each piecewise polynomial are indicated by the tick marks along the horizontalaxes. The constant trend filtering estimate is discontinuous at the knots, but we interpolate here for visual purposes. The data set istaken from the Lyman-α forest of a mock quasar spectrum (Bautista et al. 2015), sampled in logarithmic-angstrom space. We study this
phenomenon in detail in Paper II.
MNRAS 000, 1–15 (2020)
10 C. A. Politsch et al.
to the problem
minm∈Hk
n∑i=1
(f (ti) − m(ti)
)2+ γ · TV(m(k)), (29)
where m(k) is the kth derivative of m, TV(m(k)) is the to-tal variation of m(k), and γ > 0 is the model hyperparameterthat controls the smoothness of the fit. When m(k) is differen-tiable everywhere in its domain, the penalty term simplifiesto
TV(m(k)) =∫ b
a|m(k+1)(t)|dt . (30)
Avoiding the technical generalized definition of total vari-ation (see, e.g., Tibshirani 2014), we can simply think ofTV(·) as a generalized L1 norm4 for our piecewise polyno-mials that possess small discontinuities in the derivatives.Again referring back to the smoothing spline problem (24),definitions (29) and (30) reveal that trend filtering can bethought of as an L1 analog of the (L2-penalized) smooth-ing spline problem. Moreover, note that unlike smoothingsplines, trend filtering can produce piecewise polynomials ofall orders k ≥ 0.
Replacing m with its basis representation, i.e. m(t) =∑j βjhj (t), yields the equivalent finite-dimensional trend fil-
tering minimization problem5:
min{β j }
n∑i=1
(f (ti) −
n∑j=1
βjhj (ti))2
+ γ · k! · ∆tkn∑
j=k+2|βj |. (31)
The terms k! and ∆tk are constants and can therefore be ig-nored by absorbing them into the hyperparameter γ. Visualinspection of (31) reveals that trend filtering is also analo-gous to the variable-knot regression spline problem (21)—namely, by replacing the `0 norm on the basis coefficientswith an `1 norm. The advantage here is that the problemis now strictly convex and can be efficiently solved by var-ious convex optimization algorithms. Furthermore, the `1penalty still yields a sparse solution (i.e. many βj = 0),which provides the automatic knot-selection property. Let-ting β1, . . . , βn denote the solution to (31) for a particularchoice of γ > 0, the trend filtering estimate is then given by
f0(t; γ) =n∑j=1
βjhj (t), (32)
with the automatically-selected knots corresponding to thebasis functions with βj , 0, j ≥ k + 1.
The advantage of utilizing the falling factorial basis isfound by reparametrizing the problem (31) into an optimiza-tion over the fitted values m(t1), . . . ,m(tn). The problem thenreduces to
min{m(ti )}
n∑i=1
(f (ti) − m(ti)
)2+ γ
n−k−1∑i=1|∆(k+1)m(ti)| · ∆t (33)
where ∆(k+1)m(ti) can be viewed as a discrete approximation
4 We use the upper-case notation Lp , p = 1, 2 for the p-norm of a
continuous function, and `p , p = 0, 1, 2 for the p-norm of a vector.5 This may be recognized as a lasso regression (Tibshirani 1996),
with the features being the falling factorial basis functions.
of the (k + 1)st derivative of m at ti . For k = 0 the discretederivatives are
∆(1)m(ti) =
m(ti+1) − m(ti)∆t
, (34)
and then can be defined recursively for k ≥ 1:
∆(k+1)m(ti) =
∆(k)m(ti+1) − ∆(k)m(ti)∆t
. (35)
The penalty term in (33) can be viewed as a Riemann-likediscrete approximation of the integral in (30). Because of thechoice of basis, the problem has reduced to a simple gener-alized lasso problem (Tibshirani & Taylor 2011; Arnold &Tibshirani 2016) with an identity predictor matrix and abanded6 penalty matrix. This special structure allows thesolution to be computed very efficiently and with a nearlylinear time scaling—i.e. O(n) elementary operations—viathe specialized alternating direction method of multipliers(ADMM) algorithm of Ramdas & Tibshirani (2016). This al-gorithm has a linear complexity per iteration, so the overallcomplexity is O(nr) where r is the number of iterations nec-essary to converge to the solution. In the worst case scenarior ∼ n1/2, so the worst-case overall complexity is O(n1.5). Inpractice, the computations of the specialized trend filteringoptimization algorithm are highly efficient and scale to mas-sive data sets, e.g. handling data sets with n & 107 within afew minutes. See Ramdas & Tibshirani (2016) for more rigor-ous timing results. The practical and scalable computationalspeed further illustrates the value of trend filtering to astron-omy, as it is readily compatible with the large-scale analysisof one-dimensional data sets that has become increasinglyubiquitous in large sky surveys. We show a comparison inTable 1 of the computational costs associated with trendfiltering and other popular one-dimensional nonparametricmethods.
Given the trend filtering fitted values obtained by theoptimization (33) the full continuous-time representationof the trend filtering estimate follows by inverting theparametrization back to the basis function coefficients andplugging them into the basis representation (32).
3.3 Extension to heteroskedastic weighting
Thus far we have considered the simple case where the obser-vations are treated as equally-weighted in the cost functional(33). Recall from (18) the observational data generating pro-cess and define σ2
i = Var(εi) to be the noise level—the (typi-cally heteroskedastic) uncertainty in the measurements thatarises from instrumental errors and removal of systematiceffects. When estimates for σ2
i , i = 1, . . . , n accompany theobservations, as they often do, they can be used to weightthe observations to yield a more efficient statistical estimator(i.e. smaller mean-squared error). The error-weighted trendfiltering estimator is the solution to the following minimiza-tion problem:
min{m(ti )}
n∑i=1
(f (ti) − m(ti)
)2wi + γ
n−k−1∑i=1|∆(k+1)m(ti)| · ∆t, (36)
6 A banded matrix only contains nonzero elements in the main
diagonal and zero or more diagonals on either side.
Table 1. Comparison of computational costs associated with popular one-dimensional nonparametric regression methods. The compu-tational complexity column states how the number of elementary operations necessary to obtain the fitted values of each estimator (i.e.
the estimator evaluated at the observed inputs) scales with the sample size n. For trend filtering, the O(n1.5) complexity represents the
worst-case complexity of the Ramdas & Tibshirani (2016) convex optimization algorithm. In most practical settings the actual complex-ity of this algorithm is close to O(n). Variable-knot regression splines require a (nonconvex) exhaustive combinatorial search over the
set of possible knots and the complexity therefore includes a binomial coefficient term(np
)= n!/(n!(n − p)!), where p is the number of
knots in the spline. The remaining methods are explicitly solvable and the stated complexity represents the cost of an exact calculation.The O(n) complexity of wavelets relies on restrictive sampling assumptions (e.g., equally-spaced inputs, sample size equal to a power
of two). The stated computational complexity of all methods represents the cost of a single model fit and does not include the cost ofhyperparameter tuning. Gaussian process regression suffers from the most additional overhead in this regard because of the (often) large
number of hyperparameters used to parametrize the covariance function (e.g., shape, range, marginal variance, noise variance). Each of
the non-adaptive methods (linear smoothers) can be made to be locally adaptive (e.g., by locally varying the hyperparameters of themodel), but at the expense of greatly increasing the dimensionality of the hyperparameter space to be searched over.
where the optimal choice of weights is wi = σ−2i , i = 1, . . . , n.
Much of the publically available software for trend filteringallows for a heteroskedastic weighting scheme (see Section3.4).
3.4 Software
Trend filtering software is available online across variousplatforms. For the specialized ADMM algorithm of Ramdas& Tibshirani (2016) that we utilize in this work, implemen-tations are available in R and C (Arnold et al. 2014), as wellas Julia (Kornblith 2014). Matlab and Python implementa-tions are available for the primal-dual interior point methodof Kim et al. (2009), but only for equally-weighted lineartrend filtering (Koh et al. 2008; Diamond & Boyd 2016). Weprovide links to our recommended implementations in Ta-ble 2. Note that in all software implementations the trendfiltering hyperparameter is called λ instead of γ, which weuse here to avoid ambiguity with the notation for wavelengthin our spectroscopic analyses in Paper II.
3.5 Choosing the hyperparameter
The choice of the piecewise polynomial order k generally hasminimal effect on the performance of the trend filtering esti-mator in terms of mean-squared error and therefore can betreated as an a priori aesthetic choice based on how muchsmoothness is desired or believed to be present. For exam-ple, we use k = 2 (quadratic trend filtering) throughout ouranalyses in Paper II so that the fitted curves are smooth,i.e. differentiable everywhere.
Given the choice of k, the hyperparameter γ > 0 is usedto tune the complexity (i.e. the wiggliness) of the trend fil-tering estimate by weighting the tradeoff between the com-plexity of the estimate and the size of the squared residuals.Obtaining an accurate estimate is therefore intrinsically tied
Language Recommended implementation
R github.com/glmgen
C github.com/glmgen
Python cvxpy.org
Matlab http://stanford.edu/~boyd/l1_tf
Julia github.com/JuliaStats/Lasso.jl
Table 2. Recommended implementations for trend fil-tering in various programming languages. See Sec-
tion 3.4 for details. We provide supplementary R code at
github.com/capolitsch/trendfilteringSupp for selectingthe hyperparameter via minimization of Stein’s unbiased risk
estimate (see Section 3.5) and various bootstrap methods for
uncertainty quantification (see Section 3.6). Our implementationsare built on top of the glmgen R package of Arnold et al. (2014).
to finding an optimal choice of γ. The selection of γ is typ-ically done by minimizing an estimate of the mean-squaredprediction error (MSPE) of the trend filtering estimator.Here, there are two different notions of error to consider,namely, fixed-input error and random-input error. As thenames suggest, the distinction between which type of errorto consider is made based on how the inputs are sampled.As a general rule-of-thumb, we recommend optimizing withrespect to fixed-input error when the inputs are regularly-sampled and optimizing with respect to random-input erroron irregularly-sampled data.
Recall the DGP stated in (18) and let it be denoted by Qso that EQ[·] is the mathematical expectation with respect
to the randomness of the DGP. Further, let σ2i = Var(εi).
where, in the latter, t is considered to be a random compo-nent of the DGP with a marginal probability density pt (t)supported on the observed input interval. In each case, thetheoretically optimal choice of γ is defined as the minimizerof the respective choice of error. Empirically, we estimate thetheoretically optimal choice of γ by minimizing an estimateof (37) or (39). For fixed-input error we recommend Stein’sunbiased risk estimate (SURE; Stein 1981; Efron 1986) andfor random-input error we recommend K-fold cross valida-tion with K = 10. We elaborate on SURE here and refer thereader to Wasserman (2003) for K-fold cross validation.
The SURE formula provides an unbiased estimate ofthe fixed-input MSPE of a statistical estimator:
R0(γ) =1n
n∑i=1
(f (ti) − f0(ti ; γ)
)2+
2σ2df( f0)n
, (40)
where σ2 = n−1 ∑ni=1 σ
2i and df( f0) is defined in (17). A for-
mula for the effective degrees of freedom of the trend filter-ing estimator is available via the generalized lasso results ofTibshirani & Taylor (2012); namely,
df( f0) = E[number of knots in f0] + k + 1. (41)
We then obtain our hyperparameter estimate γ by minimiz-ing the following plug-in estimate for (40):
R(γ) = 1n
n∑i=1
(f (ti) − f0(ti ; γ)
)2+
2σ2df( f0)n
, (42)
where df is the estimate for the effective degrees of free-dom that is obtained by replacing the expectation in (41)
with the observed number of knots, and σ2
is an estimateof σ2. If a reliable estimate of σ2 is not available a pri-ori, a data-driven estimate can be constructed (see, e.g.,Wasserman 2006). We provide a supplementary R packageon the corresponding author’s GitHub page7 for implement-ing SURE with trend filtering. The package is built on top ofthe glmgen R package of Arnold et al. (2014), which alreadyincludes an implementation of K-fold cross validation.
Because of the existence of the degrees of freedom ex-pression (41), trend filtering is also compatible with reducedchi-squared model assessment and comparison proceduresunder a Gaussian noise assumption (Pearson 1900; Cochran1952).
Frequentist uncertainty quantification for trend filtering fol-lows by studying the sampling distribution of the estimatorthat arises from the randomness of the observational datagenerating process (DGP). In particular, most of the uncer-tainty in the estimates is captured by studying the variabil-ity of the estimator with respect to the DGP. We advisethree different bootstrap methods (Efron 1979) for estimat-ing the variability of the trend filtering estimator, with eachmethod corresponding to a distinct analysis setting. Here,we emphasize the terminology variability—as opposed to thevariance of the trend filtering estimator—since, by construc-tion, as a nonlinear function of the observed data, the trendfiltering estimator has a non-Gaussian sampling distributioneven when the observational noise is Gaussian. For that rea-son, each of our recommended bootstrap approaches is basedon computing sample quantiles (instead of pairing standarderrors with Gaussian quantiles).
We restate the assumed DGP here for clarity:
f (ti) = f0(ti) + εi, t1, . . . , tn ∈ (a, b) (43)
where E[εi] = 0. We make the further assumption that theerrors ε1, . . . , εn are independent8. The three distinct settingswe consider are:
S1. The inputs are irregularly sampledS2. The inputs are regularly sampled and the noise dis-
tribution is knownS3. The inputs are regularly sampled and the noise dis-
tribution is unknown
The corresponding bootstrap methods are detailed in Algo-rithm 1 (nonparametric bootstrap; Efron 1979), Algorithm 2(parametric bootstrap; Efron & Tibshirani 1986), and Al-gorithm 3 (wild bootstrap; Wu 1986; Liu 1988; Mammen1993), respectively. We include implementations of each ofthese algorithms in the R package on our GitHub page.
Given the full trend filtering bootstrap ensemble pro-vided by the relevant bootstrap algorithm, for any α ∈ (0, 1),a (1 − α) · 100% quantile-based pointwise variability band isgiven by
V1−α(t ′i ) =(
f ∗α/2(t′i ), f ∗1−α/2(t
′i )), i = 1, . . . ,m (44)
where
f ∗β (t′i ) = inf
g
{g :
1B
B∑b=1
1{
f ∗b (t′i ) ≤ g
}≥ β
}, β ∈ (0, 1). (45)
Analogously, bootstrap sampling distributions and variabil-ity intervals for observable parameters of the signal may bestudied by deriving a bootstrap parameter estimate fromeach trend filtering estimate within the bootstrap ensemble.For example, in Paper II we examine the bootstrap samplingdistributions of several observable light-curve parameters ofexoplanet transits and supernovae.
8 If nontrivial autocorrelation exists in the noise then a block
bootstrap (Kunsch 1989) will yield a better approximation of thetrend filtering variability than the bootstrap implementations we
Algorithm 1 Nonparametric bootstrap for random-input un-
certainty quantification
Require: Training Data (t1, f (t1)), . . . , (tn, f (tn)), hyperpa-rameters γ and k, prediction input grid t ′1, . . . , t
′m
1: for all b in 1 : B do2: Define a bootstrap sample of size n by resampling the
observed pairs with replacement:
(t∗1, f ∗b (t∗1)), . . . , (t
∗n, f ∗b (t
∗n))
3: Let f ∗b(t ′1), . . . , f ∗
b(t ′m) denote the trend filtering esti-
mate fit on the bootstrap sample and evaluated onthe prediction grid t ′1, . . . , t
′m
4: end forOutput: The full trend filtering bootstrap ensemble
{ f ∗b(t ′i )}i=1,...,m
b=1,...,B
Algorithm 2 Parametric bootstrap for fixed-input uncertaintyquantification (when noise distribution εi ∼ Qi is known a priori)
Require: Training Data (t1, f (t1)), . . . , (tn, f (tn)), hyperpa-rameters γ and k, assumed noise distribution εi ∼ Qi ,prediction input grid t ′1, . . . , t
′m
1: Compute the trend filtering point estimate at the ob-served inputs:
(t1, f0(t1)), . . . , (tn, f0(tn))
2: for all b in 1 : B do3: Define a bootstrap sample by sampling from the as-
sumed noise distribution:
f ∗b (ti) = f0(ti) + ε∗i where ε∗i ∼ Qi, i = 1, . . . , n
4: Let f ∗b(t ′1), . . . , f ∗
b(t ′m) denote the trend filtering esti-
mate fit on the bootstrap sample and evaluated onthe prediction grid t ′1, . . . , t
′m
5: end forOutput: The full trend filtering bootstrap ensemble
{ f ∗b(t ′i )}i=1,...,m
b=1,...,B
3.6.2 Bayesian
There is a well-studied connection between `1-penalizedleast-squares regression and a Bayesian framework (see, e.g.,Tibshirani 1996; Figueiredo 2003; Park & Casella 2008). Adiscussion specific to trend filtering can be found in Faulkner& Minin (2018).
3.7 Relaxed trend filtering
We are indebted to Ryan Tibshirani for a private conver-sation that motivated the discussion in this section. Trendfiltering can be generalized to allow for greater flexibilitythrough a technique that we call relaxed trend filtering9. Al-though the traditional trend filtering estimator is already
9 We choose this term because the generalization of trend filteringto relaxed trend filtering is analogous to the generalization of the
lasso (Tibshirani 1996) to the relaxed lasso (Meinshausen 2007).
Algorithm 3 Wild bootstrap for fixed-input uncertainty quan-
tification (when noise distribution is not known a priori)
Require: Training Data (t1, f (t1)), . . . , (tn, f (tn)), hyperpa-rameters γ and k, prediction input grid t ′1, . . . , t
′m
1: Compute the trend filtering point estimate at the ob-served inputs:
(t1, f0(t1)), . . . , (tn, f0(tn))
2: Let εi = f (ti) − f0(ti), i = 1, . . . , n denote the residuals3: for all i do4: Define a bootstrap sample by sampling from the fol-
lowing distribution:
f ∗b (ti) = f0(ti) + u∗i i = 1, . . . , n
where
u∗i =
{εi(1 +
√5)/2 with probability (1 +
√5)/(2
√5)
εi(1 −√
5)/2 with probability (√
5 − 1)/(2√
5)
5: Let f ∗b(t ′1), . . . , f ∗
b(t ′m) denote the trend filtering esti-
mate fit on the bootstrap sample and evaluated onthe prediction grid t ′1, . . . , t
′m
6: end forOutput: The full trend filtering bootstrap ensemble
{ f ∗b(t ′i )}i=1,...,m
b=1,...,B
highly flexible, there are certain settings in which the re-laxed trend filtering estimator provides nontrivial improve-ments. In our experience, these typically correspond to set-tings where the optimally-tuned trend filtering estimator se-lects very few knots. For example, we use relaxed trend fil-tering in Paper II to model the detrended, phase-folded lightcurve of a Kepler star with a planetary transit event.
The relaxed trend filtering estimate is defined througha two-stage sequential procedure in which the first stageamounts to computing the traditional trend filtering esti-mate discussed in Section 3.2. Recall the trend filtering min-imization problem (31). For any given order k ∈ {0, 1, 2, . . . }and hyperparameter γ > 0, let us amend our notation sothat
f TF0 (t) =
n∑j=1
β TFj hj (t) (46)
denotes the basis representation of the traditional trend fil-tering estimate. Further, define the index set
Kγ ={1 ≤ j ≤ n | β TF
j , 0}
(47)
that includes the indices of the non-zero falling factorial ba-sis coefficients for the given choice of γ. Now let β OLS
j,
j ∈ Kγ, denote the solution to the ordinary least-squares(OLS) minimization problem
min{β j }
n∑i=1
(f (ti) −
∑j∈Kγ
βjhj (ti))2
, (48)
and define the corresponding OLS estimate as
f OLS0 (t) =
∑j∈Kγ
β OLSj hj (t). (49)
MNRAS 000, 1–15 (2020)
14 C. A. Politsch et al.
That is, the OLS estimate (49) uses trend filtering to findthe knots in the piecewise polynomial, but then uses ordi-nary least-squares to estimate the reduced set of basis coeffi-cients. The relaxed trend filtering estimate is then defined asa weighted average of the traditional trend filtering estimateand the corresponding OLS estimate:
f RTF0 (t) = φ f TF
0 (t) + (1 − φ) f OLS0 (t), (50)
for some choice of relaxation hyperparameter φ ∈ [0, 1]. Re-laxed trend filtering is therefore a generalization of trendfiltering in the sense that the case φ = 1 returns the tradi-tional trend filtering estimate.
In principle, it is preferable to jointly optimize the trendfiltering hyperparameter γ and the relaxation hyperparam-eter φ, e.g. via cross validation. However, it often sufficesto choose γ and φ sequentially, which in turn adds minimalcomputational cost on top of the traditional trend filteringprocedure. Because of the trivial proximity of the fallingfactorial basis to the truncated power basis (established inTibshirani 2014 and Wang et al. 2014), it is sufficient to let
f OLS0 be the kth order regression spline with knots at the
input locations selected by the trend filtering estimator. Inheteroskedastic settings, as discussed in Section 3.3, a piece-wise polynomial or regression spline fit by weighted least-squares should be used in place of the OLS estimate (49).
4 CONCLUDING REMARKS
The analysis of one-dimensional data arising from signalspossessing varying degrees of smoothness is central to a widevariety of problems in time-domain astronomy and astro-nomical spectroscopy. Trend filtering is a modern statisti-cal tool that provides a unique combination of (1) statis-tical optimality for estimating signals with varying degreesof smoothness; (2) natural flexibility for handling practicalanalysis settings (general sampling designs, heteroskedasticnoise distributions, etc.); (3) practical computational speedthat scales to massive data sets; and (4) a single model hy-perparameter that can be chosen via automatic data-drivenmethods. Software for trend filtering is freely available on-line across various platforms and we provide links to ourrecommendations in Table 3.4. Additionally, we make sup-plementary R code available on the corresponding author’sGitHub page10 for: (1) selecting the trend filtering hyper-parameter by minimizing Stein’s unbiased risk estimate (seeSection 3.5); and (2) various bootstrap methods for trendfiltering uncertainty quantification (see Section 3.6).
ACKNOWLEDGEMENTS
We gratefully thank Ryan Tibshirani for his inspiration andgenerous feedback on this topic. This work was partiallysupported by NASA ATP grant NNX17AK56G, NASA ATPgrant 80NSSC18K1015, and NSF grant AST1615940.
