How to Make the Most Out of Evolving Information: Function Identification Using Epi-Splines Johannes O. Royset Operations Research, NPS in collaboration with L. Bonfiglio, S. Buttrey, S. Brizzolara, J. Carbaugh, I.Rios, D. Singham, K. Spurkel, G. Vernengo, J-P. Watson, R. Wets, D. Woodruff Smart Energy and Stochastic Optimization, ENPC ParisTech June 2015 This material is based upon work supported in part by the U.S. Army Research Laboratory and the U.S. Army Research Office under grants 00101-80683, W911NF-10-1-0246 and W911NF-12-1-0273 as well as by DARPA under HR0011-14-1-0060 1 / 126
Transcript
How to Make the Most Out of Evolving
Information:
Function Identification Using Epi-Splines
Johannes O. Royset
Operations Research, NPSin collaboration with L. Bonfiglio, S. Buttrey, S. Brizzolara, J.Carbaugh, I.Rios, D. Singham, K. Spurkel, G. Vernengo, J-P.
Watson, R. Wets, D. Woodruff
Smart Energy and Stochastic Optimization, ENPC ParisTechJune 2015
This material is based upon work supported in part by the U.S. ArmyResearch Laboratory and the U.S. Army Research Office under grants00101-80683, W911NF-10-1-0246 and W911NF-12-1-0273 as well as
by DARPA under HR0011-14-1-00601 / 126
Goal: estimate, predict, forecast
2 / 126
Copper prices
−10 −5 0 5 10
0.45
0.5
0.55
0.6
0.65
0.7
0.75
0.8
Months
Pric
e[U
SD
]
3 / 126
Copper prices
−10 −5 0 5 10
0.45
0.5
0.55
0.6
0.65
0.7
0.75
0.8
Months
Pric
e[U
SD
]
Real prices
E(hist+market) + σ
E(hist+market) − σ
E(hist+market)
4 / 126
Hourly electricity loads
5 / 126
Hard information: data
◮ Observations
◮ Usually scarce, excessive, corrupted, uncertain
Tuesday, October 22, 13
6 / 126
Soft information
◮ Indirect observations, external information
◮ Knowledge about
◮ structures◮ established “laws”◮ physical restrictions◮ shapes, smoothness of curves
7 / 126
Evolving information
data acquisition, information growth
8 / 126
Function Identification Problem
Identify a function that
best represents all available information
9 / 126
Applications of function identification approach
◮ energy
◮ natural resources
◮ financial markets
◮ image reconstruction
◮ uncertain quantification
◮ variograms
◮ nonparametric regression
◮ deconvolution
◮ density estimation
10 / 126
Outline
◮ Illustrations and formulations (slides 12-33)
◮ Framework (slides 34-60)
◮ Epi-splines and approximations (slides 61-77)
◮ Implementations (slides 78-85)
◮ Examples (slides 86-124)
11 / 126
Illustrations and formulations
12 / 126
Identifying financial curvesEstimate discount factor curve given instruments i = 1, 2, ..., I and
payments pi0, pi1, ..., p
iNi
at times 0 = t i0, ti1, ..., t
iNi
13 / 126
Identifying financial curvesEstimate discount factor curve given instruments i = 1, 2, ..., I and
payments pi0, pi1, ..., p
iNi
at times 0 = t i0, ti1, ..., t
iNi
Identify nonnegative, nonincreasing function f with f (0) = 1
Ni∑
j=0
f (t ij )pij ≈ 0 for all i
13 / 126
Identifying financial curvesEstimate discount factor curve given instruments i = 1, 2, ..., I and
payments pi0, pi1, ..., p
iNi
at times 0 = t i0, ti1, ..., t
iNi
Identify nonnegative, nonincreasing function f with f (0) = 1
Piecewise polynomial not automatic; must be constructed
s(x)
m1 m2 m3
x
mN-1m4 m5 m6…
Optimization over new class of piecewise polynomial functions
62 / 126
n-dim epi-splines
63 / 126
n-dim epi-splines (cont.)
Epi-spline s of order p defined on IRn with partition R = {Rk}Nk=1
is a real-valued function that
on each Rk , k = 1, ...,N, is polynomial of total degree p and
for every x ∈ IRn, has s(x) = lim infx ′→x
s(x ′)
64 / 126
n-dim epi-splines (cont.)
Epi-spline s of order p defined on IRn with partition R = {Rk}Nk=1
is a real-valued function that
on each Rk , k = 1, ...,N, is polynomial of total degree p and
for every x ∈ IRn, has s(x) = lim infx ′→x
s(x ′)
e-splpn(R) = epi-splines of order p on IRn with partition R
64 / 126
Evolving and approximating problems
minψν(s)
such that s ∈ Sν = F ν ∩ Sν
65 / 126
Evolving and approximating problems
minψν(s)
such that s ∈ Sν = F ν ∩ Sν
◮ subspace of interest Sν ⊆ e-splpν
n (Rν) ∩ F
65 / 126
Evolving and approximating problems
minψν(s)
such that s ∈ Sν = F ν ∩ Sν
◮ subspace of interest Sν ⊆ e-splpν
n (Rν) ∩ F
◮ evolving/approximate feasible set F ν
65 / 126
Evolving and approximating problems
minψν(s)
such that s ∈ Sν = F ν ∩ Sν
◮ subspace of interest Sν ⊆ e-splpν
n (Rν) ∩ F
◮ evolving/approximate feasible set F ν
◮ evolving/approximate criterion ψν
65 / 126
Function identification framework
Evolving Problems - information growth
- criterion and constraint approx.
- function approximations
Actual Problem - complete information
- infinite dimensional
- conceptual
solutions = epi-splines solutions
consistency, rates
66 / 126
Convergence of evolving problems
Epi-convergence of functionals on a metric space:
67 / 126
Convergence of evolving problems
Epi-convergence of functionals on a metric space:
Definition: {ψν : Sν → IR}∞ν=1 epi-converge to ψ : F → IR if andonly if
(i) for every sν → f ∈ lsc-fcns(IRn), with sν ∈ Sν , we havelim inf ψν(sν) ≥ ψ(f ) if f ∈ F and ψν(sν) → ∞ otherwise;
(ii) for every f ∈ F , there exists {sν}∞ν=1, with sν ∈ Sν , suchthat sν → f and lim supψν(sν) ≤ ψ(f ).
67 / 126
Convergence of evolving problems
Epi-convergence of functionals on a metric space:
Definition: {ψν : Sν → IR}∞ν=1 epi-converge to ψ : F → IR if andonly if
(i) for every sν → f ∈ lsc-fcns(IRn), with sν ∈ Sν , we havelim inf ψν(sν) ≥ ψ(f ) if f ∈ F and ψν(sν) → ∞ otherwise;
(ii) for every f ∈ F , there exists {sν}∞ν=1, with sν ∈ Sν , suchthat sν → f and lim supψν(sν) ≤ ψ(f ).
Equivalent to previous definition
67 / 126
Convergence of evolving problems
Epi-convergence of functionals on a metric space:
Definition: {ψν : Sν → IR}∞ν=1 epi-converge to ψ : F → IR if andonly if
(i) for every sν → f ∈ lsc-fcns(IRn), with sν ∈ Sν , we havelim inf ψν(sν) ≥ ψ(f ) if f ∈ F and ψν(sν) → ∞ otherwise;
(ii) for every f ∈ F , there exists {sν}∞ν=1, with sν ∈ Sν , suchthat sν → f and lim supψν(sν) ≤ ψ(f ).
Equivalent to previous definition
Also say that the evolving optimization problems epi-converge tothe actual problem.
67 / 126
Consequence of epi-convergence
If evolving problems epi-converge to the actual problem, then
lim sup
(
infs∈Sν
ψν(s)
)
≤ inff ∈F
ψ(f ).
68 / 126
Consequence of epi-convergence
If evolving problems epi-converge to the actual problem, then
lim sup
(
infs∈Sν
ψν(s)
)
≤ inff ∈F
ψ(f ).
Moreover, if sk are optimal for the evolving problemsmins∈Sνk ψνk (s) and sk → f 0, then f 0 is optimal for the actualproblem and
limk→∞
infs∈Sνk
ψνk (s) = inff ∈F
ψ(f ) = ψ(f 0).
68 / 126
When will evolving problems epi-converge to actual?
minψ(f ) minψν(s)
such that f ∈ F ⊆ F ⊆ lsc-fcns(IRn) such that s ∈ Sν = F ν ∩ Sν
Theorem: A sufficient condition for epi-convergence is
◮ ψν converges continuously to ψ relative to F
◮ Epi-splines dense in lsc functions
◮ F ν set-converges to F
◮ F is solid
69 / 126
Epi-splines dense in lsc functions
{Rν}∞ν=1, with Rν = {Rν1 , ...,R
νNν}, is an infinite refinement if
for every x ∈ IRn and ǫ > 0, there exists a positive integer ν̄ s.t.
Rνk ⊂ IB(x , ǫ) for every ν ≥ ν̄ and k satisfying x ∈ clRν
k .
70 / 126
Epi-splines dense in lsc functions
{Rν}∞ν=1, with Rν = {Rν1 , ...,R
νNν}, is an infinite refinement if
for every x ∈ IRn and ǫ > 0, there exists a positive integer ν̄ s.t.
Rνk ⊂ IB(x , ǫ) for every ν ≥ ν̄ and k satisfying x ∈ clRν
k .
Theorem: For any p = 0, 1, 2, ..., and infinite refinement {Rν}∞ν=1,
∞⋃
ν=1
e-splpn(Rν) is dense in lsc-fcns(IRn)
70 / 126
Dense in continuous functions under simplex partitioning
Definition: A simplex S in IRn is the convex hull of n + 1 pointsx0, x1, ..., xn ∈ IRn, with x1 − x0, x2 − x0, ..., xn − x0 linearlyindependent.
71 / 126
Dense in continuous functions under simplex partitioning
Definition: A simplex S in IRn is the convex hull of n + 1 pointsx0, x1, ..., xn ∈ IRn, with x1 − x0, x2 − x0, ..., xn − x0 linearlyindependent.
Definition: A partition R1,R2, ...,RN of IRn is a simplex partition ofIRn if clR1, ..., clRN are “mostly” simplexes.
71 / 126
Dense in cnts fcns under simplex partitioning (cont.)
Theorem: For any p = 1, 2, .., and {Rν}∞ν=1, an infinite refinementof IRn consisting of simplex partitions of IRn,
∞⋃
ν=1
e-splpn(Rν) ∩ C0(IRn) is dense in C0(IRn).
72 / 126
Rates in univariate probability density estimation
For convex and finite-dimensional estimation problem:
Theorem: For any “correct” soft information,
ν1/2dKL(h0||hνǫ ) = Op(1) for some hνǫ = near-optimal solution
73 / 126
Decomposition
Theorem: For every s ∈ e-splpn(R), with n > p ≥ 1 andR = {Rk}
Nk=1, there exist qk,i ∈ polyp(IRn−1), i = 1, 2, ..., n and
k = 1, 2, ...,N, such that
s(x) =n
∑
i=1
qk,i (x−i), for all x ∈ Rk .
an n-dim epi-spline is the sum of n, (n − 1)-dim epi-splines
74 / 126
Take-aways in this section
◮ Epi-splines are piecewise polynomials defined on arbitrarypartition of IRn
◮ Only lower-semicontinuity required (not smoothness)
◮ Dense in space of extended real-valued lower-semicontinuousfunctions under epi-distance (i.e., epi-splines can approximateevery lsc function to an arbitrary accuracy)
◮ Solutions of evolving problem (in terms of epi-splines) areapproximate solutions of actual problem
75 / 126
Questions?
76 / 126
Outline
◮ Illustrations and formulations (slides 12-33)
◮ Framework (slides 34-60)
◮ Epi-splines and approximations (slides 61-77)
◮ Implementations (slides 78-85)
◮ Examples (slides 86-124)
77 / 126
Implementations
78 / 126
Implementation considerations
◮ Selection of epi-spline order and composition h = exp(−s)
◮ Selection of partition: go fine!
◮ Implementation of criterion functional and soft info
79 / 126
Implementation considerations
◮ Selection of epi-spline order and composition h = exp(−s)
◮ Selection of partition: go fine!
◮ Implementation of criterion functional and soft info
Provide details for one-dimensional epi-splines of order 1
Focusing on criteria and constraints for probability densities
79 / 126
Implementation of criteria functionals
Probability densities h ∈ e-spl1(m), with meshm = {m0,m1, ...,mN},
−∞ < m0 < m1 < ... < mN <∞
80 / 126
Implementation of criteria functionals
Probability densities h ∈ e-spl1(m), with meshm = {m0,m1, ...,mN},
−∞ < m0 < m1 < ... < mN <∞
First-order epi-splines (piecewise linear):h(x) = ak0 + akx for x ∈ (mk−1,mk)
80 / 126
Implementation of criteria functionals (cont.)
Log-likelihood for observations x1, ..., xν :
log
ν∏
i=1
h(x i) =
ν∑
i=1
log(aki0 +aki x i ), where ki such that x i ∈ (mki−1,mki )
81 / 126
Implementation of criteria functionals (cont.)
Log-likelihood for observations x1, ..., xν :
log
ν∏
i=1
h(x i) =
ν∑
i=1
log(aki0 +aki x i ), where ki such that x i ∈ (mki−1,mki )
Entropy:
−
∫
h(x) log h(x)dx = −N∑
k=1
∫ mk
mk−1
(ak0 + akx) log(ak0 + akx)
81 / 126
Implementation of criteria functionals (cont.)
Log-likelihood for observations x1, ..., xν :
log
ν∏
i=1
h(x i) =
ν∑
i=1
log(aki0 +aki x i ), where ki such that x i ∈ (mki−1,mki )
Entropy:
−
∫
h(x) log h(x)dx = −N∑
k=1
∫ mk
mk−1
(ak0 + akx) log(ak0 + akx)
Both concave in epi-parameters ak0 and ak , k = 1, ...,N
81 / 126
Soft information
Nonnegativity: h ≥ 0 if ak0 + akmk−1 ≥ 0 and ak0 + akmk ≥ 0,k = 1, ...,N
82 / 126
Soft information
Nonnegativity: h ≥ 0 if ak0 + akmk−1 ≥ 0 and ak0 + akmk ≥ 0,k = 1, ...,N
Integrate to one:
∫
h(x)dx =N∑
k=1
ak0(mk −mk−1) +N∑
k=1
ak
2(m2
k −m2k−1) = 1
82 / 126
Soft information
Nonnegativity: h ≥ 0 if ak0 + akmk−1 ≥ 0 and ak0 + akmk ≥ 0,k = 1, ...,N
Integrate to one:
∫
h(x)dx =N∑
k=1
ak0(mk −mk−1) +N∑
k=1
ak
2(m2
k −m2k−1) = 1
Continuity: ak0 + akmk = ak+10 + ak+1mk for k = 1, ...,N − 1.
82 / 126
Soft information
Nonnegativity: h ≥ 0 if ak0 + akmk−1 ≥ 0 and ak0 + akmk ≥ 0,k = 1, ...,N
Integrate to one:
∫
h(x)dx =N∑
k=1
ak0(mk −mk−1) +N∑
k=1
ak
2(m2
k −m2k−1) = 1
Continuity: ak0 + akmk = ak+10 + ak+1mk for k = 1, ...,N − 1.
Log-concavity, convexity, monotonicity
82 / 126
Soft information (cont.)
Deconvolution Y = X +W : Given densities hW and hY ,
hY (y) =
∫
h(x)hW (y − x)dx =
N∑
k=1
ak0
∫ mk
mk−1
hW (y − x)dx
+
N∑
k=1
ak∫ mk
mk−1
xhW (y − x)dx for all y
83 / 126
Soft information (cont.)
Deconvolution Y = X +W : Given densities hW and hY ,
hY (y) =
∫
h(x)hW (y − x)dx =
N∑
k=1
ak0
∫ mk
mk−1
hW (y − x)dx
+
N∑
k=1
ak∫ mk
mk−1
xhW (y − x)dx for all y
Inverse problem Y = g(X ): Given vq =∫
yqhY (y)dy ,q = 1, 2, ...
vq =
∫
[g(x)]qh(x)dx ≈
N∑
k=1
Mk∑
j=1
w jk [g(x jk)]q(ak0 + akx jk)
83 / 126
Soft information (cont.)
Deconvolution Y = X +W : Given densities hW and hY ,
hY (y) =
∫
h(x)hW (y − x)dx =
N∑
k=1
ak0
∫ mk
mk−1
hW (y − x)dx
+
N∑
k=1
ak∫ mk
mk−1
xhW (y − x)dx for all y
Inverse problem Y = g(X ): Given vq =∫
yqhY (y)dy ,q = 1, 2, ...
vq =
∫
[g(x)]qh(x)dx ≈
N∑
k=1
Mk∑
j=1
w jk [g(x jk)]q(ak0 + akx jk)
Convex optimization problems
83 / 126
Software, references
◮ Papers and tutorials: http://faculty.nps.edu/joroyset
◮ Software for univariate probability density estimation
◮ Matlab toolbox http://faculty.nps.edu/joroyset/XSPL.html◮ R toolbox (S. Buttrey)
Earthquake losses, Vancouver Region*Comprehensive damage model (284 random variables)
100,000 simulations of 50-year loss
Figure 4-1: Vancouver metropolitan region.
*Data from Mahsuli, 2012; see also Mahsuli & Haukaas, 2013
94 / 126
Earthquake losses (cont.)Probability density of 50-year loss (billion CAD)
0 2 4 6 8 10 12 14 16 18 200
0.05
0.1
0.15
0.2
0.25
0.3
loss
dens
ity
95 / 126
Use fewer simulations?Using 100 simulations only and kernel estimator
96 / 126
Use fewer simulations?Using 100 simulations only and kernel estimator
0 2 4 6 8 10 12 14 16 18 200
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
loss
dens
ity
96 / 126
Using function identification approachMax likelihood of 100 simulations; second-order exp. epi-splinesEng. knowledge: nonincreasing, smooth, nonnegative support
97 / 126
Using function identification approachMax likelihood of 100 simulations; second-order exp. epi-splinesEng. knowledge: nonincreasing, smooth, nonnegative support
0 2 4 6 8 10 12 14 16 18 200
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
loss
dens
ity
100000 sample100 sample100 kernel
97 / 126
Using function identification approach (cont.)
Varying number of simulations; same soft information
![Continuous-time dynamics Bay Area Optimization Meetingfaculty.nps.edu/joroyset/docs/bayopt2018/Continuous... · 2014: Continuous-time limit of Nesterov’s method [1] Proved convergence](https://static.documents.pub/doc/80x56/5fb1536cb9183b0d97326152/continuous-time-dynamics-bay-area-optimization-2014-continuous-time-limit-of-nesterovas.jpg)
![Evolving Hash Functions using Genetic Algorithmsajiips.com.au/papers/V4.1/V4N1.4 - Evolving Hash Functions using... · hash function called "PKP Hash" by Peter.K.Pearson [5] that](https://static.documents.pub/doc/80x56/5e3486a76e7276290f0add90/evolving-hash-functions-using-genetic-evolving-hash-functions-using-hash.jpg)
