Chapter 1 Function Maximization Goals: 1. a few basic methods for function maximization 1.1 Function maximization Many statistical methods involve the maximization (or minimization) of a function of one or several variables. To begin, we consider maximizing a function of a single variable f (x) over an interval, say a<x<b or a ≤ x ≤ b. Maximization is often carried out by solving for x or xs that satisfy g (x)= f 0 (x)=0 assuming f (x) is differentiable. That is, we search for roots of the first derivative function g (x). I will discuss a few simple methods for function maximization, most of which require some smoothness on f (x) and possibly g (x).
Transcript
Chapter 1
Function Maximization
Goals:
1. a few basic methods for function maximization
1.1 Function maximization
Many statistical methods involve the maximization (or minimization) of
a function of one or several variables. To begin, we consider maximizing
a function of a single variable f (x) over an interval, say a < x < b or
a ≤ x ≤ b. Maximization is often carried out by solving for x or xs that
satisfy
g(x) = f ′(x) = 0
assuming f (x) is differentiable. That is, we search for roots of the first
derivative function g(x).
I will discuss a few simple methods for function maximization, most of
which require some smoothness on f (x) and possibly g(x).
2 Function Maximization
1.2 Direct maximization
Direct maximization is effective in a vector or matrix programming language.
It does not generalize well when f (x˜) is defined for x˜ ∈ Rp where p is larger
than 2, 3, or 4.
As an example, consider maximizing the function
f (x) =log(x)
1 + x, 0 ≤ x ≤ 5.
The basic idea is to finely divide the interval into a set of points on
which the function is evaluated. Then we find the element in the vector
of function values at which the function is maximized. Note that the
maximization is approximate. The error in the approximation is a function
of the coarseness of the grid.
# define function f(x)
f.f <- function(x) {log(x) / (1 + x)
}
# plot function
library(ggplot2)
p <- ggplot(data.frame(x = c(0.0001, 5)), aes(x))
p <- p + stat_function(fun = f.f)
p <- p + labs(title = "The function f(x) = log(x) / (1 + x)")
print(p)
1.3 Bisection (bracketing) 3
−7.5
−5.0
−2.5
0.0
0 1 2 3 4 5x
y
The function f(x) = log(x) / (1 + x)
Looking at the plot, we see that f (x) initially is increasing then slowly
decreases pass the point at which the maximum of f (x) occurs.
# a grid of x-values
x <- seq(0.0001, 5, by = 0.0001)
# evaluate the function over the grid
f.x <- f.f(x)
# determine the index of the maximum value
ind <- which(f.x == max(f.x))
# print the value of x and f(x) at the maximum
c(x[ind], f.x[ind])
## [1] 3.5911 0.2785
The max occurs at 3.5911 and the maximum value is 0.2785.
1.3 Bisection (bracketing)
This is the simplest, but slowest,, method to solve
g(x) = f ′(x) = 0.
4 Function Maximization
However, it is ”guaranteed to work” provided simple precautions are taken.
For simplicity, we assume g(x) is continuous. The idea is to find an
interval a ≤ x ≤ b on which g(x) is monotonic (either strictly increasing
or decreasing) and such that g(x) changes sign (that is, g(a)g(b) < 0).
This implies there is a unique root in this interval.
The basic idea of bisection is to sequentially halve the interval by
checking whether the root is to the left or right of the interval midpoint
and then modifying the interval appropriately. That is, if at
x0 = (a + b)/2
we have
g(a)g(x0) > 0 ⇒ g(x) has same sign at a and x0
⇒ root is to the right of x0
⇒ redefine a = x0
else if
g(a)g(x0) < 0 ⇒ g(x) changes sign between a and x0
⇒ root is to the left of x0
⇒ redefine b = x0.
The process iterates until b− a ≤ ε (a user-specified small value).
Remarks
1. By construction, if g(a)g(x0) = 0, then we know that x0 is the
root. One could build this into the routine but because of machine
roundoff it is not likely that the machine representation of g(a)g(x0)
will give you zero exactly, so the extra coding probably does not pay
off.
1.3 Bisection (bracketing) 5
2. Bisection is relatively slow because it ignores information about how
quickly g(x) changes over [a, b], that is, it does not use information
on derivatives of g(x).
3. If we let [ai, bi] be the search interval at the ith step with [a0, b0] as
the initial interval, then
bi − ai = 2−i(b0 − a0)
Given the user defined ε, we have
bi − ai = 2−i(b0 − a0) < ε
⇔ −i + log2(b0 − a0) < log2(ε)
⇔ i > log2
(b0 − a0ε
).
That is, we need approximately that
log2
(b0 − a0ε
)steps for convergence. Reducing ε by a factor of 10 (that is, adding
an additional decimal place of precision) requires an additional
log2(10) = 3.3.= 4
iterations.
4. The method is guaranteed to converge to within ε (or 0.5ε) of the
root, provided g(x) is continuous.
6 Function Maximization
Example: Quantiles of a rv X Suppose a univariate rv X has a
cumulative distribution function
F (t) = Pr(X ≤ t), (1.1)
where, for simplicity, we assume F (t) is continuous and strictly increasing.
Given 0 < α < 1, the αth percentile of X is the unique value xα so that
F (xα) = α
F (xα)− α = 0.
−3 −2 −1 0 1 2 3 4
0.0
0.2
0.4
0.6
0.8
1.0
CDF of Normal(0,1)
x
F(x
)
x.alpha = 1.64485362695147
alpha = 0.95
Given F (t), it is relatively straightforward to use bisection to compute
xα. R has routines to compute quantiles for many standard distributions,
typically using more complex approximations based on rational function
expansions and the like. The following function illustrates a bisection
evaluation of percentiles for a standard normal distribution.
1.3 Bisection (bracketing) 7
## function bisect.qnorm - standard normal quantiles
# input: alpha = probability for desired quantile
# output: xa = desired quantile with max error eps=0.001
# provided alpha is not extreme
bisect.qnorm <- function(alpha, eps = 0.001, a = -5, b = 5, sw.more = 0) {# First some error catching
if(length(alpha) > 1) {xa <- NULL
warning("alpha must be a scalar.")
return(xa)
}if((alpha < 0) | (alpha > 1)) {warning("alpha out of bounds, specify 0 < alpha < 1")
xa <- NULL
return(xa)
}if(alpha <= pnorm(a)) {warning("alpha specified less than lower bound, pnorm(a)")
xa <- NULL
return(xa)
}if(alpha >= pnorm(b)) {
warning("alpha specified greater than upper bound, pnorm(b)")
xa <- NULL
return(xa)
}if(alpha == 0.5) { # what happens if we don't have this?
xa <- 0
return(xa)
}
if (sw.more != 1) { # don't provide additional output
while ((b - a) > eps) {x0 <- a + (b - a) / 2
if ((pnorm(x0) - alpha) < 0) {a <- x0
} else {b <- x0
}}xa <- a + (b - a) / 2
return(xa)
}if (sw.more == 1) { # provide additional output for creating plot later
