Lecture Notes in Dynamic Optimization

Jorge A. BarroDepartment of EconomicsLouisiana State University

December 5, 2012

1

1 Introduction

While a large number of questions in economics can be answered by considering only the static

decisions of individuals, firms, and other economic agents, this set is by no means exhaustive. We

know that in the real world, a trip to the produce market involves a decision between how many

apples and oranges to buy today and how many apples you may want to buy tomorrow. More

generally, consumption decisions of individuals are a tradeoff between a bundle of goods today

and a bundle of goods at some time in the future. These intertemporal consumption decisions are

linked by a savings decision. Firms make intertemporal decisions as well. Consider, for example, the

decision of a CEO choosing the optimal dividend paid to stockholders. If these stockholders are paid

an excessively high dividend, the company could forego an opportunity to invest in capital or hire

workers that could increase the company’s future profits. Suppose now the firm is deciding whether

to hire new employees. The process consists of taking the time, effort, and financial resources

to advertise the opening, interview potential employees, and train the incumbent. Given these

large costs associated with adjusting the labor force, most firms would benefit from accounting for

employees’ productivity over some time horizon. You should think of several examples of these

dynamic decisions in the context of your own field.

The notes that follow are partly my own and partly from a number of resources, including Dy-

namic Economics by Jerome Adda and Russell Cooper (2003),1 Recursive Methods in Economic

Dynamics by Nancy Stokey, Robert Lucas, and Edward Prescott (1989),2 Recursive Macroeco-

nomic Theory by Thomas Sargent and Lars Ljungqvist (2004),3 and of course A First Course in

Optimization Theory by Rangarajan Sundaram.4

1Easiest.2Quite challenging.3A little bit harder than Adda and Cooper, but tons of applications.4A bit more difficult than Sargent and Ljungqvist. It’s concise, but still more than we wish to know at this point.

2

2 Dynamic Optimization

We’ll focus on the problem of maximizing a function f : D → R, where the dimension of D

possibly represents different time periods. For example, suppose you were considering the problem

of maximizing your utility over your lifetime. Just to keep things simple, assume you live for T

periods with certainty. Further, suppose that c ∈ RT is a vector of consumption bundles throughout

your lifetime. In other words, c1 represents the amount that you consume in the first period of

your life, c2 represents the amount that you consume in the second period of your life, and so on.

Finally, let B ⊆ RT+ be some convex set representing a financially feasible budget set of allocations

throughout your life. Then if U : RT → R represents your lifetime utility function, your utility

maximization problem can be stated as:

maxcU(c) subject to c ∈ B (1)

We know that if the function u satisfies certain properties, such as strict concavity, then any

solution to this problem will be unique. We could approach this problem in the traditional way.

That is, we could simply set up a Lagrangean function and take the usual first-order conditions - a

method we will call the method of direct attack. This is potentially a simple problem if T is small.

However, human beings tend to live long lives.5 Moreover, firms, such as the chemical company

DuPont have been operating in excess of two centuries. For this reason, we study an alternative

approach - the method of dynamic programming.

3 The Objective Function

This introduction to dynamic programming is example-driven in the sense that we consider the

utility-maximization problem. We can (and will) consider optimization problems with objective

functions that are more general payoff or return functions. For example, in an application within

Industrial Organization, the objective function will represent profits to a firm.

5Life is long purely in the mathematical optimization sense. Indeed, life is short, and we should enjoy everymoment of it. This point further motivates the use of more efficient optimization methods.

3

Our specification of the utility function relies on two important observations. First of all, income

volatility exceeds consumption volatility. This implies that individuals prefer to save their income

in a way that smooths consumption over time.6 Second, individuals tend to discount future utility.

We accept these as preferences that are consistent with observed behavior, and specify our utility

function as follows:

U(c1, . . . , cT ) =

T∑t=1

βt−1u(ct) (2)

where u(·) is a strictly increasing and strictly concave function, and β ∈ [0, 1) is the personal

discount factor. Also assume that limc→0 u′(c) = ∞ so that marginal utility becomes infinite as

consumption approaches zero. This will simplify the problem in ways that we will discuss later.

While the discounting feature of this utility function is immediately obvious, the consumption-

smoothing feature of the utility function perhaps is not. In order to understand the latter, suppose

that u(c) were linear. Then an individual would only be interested in maximizing the sum of lifetime

consumption, and any intertemporal consumption reallocation would not make the individual better

or worse off. By contrast, now assume the individual had strictly concave utility. Since the marginal

utility of consumption low when consumption in one period is high, the individual would likely be

better off by decreasing consumption in that period and increasing consumption in a period where

consumption was relatively low. This process of “transferring” consumption from one period to

another is called intertemporal substitution.

4 Simple Two-Period Optimization

In this section, we will consider the problem of maximizing utility over two periods, subject to

intertemporal budget constraints. Assume that we are endowed with a lump-sum of wealth a1 in

period 1.7 We can spend some and save the remainder a2 for period 2. For each unit, a, that we

save in a period, we get Ra in the following period, where R ≥ 1 is the gross interest rate. We state

6This motivated the permanent income theory of consumption developed by Irving Fisher and Milton Friedman.7Economists often reference this as the “cake-eating” problem.

4

the problem as follows:

maxc1,c2,a2,a3

u(c1) + βu(c2) (3)

subject to c1 + a2 = a1, and c2 + a3 = Ra2

While it may seem sub-optimal to save a positive amount in the final period of life (and indeed it is

sub-optimal), nothing currently states that an individual can’t borrow in the final period of life and

die with large amounts of debt. Perhaps we should add the following non-negativity constraint to

the optimization problem to ensure that the individual doesn’t run a Ponzi-scheme: a3 ≥ 0. Notice

that we are doing more work than is necessary. We could simplify this problem by eliminating

a2 from the optimization problem and combining the two per-period budget constraints to get a

lifetime budget constraint:

c1 +1

R(c2 + a3) = a1 (4)

Then we would get first-order conditions (differentiating the Lagrangean function with respect to

c1, c2, and a3, respectively):

u′(c1) = λ (5)

Rβu′(c2) = λ (6)

λ = φR (7)

and complementary slackness conditions:

a3 ≥ 0, φ ≥ 0, and φa3 = 0. (8)

where λ is the multiplier on the equality constraint and φ is the multiplier on the inequality

constraint. Let {c∗1, c∗2, a∗2, a∗3} denote the optimal solution to the optimization problem. Equations

(5) and (6) give the following intertemporal relationship:

u′(c∗1) = Rβu′(c∗2). (9)

5

Equation (9) is called the Euler equation, and it plays a fundamental role in macroeconomics and

finance.8 We’ll have more to say about this equation in the generalized T-period problem. For now,

we focus on the last remaining issue: the multiplier on the non-negativity constraint. Recall that

the multiplier on the equality constraint is interpreted as the marginal utility from increasing initial

wealth. Since utility is always increasing in the consumption goods, and demand is increasing in

wealth, it must be the case that marginal utility from increasing wealth is positive. Then from (7),

we know that λ = φR and λ > 0 implies that a∗3 = 0 from the complementary slackness conditions.

Also, since marginal utility is infinite near ct = 0 for t = 1, 2, we know that optimal consumption

will be positive in each period. This means that we can ignore any non-negativity constraints on c1

and c2. Finally, we know that the optimal solutions {c∗1(a1, R), c∗2(a1, R), a∗2(a1, R), a∗3(a1, R)} will

each depend on initial wealth a1, the interest rate R, and nothing else! These optimal solutions, or

policy functions as we will call them, can be substituted back into the objective function to get the

indirect utility function: U (c∗1(a1, R), c∗2(a1, R)). From here on, this indirect utility function will be

called the value function, denoted V (a1, R), and it can be written as the solution to the following

problem:

V (a1, R) = maxc1,c2,a2,a3

u(c1) + βu(c2) (10)

subject to c1 + a2 = a1, and c2 + a3 = Ra1

Exercise 1: Solve for the policy functions and value function of the previous exercise using

u(c) = log(c).

5 The Three-period Problem

Suppose now, that instead of having to make the decision at t = 1, that we were to add a

period t = 0 and solve the 3-period utility maximization problem. After doing all that hard work,

it would be nice if we did not have to start from scratch. Think about it...where did the a1 in the

value function come from? In the real world, we know that our current stock of wealth generally

8We can estimate the parameters of this function by assuming this equation holds true in expectation. This mo-tivated the seminal work of Hansen and Singleton (1982) in their generalized method of moments (GMM) estimationof the structural parameters.

6

Figure 1: Intertemporal Optimization

7

depends on the savings decisions that we made in the past. So wouldn’t it make sense that a1 in

the 2-period problem would just be the result of our savings decision (a1) if we had considered the

problem that started at t = 0? As it turns out, the answer is YES! Fortunately for us, we already

solved the two-period problem for any value a1. We know that because we substituted the optimal

solution into the utility function for the two-period problem, then the value function V (a1, R) will

tell us how changes in initial wealth changes the utility after accounting for the optimal decision of

the individual. In other words, we don’t need to think about all the decisions that the individual

will make in the future; it suffices to know that the individual will act optimally in each subsequent

period. This is what is known as the principle of optimality. Then instead of solving the entire

problem again, we will simply solve the following problem for any initial wealth level a0:

V0(a0) = maxa1

u(c0) + βV1(Ra1) (11)

subject to c0 = a0 − a1,

where V0 and V1 are the value functions from the 3-period and 2-period problems, respectively, a0

is the new initial stock of wealth, and the parameter R has been dropped from the value function,

since we are not concerned with that right now. Also notice that we are only maximizing over

values of a1 since we could just substitute c0 = a0 − a1 into the objective function and solve the

unconstrained problem.

Exercise 2: Using the value function from the two-period problem in Exercise 1, solve for the

value function of the 3-period problem. Then solve the entire 3-period problem from scratch using

the method of direct attack, and show that this method leads to the same value function as our

“new” method.

6 The T-period and Infinite Problem

Hopefully, you can immediately see the pattern that develops from this method. In fact, we

could have solved the one-period problem at t=2, then solved the two-period problem (solved at

8

t=1) using the value function from the one period problem, then finally the three period problem

(at t=0) using the value function from the two-period problem. This concept of backwards induction

is at the heart of dynamic programming. We can solve any T-period problem through this recursive

process. In fact, given our assumptions, we can show that by choosing a very high value of T, we

can approximate the infinitely-lived individual problem. As it turns out, our process satisfies the

conditions for a contraction mapping, and the assumptions that we made earlier satisfy sufficient

conditions for convergence of our value function through this iterative process.

6.1 The T-period Problem

For any initial endowment of wealth, a0, the finite horizon problem can be stated as follows:

V0(a0) = max{at+1}Tt=0

T∑t=0

βtu(ct) (12)

subject to ct + at+1 = Rat for t = 0, . . . , T and aT+1 ≥ 0, (13)

where the first constraint is the intertemporal budget constraint and the second constraint is the

no-Ponzi-game condition. Notice that in any period t = 0, . . . , T the problem could be restated in

terms of the value functions in each period as follows:

Vt(at) = maxat+1

{u(ct) + βVt+1(at+1)} (14)

subject to ct + at+1 = Rat for t = 0, . . . , T and aT+1 ≥ 0. (15)

When we write the equation out like we did in (12), we call this the sequential statement of the

dynamic optimization problem. By contrast, when we translate it into dynamic programming

language, like we did in (14), we call this the recursive formulation of the problem. Further, (14)

is a functional equation, and generally called a Bellman equation after Richard Bellman - a pioneer

in dynamic optimization.

9

6.2 The Infinite-Horizon Problem

You might be wondering why we care about infinitely-lived problem if people don’t actually live

forever. First of all, we tend to find that the infinitely-lived problem can accurately approximate

the finitely-lived individual’s problem and simplify quantitative analysis - especially when we don’t

care about so-called life-cycle properties of decisions. The infinite-horizon problem also makes sense

when if you think about the potential role of altruism in the savings decisions. If we care about

our children and our savings decisions are at least partially motivated by an altruistic bequest

motive, then our decisions will be accurately measured by this dynastic model. This discussion has

focused on an individual’s decisions for motivating the infinite-horizon model, but as we claimed,

we could certainly think about the decision of a firm over a much longer time horizon. We take this

motivation and generalize the infinite-horizon problem in the context of the finite-horizon problem

in the following section.

7 Mathematical Foundations of Dynamic Optimization

This section takes the logic from the previous sections and generalizes it so that we can consider

the infinite-horizon dynamic optimization problem more formally. This infinite-horizon problem

consists of a state vector, st ∈ Ds ⊆ Rm, an action vector or control vector ct ∈ Dc ⊆ Rn, a

transition function g : Ds × Dc → Ds, and a one-period reward or momentary payoff function

ρ : Ds ×Dc → R. Lots to digest here.

First consider the state vector, st, which we interpret as the agent’s “environment,” which is

outside of the control of the agent at the beginning of period t. In the savings problem, this is

the amount of assets available to the individual at the beginning of period t. The state vector

is an element of the state space Ds, which determines all possible values of st. The state vector

contains all the information available at time t needed to make an optimal decision. Examples of

state spaces include an individual’s possible asset levels, education level, health level (and health

insurance status), family composition, and employment status, or firm’s labor force, capital stock,

and productivity.

10

Next consider the control vector, ct. The set of possible actions that the economic agent can

make is characterized in the control space Dc. Notice that this set of feasible actions generally

depends on the state vector, st, so we could write Dc(st). For example, the amount that you

consume today could be limited by the funds in your bank account. More generally, the set of

controls chosen by an individual in a period could include savings, consumption, labor supply,

medical expenditures, educational investment, job search intensity, or even accept or reject a job

offer.9

Now consider the transition function, st+1 = g(st, ct), which determines our state tomorrow as a

function of our current state and the actions we take today. In the savings problem, this was simply

the budget constraint. The amount that funds that wake up with tomorrow are just the difference

between the funds we wake up with today minus the amount that we consume.10 Transition

functions generally come in the form of resource constraints. Perhaps the more important feature

of the transition function is that it maps the power set Ds×Dc back into the state space Ds. This

is an important feature that, along with other properties, will qualify this dynamic optimization

problem as a contraction mapping, or a mapping from a set back onto itself.

Finally, consider the payoff function, ρ. We initially defined this function to map values of st

and ct into R. However, we could solve the transition function for ct, redefine the function so that

ct = g(st, st+1), and substitute this into the payoff function to get: ρ(st, st+1) ≡ ρ(st, ct). Similar

to the restriction that ct ∈ Dc(st), we restrict st+1 ∈ Γ(st) and define write the Bellman equation

as follows:

V (st) = maxst+1∈Γ(st)

ρ(st, st+1) + βV (st+1) ∀ st ∈ Ds. (16)

Notice that this optimization problem has the property of stationarity, i.e., it is time-independent.

This is, of course, still a dynamic optimization problem, but the specific time of the decision is

irrelevant. This means that we can get omit of all the time subscripts and instead use a prime to

9In this case, the set of possible offers would be the state space, and “accept” or “reject” would be the set ofchoices. This idea was originally proposed by McCall (1970) and serves as the foundation of the Nobel-prize-winninglabor market model of Christopher Pissarides and Dale Mortensen.

10Assuming no outside income.

11

denote next period values as follows:

V (s) = maxs′∈Γ(s)

ρ(s, s′) + βV (s′) ∀ s ∈ Ds. (17)

Now we are tasked with determining the value function, V (s) and the associated policy function

s∗(s) that solves (17). We have the following motivating theorem from Adda and Cooper p. 27:

Theorem 1. Suppose ρ(s, s′) is real valued (which we already assumed), continuous, and bounded,

β ∈ (0, 1), and the constraint set Γ(s) is non-empty, compact, and continuous. Then there exists a

unique value function V (s) that solves (17).

Understanding this involves introducing a new functional operator T onto any function W :

R+ → R that satisfies the following condition:

(TW )(s) = maxs′∈Γ(s)

ρ(s, s′) + βW (s′) ∀ s ∈ Ds. (18)

Actually, we have already used this “T” operator once already - when we solved the 3-period problem

using the value function from the 2-period problem! Let V1 be the value function from the 1-period

problem, V2 = TV1 be the value function from the 2-period problem, . . ., Vn+1 = TVn be the value

function from the (n+1)-period problem. Suppose we use the T operator to construct the sequence of

functions {Vn}. Then it can be shown that for any initial function V1,11 limn→∞ ‖Vn−V ∗‖sup = 0,

where V ∗ is the unique value function, which we know exists from Theorem 1. This result is

exactly the motivation behind value function iteration as a computational process.

8 Computational Algorithm: The Savings Problem

8.1 Preliminary Activity

This section presents a brief computational algorithm for solving the infinitely-horizon savings

problem. Notice from the previous section that value function iteration leads to uniform conver-

11In practice, if we don’t have a good initial guess of V1, we usually choose V1(s) = 0 for all s ∈ Ds

12

gence of a sequence of functions. However, we can not show uniform convergence of one function

onto another on a computer. Instead, we must solve for the function at discrete points, generally

determined from the discrete points of the state space. In this case the state space is the assets

brought into the period. Accordingly, we can discretize the state space by choosing an ordered grid

of assets, {a1, a2, . . . , ana}, where na is the number of grid points. Then we create two zero-vectors

of size na, which will be updated with Vn+1 and Vn, respectively, in each iteration.

8.2 Value Function Iteration

We begin by specifying some very small tolerance level that will determine the approximation of

the convergence of the sequence {Vn}. Then each iteration (of the “T” operator) uses the following

steps:

1. For each i ∈ 1, 2, . . . , na, solve Vn+1(ai) = maxa′ u(rai − a′) + βVn(a′).

2. Save both the value function and optimal asset choice in a vector.

3. If ‖Vn+1 − Vn‖ is less than the pre-specified tolerance, then the program has approximately

converged

4. If ‖Vn+1−Vn‖ is not less than the tolerance, update the value function vector so that Vn+1 =

Vn and repeat from Step 1.

Notice in Step 1 that the optimization routine must solve over Vn(a′), which is only specified

at a finite amount (na) points. No big deal - we can interpolate Vn using a few different methods.

Linear interpolation, for example, will simply connect the each point Vn(ai) with a straight line.

Spline interpolation, by contrast, will fit a polynomial over the points, leading to more accurate

approximations. Unfortunately, spline interpolation is also more time-consuming. As a rule of

thumb, I usually use linear interpolation until I feel comfortable with the quality of the code, then

apply spline interpolation.

13

9 A Variety of Dynamic Programming Examples

This section provides several examples of well-known dynamic programming problems. While

we have not discussed stochastic processes in the preceding notes, we can consider them in the

context of these examples. After all, many of the intertemporal economic decisions that we make

are affected by the uncertainty of future outcomes. The models in this section are partial equilibrium

in the sense that we only consider the optimal responses of economic agents for given market values,

such as prices. By contrast, general equilibrium theory completes the analysis by determining prices

endogenously. For now, we take market values as given and proceed.

9.1 Human Capital Accumulation

This subsection presents a simple model of Ben-Porath (1970) human capital accumulation.

This model has survived the test of time and serves as the foundation for models, such as Heckman,

Locher, and Taber (1998a, 1998b, 1998c) that explain wage inequality and study the effects of

labor income taxes on educational decisions. We return to that model later. Guvenen and Kuruscu

(2006) also used the model to explain trends in the wage distribution over the last thirty years.

Kuruscu (2006) later used the model to quantify the effectiveness of on-the-job training. The idea

behind the model is that low labor earnings early in life reflect a larger percentage of time training

and learning (i.e. building a stock of human capital) while higher earnings later in life reflect low

amounts of time spent developing human capital and more time spent using the human capital in

more productive ways.

Let ht be the stock of human capital that an individual has at age t. Let it be the amount

of time that the individual spends investing in human capital, and let w be the rental rate of

this human capital. Then, the individual’s labor earnings are wht(1 − it), and the problem of the

individual is to allocate time between investing in human capital (such as training or learning) and

more productive activities as follows:

V (ht) = maxit

u(ct) + βV (ht+1) (19)

14

s.t. ct = wht(1− it) (20)

and ht+1 = A (htit)α

+ (1− δ)ht, (21)

where A and α are parameters of the human capital production function, and δ is the human

capital depreciation rate. To notice the intertemporal tradeoff, think about how changes in it affect

each of the constraints. The first constraint (current period payoff), Equation (20) shows how

consumption in period t depends on the human capital stock and rises when the individual spends

less time training. The second constraint (future payoff) shows how future human capital rises

when the individual rises when the individual invests more time in training.

9.2 Unemployment and Incomplete Markets

This subsection introduces a popular model of idiosyncratic unemployment spells and partial

insurance, as in Huggett (1993), Aiyagari (1994), and Krusell and Smith (1998). Incomplete markets

exist when individuals can not fully insure against all future outcomes. The most prevalent example

of incomplete markets is unemployment. Because of moral hazard resulting from disincentives to

job search, government or private entities will generally not pay individuals the full amount of

their lost wages in the event of job loss. This leaves individuals to self-insure. We can measure

this level of self-insurance by comparing a complete market model to the following model in which

individuals can only buy a single asset that pays out the same amount regardless of the individual’s

unemployment status.

We also introduce random outcomes captured by what we call a Markov process. Two things

we need to know about Markov processes: they capture autocorrelation (outcomes that are cor-

related over time) and they can significantly simplify our problem. A Markov process will tell us

the probability of certain outcomes tomorrow, given a certain state today. For example, if we are

employed today, there is a good chance that we will be employed tomorrow. With a certain prob-

ability, however, we might randomly transition into unemployment status. We could then consider

probabilities of outcomes given that we are in the unemployed state. This set of probabilities can

be represented in a Markov matrix like the one shown in Figure 9.2.

15

Figure 2: Markov Matrix for Employment Status

16

Let pee denote the probability of remaining employed tomorrow given that we are employed

today (top left entry in the Markov matrix in Figure 9.2), and let puu denote the probability of

remaining unemployed tomorrow given that we are unemployed today (bottom right entry in the

Markov matrix in Figure 9.2). Given this very simple stochastic process, we can include employment

status as an element of the state space, and consider the optimal savings problem of the employed

individual as follows:

V e(at) = maxat+1

u(ct) + β [peeVe(kt+1) + (1− pee)V u(kt+1)] (22)

s.t. ct = rat − at+1 + w (23)

and for the unemployed individual as follows:

V u(at) = maxat+1

u(ct) + β [(1− puu)V e(at+1) + puuVu(at+1)] (24)

s.t. ct = rat − at+1 + b, (25)

where w is the wage of an employed individual, and b is the unemployment benefit given to an

unemployed individual. It can be shown that if w > b, then the individual will save to compensate

for this incomplete insurance. In fact, at the beginning of recessions when the probability of

unemployment increases, we tend to notice spikes in individuals’ saving rates. This can be seen in

Figure 9.2, which is the historical U.S. savings rate obtained from the Federal Reserve Bank of St.

Louis data set (FRED, as it is commonly referred).

9.3 Health Shocks and Medical Expenditures

In this subsection, we present a model of a health shock and medical expenditures that is largely

credited to Michael Grossman (1972). It should be noted that the parameters and specification of

this model remain largely untested (and perhaps a good idea for future research!). Let ht denote

the stock of health of an individual at time t, and suppose that it depreciates at an accelerated rate

17

Figure 3: U.S. Savings Rate (FRED-generated graph)

18

(δt) over an individual’s lifetime. Also assume that an individual’s probability of survival depends

on the stock of health through some probability function s(ht+1), which satisfies s′(·) > 0 and

s′′(·) < 0. Further, suppose that individual’s have preferences over consumption and the quality

of their health. Finally, assume that individuals have random health shocks ε, and they purchase

medical expenditures mt that increase their stock of health. Then we can write the individual’s

optimization problem as follows:

V (at, ht, εt) = maxmt,at+1

u(ct, ht) + s(ht+1)βE [V (at+1, ht+1, εt+1)] (26)

s.t. ct = w + rat − at+1 −mt (27)

and ht+1 = (1− δt)ht + f(mt)− εt, (28)

where E in the Bellman equation is the expectation operator, w is labor income, and f(·) is a

function that determines the effectiveness of medical purchases onto health level. The original

Grossman model also included time allocation as a determinant of health quality. This is an

important consideration that can account for non-medical inputs (exercise, for example) into the

health production function.

9.4 Wage Offers and Discrete Choice Modelling

This subsection considers the simple “accept” or “reject” decision of an individual that receives

random wage offers, which is the model presented by McCall (1970). Assume that in each period,

the individual receives a random wage offer w, which arrives from some wage distribution. If she

accepts the offer, she remains employed and receives wage w forever. In this case, the payoff is

u(w)1−β . To see this, consider the geometric series u(w)

∞∑t=0

. Rejecting the offer results in receiving one

more period of the unemployment benefit b plus the discounted expected future value of the next

wage offer, w′, resulting in the following discrete-choice model:

V (w) = max

{u(w)

1− β, u(b) + βEV (w′)

}(29)

19

You might be wondering, what is the threshold wage such that an individual will accept anything

at or above the amount, and reject anything below that amount. In fact, this reservation wage is

the wage w∗ that solves the following equation:

u(w∗)

1− β= u(b) + βEV (w′), (30)

which depends on the distribution of the wage offers.

9.5 Lucas Tree Asset Pricing and Portfolio Allocation Model

Now we consider fundamental problem in financial economics - the pricing of an asset. Robert

Lucas (1978) compared this simple model to the ownership of a fruit-bearing tree. Each period,

the tree (company) produces a random quantity of fruit (dividend), dt. The individual enters each

period with some part (stock), st, of the tree and must choose the amount st+1 to purchase for next

period. Shares of the tree can be purchased or sold at price pt. Also, suppose there is a risk-free

asset at which earns return r between periods. Then the individual must allocate their portfolio

at+1 and st+1 to maximize expected utility as follows:

V (at, st) = maxat+1,st+1

u(ct) + βE [V (at+1, st+1)] (31)

s.t. ct = w + pt(st − st+1) + dtst + rat − at+1 (32)

This constraint can be modified and reinterpreted in a number of ways to address a variety of

questions in financial economics. The key here is that in equilibrium, the net supply of the risk-free

asset (or conversely, debt) in the economy must be zero. In contrast, shares of the tree (company)

must be in positive net supply, and the price times the outstanding shares of the tree determine

the value of the tree.

20

10 Conclusion

This concludes the introduction to dynamic optimization. Hopefully, you’ve learned how to

solve a dynamic optimization problem using computational techniques. By now, you should be

well-equipped to solve and simulate a variety of problems in economics using these numerical ap-

proximation methods. Always remember that there is no substitute for patience and optimism when

you’re doing computational work. Also, remember to check out economist Tony Smith’s guidelines

and recommendations for disciplined computational analysis.

11 References

References

[1] Adda, Jerome, and Russell Cooper, 2003. Dynamic Economics: Quantitative Methods and

Applications. Cambridge: MIT Press.

[2] Aiyagari, S. Rao, 1994. Uninsured Idiosyncratic Risk and Aggregate Saving. Quarterly Journal

of Economics 109 (3), 659-684

[3] Ben-Porath, Yoram, 1967. The Production of Human Capital and the Life Cycle of Earnings.

Journal of Political Economy 75(4) 352-365.

[4] Grossman, Michael, 1972. On the Concept of Health Capital and the Demand for Health.

Journal of Political Economy 80(2) 223-255.

[5] Guvenen, Fatih and Burhanettin Kuruscu, 2012. Understanding the Evolution of the US Wage

Distribution: A Theoretical Analysis. Journal of the European Economic Association 10(3)

482-517.

[6] Heckman, James J., Lance J. Lochner, and Christopher R. Taber, 1998a. Explaining Rising

Wage Inequality: Explorations with a Dynamic General Equilibrium Model of Labor Earnings

with Heterogeneous Agents. Review of Economic Dynamics 1(1) 1-58.

21

[7] Heckman, James J., Lance J. Lochner, and Christopher R. Taber, 1998b. General Equilibrium

Treatment Effects: A Study of Tuition Policy. American Economic Review. 88(2) 381-386.

[8] Heckman, James J., Lance J. Lochner, and Christopher R. Taber, 1998c. Tax Policy and

Human Capital Formation. American Economic Review. 88(2) 293-297.

[9] Huggett, Mark, 1993. The Risk-Free Rate in Heterogeneous-agent Incomplete-insurance

Economies. Journal of Economic Dynamics and Control. 17(5-6) 953-969.

[10] Krusell, Per and Anthony Smith, 1998. Income and Wealth Heterogeneity in the Macroecon-

omy. Journal of Political Economy. 106(5) 867-896.

[11] Kuruscu, Burhanettin, 2006. Training and Lifetime Income. American Economic Review 93(6)

832-846.

[12] Ljungqvist, Lars and Thomas Sargent, 2000. Recursive Macroeconomic Theory. Cambridge:

MIT Press.

[13] Lucas, Robert E. Jr., 1972. Asset Prices in an Exchange Economy. Econometrica 46(6) 1429-

1445.

[14] Lucas, Robert E. Jr., and Nancy Stokey, with Edard Prescott, 1989. Recursive Methods in

Economic Dynamics. Cambridge: Harvard University Press.

[15] McCall, J., 1970. Economics of Information and Job Search. Quarterly Journal of Economics

84(1) 113-126.

[16] Sundaram, Rangarajan, 1996. A First Course in Optimization Theory. Cambridge: Cambridge

University Press.

22