Single Agent Dynamics: Dynamic Discrete Choice Models Part I: Theory and Solution Methods Günter J. Hitsch The University of Chicago Booth School of Business 2013 1 / 113 Overview: Goals of this lecture 1. Discuss the structure and theory of dynamic discrete choice models 2. Survey numerical solution methods 3. Discuss goals of statistical inference: Reduced vs structural form of the model 4. Model estimation I Nested fixed point (NFP) estimators I MPEC approach I Heterogeneity and unobserved state variables 2 / 113
Transcript
Single Agent Dynamics: Dynamic Discrete ChoiceModels
Part I: Theory and Solution Methods
Günter J. HitschThe University of Chicago Booth School of Business
2013
1 / 113
Overview: Goals of this lecture
1. Discuss the structure and theory of dynamic discrete choice models2. Survey numerical solution methods3. Discuss goals of statistical inference: Reduced vs structural form of
the model4. Model estimation
I Nested fixed point (NFP) estimatorsI MPEC approachI Heterogeneity and unobserved state variables
Numerical Solution of a Dynamic Decision ProcessApproximation and InterpolationNumerical Integration
The Durable Goods Adoption Problem (Cont.)
References
4 / 113
Elements of a Markov Decision Process
I Time: t = 0, 1, . . . (infinite horizon)I A set of states, s 2 S
I S is the state space
I A set of actions, a 2 A(s)
I Set of feasible actions may depend on the current state, althoughthis is more general than what we typically need in dynamic discretechoice
5 / 113
Elements of a Markov Decision Process
I A decision maker chooses some action a 2 A(s) in period t, giventhe current state s, and then receives the utility U(s, a)
I The state evolves according to a probability distribution p(·|s, a)I If S is discrete, then Pr{s
t+1 = j|st
, a
t
} = p(j|st
, a
t
) for j 2 SI If S is continuous, then p(·|s
t
, a
t
) is the conditional density of st+1
6 / 113
Elements of a Markov Decision Process
I The decision process is called “Markov” because the utility functionand transition probabilities depend only on the current state andactions, not on the whole history of the process
I Utilities, U(s, a), and transition probabilities, p(·|s, a) are stationary,i.e., do not depend on the time period t when an action is taken
I Note: this does not imply that the time series process {st
}1t=0 is
stationary, and vice versa, stationarity of {st
}1t=0 is not required in
this class of modelsI Example: A decision make gets tired of a product as time progresses.
Could this be captured by a Markov Decision Process? — Answer:yes
IU
t
(s, a) = �t is not allowedI Instead let S = {0, 1, 2, . . . }, and for any s 2 S let p(s+ 1|s, a) = 1,
i.e. s
t+1 = s
t
+ 1. Then define
U(s, a) = �s
7 / 113
Actions, decision rules, and policies
I We examine decision problems where the decision maker usescurrent (and possibly past) information to choose actions in order tomaximize some overall objective function
I Decision rules: dt
: S !S
s2S A(s), dt
(s) 2 A(s)
I Decision rules associate a current state with a feasible action
I Policies, strategies, or plans: ⇡ = (d0, d1, . . . ) = (dt
)
1t=0
I A policy is a sequence of decision rules—a plan on how to chooseactions at each point in time, given the state at that point in time, s
t
8 / 113
Expected total discounted reward
I A given policy ⇡ = (dt
)
1t=0 induces a probability distribution on the
sequence of states and actions, (st
, at
)
1t=0
I For a given policy ⇡, the expected total discounted reward (utility) isgiven by
v⇡(s0) = E⇡
" 1X
t=0
�tU(st
, dt
(st
))|s0
#
I Future utilities are discounted based on the discount factor0 � < 1 (in finite horizon problems we could allow for � = 1)
9 / 113
Expected total discounted reward
I We will assume that v⇡ is well-definedI One necessary condition if the state space is discrete: The series
inside the expectation convergesI Ensured if U is bounded, because 0 � < 1
I Note, however, that boundedness is not a necessary condition forconvergence
I If the state space is continuous, certain technical conditions need tohold for the expectation to exist (see Stokey and Lucas 1989)
10 / 113
Optimal policies
I Let ⇧ be the set of all policies. ⇡⇤ 2 ⇧ is optimal if
v⇡⇤(s) � v⇡(s), s 2 S
for all ⇡ 2 ⇧I The value of the Markov decision problem is given by
v⇤(s) = sup
⇡2⇧{v⇡(s)}
I We call v⇤ the (optimal) value functionI An optimal policy ⇡⇤ is a policy such that
v⇡⇤(s) = v⇤(s) = max
⇡2⇧{v⇡(s)}
11 / 113
Bellman Equation
I Intuition:I Due to the stationarity of the utilities and transition probabilities, and
given that there is always an infinite decision horizon, the (optimal)value function should only depend on the current state, but not on t
I Hence, for any current state s, the expected total discounted rewardunder optimal decision making should satisfy the relationship
v⇤(s) = sup
a2A(s)
⇢U(s, a) + �
Zp(s0|s, a)v⇤(s0)ds0
�
12 / 113
Bellman Equation
I The equation above is called a Bellman equation
I Note that the Bellman equation is a functional equation withpotential solutions of the form v : S ! R
I More formally, let B = {f : S ! R : kvk < 1} be the metric spaceof real-valued, bounded functions on S. kfk = sup
s2S |f(s)| is thesupremum-norm. Define the operator L : B ! B,
Lv(s) ⌘ sup
a2A(s)
⇢U(s, a) + �
Zp(s0|s, a)v(s0)ds0
�
I A solution of the Bellman equation is a fixed point of L, v = Lv
13 / 113
Questions to be addressed
1. Is a solution of the Bellman equation the optimal value function?2. Is there a solution of the Bellman equation, and how many solutions
exist in general?
14 / 113
The main theorem
Theorem
If v is a solution of the Bellman equation, v = Lv, then v = v⇤.Furthermore, under the model assumptions stated above a (unique)
solution of the Bellman equation always exists.
I The proof of this theorem is based on the property that L is acontraction mapping. According to the Banach fixed-point theorem,a contraction mapping (defined on a complete metric space) has aunique fixed point. The contraction mapping property cruciallydepends on the additive separability of the total discounted rewardfunction, and in particular on discounting, � < 1
15 / 113
Optimal policies
I Assume that the supremum in the Bellman equation can beattained, and define the decision rule
d⇤(s) 2 argmax
a2A(s)
⇢U(s, a) + �
Zp(s0|s, a)v⇤(s0)
�
I Let ⇡⇤= (d⇤, d⇤, . . . ) be a corresponding policy
Theorem
Let ⇡⇤be as defined above. Then ⇡⇤
is an optimal policy, such that
v⇡⇤= v⇤
16 / 113
Optimal policies
I Note that the theorem on the previous slide shows us how to find anoptimal policy
I In particular, note that the theorem shows that if the supremum inthe Bellman equation can be attained, then there is a stationary
optimal policy
17 / 113
Optimal Policies: Notes
I A sufficient condition for the existence of an optimal policy isI A(s) is finiteI Satisfied (by definition) in dynamic discrete choice problems
I Decision rules that depend on the past history of states and actionsdo not improve the total expected reward
I Hence, basing decisions only on current, not past information, is notrestrictive in Markov decision processes
Numerical Solution of a Dynamic Decision ProcessApproximation and InterpolationNumerical Integration
The Durable Goods Adoption Problem (Cont.)
References
19 / 113
Kellogg’s Raisin Bran, 20 oz pack size
●
●
●
●
●
●●
●
●
●●●
●
●
●●●●●●●●
●
●●●
●●
●
●
●●●●
●●●●
●
●●●●●●●●
●●●●●●●●●
●
●●●
●●
●●●●●●●
●
●●●●
●
●
●●●●
●
●●●●●●●●●●
●
●●●
●
●
●●●●●●
●
●●●●●●●●
●
●●
●●●
●
●
●
●●●●
●●●
●
●
●
●
●●●●
●
●●●●●●●
●
●
●●●●●●●●
●
●●
●
●●●●●
●
●●●●●●
●
●●●●
●
●●●●
●●●
●
●
●●●
●
●●●●●●●●●●●
●
●●●●●●
●●
2008 2009 2010 2011 2012
2.0
2.5
3.0
3.5
4.0
Pric
e
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
2008 2009 2010 2011 2012
0.0
0.5
1.0
1.5
2.0
Log
(bas
e 10
) uni
ts
20 / 113
Overview: Sales promotions
I Many CPG products are sold at both regular (base) prices and atpromotional prices
I We often observe sales spikes during a promotionI Could be due to brand switching or increased consumptionI Could be due to stockpiling if the good is storable
I Develop a simple model of stockpiling—storable goods demandI Goal: predict how demand is affected by the distribution of prices at
which the storable good can be bought
21 / 113
Model outline
I At the beginning of each period a consumer goes to a store andchooses among K different pack sizes of a storable good
I Pack size k contains n
k
units of the product (nk
2 {1, 2, . . . })I No purchase: n0 = 0
I The choice in period t is at
2 A = {0, 1, . . . ,K}I i
t
is the number of units in the consumer’s inventory at thebeginning of period t
I After the shopping trip, if at
= k the consumer has it
+ nk
units ofthe product at her disposal
I The consumer has a consumption need of one unit in each periodI Will consume one unit if i
t
+ n
k
� 1, and will not consume otherwise
22 / 113
Evolution of inventory
I Consumer can store at most I unitsI Conditional on i
t
and at
= k, inventory evolves as
it+1 =
8<
:
0 if it
+ nk
= 0
it
+ nk
� 1 if 1 it
+ nk
II if i
t
+ nk
> I
I Note: Units in excess of I + 1 can neither be consumed nor stored
I The evolution of inventory is deterministic, and we can writeit+1 = �(i
t
, at
)
23 / 113
Utilities and disutilities
I The purchase utility (rather: disutility) from buying pack size k is�↵P
kt
� ⌧(nk
)
IP
kt
is the price of pack size k and ↵ is the price sensitivity parameterI
⌧(n
k
) is the hassle cost of purchasing (or transporting) pack size k
I Outside option: P0t = 0, ⌧(n0) = ⌧(0) = 0
I The consumption utility, given that there is at least one unit toconsume (i
t
+ nk
� 1) is �, and 0 otherwiseI Inventory holding cost: c(i
t+1)
Ic(0) = 0
24 / 113
Notation
I Define xt
⌘ (it
, Pt
), where Pt
= (P1t, . . . , PKt
)
I Define utility as a function of xt
and as a function of the chosenpack size a
t
= k :
uk
(xt
) =
8<
:
�↵Pkt
� ⌧(nk
) + � � c(it+1) if k 6= 0
� � c(it+1) if k = 0 and i
t
� 1
0 if k = 0 and it
= 0
25 / 113
Distribution of prices
I The price vectors Pt
are drawn from a discrete distributionI P
t
can take one of L values {P (1), . . . , P (L)} with probabilityPr{P
t
= P (l)} = ⇡l
I In this specification prices are i.i.d. over timeI Generalization: Prices be drawn from a Markov process where
current prices depend on past price realizations
26 / 113
State vector and utility function
I A random utility term enters the consumer’s preferences over eachpack size choice
I ✏t
= (✏0t, . . . , ✏Kt
), where ✏kt
is iid Type I Extreme Value distributedI
g(✏) is the pdf of ✏t
I The state of this decision process is st
= (xt
, ✏t
)
I The (full) utility function of the consumer:
U(st
, at
= k) = uk
(xt
) + ✏kt
27 / 113
State evolution
I The evolution of xt
is given by the pmf f :
f(i, P (l)|xt
, at
) ⌘ Pr{it+1 = i, P
t+1 = P (l)|xt
, at
}= I{i = �(i
t
, at
)} · ⇡l
IDefined for 0 i I, 1 l L
I Then the evolution of the state, s = (x, ✏), is given by
p(st+1|st, at) = f(x
t+1|xt
, at
) · g(✏t+1)
28 / 113
Discussion
I The storable goods demand model has a specific structure due tothe way the random utility terms ✏
kt
enter the problem:I The state has two separate components, x
t
and ✏
t
I The utility function has the additive form
U(s
t
, a
t
= k) = u
k
(x
t
) + ✏
kt
I The transition probability factors into two separate components:
p(s
t+1|st, at
) = f(x
t+1|xt
, a
t
) · g(✏t+1)
I In the next section we will discuss Markov decision processes withthis special structure
Numerical Solution of a Dynamic Decision ProcessApproximation and InterpolationNumerical Integration
The Durable Goods Adoption Problem (Cont.)
References
30 / 113
Structure of the decision problem
I A decision maker (typically a consumer or household) choosesamong K + 1 alternatives (goods or services),a 2 A = {0, 1, . . . ,K}, in each period t = 0, 1, . . .
I Alternative 0 typically denotes the no-purchase option
I The state vector, s 2 S, has the following structure:
s = (x, ✏), where x 2 X, ✏ 2 RK+1
I The transition probability is p(st+1|st, at)
31 / 113
Main assumptions
1. Additive separability. The utility from choosing action at
= j instate s
t
= (xt
, ✏t
) has the following additive structure:
U(st
, at
) = uj
(xt
) + ✏jt
2. Conditional independence. Given xt
and at
, current realizations of ✏t
do not influence the realizations of future states xt+1. Hence, the
transition probability of x can be written as f(xt+1|xt
, at
)
3. iid ✏jt
. ✏jt
is iid across actions and time periods.I
g(✏) is the pdf of ✏ = (✏0, . . . , ✏K), and we typically assume that thesupport of ✏
j
is RI We could allow for g(✏|x), although this more general specification is
rarely used in practice
32 / 113
Discussion of model structure
I While x can contain both observed and unobserved components, ✏ isassumed to be unobservable to the researcher
I✏
j
is an unobserved state variable
I Additive separability and the assumptions on ✏j
will allow us torationalize any observed choice, and thus ensure that the likelihoodfunction is positive on the whole parameter space
I Given the conditional independence and iid assumptions, thetransition probability can be written as follows:
p(st+1|st, at) = f(x
t+1|xt
, at
) · g(✏t+1)
33 / 113
Decision rules and rewards
I In each period, the decision maker chooses an action according to adecision rule d : X⇥ RK+1 ! A, a
t
= d(xt
, ✏t
).
I The expected present discounted value of utilities under the policy⇡ = (d, d, . . . ) is
v⇡(x0, ✏0) = E" 1X
t=0
�tU(xt
, ✏t
, d(xt
, ✏t
))|x0, ✏0
#.
34 / 113
Optimal behavior and the Bellman Equation
I Given that A is finite, there is an optimal decision rule d⇤(x, ✏)
I The associated value function satisfies
v(x, ✏) = max
j2A{u
j
(x) + ✏j
+ �E [v(x0, ✏0)|x, a = j]}
= max
j2A
⇢uj
(x) + ✏j
+ �
Zv(x0, ✏0)f(x0|x, j)g(✏0) dx0d✏0
�
35 / 113
The expected value function
I Goal:I Characterize optimality condition as a function of x onlyI ) faster computation of model solution
I Define the expected or integrated value function,
w(x) =
Zv(x, ✏)g(✏)d✏
I w(x) is the value that the decision maker expects to receive beforethe unobserved states, ✏, are realized
I Using the definition of w, we can re-write the value function:
v(x, ✏) = max
j2A
⇢uj
(x) + ✏j
+ �
Zw(x0
)f(x0|x, j) dx0�
36 / 113
The expected value function
I Taking the expectation on both sides of the previous (last slide)equation with respect to ✏, we obtain the integrated Bellman
equation
w(x) =
Zmax
j2A
⇢uj
(x) + ✏j
+ �
Zw(x0
)f(x0|x, j) dx0�g(✏)d✏
I The right-hand side of this equation defines a contraction mapping,� : B ! B, on the space of bounded functions, B
I Thus, the equation has a unique solution which must equal theexpected value function, w = �(w)
I Note the role of the conditional independence and iid ✏ assumptionsin the derivation of this equation
37 / 113
Blackwell’s sufficient conditions for a contraction
Theorem
Let B = {f : S ! R : kfk < 1} be the metric space of real-valued,
bounded functions on S. Suppose the operator L : B ! B satisfies the
following two conditions:
(1) Monotonicity: If f, g 2 B and f(x) g(x) for all x 2 S, then
(Lf)(x) (Lg)(x) for all x 2 S.
(2) Discounting: There is a number � 2 (0, 1) such that
(L(f + a))(x) (Lf)(x) + �a for all f 2 B, a > 0, and x 2 S.
Then L is a contraction mapping with modulus �.
38 / 113
� is a contraction mapping
Proof.
If g h, thenZ
g(x0)f(x0|x, j) dx0
Zh(x0
)f(x0|x, j) dx0
for all x 2 X and j 2 A. It follows that �f �h. Let a � 0. Then
�
Z(g(x0
) + a)f(x0|x, j) dx0= �
Zg(x0
)f(x0|x, j) dx0+ �a.
Hence(�(g + a))(x) = (�g)(x) + �a.
39 / 113
Choice-specific value functions
I Using the definition of the expected value function, w(x), define thechoice-specific value functions
vj
(x) = uj
(x) + �
Zw(x0
)f(x0|x, j) dx0
I The full expected PDV of utilities from choosing j is vj
(x) + ✏j
40 / 113
Choice-Specific value functions
I The choice-specific value functions characterize the model solution
I The value function can be expressed in terms of these choice-specificvalue functions,
v(x, ✏) = max
j2A{v
j
(x) + ✏j
}
I Therefore, w(x) =
Rv(x, ✏)g(✏)d✏ is sometimes also called the
EMAX function:
w(x) =
Zv(x, ✏)g(✏)d✏ =
Zmax
j2A{v
j
(x) + ✏
j
} g(✏)d✏
I Furthermore, the optimal decision rule satisfies
d⇤(x, ✏) = k , vk
(x) + ✏k
� vj
(x) + ✏j
for all j 2 A, j 6= k
41 / 113
Recap
I The previous slides show how the solution of this class of dynamicdecision problems can be simplified:
1. Solve for the expected value function, w(x)
2. Calculate the choice-specific value functions, vj
(x)
3. Calculate the value function and optimal policy from thechoice-specific value functions
I The key step here is 1: By “integrating out” ✏, we can reduce thedimensionality of the problem by K + 1. This saves huge amounts ofcomputing time!
42 / 113
Conditional choice probabilities (CCP’s)
I The additive separability assumption on the utility function U(s, a)implies that the expected PDV of choosing j is additively separablein v
j
(x) and the unobserved state, ✏j
I Thus, once vj
(x) is known, the model predictions—choiceprobabilities—can be derived just as the model predictions of astatic discrete choice model:
Pr{a = k|x} = Pr{vk
(x) + ✏k
� vj
(x) + ✏j
8j 6= k}
I We call these the conditional (on the state x) choice probabilities(CCP’s)
I Frequently, we assume that ✏j
has the Type I Extreme Valuedistribution. Then, choice probabilities have the logit form
Pr{a = k|x} =
exp(vk
(x))P
K
j=0 exp(vj(x))
43 / 113
Technical note
I Let �1, . . . , �N be some numbers and let each of the N randomvariables ✏
j
be i.i.d. Type I Extreme Value (Gumbel) distributedI The cdf of each ✏
j
is G(✏
j
) = exp(� exp(�(✏
j
� µ)))
I E(✏j
) = µ+ �, where � ⇡ 0.57722 is Euler’s constantI In many applications, ✏
j
is assumed to be drawn from the standardGumbel distribution, µ = 0
I If g(✏) =Q
N
j=1 g(✏j), we obtain the closed-form expression
Z ✓max
j=1,...,N{�
j
+ ✏j
}◆g(✏)d✏ = µ+ � + log
0
@NX
j=1
exp(�j
)
1
A
44 / 113
Technical note
I Hence if ✏j
is i.i.d. Type I Extreme value distributed, then theexpected value function can be expressed as
w(x) = µ+ � + log
0
@X
j2Aexp
✓uj
(x) + �
Zw(x0
)f(x0|x, j) dx0◆1
A
I Hence, the integration over ✏ can be performed analyticallyI To get rid of the constant term µ+ � in the equation above, simply
assume µ = ��, in which case ✏j
is centered at 0
45 / 113
Computing the expected value function w
I Use a form of value function iterationI The algorithm works because of the contraction mapping property of
the operator �I Note:
I Policy iteration or modified policy iteration are better algorithms toobtain the value function corresponding to a Markov decision process
I However, these algorithms are not applicable here because wecalculate the expected value function, w, not the value function v
46 / 113
Computing w : The algorithm
1. Start with some w(0) 2 B2. For each x 2 X, calculate
w(n+1)(x) =
Zmax
j2A
⇢uj
(x) + ✏j
+ �
Zw(n)
(x0)f(x0|x, j) dx0
�g(✏)d✏
3. If��w(n+1) � w(n)
�� < "w
, then proceed to 4. Otherwise, return to 2.4. Calculate the (approximate) choice-specific value functions
v(n+1)j
(x) = uj
(x) + �
Zw(n+1)
(x0)f(x0|x, j) dx0
47 / 113
Computing w : Speed of convergence
I For any starting value w(0) 2 B, w(n) ! w in the sup-norm (k·k)I The rate of convergence is O(�n
) :
lim sup
n!1
��w(n) � w��
�n
< 1
I That is, w(n) converges geometrically at the rate �I Correspondingly, in marketing applications problems with short time
intervals (daily or weekly decision making) the algorithm requires alarger number of iterations to find a solution with a given accuracy,because � will be close to 1
48 / 113
Literature background
I The model structure discussed above closely follows John Rust’s(1987) “Optimal Replacement of GMC Bus Engines: An EmpiricalModel of Harold Zurcher” paper
I Harold Zurcher was the superintendent of maintenance at theMadison Metropolitan Bus Company
I Other early key contributionsI Choice of occupations — Miller (1984)I Fertility choice (number and timing of children) — Wolpin (1984)I Option value of patents — Pakes (1986)
)I v then becomes a finite-dimensional object, v(s
i
) = vi
, and can bestored in computer memory using L floating-point numbers
67 / 113
Discretization
I One way to proceed is to also discretize the transition probability:
pi
(s, a) = Pr{s0 = si
|s, a} =
Z
Bi
p(s0|s, a)ds0,
or: pi
(s, a) =p(s
i
|s, a)P
L
l=1 p(sl|s, a)
I Here, Bi
is an appropriately chosen partition of the state space andsi
2 Bi
I For a given a, the evolution of the state can now be described by astochastic matrix, P a with (k, i)-element p
i
(sk
, a)
I One can also discretize the set of actions,S
s2S A(s)
68 / 113
Discussion
I Although conceptually straightforward, the discretization discussedon the previous slide may neither deliver a fast algorithm nor anaccurate solution
I Note that taking the expectation can involve up to L = |S|multiplications, which can be much larger than the number ofmultiplications using alternative methods, such as quadrature andinterpolation
69 / 113
Interpolation
I Interpolation methods record the function values at a finite numberof points (the table values), and use some algorithm to evaluate thefunction for any point that is not in the table:
1. Discretize the state space into a grid G, as before, and record thevalues of v at each point s
i
2 G2. Instead of assuming that the state can only take on values in G, we
now try to find a method to evaluate v(s) at some arbitrary points 2 S
70 / 113
Interpolation
I Linear interpolation in one dimension (D = 1) :
1. Find the index i such that x
i
s < x
i+1
2. Calculate ! = (s� x
i
)/(x
i+1 � x
i
)
3. The interpolated value is
bv(s) = v
i
+ !(v
i+1 � v
i
)
71 / 113
Interpolation: Example
0 2 4 6 8 10!20
!10
0
10
20
30
40
s
v
72 / 113
Bilinear interpolation
I We can extend the linear interpolation algorithm to more than onedimension
1. In two dimensions, s = (s1, s2), find the indices i and j such thatx
i1 s1 < x
i+1,1 and x
j2 s2 < x
j+1,2
2. Calculate !1 = (s1 � x
i1)/(xi+1,1 � x
i1) and
!2 = (s2 � x
j2)/(xj+1,2 � x
j2)
3. The interpolated value is
bv(s) = y0 + !1(y1 � y0)
y0 = v
ij
+ !2(vi,j+1 � v
ij
)
y1 = v
i+1,j + !2(vi+1,j+1 � v
i+1,j)
I This procedure is called bilinear interpolation
I Note that bilinear interpolation does not yield a linear functionsurface!
73 / 113
Bilinear interpolation: Example
02
46
810
0
5
10
4.5
5
5.5
6
6.5
7
7.5
8
8.5
9
s1
v
s2
74 / 113
Other interpolation methods: Simplicial interpolation
I Partition the state space into trianglesI Interpolate using linear surfacesI Example for D = 2
1. Partition the state space into rectangles as before2. Further subdivide each rectangle into two triangles3. Interpolate using the unique linear function that coincides with v
i
ateach of the three vertices of the triangle
75 / 113
Other interpolation methods: Splines
I A spline of order N is a function that is a polynomial of degreeN � 1 on each interval (or rectangle) defined by the grid
I Furthermore, we require that is CN�2 on all of the grid, not justwithin each interval or rectangle
I That is, we require that is smooth not only within, but also at theboundary of each interval or rectangle
I Example: A cubic spline is of order N = 4. It has continuous firstand second derivatives everywhere. For D = 1, it can be representedas
(s) = a0i + a1is+ a2is2+ a3is
3
within each interval [xi
, xi+1]
I The requirement that is CN�2 imposes coefficient restrictions forthe polynomials across intervals/rectangles
76 / 113
Splines
I Advantage of splines: allows for continuous or smooth derivativesI This can be important in games with continuous controls to ensure
continuity of best-reply functions and thus existence of apure-strategy equilibrium
77 / 113
Approximation using orthogonal polynomials
I Instead of first discretizing the state space and then approximatingv(s) using interpolation, we try to approximate v(s) globally using alinear combination of easy-to-evaluate functions
I We start with a set of basis functions, �i
: S ! R, i = 1, . . . ,K.Here, S ⇢ RD
I Each �i
is a polynomial (polynomials are easy to evaluate)I To simplify the exposition, let D = 1 and consider approximations
on the interval [�1, 1]
78 / 113
Approximation using orthogonal polynomials
I We will only discuss Chebyshev polynomials, which are particularlyuseful in many applications
I The Chebyshev polynomial of degree n is defined as
Tn
(x) = cos(n · cos�1(x))
I Even though this doesn’t look like a polynomial, it is. An easier wayto calculate T
n
(x) is to set T0(x) = 1, T1(x) = x, and then forn � 2 recursively calculate
Tn
(x) = 2xTn�1(x)� T
n�2(x)
79 / 113
Chebyshev polynomials, degree n = 0, . . . , 5
!1 0 10
1
2n = 0
x!1 0 1
!1
0
1n = 1
x!1 0 1
!1
0
1n = 2
x
!1 0 1!1
0
1n = 3
x!1 0 1
!1
0
1n = 4
x!1 0 1
!1
0
1n = 5
x
80 / 113
Approximation using orthogonal polynomials
I Define the inner product hf, gi between two functions f and g on[�1, 1] as
hf, gi =Z 1
�1
f(x)g(x)p1� x2
dx
I We say that f and g are orthogonal to each other if hf, gi = 0
I One can verify that Chebyshev polynomials are mutually orthogonal,hT
n
, Tm
i = 0 for n 6= m
I Goal: Choose a linear combination of Chebyshev polynomials toapproximate some arbitrary function,
v(s) ⇡KX
i=0
✓i
Ti
(s)
81 / 113
Chebyshev regression algorithm
I An algorithm to find a good approximation for f on some interval[a, b] is as follows:
1. Choose the degree N of the polynomial and the number ofinterpolation nodes M � N + 1
2. Compute each interpolation node:
⌫
i
= � cos
✓2i� 1
2M
⇡
◆, i = 1, . . . ,M
3. Adjust these nodes to the interval [a, b] :
⇠
i
= (⌫
i
+ 1)
(b� a)
2
+ a
4. Calculate the Chebyshev coefficients:
✓
k
=
PM
i=1 f(⇠i)Tk
(⌫
i
)
PM
i=1 Tk
(⌫
i
)
2k = 0, . . . , N
82 / 113
Regression with orthogonal regressors
I Consider the regression y = X� + ✏, where xj
denotes the jth
column of XI Suppose the regressors are orthogonal:
hxj
, xk
i = x0j
xk
= 0
I Then
XTX =
2
6664
x01x1 0 · · · 0
0 x02x2
.... . .
...0 · · · x0
N
xN
3
7775
83 / 113
Regression with orthogonal regressors
I Let b = (XTX)
�1XT y
I Thenbk
=
x0k
y
x0k
xk
=
Pi
xik
yiP
i
xik
xik
I Note the form of the Chebyshev coefficients:
✓k
=
Pi
v(⇠i
)Tk
(⌫i
)Pi
Tk
(⌫i
)
2
I Hence the name “Chebyshev regression”!I Difference to regular regression: the Chebyshev regression algorithm
chooses the values of the regressors/nodes (in a smart fashion, tominimize approximation error)
84 / 113
Approximation using orthogonal polynomials
I Given the Chebyshev coefficients calculated in the Chebyshevregression algorithm, we can approximate v(x), x 2 [a, b] by
v̂(x) =
NX
k=0
✓k
Tk
✓2
x� a
b� a� 1
◆
I If v is Ck, k � 1, and v̂N
(x) is a degree N Chebyshevapproximation, then
lim
N!1sup
x
|v̂N
(x)� v(x)| = 0
Iv̂
N
converges uniformly to v if v is (at least) continuouslydifferentiable
I For non-differentiable v’s, Chebyshev regression will generally notprovide a good approximation
85 / 113
Multidimensional approximation
I Suppose we have basis functions Tid
(xd
), i = 1, . . .Kd
, for each ofd = 1, . . . , D dimensions
I Goal: construct a basis for functions of (x1, . . . , xD
)
I Collect all functions of the form
(x1, . . . , xD
) =
DY
d=1
Tidd(xd
), 1 id
Kd
I The collection of all these N =
QD
d=1 Kd
functions is called theD-fold tensor product basis
I Important application: D-dimensional Chebyshev approximation
86 / 113
Discussion
I What are the relative merits of approximations using orthogonalpolynomials versus interpolation methods such as bilinearinterpolation?
I Chebyshev coefficients are easy to compute if there is only a smallnumber of nodes M , while finding values for each grid point s
i
2 Gcan be very time-intensive
I If the true function v is not smooth or displays a lot of curvature,then a Chebyshev approximation will be poor unless N and M arelarge, in which case there is no more speed advantage overinterpolation
I Note that approximation using grids and interpolation is essentially anon-parametric method, while approximation using polynomials offixed degree N is a parametric method
87 / 113
Discussion: The curse of dimensionality
I In practice, the biggest problem when we try to find a solution of aMarkov decision problem involves computation time
I If we discretize each axis of the space into M points, then the totalnumber of grid points is MD
I Hence, the number of points at which the value function has to becalculated rises exponentially in D
I Even worse, the number of floating point operations to evaluate V
also rises exponentially for many methods
88 / 113
Newton-Cotes formulas
I Easy to understand, but ineffective compared to Gaussianquadrature or Monte Carlo methods
I Example: Midpoint ruleI Consider the interval [a, b], choose N and define the step-size
h = (b� a)/N
I Let x
i
= a+ ih� h/2. x
i
is the midpoint in the interval[a+ (i� 1)h, a+ ih]
I Approximate the integral of f :
Zb
a
f(x)dx ⇡ h
NX
i=1
f(x
i
)
I In practice, we can typically calculate the integral with much higheraccuracy given the same number of operations using alternativemethods
89 / 113
Gaussian quadrature
I Gaussian quadrature rules are formulas for integrals of the functionW (x)f(x) for some specific W (x) :
Zb
a
W (x)f(x)dx ⇡NX
i=1
wi
f(xi
)
I The wi
’s are weights, and the xi
’s are called nodes or abscissasI This looks exactly like a Newton-Cotes formula, however, the main
idea of Gaussian quadrature is to choose the weights and nodes suchthat the approximation is very close to the true integral even if N issmall
I Weights and nodes are chosen so that the approximation is exact forlow-order polynomials
90 / 113
Gaussian quadrature
I Given a function W (x) and a choice of N, there are algorithms tocompute the quadrature nodes x
i
and the corresponding weights wi
I There are different quadrature rules, distinguished by the choice ofthe function W (x)
I Each rule makes use of the fact that the integrand can be factoredas W (x)f(x) with W (x) known
91 / 113
Gaussian quadrature
I Consider the important case of Gauss-Hermite quadrature:
W (x) = exp
��x2
�
I For the corresponding nodes xi
and weights wi
,
Z 1
�1exp
��x2
�f(x)dx ⇡
NX
i=1
wi
f(xi
)
I Hence,
Z 1
�1
1p2⇡�
exp
�1
2
✓x� µ
�
◆2!f(x)dx
⇡ 1p⇡
NX
i=1
wi
f(µ+
p2�x
i
)
92 / 113
Gaussian quadrature
I Gauss-Hermite quadrature yields a very accurate approximation tointegrals involving a normal pdf for a small number of quadraturenodes N
I Gaussian quadrature can be extended to higher dimensions, D > 1
I Unfortunately, there is a curse of dimensionality in D
I For large D, Monte Carlo integration is the only feasible integrationmethod
93 / 113
Monte Carlo integration
I Basic idea: Let (Xn
)
n�0 be a sequence of random variables withdensity p(x)
I Then we can simulate the integral
Zf(x)p(x)dx ⇡ 1
N
NX
n=1
f(Xi
)
I In fact,1
N
NX
n=1
f(Xi
)
a.s.�!Z
f(x)p(x)dx
I Methods such as importance sampling improve on the efficiency ofthis crude simulator
Numerical Solution of a Dynamic Decision ProcessApproximation and InterpolationNumerical Integration
The Durable Goods Adoption Problem (Cont.)
References
95 / 113
Code documentation
I The solution algorithm is coded in MATLABI Run Main.m to compute and display the solution of the durable
goods adoption problemI Solutions are obtained using both bilinear interpolation and
Chebyshev regressionI All information (grid points, table values, Chebyshev coefficients, ...)
is contained in MATLAB structures (w_interp, w_cheb, etc.)
96 / 113
Main.m
1. Sets values for the model parameters2. Calls Gauss_Hermite_multidim to calculate Gauss-Hermite
quadrature weights and nodes3. Defines and initializes the bilinear and Chebyshev approximation
routines4. Calls value_function_iteration to solve the model using both
methods5. Calls calculate_choice_probabilities to predict the CCP’s6. Defines a grid on which to graph the solutions and calls
transfer_representation to evaluate the model solutions on thisgrid
7. Plots the expected value functions, conditional choice probabilities,and differences in the CCP’s predicted by the interpolation andChebyshev approximation solutions
97 / 113
value_function_iteration
I Uses value function iteration to calculate the expected valuefunction w
I Works for both bilinear interpolation and Chebyshev approximation
98 / 113
Bellman_operator
1. Bilinear interpolation:I Iterates through the state space and calculates the expected value
function for each stateI The expected future value is obtained using quadrature and bilinear
interpolation
2. Chebyshev approximation:I Algorithm can be efficiently expressed using matrix operationsI Utilize linearity of quadrature and Chebyshev approximation to
pre-compute many terms needed to update the expected valuefunction
Numerical Solution of a Dynamic Decision ProcessApproximation and InterpolationNumerical Integration
The Durable Goods Adoption Problem (Cont.)
References
108 / 113
Theory of dynamic decision processes
I Bertsekas, D. P. (2005): Dynamic Programming and Optimal Control: 2ndEdition (Volumes 1 and 2). Athena Scientific
I Puterman, M. L. (1994): Markov Decision Processes. Wiley-InterscienceI Rust, J. (1996): “Numerical Dynamic Programming in Economics,” in Handbook
of Computational Economics, ed. by H. M. Amman, D. A. Kendrick, and J.Rust: Elsevier Science B.V., 619-729
I Stokey, N. L., R. E. Lucas, and E. C. Prescott (1989): Recursive Methods InEconomic Dynamics. Harvard University Press
109 / 113
Dynamic discrete choice models
I Aguirregabiria, V. and Mira, P. (2010): “Dynamic discrete choice structuralmodels: A survey,” Journal of Econometrics, 156, 38-67
I Miller, R. A. (1984): “Job Matching and Occupational Choice,” Journal ofPolitical Economy, 92(6), 1086-1120
I Pakes, A. (1986): “Patents as Options: Some Estimates of the Value of HoldingEuropean Patent Stocks,” Econometrica, 54(4), 755-784
I Rust, J. (1987): “Optimal Replacement of GMC Bus Engines: An EmpiricalModel of Harold Zurcher,” Econometrica, 55(5), 999-1033
I Rust, J. (1994): “Structural Estimation of Markov Decision Processes,” inHandbook of Econometrics, ed. by R. F. Engle, and D. L. McFadden. Londonand New York: Elsevier North-Holland, 3081-3143
I Wolpin, K. I. (1984): “An Estimable Dynamic Stochastic Model of Fertility andChild Mortality,” Journal of Political Economy, 92(5), 852-874
110 / 113
Numerical analysis
I Judd, K. L. (1998): Numerical Methods in Economics. Cambridge: MIT PressI Press, W. H., S. A. Teukolsky, W. T. Vetterling, And B. P. Flannery (2007):
Numerical Recipes: The Art of Scientific Computing. Cambridge University PressI Rust, J. (1996): “Numerical Dynamic Programming in Economics,” in Handbook
of Computational Economics, ed. by H. M. Amman, D. A. Kendrick, and J.Rust: Elsevier Science B.V., 619-729
I Ueberhuber, C. W. (1997): Numerical Computation: Methods, Software, andAnalysis (Volumes 1 and 2). Springer Verlag
I Benítez-Silva, H., G. Hall, G. J. Hitsch, G. Pauletto, and J. Rust (2001): “AComparison of Discrete And Parametric Approximation Methods ForContinuous-State Dynamic Programming Problems,” manuscript
111 / 113
The Storable Goods Demand Problem
I Erdem, T., Imai, S., and Keane, M. P. (2003): “Brand and Quantity ChoiceDynamics Under Price Uncertainty,” Quantitative Marketing and Economics, 1,5-64
I Hartmann, W. R. and Nair, H. S. (2010): “Retail Competition and the Dynamicsof Demand for Tied Goods,” Marketing Science, 29(2), 366-386
I Hendel, I. and Nevo, A. (2006): “Measuring the Implications of Sales andConsumer Inventory Behavior,” Econometrica, 74(6), 1637-1673
112 / 113
The Durable Goods Adoption Problem
I Dubé, J.-P., Hitsch, G. J., and Chintagunta, P. (2010): “Tipping andConcentration in Markets with Indirect Network Effects,” Marketing Science,29(2), 216-249
I Goettler, R. and Gordon, B. (2011: “Does AMD Spur Intel to Innovate More?,”Journal of Political Economy, 119(6), 1141-1200
I Gordon, B. R. (2009): “A Dynamic Model of Consumer Replacement Cycles inthe PC Processor Industry,” Marketing Science, 28(5), 846-867
I Gowrisankaran, G. and Rysman, M. (2012): “Dynamics of Consumer Demand forNew Durable Goods,” Journal of Political Economy, 120(6), 1173-1219
I Nair, H. S. (2007): “Intertemporal Price Discrimination with Forward-LookingConsumers: Application to the US Market for Console Video-Games,”Quantitative Marketing and Economics, 5, 239-292
I Melnikov, O. (2013): “Demand for Differentiated Durable Products: The Case ofthe U.S. Computer Printer Market,” Economic Inquiry, 51(2), 1277-1298
I Song, I. and Chintagunta, P. K. (2003): “A Micromodel of New ProductAdoption with Heterogeneous and Forward-Looking Consumers: Application tothe Digital Camera Category,” Quantitative Marketing and Economics, 1,371-407
![Antony and Cleopatra [James F. Bellman, Kathryn Bellman]](https://static.documents.pub/doc/80x56/55cf9761550346d03391502a/antony-and-cleopatra-james-f-bellman-kathryn-bellman.jpg)
