Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Mixing it up: Discrete and Continuous OptimalControl for Biological Models
Suzanne Lenhart
University of Tennessee, Mathematics DepartmentNational Institute for Mathematical and Biological
Synthesis
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Dedicated to the memory of Sonia Kovalevsky
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Outline
1. Background to Optimal Control of ODEs/Introduction
2. Cardiopulmonary Resuscitation Model (CPR)
3. Parabolic PDE Model for Rabies in Raccoons
4. Integro-difference Models for Dispersal, Growth and Harvest
5. Dynamic Fishery Problem
SHOW key mathematical features that make each of theseapplications different
Mathematical results can tell us something new about thebiological situations!
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Work supported by the National Science Foundation and
Oak Ridge National Laboratory and
website www.nimbios.org
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Deterministic Optimal Control- ODEs
time dependent control u(t)
Find piecewise continuous control u(t) and associated statevariable x(t) to maximize
max
∫ T
0
f (t, x(t), u(t)) dt
subject to
x ′(t) = g(t, x(t), u(t))
x(0) = x0 and x(T ) free
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Optimal Control and Pontryagin’s Maximum Principle
Pontryagin and his collaborators developed optimal control theoryfor ordinary differential equations about 1950.
Pontryagin’s KEY idea was the introduction of the adjoint variableswhich link the differential equations to the objective functional.
Converted problem of finding an optimal control to maximize theobjective functional subject to dynamic equations to maximizingthe Hamiltonian pointwise.
The EXISTENCE of an optimal control must be proven first beforeusing this principle (using some type of compactness argument).
The four examples will show the use of PMP ideas in new waysand extensions for new applications.
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Pontryagin Maximum Principle (PMP)If u∗(t) and x∗(t) are optimal for above problem, then there existsadjoint variable λ(t) s.t.
H(t, x∗(t), u(t), λ(t)) ≤ H(t, x∗(t), u∗(t), λ(t)),
for all u, at each time, where Hamiltonian H is defined by
H(t, x(t), u(t), λ(t)) = f (t, x(t), u(t)) + λg(t, x(t), u(t)).
and
λ′(t) = −∂H(t, x(t), u(t), λ(t))
∂x
λ(T ) = 0 transversality condition
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Example 1 - Cardiopulmonary Resuscitation (CPR)
Each year, more than 250,000 people die from cardiac arrest in theUSA alone. Despite widespread use of CPR the survival of patientsrecovering from cardiac arrest remains poor.
The rate of survival for CPR performed out of the hospital is 3%,while for patients in the hospital, the rate of survival is 10-15%.
Here, we consider a model for CPR allowing chest and abdomencompression and decompression.
We apply the optimal control strategy for improving resuscitationrates to a validated circulation model developed by Babbs.Reference: Babbs, Circulation 1999.
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Heart Diagram
Figure 23-11 Valvular structures of the heart. The atrioventricular valves are in an open position, and the semilunar
valves are closed. There are no valves to control the flow of blood at the inflow channels (i.e., vena cava and
pulmonary veins) to the heart.
Copyright © 2005 Lippincott Williams & Wilkins. Instructor's Resource CD-ROM to Accompany Porth's Pathophysiology: Concepts of Altered Health States, Seventh Edition.
Superior
vena cavaAortic valve
Pulmonary
veins
Inferior
vena cava
Mitral
valve
Pulmonic
valve
Tricuspid
valvePapillary
muscle
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Diagram of Circulation Model
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
State Variables
As controls, we choose the the pattern of the external pressure onthe chest and on the abdomen. The pressure state variables are asfollows:P1 pressure in abdominal aortaP2 pressure in inferior vena aortaP3 pressure in carotid arteryP4 pressure in jugular veinP5 pressure in thoracic aortaP6 pressure in right heart, superior vena cavaP7 pressure in thoracic pump and left heart.
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
The chosen CPR model consists of seven difference equations, withtime as the discrete underlying variable.At the step n, when time is n∆t , the pressure vector is denoted by:
P(n) = (P1(n),P2(n), ...,P7(n)).
We assume that the initial pressure values are known, when n = 0.To make the chest pressure profiles medically reasonable, assumei.e., ui (0) = ui (N − 1).
u1 = (u1(0), u1(1), ..., u1(N − 2), u1(0)),
u2 = (u2(0), u2(1), ..., u2(N − 2), u2(0)),
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Difference Equations Model
for n = 1, 2, ...,N − 1 (in vector notation)
P(1) = P(0) + T1(u1(0)) + T2(u2(0)) + ∆tF (P(0)), (1)
P(n + 1) = P(n) + T1(u1(n) − u1(n − 1)) (2)
+T2(u2(n) − u2(n − 1)) + ∆tF (P(n)), (3)
T1(u1(n)) = (0, 0, 0, 0, tpu1(n), tpu1(n), u1(n)),
T2(u2(n)) = (u2(n), u2(n), 0, 0, 0, 0, 0).
Interactions between compartments in function F
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Show function F (P(n)) by some of its seven components:
1
cjug
[
1
Rh
(P3(n) − P4(n)) −1
RjV (P4(n) − P6(n))
]
1
cao
[
1
Ro
V (P7(n) − P5(n)) −1
Rc
(P5(n) − P3(n))
]
+1
Ra(P5(n) − P1(n)) −
1
Rht
V (P5(n) − P6(n))
]
where the valve function is defined byV (s) = s if s ≥ 0V (s) = 0 if s ≤ 0.
Three valves: between compartments 4 - 6, 5 - 7, and 5 - 6.
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Goal
Choose the control set U ⊂ ℜ2N , defined as:
U = (u1, u2)|ui (0) = ui(N − 1)
−Ki ≤ ui (n) ≤ Li , i = 1, 2, n = 0, 1, . . . ,N − 2.
NEED positive and negative values due to compression anddecompression!
We define the objective functional J(u1, u2) to be maximized
N∑
n=1
[P5(n) − P6(n)] −
N−2∑
n=0
[B1
2u21(n) +
B2
2u22(n)] (4)
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Key Feature
The calculation of the pressures at the next time step requires thevalues of the controls at the current and previous time steps. Weuse extension of the discrete version of PMP.
Use derivative of the map from controls-to-states to form thesensitivity equations. Use the sensitivity operator and the theobjective functional to find the adjoint system.
Use the adjoint system to simplify the quotient below and obtainOC characterizations
0 ≥ limǫ→0+
J(u∗ + ǫl)− J(u∗)
ǫ
Numerical method: iterative method with forward-backward sweeps
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Lifestick
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Optimal Controls for Lifestick
0 2 4 6 8 10 12
−20
0
20
40
60
Time (s)
Che
st C
ontr
ol (
mm
Hg)
0 2 4 6 8 10 12
−20
0
20
40
60
80
100
120
Time (s)
Abd
omin
al C
ontr
ol (
mm
Hg)
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Concluding Remarks about CPR
We can increase the pressure difference across the thoracic aortaand the right heart by about 25 percent.
We received a US government patent for the idea of optimalcontrol of CPR models. (Vladimir Protopopescu and Eunok Jung)
This procedure with RAPID compression and decompression cycleshas recently been recommended by several medical groups. (use adevice)
Reference: IMA Journal Mathematical Medicine and Biology 25,2008, 157-170.
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Optimal Control of PDEs
There is no complete generalization of Pontryagin’s MaximumPrinciple in the optimal control of PDEs.
After setting up a PDE with a control in a specifed set and anobjective functional, proving existence of an optimal control is afirst step.
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Necessary Conditions
To derive the necessary conditions, we need to differentiate themap
control → objective functional
Note that the state contributes to the objective functional, so wealso must differentiate the map
control → state
The “sensitivity” is the derivative of the control-to-state map. Thesensitivity solves a PDE, which is linearized version of the statePDE.
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
How to Find and Use the Adjoint Function
The formal adjoint of the operator in the sensitivity PDE is found.
Transversality Condition: final time condition λ = 0 at t = T
Nonhomogeneous term
∂integrand of J
∂state
Differentiate the objective functional J(control) with respect tothe control.
Use the adjoint problem and the sensitivity problem to simplify andobtain the explicit characterization of an optimal control.
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Example 2 - Rabies in Raccoons
Rabies is a common viral disease.Transmission occurs through the bite of an infected animal.Raccoons are the primary terrestrial vector for rabies in theeastern US.Vaccine is distributed through food baits.(preventative)Medical and Economic Problem -death to humans andlivestock and COSTS.
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Epidemic Model - S susceptibles, I infecteds and R immune
L1S = b(S + R) − µ1S − βSI − avS ,
L2I = βSI − µ2I ,
L3R = −µ1R + avS ,
Lku ≡∂u
∂t−
2∑
i ,j=1
(akij(x)uxi
)xj+
2∑
i=1
(bki (x)(u))xi
With initial conditions and no-flux boundary conditions onQ = Ω × (0,T )
References: R. Miller Neilan and Lenhart, preprint, Ding et al. J.Bio. Dynamics 2007, Asano et al. Math Biosc. Eng. 2008
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Optimal Control Problem
Define the class of admissible controls as
V = v ∈ L∞(Q) |v : Q → [0, vmax].
Optimal Control Problem: Characterize the optimal vaccinationv∗(x , t) ∈ V which minimizes the number of infected raccoons andthe costs of vaccination.
J(v∗) = minv∈V
∫
Q
(
I (x , t) + Cv(x , t)2)
dxdt
subject to state equations with initial and boundary conditions.
Key Feature
Inclusion of detailed spatial-temporal heterogeneous environmentfor the distribution of vaccine baits.
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Numerical Results -Infecteds, Domain with River, Forestand Urban Areas
Figure: Constraints on movement due to environment, and control startsat t = 21
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Numerical Results -Control Vaccine Baits
Figure: Put baits in the best location to reduce the spread of disease andminimize cost
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Example 3 - Integro-difference Models for Dispersal,Growth and Harvest
Integro-difference equations are discrete-time models thatpossess many of the attributes of continuous-timereaction-diffusion equations. They arise naturally inpopulation biology as models for organisms with discretegenerations and well-def. growth and dispersal stages (Kot,JMB 1992, Neubert, Kot, Lewis TPB 1995).
Integro-difference equations do a better job of estimatinginvasion speeds, Mark Kot, Do Invading Organisms Do theWave? Canadian Applied Math Quarterly, 2003
For populations with continuous spatial dispersal and discrete
generations, like insects, birds, certain plants
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Growth, Dispersal and Harvest Model
Integro-difference model for density of the population:
Nt+1(x) = (1 − αt(x))
∫
Ω
k(x , y)f (Nt(y), y) dy ,
where t = 0, 1, . . . ,T − 1. The state variable N and the harvestcontrol α:
N = N(α) = (N0(x),N1(x), . . . ,NT (x))
α = (α0(x), α1(x), . . . , αT−1(x)) ,
where N0(x) is known at start.
NOTICE the order... growth f , dispersal, and harvest!
Different order: growth, harvest and then dispersal
Nt+1(x) =
∫
Ω
k(x , y)(1 − αt(y))f (Nt(y), y)dy (5)
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Various Stages
Space
Space
Space
GrowthSedentary Stage
Dispersal Stage
t (x) tN (y)
f(Nt (x)) f(Nt (y))
k(x,y)
N
k(x,x)
Nt+1(x) =∫
Ωk(x , y)f (Nt(y)) dy
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Maximize the Profit, with order - growth, dispersal andharvest
The objective functional (revenue less cost) is
J(α) =
T−1∑
t=0
∫
Ω
[
Ate−δtαt(x)
∫
Ω
k(x , y)f (Nt(y), y)dy − BtV (αt(x))
]
dx
where N = N(α), At price factor
e−δt discount factor, f concave, V convex
Money now is worth more than money later.
Reference: Joshi, Gaff, and Lenhart, Optimal Control Applications andMethods, 2006, Zhong and Lenhart, Systems Theory Proceeding Volume,2009.
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Hybrid Model
Optimal Control of this hybrid model combines discrete PontryaginMaximum Principle and techniques for continuous spatial variables(like control of PDE’s).
This combination of techniques is novel. Our results on thisharvesting system are the first results on optimal control for thistype of hybrid system.
Key Feature
Order of events is crucial due to discrete time growth, dispersal,and harvest. We use a mixture of discrete and continuous controlanalysis techiques
Used four types of kernels, different growth functions, and order ofevents
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Harvest Rate for Finite Range Kernel
0.00 0.33
0.67 1.00
Space0
2
4
68
Time
0.00
0.05
0.10
0.15
0.20
0.25
0.30
Har
vest
Figure: Optimal spatial temporal harvest to maximize profit
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Example 4 - Fishery Model
Neubert (Ecology Letters, 2003)
Examined steady state of this population PDE
Nt = DNxx + rN
(
1 −N
K
)
− qE (x)N
Population spreads out by diffusion and has logistic growth andharvest
Steady state
0 = DNxx + rN
(
1 −N
K
)
− qE (x)N, 0 < x < L.
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Example 4 - Fishery Model
No-take marine reserves may be a part of optimal harveststrategy designed to maximize yield.
Marine reserves can protect habitat and defend endangeredstock from overexploitation.
The establishment of marine reserves is controversial in fisherymanagement.
Marine reserves = areas where harvesting is not allowed
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Neubert’s work
rescaled equation u′′ + u(1 − u) − h(x)u = 0
u = 0 at the boundary x = 0 and x = L
control h
max yield∫ L
0h(x)u(x)dx
Depending on length of domain, marine reserves are part ofoptimal harvesting strategy.
For large length, there are many intervals of noharvest(reserve), leading to ‘chattering’ (rapid changes up anddown).
For small length, there is one reserve in the middle.
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Convert to First Order System and Use PMP
u′′ + u(1 − u) − h(x)u = 0
Let w = u′ and then you obtain with control h
u′ = w
w ′ = u′′ = −u(1 − u) + hu
0 LHarvest Reserve
No harvestHarvest
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Parabolic Case
This work extends the work of Neubert
includes both time and space
multi-dimensional spatial domain
non-steady state
Key Feature
The presence of marine reserves in harvesting strategy arising outof the maximizing yield problem.
We have a complete rigorous analysis of bang-bang control case inPDE model with a surprising solution!
We have completed the analysis for general semilinear parabolicPDE in a multidim.domain. Here we present a simpler case.
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Parabolic Fishery Model
Our fishery model in domain Q = Ω × (0,T ) with Ω ⊂ Rn is :
ut = ∆u + u(1 − u) − hu in Q (6)
with initial and boundary conditions:
u(x , 0) = u0(x) for x ∈ Ω
u(x , t) = 0 on ∂Ω × (0,T )
u represents fish population
h represents CONTROL, the proportion to be harvested
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Goal
We seek to maximize the objective functional over h ∈ U:
J(h) =
∫ T
0
∫
Ω
e−δthu dx dt (7)
where U = h ∈ L∞(Q) : 0 ≤ h(x , t) ≤ M ≤ 1 is class of controlsand e−δt represents a discount factor with interest rate δ.
This problem is linear in the control. Expect bang-bang or singularoptimal control.
References: Joshi, Herrera, Lenhart, Neubert, Natural ResourceModeling, 2009 and Ding and Lenhart, NRM 2009
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Existence of an Optimal Control
Solution space u in V = L2(O,T ,H10 (Ω)) with
ut in L2(0,T ;H−1(Ω)) Note u > 0 in Q.
Theorem
There exists an optimal control h∗ maximizing the functional J(h)over U.
idea of proof
Choose a maximizing sequence hn in U.
Use apriori estimates.
Use weak convergence results.
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Derivation of Optimality System
Theorem
The mapping h → u = u(h) is differentiable in the following sense:
u(h + ǫl) − u(h)
ǫ ψ
weakly in V as ǫ→ 0 for any h ∈ U and l ∈ L∞(Q) s.t.
(h + ǫl) ∈ U for ǫ small. The sensitivity ψ satisfies:
ψt = ∆ψ + ψ − 2uψ − hψ − lu in Q
ψ(x , 0) = 0 for x ∈ Ωψ(x , t) = 0 on ∂Ω × (0,T )
(8)
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Characterization of Optimal Control
Theorem
Given an optimal control h∗ and corresponding solution u∗ = u(h∗)there exists a weak solution λ with satisfying the adjoint equation:
−λt − ∆λ− λ+ 2u∗λ+ h∗λ+ δλ = h∗ in Q
λ(x , t) = 0 on ∂Ω × (0,T ).(9)
and transversality condition λ(x ,T ) = 0 for x ∈ Ω.
And furthermore the characterization of an OC:
h∗(x , t) =
0 if λ(x , t) > 11+δ2
if λ(x , t) = 1M if λ(x , t) < 1
(10)
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Optimal Control for Unexploited Stock Initial Condition
Figure: Final time 4, length of domain 4, discount .2
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Optimal Control for Different Initial Conditions
Figure: Left IC -unexploited stock, Right IC -overexploited stock (needmore protection)
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Optimal Control for Different Discount Rates with ICUnexploited Stock
Figure: Obtain a marine reserve in the interior of the domain , whilemaximizing yield!
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Conclusions
Key Features
1. CPR - two previous time steps feeding into the state at next step
2. Rabies - spatial and temporal control in heterogeneousenvironment
3. Integro-difference - discrete and continuous features andORDER of events
4. Fishery PDE - bang-bang control with interesting outcome!
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Collaborators
1.CPR: Vladimir Protopopescu (ORNL), Eunok Jung (Konkuk U.,S. Korea), Charlie Babbs (Purdue U.)
2. Rabies in Raccoons: Rachel Miller Neilan (LSU), Wandi Ding(Middle Tenn. State U.), Les Real (Emory), Erika Asano (U. of S.Florida, St. Petersburg), Lou Gross (UT)
3. Integro-difference Equations: Peng Zhong (UT), Hem Raj Joshi(Xavier U.), Holly Gaff (Old Dominion U.), Hongwei Lou (Fudan)
4. Fishery Models: Hem Raj Joshi (Xavier U.), Mike Neubert(Wood’s Hole), Ta Herrera (Bowdoin), Wandi Ding (UT)
Mixing it up: Discrete and Continuous Optimal Control for Biological Models
Upcoming OC Talks and Poster