22
Structure and Synthesis of Robot Motion Introduction to Optimal Control Subramanian Ramamoorthy School of Informatics 12 February, 2009
Structure and Synthesis of Robot Motion
Introduction to Optimal Control
Subramanian RamamoorthySchool of Informatics
12 February, 2009
In this Lecture
• We will begin with basic notions of extrema of functions and its variational equivalents
• Then we will pose the basic optimal control problem
• And look at a few versions of this problem
Note/Warning: This lecture is drawn from many sources and there may be notational inconsistencies between parts (e.g., I use the prime symbol for derivative and transpose at various places) – please use context to disambiguate!
12/02/2009 Structure and Synthesis of Robot Motion 2
Recap from Calculus: Extrema of Function
12/02/2009 Structure and Synthesis of Robot Motion 3
Similar Question for Paths
12/02/2009 Structure and Synthesis of Robot Motion 4
A More Interesting Question about Paths
12/02/2009 Structure and Synthesis of Robot Motion 5
What is the Derivative of a Functional?
12/02/2009 Structure and Synthesis of Robot Motion 6
Functional maps y to a real value
Functional maps z to delta I
Moving Towards Solution of the Variational Problem
12/02/2009 Structure and Synthesis of Robot Motion 7
Vanishes at end points
Solving the Brachistochrone Problem
12/02/2009 Structure and Synthesis of Robot Motion 8
A ball will rolldown the cycloid faster than the straight line!
The Optimal Control Problem
• Given a dynamical system with states and controls
• Find a policy or sequence of control actions u(t) up to some final time
• Forcing the state to go from an initial value to a final value
• While minimizing a specified cost function
The resulting state trajectory x(t) is an optimal trajectory
Remarks:
• Certain combinations of cost functions and dynamical systems yield analytical solutions
• Often, the control policy can be described as a feedback function or control law
12/02/2009 Structure and Synthesis of Robot Motion 9
The Optimal Control Problem
12/02/2009 Structure and Synthesis of Robot Motion 10
The Optimal Control Problem
12/02/2009 Structure and Synthesis of Robot Motion 11
Necessary Conditions for Optimality
12/02/2009 Structure and Synthesis of Robot Motion 12
Necessary Conditions for Optimality
12/02/2009 Structure and Synthesis of Robot Motion 13
Necessary Conditions for Optimality
12/02/2009 Structure and Synthesis of Robot Motion 14
Equivalent to Euler-Lagrange equations,although this is fordynamic optimization
= 0
Sufficient Conditions for Optimality
12/02/2009 Structure and Synthesis of Robot Motion 15
This is a local condition – only true on the optimal trajectory.In nonlinear systems, may be untrue with large deviations.
We will now look at an alternate approach to sufficiency…
Hamilton-Jacobi-Bellman Equation
12/02/2009 Structure and Synthesis of Robot Motion 16
Hamilton-Jacobi-Bellman Equation
12/02/2009 Structure and Synthesis of Robot Motion 17
Understanding the HJB Equation
• It is a sufficient condition for optimality– Not a necessary condition
– i.e., there may be value functions that do not have the differentiability properties to be a solution of HJB equation but are still optimal
• Semantics of value function:– It is a hypersurface of minimum cost in the possible state space
within the given time interval
– Each entry for the value function is associated with a specific optimal trajectory and control sequence – an extremalcorresponding to the initial condition
12/02/2009 Structure and Synthesis of Robot Motion 18
The Value Function
12/02/2009 Structure and Synthesis of Robot Motion 19
The Linear Quadratic Regulator
12/02/2009 Structure and Synthesis of Robot Motion 20
Can we Deal with Task Constraints?
12/02/2009 Structure and Synthesis of Robot Motion 21
Summary
• This lecture presents an overview of the core of optimal control theory
– In the next lecture, we will look more closely at how these equations can be numerically solved, with examples
• There are many more extensions that we can’t cover in a single lecture:
– Stochastic version of optimal control (same idea but need to bring in expectations)
• Closely related to reinforcement learning
• Becomes much harder in continuous time
12/02/2009 Structure and Synthesis of Robot Motion 22
