Post on 03-Jan-2016
transcript
Solving Large Markov Decision Processes
Yilan Gu
Dept. of Computer Science
University of Toronto
April 12, 2004
2
Outline
• Introduction: what’s the problem ?• Temporal abstraction • Logical representation of MDPs• Potential future directions
3
Markov Decision Processes (MDPs)• Decision-theoretic planning and learning problems are often
modeled in MDPs.• An MDP is a model M = < S, A, T, R > consisting
a set of environment states S, a set of actions A, a transition function T: S A S [0,1]
T(s,a,s’) = Pr (s’| s,a), a reward function R: S A R .
• A policy is a function : S A.• Expected cumulative reward -- value function V: S R . The Bellman Eq.: V(s) = R(s, (s)) + s’T(s, (s),s’) V(s’)
4
MDP Example
S = {(1,1), (1,2), …,(8,8)}
A = {up, down, left, right}
e.g.,
T((2,2),up,(1,2)) = 0.8, T((2,2),up,(2,1))=0.1,
T((2,2),up,(2,3)) = 0.1,
T((2,2),up,s’) = 0 for s’(1,2), (2,1), (2,3)
……
R((1,8)) = 1, R(s)= -1 for s (1,8).
Fig. The 8*8 grid world
1 2 3 4 5 6 7 8
0.1 0.1
up0.8
(2,2)
Notice: explicit representation of the Notice: explicit representation of the modelmodel
1 2 3 4 5 6 7 8
+1
5
Conventional Solution Algorithms for MDPs
• Goal: looking for optimal policy * so that V*(s) = V*(s) V(s) for all sS and • Conventional algorithms
Dynamic programming: value iteration and policy iteration, Decision tree search algorithm, etc. Example: Value iteration
Beginning with arbitrary V0; In each iteration n>0: for every s S ,
Qn(s,a) := R(s, a) + s’T(s, a, s’) Vn-1 (s’) for any a;
Vn(s) := max a Qn(s,a) ;
When n , Vn(s) V*(s).• Problem: it does not scale up!
6
Solving Large MDPs (Part I)
• Temporal abstraction approaches (basic idea) Solving MDPs hierarchically Using complex actions or subtasks to
compress the scales of the state spaces
• Representing and solving MDPs in a logical way (basic idea) Logically representing environment features Aggregating ‘similar’ states Representing effect of actions compactly by using logical structures,
and eliminating unaffected features during reasoning
7
Options (Macro-Actions)
Partition {S1 , S2 , S3 , S4 } A macro-action -- a local policy i : Si A on region Si
E.g., EPer (S1) = {(3,5),(5,3)} Discounted transition model Ti: Si {i} Eper(Si) [0,1] Discounted reward model Ri: Si {i} R
Example
1 2 3 4 5 6 7 8
S1
1 2 3 4 5 6 7 8
8
Abstract MDP M’= < S’, A’, T’, R’>
• S ’= Eper(Si ), e.g., {(4,3),(3,4),(5,3),(3,5),(6,4),(4,6), (5,6),(6,5)} .
• A’= Ai , where Ai is a set of macro-actions on region Si .
• Transition model T’: S ’ A’ S ’ [0,1] T’(s, i, s’) = Ti(s, i,s’) if s Si , s’ Eper(Si ); T’(s, i, s’) = 0 otherwise.
• Reward model R’: S ’ A’ R R’(s, i ) = Ri(s, i ) for any s’ Eper(Si ).
9
Other Temporal Abstraction Approaches
• Options [Sutton 1995; Singh, Sutton and Precup 1999] Macro-actions [Hauskrecht et al. 1998; Parr 1998]
– Fixed policies
• Hierarchical abstract machines (HAMs) [Parr and Russell 1997; Andre and Russell 2001,2002]– Finite controllers
• MAXQ methods [Dietterich 1998, 2000]– Goal-oriented subtasks
• Etc.
10
Solving Large MDPs (Part II)
• Temporal abstraction approaches (basic idea) Solving MDPs hierarchically Using complex actions or subtasks to compress the scale of the
state spaces
• Representing and solving MDPs logically (basic idea) Logically representing environment features Aggregating ‘similar’ states Representing effect of actions compactly by using
logical structures, and eliminating unaffected features during reasoning
11
First-Order MDPs
• Using the stochastic situation calculus to model decision-theoretic planning problems
• Underlying model : first-order MDPs (FOMDPs)
• Solving FOMDPs using symbolic dynamic programming
12
Stochastic Situation Calculus (I)
• Using choice axioms to specify possible outcomes ni(x) of
any stochastic action a(x) Example: choice(delCoff(x),a) a = delCoffS(x) a = delCoffF(x)
• Situations: S0 , do(a,s)
• Fluents F(x,s) – modeling environment features compactly
Examples: office(x,s), coffeeReq(x,s), holdingCoffee(s)
• Basic action theory: Using successor state axioms to describe the effect of
the actions’ outcomes on each fluent coffeeReq(x,do(a,s)) coffeeReq(x,s) a = delCoffS(x)
13
Stochastic Situation Calculus (II)
• Asserting probabilities (may be depended on conditions of
current situation
Example: prob(delCoffS(x), delCoff(x), s) = case [hot, 0.9; hot, 0.7]
• Specifying rewards/costs conditionally
Example: R(do(a,s)) =
case[ x. a = delCoffS(x) , 10; x. a = delCoffS(x) , 0]
• stGolog programs, policies
proc (x)
if holdingCoffee then getCoffee else (
?(coffeeReq(x)) ; delCoffee(x))
end proc
14
Symbolic Dynamic Programming
• Representing value function Vn-1(s) logically
case [1 (s) ,v1 ; … ; m (s) ,vm ]
• Input: the system described in stochastic SitCal and Vn-1(s)
• Output (also in case format): – Q-functions
Q n(a(x),s)= R(s)+ i prob(ni(x),a(x),s) Vn-1(do(ni(x),s))
– Value function Vn(s)
Vn(s) = ( a)( b) Q n(a,s) Q n(b,s)
15
Other Logical Representations
• First-order MDPs [e.g., Boutilier et al. 2000; Boutilier, Reiter and Price 2001]
• Factored MDPs [e.g., Boutilier and Dearden 1994, Boutilier, Dearden and Goldszmit 1995; Hoey et al. 1999]
• Relational MDPs [e.g., Guestrin et al. 2003]
• Integrated Bayesian Agent Language (IBAL) [Pfeffer 2001]
• Etc [e.g., Bacchus 1993, Poole 1995].
16
Our Attempt: Combining temporal abstraction with logical representations of MDPs.
17
Motivation
cityB
cityA
living(X, houseA)
houseB
inCity(Y,cityA)
18
Prior Work
• MAXQ approaches [Dietterich 2000] and PHAMs method [Andre and Sutton 2001]– Using variables to represent state features– Propositional representations
• Extending DTGolog with options [Ferrein, Fritz and Lakemeyer 2003]– Specifying options with the SitCal and Golog programs– Benefit: reusable when entering the exact same region– Shortage: options are based on explicit regions, and
therefore not reusable under ‘similar’ regions
19
Our Idea and Potential Directions
• Given any stGolog program (a macro-action schema) Example: proc getCoffee(X) if holdingCoffee then getCoffee else ( while coffeeReq(X) do delCoffee(X) ) end proc
• Basic Idea – inspired by macro-actions [Boutilier et al 1998]:
Analyzing the macro-action to find what has been affected by the macro-action
Example: holdingCoffee, coffeeReq(X)
Preprocessing discounted transition and reward models Example: tr(holdingCoffee coffeeReq(X) , getCoffee(X),
holdingCoffee coffeeReq(X) )
20
(Continue)
using and re-using macro-actions as primitive actions
• Benefit: Schematic Free variables in the macro-actions can represent a
class of objects which have same characteristics Even for infinite objects Reusable in similar regions, other than the exact region
THE END
Thank you!