1
Adaptive Value Estimate Updates in Generalized Policy Iteration Dynamic Programming for Reinforcement Learning John Ahmed Dellas 1 , Joseph Anderson, PhD 2 1 Rutgers University-NB, 2 Salisbury University John Ahmed Dellas Rutgers University-New Brunswick [email protected] 609-608-2057 Contact 1. F. Wang, H. Zhang and D. Liu, "Adaptive Dynamic Programming: An Introduction," in IEEE Computational Intelligence Magazine, vol. 4, no. 2, pp. 39-47, May 2009. 2. Sutton and Barto, “Reinforcement Learning: An Introduction”, 2 nd Ed References Most modern “adaptive” methods of accelerating traditional dynamic programming policy iteration algorithms rely on using a neural net to approximate the value function [1]. In this proposal, we instead focus on accelerating DP by adaptively selecting the order in which state value estimates are updated using other reinforcement learning algorithms so as to minimize convergence time. More specifically, we tackle the classical DP-RL problem of finding the best policy iteration strategy by exploring the use of a Monte Carlo Reinforcement Learner to accelerate the convergence of Dynamic Programming Policy Iteration algorithm. The MC Learner is tasked with finding the optimal “sweep” through the state space during policy evaluation. We analyze the shortcomings of using an MC method for this purpose as originally proposed and present alternatives to extending the effectiveness of a crude proof-of-concept algorithm implemented and tested on a classical DP-RL problem. Abstract 1. Bellman equation for the state value function. 4. Left: Random policy after 1 iteration. Right: Optimal policy after convergence. Introduction Reinforcement Learning refers to a set of algorithms, techniques and methods used to train some agent to accomplish a goal-driven task by having the agent maximize a reward that it receives by interacting with its environment. This interaction is typically represented by 3 signals which pass between agent and environment: state, action and reward. The state signal can be thought of as an indication of where the agent is in the set of all possible arrangements of its environment (a state in a game of chess might refer to an arrangement of the pieces on the board) that can be reached by performing some actions (the second signal, passed from agent to environment). The agent accomplishes its task by following a policy. A policy, denoted using the lowercase pi, is a mapping from the set of all actions (referred to as the action space) to probabilities that the agent selects that action given that it is currently at a particular state. More specifically, the goal of most RL algorithms to obtain the optimal policy (optimality in general is denoted with a subscript “*”), which is the only policy that is deterministically greedy with respect to its value function. There are two flavors of value functions: state value functions (denoted with a “v”), and action value functions (denoted with a “q”). A state value function of a policy gives the expected return by starting at a particular state and following the policy associated with it thereafter (action value functions are identical except that instead we start at a state-action pair). Return is defined as cumulative function of the rewards experienced after a time step t, and is typically denoted G t . The value functions of the optimal policy are called the optimal value functions, and are the unique functions which satisfy the Bellman Optimality equations, which are just the Bellman equations for the optimal policy (and can therefor be written in a special form without reference to any specific policy). The Bellman equation for the state value function (1) gives the value of one state in terms of the values of its successor states. Dynamic Programming, as applied to RL, refers to a collection of algorithms used for solving RL problems when the dynamics of the environment are completely known. The most popular DP-RL algorithm, policy iteration, is the focus of this research. Policy Iteration (outlined in (2)) works by turning the Bellman equation into an update rule and exploiting the fact that iterating value function estimates produced by the rule will converge to the true value function of a given policy at infinity. Standard policy iteration has two main components: policy evaluation and policy improvement. Policy evaluation is the calculation of the value function of a policy given as input. Policy improvement is the creation of a new policy that is greedy with respect to the newly computed value function to produce a better policy which can then be fed back into policy evaluation until the optimal policy is produced. Policy comparison is defined as: Note that the order in which state values are updated during policy iteration significantly influences the rate of convergence of the algorithm [2]. Generalized policy iteration considers versions of policy iteration in which state value updates are asynchronous and intermixed with policy improvements. • Traditional implementation of DP-PI has arbitrary state space sweep. • However, sweep selection can significantly influence convergence. • The task of selecting sweeps through the state space during DP policy evaluation can be modeled as a FMDP! • States: States of the underlying MDP • Actions: The decision of which state to evaluate next • Rewards: The sum of the differences between the previous state value estimate and the new state value estimate across estimates which have yet to fall beneath the threshold • Discounting Rate: gamma = 1 (episodic) • An episode in the task of finding the best sweep would be n policy evaluations before episode termination. Using a “supervising” reinforcement learning algorithm, we can intelligently select which states we choose to update next, and reward update sweeps that maximize the rate of convergence of the underlying DP algorithm. Originally, we intended to implement this supervisor with a Monte Carlo Learning method. However, it became clear with time that a MC method would be inappropriate, and was dropped in favor of a Temporal Difference learning method based approach. Performance Evaluation Improvements and Conclusions Approach 2. Right: A set of illustrations of Generalized policy iteration In action 3. Top: Standard Policy Iteration Right: Value Iteration, an example of GPI. Used in Jack’s Car Rental Prob. . • Memory Complexity is not sustainable for problems of any appreciable size • Look into non-tabularized, approximation based methods for value function representation, such as neural nets. • DP sweeps are not an inherently episodic task • MC methods are most appropriate for episodic tasks • Needing to arbitrarily stop DP execution to perform the backwards computation required for MC action value updates is both silly and slows the convergence of the algorithm • Using an on-line learning algorithm such as TD Learning allows for experiential learning with bootstrapping (no costly backwards return computations like in MC methods). • Crude implementation for Jack’s Car Rental problem does not allow for experimentation with multiple policy evaluation steps before a policy improvement • Full implementation would require more complex reward signal • Eventually, we will tackle more interesting RL problems • Specifically problems with asymmetric action spaces We rely on a classical DP problem known as Jack’s Car Rental problem to evaluate crude adaptive sweep performance against traditional arbitrary sweeps. • Max Number of Cars at each location: 5 • 10$ for rental, -1$ for transportation • Max Number of cars transported overnight: 2 • Discounting Rate: 0.9 • Car transactions per day modeled by Poisson random variables (# of cars returned at location 1: mean 3; # of cars rented at location 1 and returned at location 2: mean 1; # of cars rented at location 2: mean 4)
![Page 1: ] À s o µ ] u h ] v ' v o ] Ì W } o ] Ç / ] } v Ç v u ] W ...faculty.salisbury.edu/~ealu/REU/Projects_File/2018/... · ] À s o µ ] u h ] v ' v o ] Ì W } o ] Ç / ] } v Ç](https://reader034.documents.pub/reader034/viewer/2022042712/5f9732840bda4b422f7d832b/html5/thumbnails/1.jpg)
![o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }](https://static.documents.pub/doc/80x56/6117f9be3785f80bcd0d85f2/o-e-v-o-oe-u-v-v-d-z-v-2021-3-5-d-z-o-u-z-o-oe-v-p-.jpg)
![v o Ç ] } ( Z o } } µ ] v v / D } o ] Z Ç o Z o } } µ ] v Ç > …...v o Ç ] } ( Z o } } µ ] v v / D } o ] Z Ç o Z o } } µ ] v Ç > rD^ lD^ µ ] v P < ] v Æ } r^](https://static.documents.pub/doc/80x56/5f09ae7f7e708231d42801b6/v-o-z-o-v-v-d-o-z-o-z-o-v-v-o-.jpg)
![t Z P Ç Á Z ( } P } o ( } v & ] Ç U } o Ç } v } ( Z Ç } o ...](https://static.documents.pub/doc/80x56/616c3c1302fadb50aa4a3d07/t-z-p-z-p-o-v-amp-u-o-v-z-.jpg)
![Learn Angular 2 · d Z Á Ç } o v v Ç v Á Z v } o } P Ç ] Ç ] v P } i X ^ } ] v Z ] Ç ]](https://static.documents.pub/doc/80x56/5b505fb97f8b9a2f6e8e5af3/learn-angular-2-d-z-a-c-o-v-v-c-v-a-z-v-o-p-c-c-v-p-i-x-.jpg)
![} v Ç ] o Ç d ì ï ( } z o z í î Z zD ] o } v Ç ] o Ç d ì ð ......} v Ç ] o Ç d ì ï ( } z o z í î Z zD ] o } v Ç ] o Ç d ì ð ... ... 3, , 3d, , &](https://static.documents.pub/doc/80x56/60358cc77dbdfe67ff50c665/-v-o-d-z-o-z-z-zd-o-v-o-d-v.jpg)
![2008 Annual Report - s2.q4cdn.com · July 2008 } v À u ] v ] v P Ç } µ } o v U v Á o v P Ç Minimize environmental ] u Ç } µ } o o } v v recycling Reduce u ] ] } v Ç µ µ](https://static.documents.pub/doc/80x56/6036ddc33def47125e58e486/2008-annual-report-s2q4cdncom-july-2008-v-u-v-v-p-o-v-u-v.jpg)
![] v P ] Ç U ] P v ] Ç X ^ Z } Á Ç } µ o ( ] v o o } u } o ... · & µ Ç } u u µ v ] } v W } v vd Ç } v v d } v Ç ^ Á ] P o } ( Z t l W } v v d v v ~ Z } u ô ñ õ r ï](https://static.documents.pub/doc/80x56/5ecbd29a6561874566511c26/-v-p-u-p-v-x-z-o-v-o-o-u-o-.jpg)
![Stroud District Playing Pitch Strategy Final Strategy June ... · ^ } µ ] ] W o Ç ] v P W ] Z ^ P Ç W & ] v o ^ P Ç ^ } µ ] ] W o Ç ] v P W ] Z ^ P Ç W & ] v o ^ P Ç](https://static.documents.pub/doc/80x56/5f4d47452b8d3e336c3a6031/stroud-district-playing-pitch-strategy-final-strategy-june-w-o-.jpg)
![u o , ] P Z À ] o ] o ] Ç ( } v Ç o ] ] } v ] v o } µ } K ... · w z Ç ] o u s ] µ o u o } µ u } v Ç } u ] v ] } v o ] ] } v z } À Ç < ] ~ z< z } À Ç k z ] } v z µ v](https://static.documents.pub/doc/80x56/5f50de8d05566a226e0f2db7/u-o-p-z-o-o-v-o-v-v-o-k-w-z-o-u-s.jpg)
![W> ^ WZ/Ed W ] v , ] } Ç Y µ ] } v v ] W Z } } z z z z z z ... · ,/W W ] À Ç W } o ] Ç / l v } Á o P Z u Ç u ] o Z ] } Ç u Ç } } v o ] v Z } µ P Z D o o Ç [ } u µ Ç](https://static.documents.pub/doc/80x56/5f6bba30ba82fe634d732382/w-wzed-w-v-y-v-v-w-z-z-z-z-z-z-z-w-w-.jpg)
![Settlement and Locality Names - calico.scot · Z ] v } µ ( o Ç ( } Ç o o Ç o } µ v Z Ç](https://static.documents.pub/doc/80x56/6018930bde431d576d0a73fa/settlement-and-locality-names-z-v-o-o-o-o-v-z-.jpg)
![Registration System.pdfKE>/E Z '/^dZ dKE ó ñ X } v ] v µ } ] o } ( Z o v ~ ] ( v Ç X W µ o ] l Ç ~< Ç ] o } ( Z o v ] } ] } v o X / ( o v P v µ o ] l Ç U o } v ] v µ](https://static.documents.pub/doc/80x56/60827636c939660b7719de14/registration-kee-z-dz-dke-x-v-v-o-z-o-v-v-.jpg)
![Rwanda W µ Ç ] v ] v µ } v o µ } v ] o ] v€¦ · RES-Research Studies Z ] o ] v ] v µ } v ^ Ç u W Z Á v ^ µ Ç Rwanda W µ Ç ] v ] v µ } v o µ } v ] o ] v](https://static.documents.pub/doc/80x56/605e87e7e0d342534224262d/rwanda-w-v-v-v-o-v-o-v-res-research-studies-z-o-v-.jpg)
![Z ] v P K ] } v ( Ks/ r í õ > } l r } Á v d Z d } Ç } t Ç ... · ^ ] v P & } u , } u d } t } l, o Z ^ ( Ç } u v Ç, o Z ^ ( Ç & ] o ] ] , o Z v ^ ( Ç ( Z Z ] v P , } u í](https://static.documents.pub/doc/80x56/6024fc28dd056f62ac65bbd6/z-v-p-k-v-ks-r-l-r-v-d-z-d-t-v-p-.jpg)
![E µ v P D ] o ] Ç } u u µ v ] Ç ] } Ç o v P v](https://static.documents.pub/doc/80x56/6255b6575a6e0840a9430ef5/e-v-p-d-o-u-u-v-o-v-p-v.jpg)
![Z } v ] v P Z } } o t h v v ] v P > ] ] o ] ] U Z } v ] ] o ] ] v W ] o ] ] · 2020-06-15 · • o ] v ] o o Ç Æ u o Ç À µ o v o • o ] v ] o o Ç À µ o v o •> ] À ] v](https://static.documents.pub/doc/80x56/5f2b2a8118f6f166586134f4/z-v-v-p-z-o-t-h-v-v-v-p-o-u-z-v-o-v-w-o-2020-06-15.jpg)
![BTSG Green Bond at Towards the Sustaibable …...^ µ ] v ] o ] Ç d^ ' } µ K µ Z } Á µ ] v ] o ] Ç d^ ' } µ ' v } v W v ] } v ñ ^ } } ^ µ ] v ] o ] Ç W } o ] Ç _ ] v o (](https://static.documents.pub/doc/80x56/5fcd998c78d6137a2f250635/btsg-green-bond-at-towards-the-sustaibable-v-o-d-k-.jpg)