+ All Categories
Home > Documents > Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

Date post: 28-Feb-2022
Category:
Upload: others
View: 5 times
Download: 0 times
Share this document with a friend
30
Transcript
Page 1: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

����������� ���� �������������������� �! #"$��� �%� &('��#"*) +�,-�.�/�0�1"*) ) ���� �!������2� 34) 5768��9" �0�:�0�;34"$5.����=<>"?�:��3 ���@�A%�+�B�

,-�.�*C����@)

DFEHGJILKNMPO�ERQTS�QUEAVWEYX�E[Z\M]>E[K^K^I_O-`�S�Q/acb0I_ded2f

g�ILVWE[KhGJijI_O.GkaYl*m�V;ILK8EHG8Mea�O npopIqnhIcE[KNrtsuEYO XwvTMPO�E[O rqMPEYdyxzO|{.MPO|ILILKJMeO|{�}Q*KNMPO r~ILGJa�O���O M��.IqKNn^M�G��.}:Q*KJMeO rLI~GJa�Oy}|���������H�Y�

� I_rts O Mer_E[d�opILV�aYKhG4�k�������t�����

��� { � n�G_};�A�����

�?�:�h�����+���_�^ L¡~¢£ ¥¤?¦h§¥¨�©¥�h�+ª�+«$¦h¢£¬�ª.�_��­\�+¬®¬°¯;�~�±¢® ~²³��¨´�^ ¶µ·�+¡~§

�L©~�^ ¥�hªW¸º¹�»^¼^½¿¾hÀhÁ�ÂeÀhÃ^Ä�ÃÅ ¦ÇÆcª�¸º¹�»^¼^½¿¾hÀhÁ�ÂeÃ^Ä�¼h¹

È ��É8­\�h�2¡~��ª�¡¥Ét ~¦h«$¢£²;�~�±�h¤h�±¦h«$«$¢£ ~¤¥Ê~¢£ NËh�+ J¨´�h��Éz¨�©¥���h��É_Ê

Page 2: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

����������� �

� ©¥�*�L§¥�±�_�^���/� Å ¨�©~¢®���q¦��_���?¢£� ¨´��¢£¬£¬£§L�´¨´�±¦�¨´�/¨�©~�*¢£«$�_�h�±¨�¦h ~²³�U� Å §~��¢£ ¥¤k�´¨´�±§~²³¨�§~�±¦h¬:���+��§~¬ ¨�� ¢£ 4¡¥Ét ~¦h«$¢£²�~���h¤h�2¦h«$«$¢£ ¥¤p¦h¬ ¤h�h�±¢£¨�©~«$��µ�����²³�^ ~��¢£¡~��� ¨�©~� �~��� �L¬ �+« � Å ¦��~�~����Æt¢£«/¦�¨�¢£ ¥¤p�h�~¨�¢£«$¦h¬|�´¨´�2¦�¨´��¤^¢ �+� Å �h�T¨�©¥��L¦�¨�²2©0�´���±Ë8¢£²³�@� Å ²�§L�´¨´�^«T���±��¦�¨�¦/������Ë8¢®²³�@�´¨�¦�¨�¢ �^ ¶µ��|§~��¨´�^«T���±���´¨´�t²±©L¦h�´¨�¢£²�¦h¬£¬ É ¦����2¢ Ëh�@¦�¨W¨�©~�@�´¨�¦�¨�¢ �^ k¦h ~¡­|¦h¢ ¨B¨´���c�T�´����Ëh�+¡��A¢® ~²�§¥���±¢® ¥¤U¦z­|¦h¢ ¨�¢£ ¥¤ ²³�^�´¨�¦h ~¡�¦�´����Ët¢£²³�T²³�^�´¨�µ��8���±Ë8¢£²³�T� Å ²�§~�´¨´�^«$���±�y¢£�;�c��� Å �h�±«T�+¡¢£ k¤h���^§¥�q��� Å ¦��¥Æ8�+¡p�´���±Ë8¢£²³�T²�¦��L¦h²�¢£¨�Éhµ����$¢£ NËh�+��¨�¢ ¤^¦�¨´�T¨�©¥�T�´¨´�±§~²³¨�§~����� Å ²³�^�´¨ Å §~ ~²³¨�¢ �^ ~�;¦h ~¡p�+�´¨�¦��q¬£¢£��©�´�^«T��¨�©¥���h����¨�¢£²�¦h¬A���+��§~¬£¨���¢£ ~²�¬£§L¡~¢£ ¥¤?«T�^ ¥�h¨´�^ ~¢®²�¢ ¨�Éz� Å ¨�©¥�yË�¦h¬£§¥� Å §~ L²³¨�¢ �^ ~��µ � ©¥�+ ��¥­:��§~���y�^§¥��¦h¡L¦��~¨�¢ Ëh�¡¥Ét ~¦h«$¢£²*�~�±�h¤h�±¦h«$«$¢£ ~¤�«T�^ ¥�h¨´�^ ¥��¦h¬£¤h�h�±¢ ¨�©~«9¨�©L¦�¨/§~�´�+�U�´¨´�±§~²³¨�§¥�±� ¨´���~���+�´���±Ëh��«T�^ ¥�h¨´�^ ~¢®²�¢ ¨�É#� Å ¨�©¥��+�´¨�¢£«/¦�¨´�+�;¦�¨��+¦h²2©�¢ ¨´���±¦�¨�¢ �^ ~��¨´� ¦��~�~����Æt¢®«$¦�¨´��¨�©~�@Ë�¦h¬£§¥� Å §~ L²³¨�¢ �^ ~��µ��¥¢£ ~²³�@¨�©~���~�±� �L¬ �+«-­�¢ ¨�©0©~�^«T�h¤h�³Â ¥���^§~�:²�§~�´¨´�^«$���±�|²�¦h ��c�;�´�^¬ Ëh�+¡U�h�~¨�¢£«$¦h¬£¬£É �N­:�;©~¦�Ëh�B¦�«T�+¦h L�1� Å ²³�^«$�L¦��±¢£�´�^ U¨´� ��ËǦh¬£§L¦�¨´�;�^§~�:©¥�+§~�±¢£�´¨�¢£²�µ� ¢£ ~¦h¬£¬£É �8­:�@²³�^«T�q¦����y�^§¥�W¦h¬£¤h�h�±¢ ¨�©~« ¨´�/²�¬®¦h����¢£²�¦h¬ Å �h��­|¦��±¡ ¡¥Ét ~¦h«$¢£²��L���h¤h�±¦h«$«$¢® ¥¤T«T��¨�©¥�t¡~��µ

Page 3: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� �

�  �¨�©L¢£�y�L¦��c����­\�*¢£¡¥�+ J¨�¢ Å Ép¨�©¥�*¢£«T�c�h��¨�¦h ~²³�U� Å §~��¢£ ~¤��´¨´�±§~²³¨�§~�±¦h¬����+�±§~¬ ¨��@¢®  Å �h��­:¦��2¡�¡¥Ét ~¦h«$¢£²$�~����¤h�±¦h«$«/¢£ ¥¤B¦h¬ ¤h�h�±¢ ¨�©L«$��µ�� ¬£¦h����¢£²�¦h¬ Å �h��­|¦��±¡T¡¥Ét ~¦h«$¢£²\�~���h¤h�±¦h«/«$¢£ ¥¤�«T��¨�©¥�t¡~���+�´¨�¢£«/¦�¨´� ¨�©¥� ËǦh¬®§¥� ¦�¨.¨�¢£«T�� � Å �c�+¢£ ~¤$¢£ z�+¦h²±©k�´¨�¦�¨´��¢® ~¡¥���c�+ ~¡¥�+ J¨�¬ ÉT� Å  ¥�+¢ ¤^© �c�h�±¢£ ¥¤T�´¨�¦�¨´�+��µ���§¥����¨´�±§~²³¨�§¥�±¦h¬ Å �h��­|¦��±¡¡~É8 ~¦h«/¢£²;�~����¤h�±¦h«$«/¢£ ¥¤�¦h¬ ¤h�h�2¢ ¨�©~« §~���+�@¨�©¥�/�~���h�c����¨�¢ �+�B� Å ¨�©~�/Ë�¦h¬£§¥� Å §~ ~²³¨�¢ �^ �¨´�k«$¦��h�*¢£  Å �����+ ~²³�+�@� Å ¨�©¥�/Ë�¦h¬£§¥�$� Å�´¨�¦�¨´�+�;¨�©~¦�¨�­\� ©~¦+Ëh�  ¥�h¨WËt¢£��¢ ¨´�+¡ Å ���^« ¨�©¥�@Ë�¦h¬£§¥�+��� Å  ¥�+¢ ¤^© �_�h�2¢£ ¥¤T�´¨�¦�¨´�+��µ ���?²³�^ ~��¢£¡~����¨�©¥�@�L��� �L¬ �+«w� Å�h�~¨�¢£«/¢��+¢£ ¥¤U¨�©¥�$�´����Ët¢£ ¥¤*� Å �L¦�¨�²2©¥�+�y� Å ²�§~�´¨´�^«T���2����Ëh���@¨�¢®«T�hµ � ©~�?�±¢£«T�L¬£¢®²�¢ ¨�Éz� Å ¨�©~¢£�;�~��� �L¬£�+« �+ L¦��L¬ �+�§~�:¨´�?¤^¦h¢£ z¦h z§~ ~¡¥���±��¨�¦h ~¡~¢£ ¥¤@� Å ¨�©¥�B�´¨´�±§~²³¨�§~���W� Å ¨�©¥���L��� �L¬ �+« �JÉ/�+�´¨�¦��L¬£¢£�±©~¢£ ¥¤y�±�+��§~¬ ¨��\�^ z¨�©¥����©~¦��c��� Ũ�©¥� ËǦh¬®§¥� Å §~ ~²³¨�¢ �^ L��µ � ©¥� ²�¦h�´�?� Å ©¥�^«T�h¤h�+ ¥���^§L�;²�§L�´¨´�^«T���±��²�¦h  �_� �´�^¬ Ëh�+¡k�h�~¨�¢£«/¦h¬£¬ Éz§~��¢£ ¥¤ �L¦h² �8­:¦��2¡¡¥Ét ~¦h«$¢£²W�~���h¤h�±¦h«/«$¢£ ¥¤y­�©~¢£²±©U¤^¢ Ëh�+�|§~�\¨�©¥��¦��L¢£¬£¢ ¨eÉ?¨´� ��Ë�¦h¬£§~¦�¨´�;¨�©¥�;�c��� Å �h�2«$¦h ~²³�W� Å �^§¥�:«$��¨�©¥�8¡L��µ � ©¥���Ëh�+ J¨�§~¦h¬�¤h�^¦h¬Y� Å ¨�©¥� ���+�´�+¦��±²2©k¢®�W¨´�z¡¥��Ëh�+¬ �h��¦��~�~�±�+Æ¥¢£«$¦�¨�¢ �^ ~��¨�©~¦�¨�²�¦h  �_� §~�´�+¡0¨´�z�~���t¡~§~²³�@²³�^«T�q§¥¨�¦Ç¨�¢ �^ ~¦h¬®¬ É�¨´�±¦h²³¨�¦��q¬ �/¦h¬ ¤h�h�±¢£¨�©~«$� Å �h�yËh�+²³¨´�h�´Â�Ë�¦h¬£§¥�+¡��~��� �L¬£�+«$�B��§~²2©�¦h�y¨�©¥�T�L��� �L¬ �+« ­W¢£¨�©�©¥��¨´�����h¤h�+ ¥���^§L�²�§~�´¨´�^«$���±��¸ �L§~¨ ­\�y¡¥�U ¥�h¨�²³�^ ~��¢£¡~����¨�©~¢£�|¤h�+ ¥���2¦h¬£¢��+¦�¨�¢ �^ ¢® z¨�©~¢£�|�L¦��c����½2µ

� ©~� �L¦�¨�²2©p�´����Ët¢£²³���L��� �L¬ �+« ²³�^ ~��¢£��¨��W� Å ²�§~�´¨´�^«T���±���´¨´�t²±©L¦h�´¨�¢£²�¦h¬£¬ É0¦����±¢ Ët¢£ ¥¤/¦�¨B¦*�´����Ët¢£²³�?�´¨�¦�¨�¢ �^ p¦�¨¡~¢£�±²³����¨´�y¨�¢£«T��¢£ N¨´���±ËǦh¬£����Ëh����¦ �L ~¢£¨´�y©¥�h�±¢����^ ��q¢£ ~²�§¥���2¢£ ¥¤?¦T­:¦h¢£¨�¢£ ¥¤/²³�^�´¨;¦h ~¡¦U�´����Ët¢£²³�@²³�^�´¨�µ � ©¥� �L¦h��¢£²¡¥�+²�¢£�±¢ �^ �¢£��­W©¥��¨�©~���B�h�@ ¥�h¨y¦ �L¦�¨�²2©�� Å ²�§~�´¨´�^«T���2�y��©¥�^§~¬®¡ �c�T�´����Ëh�+¡�¢ Å ¢£¨�¢®�B¬ �+���B¨�©~¦h p¨�©~�/��¢����T� Å ¨�©¥��´����Ët¢£²³��²�¦��L¦h²�¢ ¨eÉhµ � ©8§~� �8¨�©¥������¢®�\¦�¨´�±¦h¡¥����! �_��¨e­\���+ z²�§~��¨´�^«T���|­:¦h¢ ¨�¢® ¥¤�²³�^�´¨�� ¦h L¡/¨�©¥��²³�^�´¨|� Å �´���±Ë8¢£ ~¤�¦�L¦�¨�²2©¶µ � ©¥�@�±¦h L¡¥�^«$ ¥�+���;¢£ �¨�©¥� «$�8¡¥�+¬[²³�^«T�+� Å ���^« ¨�©~���³Æt�h¤h�+ ¥���^§~�B�´¨´�t²±©L¦h�´¨�¢£²  ¥�^ ~�´¨�¦�¨�¢ �^ L¦���ɦ����±¢ Ë�¦h¬�~���t²³�+����µ ���B¦h���±§~«T�B�´¨�¦�¨�¢ �^ ~¦��±É/¡¥��¨´���±«$¢® ~¢£�´¨�¢£²��´����Ët¢£²³��²³�^�´¨|�_��� �L¦�¨�²2© ¦h ~¡*­:¦h¢£¨�¢£ ¥¤�²³�^��¨ �_���|²�§~�´¨´�^«$���+µ

� ©~����� ¦������´��Ëh���±¦h¬¥Ë�¦��±¢£¦�¨�¢£�^ ~�Y� Å ¨�©¥� �L��� �L¬ �+«j¡¥���c�+ ~¡~¢£ ~¤W�^ T­W©~��¨�©¥���.�h�� ¥�h¨.­:��¦h����§~«T�|¨�©~¦�¨�²³���±¨�¦h¢£ Ë�¦��±¢£¦��L¬ �+�U¦h ~¡ «T�t¡¥�+¬��L¦��2¦h«T��¨´���±� ¦����p ¥�^ ~�´¨�¦�¨�¢£�^ ~¦���É��h� �´¨´�t²±©~¦h�´¨�¢®²�µ �   ¨�©¥�k²�¦h���k­�©¥�����p¦����±¢ Ë�¦h¬£�$� Ų�§~�´¨´�^«$���±� �W�´���±Ëh���z¦�ËǦh¢®¬£¦��L¢£¬£¢£¨�É � �´����Ët¢£²³�p²³�^�´¨U� Å ¦ �L¦�¨�²2© ¦h ~¡F�c�^���±¢ �L¬ É4�h¨�©¥���z«T�8¡~�+¬W�L¦��2¦h«T��¨´���±�z¦����¦h¬£¬��´¨´�8²2©~¦h�´¨�¢£²����^§¥�/�^§~¨�²³�^«T���´�L¦h²³� �_�+²³�^«$�+�U« §~¬ ¨�¢£¡L¢£«T�+ ~��¢£�^ ~¦h¬ºµ ����²³�^§~¬®¡F¦h¬£�´�4²³�^ ~��¢£¡~���U�~�±� �L¬ �+«$�­W©¥���±�y²�§~�´¨´�^«T���±� �_�+¬£�^ ¥¤?¨´�/¡~¢"!c�����+ J¨�²�¬£¦h���´�+��µ � ©¥�+�´�BË�¦��±¢£¦�¨�¢ �^ ~���~���t¡~§~²³�B�´¨�¦�¨´� �~�^§¥¨�²³�^«$��¦h ~¡ ¡¥�+²�¢®��¢ �^ �´�L¦h²³�+� ¨�©~¦�¨?¦����z«�§L¬ ¨�¢£¡~¢£«$�+ ~��¢ �^ ~¦h¬ �A«$¦��t¢£ ¥¤�¨�©~�U�~�±� �L¬ �+« ¢® N¨´�±¦h²³¨�¦��q¬ É�²³�^«T�L¬ �³Æcµ � �h�T��§~²±©��~��� �q¬ �+«$� �­:� ­:�^§~¬£¡k©~¦�Ëh� ¨´�z���+���h��¨�¨´�z¦��L�~���+Æ¥¢£«$¦�¨�¢£�^ �¨´�+²2©~ ~¢$#8§¥�+���L¦h�´�+¡k�^ k¨�©¥� �~�±¢£ ~²�¢£�L¬ �+� � Å�Å �h��­|¦��±¡p¡¥Ét ~¦h«$¢£²�~���h¤h�2¦h«$«$¢£ ¥¤T¸ �´����%:����¨��´�&��¦h��' � ��¢£¨���¢��t¬£¢£�\¸ � ¼h¼h¹^½ � �t§¥¨´¨´�^ (')%|¦���¨´�/¸ � ¼h¼hÁ^½´½ �8­W©~¢£²2©?§~�´�|¦��~�~����Æt¢£«/¦�¨�¢ �^ ~�� Å ¨�©¥� ËǦh¬®§¥� Å §L ~²³¨�¢ �^ 4¨´��«T��Ëh� Å �h�±­:¦��±¡F¢£ 4¨�¢£«T� �|§¥�¶¡~¦�¨�¢£ ¥¤k�+�´¨�¢£«$¦�¨´�+�$� Å ¨�©¥� Ë�¦h¬£§¥� Å §~ ~²³¨�¢£�^ �¦h�T¨�©¥�¦h¬ ¤h�h�±¢£¨�©~« �~���t²³���+¡~��µ

�  $¨�©~¢®�.�L¦��c�����N­:����¨�§~¡¥É?¦@��¢£«$�L¬ ����²�¦h¬£¦��\Ëh���±��¢ �^ /� Å ¦@�´¨´�t²±©L¦h�´¨�¢£²�� �L¦�¨�²±©*¡~¢£�´�q¦�¨�²±©~¢£ ~¤B�~���t²³�+��� �N­�©¥�����­:�?²�¦h ��+¦h��¢£¬ É� �L¨�¦h¢£ k�h�~¨�¢£«$¦h¬��´�^¬£§~¨�¢ �^ ~�;§~��¢£ ¥¤z²�¬£¦h���±¢£²�¦h¬Y¨´�+²±©~ ~¢�#J§¥�+�+µ ���$§~�´�?¨�©¥�?�h�~¨�¢£«/¦h¬Y���^¬£§¥¨�¢ �^ 0¨´���Ë�¦h¬£§~¦�¨´�?¨�©¥�*#J§~¦h¬®¢ ¨�É� Å ¡~¢"!c�����+ N¨�¦��~�~�±�+Æ¥¢£«$¦�¨�¢ �^ k«T��¨�©¥�t¡~�;­W©~¢£²2©0²³�^§L¬£¡ �c�?¦��~�L¬£¢£�+¡¨´� «T�h���?²³�^«T�L¬ �³Æ�~��� �q¬ �+«$��µ ����¨�©¥�+ �²³�^«T�q¦����y¨�©¥�y�h�L¨�¢£«$¦h¬H�´�^¬®§¥¨�¢ �^  ¨´�$¨e­\�*²�¬£¦h���´�+�W� Å�Å �h��­|¦��±¡0¡¥É8 L¦h«$¢£²;�~���h¤h�±¦h«/«$¢£ ¥¤¨´�+²±©L ~¢$#8§¥�+��¸ �c��� Å �h�±«T�+¡ ¢£  ¨�©~��²³�^ N¨´�³Æt¨0� Å �L L¢ ¨´��©¥�h�±¢����^  �L��� �L¬ �+«$��½/§L��¢£ ¥¤�¡~¢£�±²³����¨´��ËǦh¬®§¥� Å §L ~²³¨�¢ �^ ¦��~�~�±�+Æ¥¢£«$¦�¨�¢ �^ ~�+µ � ©¥���~�±�´¨ ²�¬£¦h��� §~�´�+��� �L�´����Ë�¦�¨�¢ �^ ~� � Å ¨�©~�/Ë�¦h¬£§¥�*� Å �c�+¢£ ¥¤�¢£ �¦0��¨�¦�¨´�*¨´��§¥�¶¡~¦�¨´�U¨�©¥�

Page 4: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� ¾

�+�´¨�¢£«/¦�¨´�:� Å ¨�©¥�\Ë�¦h¬£§¥� Å §L ~²³¨�¢ �^ ¶µ � ©¥�|�´�+²³�^ ~¡?¡¥�8�+�A¨�©¥� ��¦h«$� � �L§¥¨[¦h¬®�´�;§L�´�+�A¨�©~�:�´¨´�2§~²³¨�§¥���:� Å ¨�©¥� Å §L ~²³¨�¢ �^ ¨´� ¢£«T�c�^�´�@¦h¡L¡~¢ ¨�¢ �^ ~¦h¬[§¥�¶¡~¦�¨�¢£ ¥¤U�´¨´���L�;¨´� «$¦h¢£ J¨�¦h¢£ ¨�©~¢®� Å §~ ~²³¨�¢ �^ Hµ � ©¥� Ëh���±��¢£�^ �� Å ¨�©~� ¦h¬ ¤h�h�2¢ ¨�©~« ¨�©~¦�¨«$¦h¢£ J¨�¦h¢£ ~�;¨�©¥�$��¨´�±§~²³¨�§¥���?� Å ¨�©¥� Å §L ~²³¨�¢ �^ k�~���t¡~§~²³�+�;�������h�2��¨�©~¦�¨y¦��~�c�+¦���¨´� �_�T¦h p�h�±¡~���B� Å «$¦�¤^ L¢ ¨�§~¡¥���«$¦h¬®¬ ��� ¨�©~¦h  ¨�©¥�@¨´�+²±©~ ~¢�#J§¥�B¨�©L¦�¨W¡¥�8�+�� ¥�h¨W¢£«$�_�^������¨´�±§~²³¨�§¥���hµ ��� Å ���+¬H¨�©~¦�¨�¨�©~¢£���³Æ8�c���±¢£«$�+ N¨ �´����Ëh�+�W¨´�©~¢ ¤^©L¬£¢ ¤^©J¨�¨�©¥��¢£«T�c�h��¨�¦h ~²³�;� Å ¢£¡¥�+ J¨�¢ Å É8¢® ¥¤@¦h ~¡U�³Æ8�q¬ �^¢ ¨�¢£ ¥¤��~��� �L¬£�+«��´¨´�2§~²³¨�§¥���;¢£ /¨�©~���+�´¨�¢®«$¦�¨�¢ �^ *� Å Ë�¦h¬£§¥�Å §~ ~²³¨�¢ �^ ~�+µ

��§~� �L¦h�±¢£²/�L��� �L¬ �+« ©~¦h�T¦ Å ¦h¢£�±¬ Ép�±¢£²2©4©L¢£�´¨´�h��É�¢® �¨�©¥� ¬£¢£¨´���±¦�¨�§¥���hµ � ¨ ­|¦h� �~�±��¨��´¨�§~¡L¢ �+¡ �8É È �^�´¨´�+ ¸ � ¼h¹^Äh½$¸ �����/¦h¬£�´� � �+¡~©~¢�¸ � ¼hÁ��J½ � È �^�´¨´�+ F¸ � ¼^Ä�Ã^½´½ ��­W©¥� ¡¥�+�±²³�±¢ �c�+¡0¦ ²�§L�´¨´�8¡L¢£¦h p­W©¥� ¡~¢£���L¦�¨�²±©¥�+��¨´�2§~² �t�­W©¥�+ ~��Ëh���$¨�©¥� 8§~« �c���?� Å ­:¦h¢ ¨�¢® ¥¤��L¦h���´�+ ~¤h���±�?�³Æt²³���+¡L�/¦�²³����¨�¦h¢£  ¨�©¥���+��©¥�^¬®¡¶µ��;� � ' �t��� Å �����F¸ � ¼^Ä�Ã^½­:�����@¨�©¥� �~�±�´¨�¨´�U�~����Ëh�@¨�©~¦�¨;¢£ �´¨´�+¦h¡¥É��¨�¦�¨´��¨�©¥�@�h�~¨�¢£«$¦h¬H¡~�+²�¢£��¢ �^  �2§~¬ �B� Å ¨�©~¢£�W�´É8��¨´�+«-­|¦h��«T�^ ~�h¨´�^ ¥�¦h ~¡�¨�©¥����� Å �h����©~¦h¡0¦/²³�^ J¨´���^¬Y¬£¢£«$¢ ¨���¨´�±§~²³¨�§¥��� �q¨�©¥����� �8É �~����Ë8¢£ ~¤T¨�©¥�@�h�~¨�¢®«$¦h¬£¢ ¨eÉ*� Å È �^�´¨´�+ �� �;²�§~�´¨´�t¡~¢£¦h ¶µ� ©¥��É0¦h����§~«T� ¨�©¥�?­|¦h¢ ¨�¢£ ¥¤z²³�^�´¨B�c���;²�§L�´¨´�^«T���y¢£��¦h �¢£ ~²³�±�+¦h��¢£ ¥¤ Å §~ L²³¨�¢ �^ �� Å ¨�©¥�T J§L« �c���;� Å ²�§~�´¨´�^«$���±�­|¦h¢ ¨�¢£ ¥¤�¢£  ¨�©¥� #8§¥�+§¥�hµ ���+¢£��� '��\¬£¢£� ��¦ ¸ � ¼^Ä�¹^½ ¦h����§L«T�p¨�©~¦�¨ ¨�©¥��­|¦h¢ ¨�¢£ ¥¤�²³�^�´¨�c��� ²�§~�´¨´�^«$����¢£� ¦Å §~ ~²³¨�¢ �^ A¸��B½;¢ Å ¦z²�§~�´¨´�^«T���B©~¦h� �c���+ 0­:¦h¢£¨�¢£ ¥¤*¨�¢£«T� �@µ � ©~��É���©¥��­j¨�©L¦�¨;¨�©~� �h�L¨�¢£«$¦h¬Y�´����Ët¢£²³�?�_�^¬®¢£²³É¢£� ¨´�*�´�+ ~¡ ¨�©~�@Ëh�+©~¢£²�¬£�@¢ Å ¨�©¥�@�´����Ëh���;¢£�W¦�ËǦh¢®¬£¦��L¬ �y¦h ~¡�¨�©¥�@«$¦��±¤^¢£ ~¦h¬¶­|¦h¢ ¨�¢£ ¥¤$²³�^��¨;¢£�W¦�¨�¬ �+¦h�´¨�¦h�W¬£¦���¤h� ¦h�¨�©¥�U�h�~¨�¢£«$¦h¬\¬ �^ ¥¤�±§L �¦+Ëh���±¦�¤h�²³�^��¨�µ � ©~¢£�y¢®�y¨´���±«T�+¡4¦������������������������� !�#"%$���¢£ �²³�^ J¨´�±¦h�´¨ ¨´�0¨�©~�/²³�^ N¨´�±�^¬¬£¢£«/¢ ¨|�_�^¬®¢£²³É/�~�±��Ëh�+¡��8É&�;� �0¦h ~¡ �t��� Å �����¥µ � ©¥� �q¦h��¢£²B«T�t¡¥�+¬H¢£��¤h�+ ¥���±¦h¬£¢����+¡�¢£ '�;� �4¸ � ¼^Ä�Á(��½W¨´�/¢® ~²�¬£§~¡¥��´­W¢£¨�²±©~¢£ ~¤k²³�^�´¨����|¦h L¡�©~¦h� �c���+ �¦��~�L¬£¢ �+¡4¨´��¨�©¥���¨�§~¡¥É�� Å ¦p¨�­:��Â�¨´���±«$¢® ~¦h¬���©8§¥¨´¨�¬ � �´Ét�´¨´�+«9­W©~����� �^ ¥��h�@¨e­\��Ëh�+©~¢£²�¬ �+�y²³Ét²�¬ � �c��¨e­\���+ 4¦z�L¦h¢ �y� Å ¨´���±«/¢£ ~¦h¬£�T¸ � ¤^ ~¦h¬£¬�' È �^¬ �+��¦�� � ¼^Ä�¾ � %|¦��± ¥��¨´¨ � ¼^Ä�à � � ¤^ L¦h¬£¬�'

È �^¬ �+�±¦�� � ¼^Ä)� �*�;� � � ¼^Ä�Á,+ �����+¢£�±� � ¼hÁ � �*�;� � ' �t²±©L«$¢£¡¥¨ � ¼hÁ^Äh½2µ�� L²³�:¨�©~�|��¨´�±§~²³¨�§¥���:� Å ¦h  �h�~¨�¢®«$¦h¬8�_�^¬®¢£²³Éy¢£� �8 ~��­W ��h¨�©~�:�L�±¢£«$¦���É��~�±� �L¬ �+«j¢£�Y�^ ¥�|� Å ¡¥��¨´���±«/¢£ ~¢£ ¥¤�¨�©¥�

�³Æt�_�+²³¨´�+¡z²³�^�´¨�� Å �h�|¦@¤^¢ Ëh�+ z²³�^ J¨´���^¬c�´¨´�±¦�¨´��¤hÉ �~¦h ~¡U¨�©¥�+ *§~��¢£ ~¤@¨�©~¢£� Å §L ~²³¨�¢ �^ U¨´� �L L¡$¨�©¥�;�h�~¨�¢£«$¦h¬q²³�^ N¨´�±�^¬�´¨´�±¦�¨´��¤hÉhµ � ©~�����$©~¦h�@¡~��Ëh�+¬ �h�c�+¡�¦h ��³Æt¨´�+ ~��¢ Ëh�*¬£¢ ¨´���±¦�¨�§¥�±�T�^ ��´¨´�±¦�¨´��¤^¢ �+� Å �h���´¨´�+¦h¡¥É��´¨�¦�¨´���L§~¬�� #J§~�+§¥�+� ��c��¤^¢£ ~ ~¢® ¥¤­W¢ ¨�©�¨�©¥�z�h�±¢ ¤^¢® ~¦h¬1�q¦��_��� �8É %|¦h¢£¬ ��É ¸ � ¼hÀ��J½ ¸ ­W©~¢£²2©�¡L¢£¡� ~�h¨?²³�^ L��¢£¡¥���T¦p²³�^ N¨´�±�^¬ �´¨´�±¦�¨´��¤hÉ~½2µ�8��Ëh���±¦h¬�¦h§¥¨�©¥�h�±�U©~¦+Ëh�k��§¥¤h¤h�+�´¨´�+¡ ¡~¢"!c�����+ J¨T¨�É8�c�+�$� Å ²³�^ N¨´�±�^¬W�´¨´�±¦�¨´��¤^¢£�+� ��«T�h¨�¢ Ë�¦�¨´�+¡ �8É#¡~¢"!c�����+ N¨U²³�^�´¨Å §~ ~²³¨�¢ �^ ~��¨�©~¦h 4¨�©¥�^�´�²³�^ L��¢£¡¥�����+¡4¦��c��Ëh�hµ.-W�+§¥¨�� ¸ � ¼h¹^Äh½?¢® N¨´���t¡~§~²³�+¡#¦k¬ ��­:���$¬£¢£«/¢ ¨@¨´�p¨�©¥� �L¦�¨�²2©���¢����¦h ~¡ ¨´���2«T�+¡¨�©¥�@���+��§~¬ ¨�¦/¤h�+ ¥���±¦h¬ �L§~¬��*�´����Ët¢£²³�y�±§~¬£�hµ/�Y��­:�+¬£¬\¸ � ¼hÁhÀ^½W­|¦h��¨�©¥� �L�±�´¨ ¨´�U¢® N¨´���t¡~§~²³�B¤h�+ ~���±¦h¬©¥�^¬£¡L¢£ ¥¤y¦h ~¡*²�¦h ~²³�+¬£¬£¦�¨�¢£�^ U�´¨´�2¦�¨´��¤^¢ �+� �t­W©¥�����;¦@Ëh�+©L¢£²�¬ ��¡~���L¦���¨�§¥����«$¢ ¤^©J¨ �c��²�¦h ~²³�+¬®¬ �+¡ Å �h�|¦ �¥Æt�+¡$�c���±¢£�8¡� Å ¨�¢£«T�;¢ Å ¦@¡L¢£�´�L¦�¨�²2©/�±§L¬ �W©~¦h�: ¥�h¨ �c���+ *��¦�¨�¢£� �L�+¡$­W¢ ¨�©L¢£ $¦y�c���±¢£�8¡T� Å ¨�¢£«T� ¸ �±�%0~�+²³¨�¢£ ¥¤ � Å �h�:�³Æ¥¦h«T�L¬ � �8¨�©¥�¢£ ~¦��q¢£¬£¢ ¨eÉT¨´� �h�����0¦T¡¥�±¢£Ëh���|�±¢ ¨´¨�¢£ ¥¤T¦h ~¡z­|¦h¢ ¨�¢£ ¥¤J½2µ1�Y��­\�+¬£¬ '32�§~« �L¬ ��¨y¸ � ¼hÁh¹^½ �~���+�´�+ J¨�¦?¤h�+ ¥���±¦h¬ �~§L ~¢ �~�+¡Å �±¦h«T��­:�h� � Å �h��¨�©¥��¦h L¦h¬ É8�±¢£� � Å ¦ �~�±�^¦h¡�²�¬£¦h����� Å ¸ � ¦�� �h��Ët¢£¦h q½W¡~¢£�´�L¦�¨�²2©�c�^¬£¢£²�¢£�+� �~¦h����§~«/¢£ ¥¤$�´¨�¦�¨�¢ �^ L¦���É ��´¨´�t²±©~¦h��¨�¢£²T¡¥�+«$¦h ~¡L��µ54.Æ¥²³�+¬£¬ �+ J¨�����Ë8¢£��­W�;� Å ¨�©~¢£�;¬£¢ ¨´���2¦�¨�§¥���?¦����$²³�^ N¨�¦h¢® ¥�+¡�¢£ ��|©~¦h§~¡~©~��É ' � �+«T�q¬ ��¨´�^ ¸ � ¼hÁhÃ^½W¦h ~¡ � �+¡~©~¢�¸ � ¼hÁ��J½2µ

�8�c���±¦h �+¦ '76 �h��Ët¢£²±©�¸ � ¼h¼��J½|¦h L¡ �t�_���2¦h  �+¦ '76 �h��Ët¢£²±©�¸ � ¼h¼h¹^½ �´¨�§~¡¥É$¨�©~�;« §~¬ ¨�¢ �L���8¡L§~²³¨1��¢£ ~¤^¬ ��¬£¢®  �

Page 5: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� Ã

�~��� �q¬ �+« �8É ¡¥��¨´���±«$¢£ L¢£ ¥¤ Å ��� #J§~�+ ~²�¢ �+�/¦�¨*­W©~¢®²±©F����Ëh���±¦h¬W�L���8¡L§~²³¨��/©~¦�Ëh�0¨´� �c�0��©L¢ �~�c�+¡#¨´��«$¢® ~¢£«$¢����¨´�±¦h ~���_�h�±¨�¦�¨�¢ �^ �¦h ~¡�¢£ JËh�+ J¨´�h��É�²³�^�´¨���µ � ©~��Ék¡¥��Ëh�+¬£�h��¢£ J¨´��¤h���@¦h ~¡�«$¢ Æ8�+¡�¢£ J¨´��¤h����¬®¢£ ¥�+¦����~���h¤h�±¦h«/«$¢£ ¥¤«T�t¡¥�+¬£��µ %:����¨�¦��&�+¢ ' �8�c���±¦h  �+¦0¸ � ¼h¼h¼(��½�¦h ~¡ %:����¨�¦��&�+¢ ' �t�_���2¦h  �+¦¸ � ¼h¼h¼,+±½�²³�^ ~��¢®¡¥����¨�©¥� ��©~¢ �L«$�+ N¨�� � Å�~���t¡~§~²³¨�� Å ���^« ¦h ��h�±¢ ¤^¢£ p¨´�0¦¡¥�+�´¨�¢£ L¦�¨�¢ �^ �¨�©¥���^§~¤^©��^ ¥�$�h�@����Ëh���±¦h¬1¢£ J¨´���±«T�+¡L¢£¦�¨´�/ ¥�t¡¥�+� �A¦h L¡��~���+�´�+ J¨�´��Ëh���±¦h¬|²�¬£¦h�±�´�+� � Å ©¥�+§¥�±¢®�´¨�¢£²$¦h¬ �h�2¢ ¨�©~«$� ¢£ ~²�¬£§L¡~¢£ ¥¤ ¡¥�+²³�^«T�c�^��¢£¨�¢ �^ �� Å ¨�©¥�*��� #J§¥�+ L²³�*� Å ¬£¢£  �t� ��¦h �4 � � ¨�É8�c� ���^¬£§¥¨�¢ �^ 0¦h ~¡�¦*¡¥Ét ~¦h«$¢£²B�L���h¤h�±¦h«$«$¢® ¥¤� �L¦h�´�+¡0©¥�+§¥�±¢£��¨�¢£²�µ 2W��­\��Ëh�������_�h¨�©k«T�t¡¥�+¬£��¦h����§~«T� ��§¥�~�q¬ ɦh ~¡4¡¥�+«$¦h ~¡��±¦�¨´�+�?¦���� ¡¥��¨´���±«/¢£ ~¢£�´¨�¢®²T¦h ~¡�²³�^ L�´¨�¦h N¨?��Ëh���$¨�¢£«T�hµ �   %:����¨�¦��&�+¢ ��%:����¨��´�&��¦h�(' �8�c���±¦h  �+¦¸º¾�»h»h»N½�¨´�+²±©L ~¢$#8§¥�+�@� Å  ¥�+§¥����Â�¡¥Ét ~¦h«$¢®²T�~���h¤h�±¦h«/«$¢£ ¥¤�¦��±�*¢£«$�L¬ �+«T�+ J¨´�+¡�¨´�p¦��~�~����Æt¢®«$¦�¨´�*¦k�´¨´�t²±©~¦h��¨�¢£²��« §~¬ ¨�¢ �~�±�8¡~§L²³¨\Ëh���±�±¢ �^  � Å ¨�©¥�y�~��� �q¬ �+« µ

� ©L¢£�$�L¦��c���*«$¦��h�+�*¨�©¥� Å �^¬£¬ ��­W¢£ ¥¤�²³�^ J¨´�±¢ �q§¥¨�¢ �^ ~��µ � ½ ���p��©¥��­ ¨�©~¦�¨*¨�©¥�0ËǦh¬£§~� Å §~ ~²³¨�¢ �^  Å �h�*¨�©¥��~��� �q¬ �+« ¢£��«T�^ ¥�h¨´�^ ¥�hµ � ©~¢£�W�~���h�c����¨eÉz©L¦h� �c���+ k��©¥��­� �¸ �´��� � Å �h���³Æ¥¦h«T�L¬ � � �1§~¨´���±«$¦h #¸ � ¼h¼��J½´½ Å �h�;¨�©¥��~��� �q¬ �+« ­�¢ ¨�©#¢£  �L ~¢£¨´�/²�¦��L¦h²�¢ ¨eÉ � �L§~¨�­:� ©~¦+Ëh� ¥�h¨T�´���+ �¢£¨ Å �h�?¨�©¥� �L ~¢ ¨´�z²�¦��L¦h²�¢ ¨eÉ4²�¦h�´�hµ#¾^½ ���¦h¬£�´���©¥��­ ¨�©L¦�¨�¨�©¥��ËǦh¬®§¥� Å §~ ~²³¨�¢ �^ 0¢£���0Â�²³�^ JËh�³Æ �c­W©~������� ¢®��¨�©¥� ²�¦��L¦h²�¢ ¨�ÉhµBÃ^½ 6���¢® ¥¤��0Â�²³�^ NËh�³Æ¥¢ ¨eÉ �¶­\�?¦����¦��L¬ �z¨´��¦h¬£�´�p�+�´¨�¦��q¬£¢£��©#«T�^ ¥�h¨´�^ L¢£²�¢ ¨�É�� Å ¨�©¥� ¡~¢®�´�L¦�¨�²±© Å §~ ~²³¨�¢ �^  ��¦p���+��§~¬ ¨ ¨�©~¦�¨T¢£� ­:�+¬£¬  �t ¥��­W  Å �h�T¨�©¥�¢£  �q ~¢ ¨´��²�¦��q¦h²�¢ ¨�É ²�¦h�´� �q§¥¨ ©~¦h�  ¥�h¨ �c���+ z�+�´¨�¦��L¬£¢®��©¥�+¡ Å �h� ¨�©~� �L ~¢ ¨´�B²�¦��L¦h²�¢ ¨eÉ ²�¦h�´�hµ1�J½ � ¢£ ~¦h¬®¬ É �J­:�y��©¥��­¨�©~¦�¨y¦ Å �h��­|¦��±¡p¡¥É8 L¦h«$¢£²��~���h¤h�±¦h«$«/¢£ ¥¤ ¦h¬ ¤h�h�±¢ ¨�©L« ¨�©L¦�¨B�³Æ8�L¬£�^¢ ¨��;¨�©¥�$�´¨´�±§~²³¨�§~����� Å ¨�©¥�?Ë�¦h¬£§¥� Å §L ~²³¨�¢ �^ ¸ ¢£ ��q¦���¨�¢£²�§~¬£¦��T«T�^ ~�h¨´�^ ~¢£²�¢ ¨eÉ¥½$¡¥�±¦h«$¦�¨�¢£²�¦h¬®¬ É�¢£«$�~����Ëh�+�/�´�^¬£§¥¨�¢ �^  #8§~¦h¬£¢£¨�É4­W©¥�+ �²³�^«T�L¦��±�+¡F¦�¤^¦h¢£ ~��¨$¨�©¥��h�~¨�¢£«/¦h¬¶�´�^¬£§¥¨�¢ �^ Hµ

��� �c��¤^¢£  �JÉy¡¥� �q ~¢£ ¥¤�¨�©~� �L¦h�±¢£²��~�±� �L¬ �+«j¢£ @�´�+²³¨�¢£�^  � µ �8�+²³¨�¢£�^  ¾W�~����Ë8¢£¡~�+�A¨�©¥���h�±��¨�¢£²�¦h¬8���+��§~¬ ¨��[�^  ¨�©¥�©¥�^«T�h¤h�+ ~���^§~� �L¦�¨�²±©*�´����Ët¢£²³�W�L��� �L¬ �+«�¢® ~²�¬£§~¡~¢® ¥¤B�´¨´�±§~²³¨�§¥�2¦h¬~�~���h�c����¨�¢£�+�.� Å ²³�^�´¨ Å §~ L²³¨�¢ �^ ~�1¦h L¡/¦��~���8� Å � Ũ�©¥�B²³�^ J¨´���^¬c¬£¢£«$¢ ¨:�´¨´�±§~²³¨�§~���W� Å ¨�©¥�B�´����Ët¢£²³���L���8²³�+�±��µ �  z�´�+²³¨�¢ �^ Ã@­:��¢® N¨´���t¡~§~²³�;�^§¥�|«T�^ ¥�h¨´�^ ¥�y¦h¡L¦��~¨�¢ Ëh�¡¥Ét ~¦h«$¢£²$�~���h¤h�±¦h«$«/¢£ ¥¤0¦h¬£¤h�h�±¢ ¨�©~«>¦h ~¡�²³�^«$�L¦����z¢ ¨@¨´�k¨�©¥�z²�¬£¦h����¢£²�¦h¬ Å �h��­:¦��2¡�¡¥Ét ~¦h«$¢®²T�~���h¤h�±¦h«/«$¢£ ¥¤¦h¬ ¤h�h�±¢£¨�©~« µ � ©¥�B�³Æt�c���±¢£«T�+ J¨�� ¦h ~¡ �±�+��§~¬ ¨�� ¦�����¡¥�+��²³�±¢ �c�+¡z¢£  �´�+²³¨�¢£�^ &�¥µ

� ������ ���� ����������������

�  ¨�©~¢£�W�´�+²³¨�¢ �^ ­:� Å �h�±«$¦h¬®¬ É*¢® N¨´���t¡~§~²³�B¨�©~�y�~��� �L¬£�+« Å �h�;©¥�^«T�h¤h�+ ¥���^§L��²�§~��¨´�^«T���±�W¦h L¡­:��¡¥��Ëh�+¬ �h�0¨�©¥��L¦h��¢®²;«$�8¡¥�+¬c­�©~¢£²±©z¢£� §L�´�+¡ Å �h� ¨�©~�B���+�´¨ � Å ¨�©~�B�L¦��c���+µ ���@�´¨�¦�¨´�y¨�©~�y¦h����§~«T�L¨�¢ �^ ~� �t¡¥� �L ¥��¨�©¥�y ~�h¨�¦�¨�¢ �^ ¦h ~¡ ¨�©~� Å §~ L²³¨�¢ �^ ~� ¦h���´�t²�¢£¦�¨´�+¡�­W¢ ¨�©z¨�©¥�@«$�8¡¥�+¬�µ

���*²³�^ ~��¢£¡¥���@¨�©¥�$�~��� �q¬ �+« � Å ²�§~�´¨´�^«T���±��¦����±¢ Ët¢£ ¥¤ ¦�¨@¦0�´¨�¦�¨�¢ �^ �­:¦h¢ ¨�¢® ¥¤ ¨´���c�$�´����Ëh�+¡¶µ�|§~��¨´�^«T���±�¦����/������Ëh�+¡�¢£ p¤h���^§~�L� �8Ék¦������Ëh���y­W¢ ¨�© �q ~¢ ¨´�T�´����Ët¢£²³�/²�¦��L¦h²�¢£¨�Éhµ � ©¥�T�´����Ët¢£²³�$� Å ²�§~�´¨´�^«T���±��²�¦h ��t²�²�§¥�¦�¨¡~¢£�±²³����¨´���c�^¢£ N¨�� ¢®  ¨�¢®«T��²�¦h¬£¬ �+¡ ¡¥�+²�¢£�±¢ �^  ���c�8²2©~� ¦h ~¡ ­:����ËǦh¬£§L¦�¨´��¨�©¥��«T�t¡¥�+¬ Å �h� �L ~¢ ¨´�+¬ É «$¦h NÉ¡¥�+²�¢£�±¢ �^ ����c�t²±©~� ¸ �L ~¢£¨´�/¨�¢£«T�z©¥�h�±¢$���^ q½2µ������±¢ Ë�¦h¬£�@� Å ²�§~�´¨´�^«T���2���t²�²�§¥�T¦�¨ ¨�©¥�*������Ë8¢®²³�*�´¨�¦�¨�¢£�^ 4¦�¨?�+¦h²2©¨�¢£«T�z�c���±¢ �t¡¶Ê\­\� ¦h����§L«T�z�^ ~¬ É�¨�©L¦�¨T¨�©¥�  J§~« �c���?� Å ¦����±¢£ËǦh¬£�?¢£ 4�+¦h²2©�¨�¢£«T�z�c���±¢ �t¡�¢®�T¢£ ~¡¥���c�+ ~¡¥�+ J¨@� Å

Page 6: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� �

�h¨�©¥���;¨�¢£«T���_���2¢ �8¡L��µ ���L¦���¨ Å ���^« ¦����±¢ Ë�¦h¬£��¦h ~¡Ë�¦��±¢£¦��L¬£�+� ¨�©~¦�¨�¡¥���c�+ ~¡ �^ �¦��±�±¢ Ë�¦h¬£� �L¦h¬£¬A�h¨�©¥���;ËǦ��±¢®¦��L¬ �+� �Å §~ ~²³¨�¢ �^ ~�B¦h L¡��L¦��±¦h«T��¨´���2�@¦����/¦h�±��§~«T�+¡�¨´� �c�$¡¥��¨´���±«$¢£ L¢£�´¨�¢£²�µ � ©¥�T�L��� �L¬ �+« ²³�^ ~�±¢£�´¨��B� Å ¡¥��¨´���±«/¢£ ~¢£ ¥¤�h�~¨�¢£«/¦h¬��´����Ët¢£²³�T�c�^¬£¢£²�¢ �+�;¨´�k«$¢£ ~¢£«/¢�����¨´�h¨�¦h¬\¡~¢£��²³�^§L N¨´�+¡�²³�^��¨�� �8É �q ~¡~¢£ ¥¤z¦ ¨´�±¦h¡~�³Â���! �_��¨e­\���+ ��´����Ët¢£²³�¦h ~¡ ­|¦h¢ ¨�¢£ ~¤?²³�^��¨���µ� ����� ��� ��

��� �|�^�´¨�¨´�$�c��� Å �h�±«w¦ �L¦�¨�²2©��´���±Ë8¢£²³�hµ � �Y�+ ~¦h¬£¨�Éz�c����¨�¢£«T�B�c���±¢ �t¡*�c����²�§~�´¨´�^«T��� Å �h��­|¦h¢ ¨�¢£ ~¤$¦�¨�¨�©¥�y�´¨�¦�¨�¢ �^ Hµ� � �t����Ë8¢®²³�y²�¦��L¦h²�¢ ¨�É �_«$¦ÇÆ¥¢£«�§L«- J§L« �c��� � Å ²�§~�´¨´�^«T���±�W¨�©~¦�¨W²�¦h  �c�y�´���±Ëh�+¡�¢£  ¦ �L¦�¨�²2©¶µ � �\¬£¦h ~ ~¢® ¥¤ ©~�h�±¢����^  Å �h��¨�©¥�@«$�8¡¥�+¬�µ�� ��¢£�±²³�^§~ N¨ Å ¦h²³¨´�h�+µ

����¦h�±��§~«T�y²³�^ ~��¨�¦h N¨W²³�^��¨W�L¦��±¦h«$��¨´���±� � Å �L¦�¨�²±©0�´����Ët¢£²³�y¦h ~¡ ­|¦h¢ ¨�¢£ ~¤T�_����²�§L�´¨´�^«T���+µ�� ��� ����� � ��� � � ������� ���

��� ¦h����§~«T�y¦h �³Æ8�h¤h�+ ¥�^§L�W�´¨´�t²±©~¦h��¨�¢£²y�~�±�8²³�+����� Å ¦����2¢ ËǦh¬®��µ��H��¨�ª

��� ¸�������� �!���#"!�%$&$&$&���('.½)�

­W©¥���±�*�(+-,/.w¢£��¦/���+¦h¬£¢��+¦�¨�¢ �^ � Å ¨�©¥�@ 8§~« �c����� Å ²�§~��¨´�^«T���±�;¦����±¢ Ët¢£ ¥¤T¦�¨W¨�©~�@�´¨�¦�¨�¢ �^ k¦�¨W¡¥�+²�¢£�±¢ �^  ���c�8²2©� �~¦h ~¡0. ¢£�|¨�©¥�@�´��¨W� Å ¦h¬£¬H�_�^�±��¢ �L¬ ��¦����±¢£ËǦh¬£�+µ � ¢®� ¦T�L¦���¨�¢£²�§L¬£¦���¢£ ~�´¨�¦h L²³�B� Å ¦$²³�^«$�L¬ ��¨´�@�´��¨W� Å ¦����2¢ ËǦh¬®��µ

���$²�¦h � ¥��­ ¡¥� �L ¥�?¦z�´¨�¦h ~¡~¦��±¡0�~��� �q¦��L¢£¬£¢ ¨eÉ �´�L¦h²³�¸213�540�76�½ �A­W©~�����81j¢£��¨�©¥�T��¦h«T�q¬ ���´�q¦h²³� � Å ¦h¬£¬� � � �94 ¢£��¨�©~�;:_Â�¦h¬ ¤h� �~�±¦ ¡¥� �q ¥�+¡k��Ëh����1 �A¦h ~¡�<-¢£�B¦z�L��� �L¦��L¢£¬®¢ ¨�É«T�+¦h�±§¥���$¡¥� �L ~�+¡0��Ëh���=4 µ ���$��� Å ���¨´�>47¦h� ¨�©¥� ¢£  Å �h�±«/¦�¨�¢ �^  �~�+¬®¡¶µ ��� ¤h� Å §¥��¨�©¥���?¦h ~¡4¬ ��¨�4�+ �c�U¨�©¥� ¢£  Å �h�±«$¦�¨�¢ �^ ���§ �¥Â �~�+¬®¡�¦�¨?¨�¢£«T�*¨ ������~�±�+�´�+ N¨�¢® ¥¤p¨�©¥���´��¨$� Å ¦h¬£¬|�c�^����¢ �L¬ �*��Ëh�+ N¨��U§¥��¨´��¨�¢£«T� � µ �t¢£ L²³�*­:��©~¦�Ëh�04�+@?A4�+CBD���\¨�©~� �~���t²³�+���E%4�+5F '+CGD� ¢£� ¦ �L¬ ¨´�±¦�¨�¢ �^ Hµ

���*¡¥� �L ¥�;HI+�ª91KJL. ��§~²2©�¨�©~¦�¨*HI+±¸ � ½ � �(+ �[¨´���c�T¨�©¥�$�±¦h L¡¥�^« Ë�¦��±¢£¦��L¬ �?¨�©~¦�¨�¡¥��¨´���±«$¢£ ~�+�B¨�©¥�¦����±¢£ËǦh¬£� ¦�¨W¨�¢£«T�B¨�µ � ©~�B�~���t²³�+���3E�HI+5F '+CGD� ¢£�|�^§¥�W��¨´�8²2©~¦h�´¨�¢£²y¦��±�±¢ Ë�¦h¬c�~���t²³�+����µ

-���­ �_­:��¦��������+¦h¡¥É ¨´�U¡~� �L ¥�y�^§¥�;�´¨�¦�¨´�?ËǦ��2¢£¦��L¬ �hµ � ©¥�y�q©NÉt��¢£²�¦h¬ #J§~¦h J¨�¢ ¨eÉ*¨�©L¦�¨�¨�©¥� �´¨�¦�¨´�?¡¥�+��²³�±¢ �c�+�¢£�:¨�©¥�y 8§~« �_���|� Å ²�§~�´¨´�^«T���±��­:¦h¢ ¨�¢® ¥¤T¦�¨ ¨�©¥�B�´¨�¦�¨�¢ �^  ¨´� �c���´����Ëh�+¡¦�¨�¦T�´�c�+²�¢ �L�+¡/¨�¢£«$�;���c�t²±©¶µ ����¬ ��¨NM�c�B¨�©¥�y�´��¨�� Å ¦h¬£¬£��­|¦��L¬ �y�´¨�¦�¨´�+��µ

O +.ªP1QJRMP$

Page 7: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� 
O + � � ©¥�T�±¦h ~¡¥�^« Ë�¦��±¢£¦��q¬ �?¨�©~¦�¨@¡¥��¨´���±«/¢£ ¥�+��¨�©¥�/ 8§~« �c���B� Å ²�§~��¨´�^«T���±�y¦�¨y¨�©~�/�´¨�¦�¨�¢ �^ �¦�¨¨�¢®«T� � µ

� + � �j���+¦h¬®¢��+¦�¨�¢ �^ z� Å ¨�©¥�@ 8§~« �_��� � Å ²�§~�´¨´�^«$���±�W¦�¨�¨�©~�y�´¨�¦�¨�¢ �^ �¦�¨W¨�¢£«T� � µ

� ©¥�U�´¨�¦�¨´�U� Å ¨�©¥�*�´Ét�´¨´�+«>¦�¨�¨�¢£«T� � � � + ��¢£�y«T�+¦h�±§¥���+¡�¦�¨ ¡¥�+²�¢£��¢£�^ ����c�8²2© � µ ���U¦h����§L«T�$¨�©~¦�¨?¦����±¢ Ë�¦h¬£��t²�²�§¥� ¦�¨:¨�©~�;�´¨�¦�¨�¢£�^ ��´§~��¨ �c� Å �h���W¨�©¥�B¡¥�+²�¢®��¢ �^ U���_�t²±©z¦h ~¡*�´����Ët¢£²³�;�8²�²�§~�±���´§~��¨:¦ Å ¨´��� ¨�©¥��¡¥�+²�¢£��¢ �^ U���c�8²2©¶µ� ����� �(� � �

��� ¡¥� �L ¥��¨�©¥�@¡~�+²�¢£��¢ �^ zË�¦��±¢£¦��L¬£�+� ¦h� Å �^¬®¬ ��­W�+ª

� + � � � ¢ Å ¦$�´����Ët¢£²³�y¢£�|�c��� Å �h�±«T�+¡ ¦�¨�¨�¢®«T� �» �h¨�©¥����­�¢£�´�� � E � �%$%$%$�� � '��9� F

���?¡¥� �L ¥��¨�©¥��¡¥�+²�¢®��¢ �^  �±§L¬ �+� �8É � +:ª#MQJ E�» � � F �¶­W©~�����;E�» � � F$¢£�W�^§¥��¦h²³¨�¢ �^ p�´�q¦h²³��­W¢£¨�©¦h²³¨�¢ �^  � Å �h��´����Ët¢£²³�@¦h L¡�» Å �h�� ¥�U�´����Ët¢£²³�hµ ���?¡¥� �L ~�B�^§¥��¡¥�+²�¢®��¢ �^  �±§L¬ �+� ¨´�U¡~���_�+ L¡z�^ ~¬ É*�^ �¨�©¥��²�§¥�����+ J¨W�´¨�¦�¨´�?¦h ~¡ ¥�h¨��^ ¨�©¥�y©~¢£�´¨´�h�±ÉU� Å �´¨�¦�¨´�+��µ �j����¨�� Å ¡~�+²�¢£��¢ �^ z�±§L¬ �+�|��Ëh���;¨�©¥�B¨�¢£«T�y©~�h�±¢����^ ¢£� ¦$�_�^¬®¢£²³É���ª

� � ¸���� ������ �%$&$&$&�����'��9� ½)$� ©¥�y����¨�� Å ¦h¬®¬¶�c�^¬£¢£²�¢ �+����¢£��¡~�+ ¥�h¨´�+¡���µ��� � ��� ������� � � � �

� ¢£Ëh�+ U¨�©L¦�¨|¦�¨ ¨�¢®«T� � �8­\��­:�����B¢£ U��¨�¦�¨´� O + ¸ � ½1¦h L¡U­:�B²±©¥�^����¦h²³¨�¢£�^  � + �¥¦h ~¡*¤^¢ Ëh�+ *¨�©~¦�¨|¨�©¥�B¦����±¢ Ë�¦h¬£�¦�¨�¨�¢£«$� � �)� ¦����*HI+CBD��¸ � ½ �~¨�©¥�+  ­:�@²�¦h ¡¥���2¢ Ëh�B¨�©¥�@�´¨�¦�¨´��ËǦ��2¢£¦��L¬ �B¦�¨�¨�¢£«$� � �)� µ

O +CBD��¸ � ½ �"! O +�¸ � ½$# � � +&% B � HP+CBD��¸ � ½ ¸ � ½

Å �h��¦h¬£¬ � , 1�µ � �h�W¦ ¤^¢ Ëh�+  � , 1 �t­\�y²�¦h z¤h�+ ¥���2¦�¨´�y¨�©¥�B�´¨�¦�¨´�y�~�±�8²³�+���IE O'�+ F ' +CG §~��¢£ ~¤z¸ � ½:¦h²�²³�h�±¡L¢£ ¥¤?¨´��´�^«T�@�_�^¬®¢£²³É(��µ

��� ¦����y ¥��­ ���+¦h¡¥É*¨´�/¢£ J¨´���t¡~§~²³�B¨�©¥�B¨´�±¦h L��¢ ¨�¢ �^ z�~�±� �L¦��L¢£¬£¢£¨�¢ �+��ª

) +�¸+*P½ � <-,/.10 ¸�HI+ � *P½2 +CBD� ¸ � +CBD�43 � + � � + ½ � <-,/.10 ¸ O +CBD� �5� +CBD�3 � + � � + ½

2 +CBD��¸76 3 � ��»N½ � � ) +CBD��¸768# � ½96�: �» �h¨�©~����­W¢£�´� ¸º¾^½

Page 8: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� ¹

2 +CBD��¸76 3 � � � ½ ��� � ) +CBD��¸768½ ��� � � 6 : ») +CBD��¸76�#F¸ � # ��½´½ ��� ��� 6 : � # �» �h¨�©¥����­W¢®�´�

¸ºÃ^½

<D+CBD��¸ � +CBD� 3 � +7� � +e½ � ��� G��� �� 2 +CBD��¸76 3 � +7� � +e½ ¸��J½

� � � �����'��� ��� �������� §~�´�B¨�©¥� Å �^¬£¬ ��­W¢£ ¥¤T²³�^�´¨ Å §~ L²³¨�¢ �^ ~��ª

� +±¸ � +5� � +P½ � ²³�^��¨�¢£ ~²�§~�����+¡z¢£ z�c���±¢ �t¡ � �¥¤^¢ Ëh�+ 0�´¨�¦�¨´� � +�¦h ~¡�´���±Ë8¢£²³�y¡~�+²�¢£��¢ �^  � +� � � + � A¸ � +�# � � +�½ B� ' ¸ � ' ½ � ¨´���2«$¢£ ~¦h¬¶²³�^�´¨ Å §L ~²³¨�¢ �^ � ¸ O ½ � «/¢£  ��������� � '��9�+CG + � +±¸ O +7��� �+ ¸ O +�½´½ � ' � '\¸ O 'Y½"!

��� ²�¦h¬£²�§~¬£¦�¨´� � + ¢£ ��c���±¢ �t¡ � µ � ©¥�*� � �´�+²³¨�¢ Ëh� Å §~ ~²³¨�¢£�^  � ¸ O ½@¢£��¨�©¥�*�³Æ8�c�+²³¨´�+¡4¨´�h¨�¦h¬|¡~¢£��²³�^§~ J¨´�+¡4²³�^�´¨«$¢£ L¢£«$¢����+¡U��Ëh����¦h¬£¬c�c�^¬£¢£²�¢ �+� ��µ

� �/��¢£«$�L¬£¢ Å É$ ¥�h¨�¦�¨�¢ �^ �­\�y¬ ��¨�ª

� +±¸ � +7� � +e½$# � +±¸ � +7� � +e½ � ��% �'& 2 +CBD��¸ �)( 3 � +7� � +e½+* +CBD��¸ �)( ½)$� ¢£ ~¦h¬£¬£É � ­\�p¡¥� �L ¥�k¨�©¥�kËǦh¬®§¥� Å §~ ~²³¨�¢ �^ L�/¨´� �c�0¨�©¥� Å §~ ~²³¨�¢ �^ ~�,*�+ � Å �h� � � » � � �%$&$&$&� �B��§L²±© ¨�©~¦�¨z¨�©¥�Å �^¬£¬ ��­�¢£ ¥¤?���+²�§¥�±�±¢ Ëh�y�´��¨�� Å � #J§L¦�¨�¢ �^ ~��©¥�^¬®¡~��ª

* + ¸ � + ½ � «$¢£ - � E � + ¸ � + � � + ½ � � ! * +CBD� ¸ � +CBD� ½ 3 � + � � + %2F ¸ºÀ^½

Å �h� � � » � � �%$&$&$&� # � µ � ©¥�+�´�y¦��±�y¨�©¥�B�h�~¨�¢£«/¦h¬£¢ ¨�É*� #8§~¦�¨�¢ �^ ~� ­�©~¢£²±© ²�¦h 0¦h¬£�´� �c�B­��±¢£¨´¨´�+  ¢£ z¨�©¥� Å �h�±« ª

* + ¸ � + ½ � «$¢£ - ��� � � + ¸ � + � � + ½ � � % �.& 2 +CBD� ¸ �/( 3 � + � � + ½+* +CBD� ¸ �/( ½'0 12 ¸º¹^½

� «$¢£ - � E � +�¸ � +7� � +e½ F ¸�Äh½

Å �h� � � » �%$&$&$&� # � µ ��§¥�|¤h�^¦h¬H¢£�|¨´�$�´�^¬£Ëh�B¨�©¥���h�~¨�¢£«$¦h¬£¢£¨�É/� #8§~¦�¨�¢ �^ L�|­W©~¢£²2©*¢®�|� #J§~¢£ËǦh¬ �+ J¨:¨´�/�´�^¬ Ët¢£ ¥¤ ¨�©¥�� � �´�+²³¨�¢ Ëh� Å §~ ~²³¨�¢ �^  � ¸ O ½2µ � �h�@¦��²�¦h¬®¦��@�´¨�¦�¨´�$Ë�¦��±¢£¦��L¬£� ¨�©L¢£��²�¦h  �c�T�´�^¬ Ëh�+¡��h�~¨�¢®«$¦h¬£¬ É�§L��¢£ ¥¤z¡¥Ét ~¦h«$¢£²�~���h¤h�2¦h«$«$¢£ ¥¤?¨´�+²2©~ ~¢$#8§¥�+��µ

Page 9: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� Ä

� ��� � ��� ������� ����� ����

�  z¨�©~¢£� �´�+²³¨�¢ �^ �­\�B�+��¨�¦��L¬£¢£��© ���^«T�y¨�©¥���h����¨�¢®²�¦h¬¶���+��§~¬£¨�� Å �h��¨�©~�B�~��� �L¬ �+« ­�¢ ¨�© ©¥�^«T�h¤h�+ ¥���^§L�W²�§~�´¨´�^«$���±� �¢£  ¨�©~� Å �h�±«-� Å �´¨´�2§~²³¨�§¥�±¦h¬A�~���h�c����¨�¢ �+��µ � ©¥� «$¦�¿�h���~���h�c����¨�¢ �+�W¦������´¨�¦�¨´�+¡p¦h ~¡��³Æ8�q¬£¦h¢£ ¥�+¡¢® �´�+²³¨�¢ �^ p¾8µ �¦h¬ �^ ¥¤�­�¢ ¨�©��´�^«T� ¨´�+²2©~ ~¢£²�¦h¬ �L���h�c����¨�¢ �+��µ �8�+²³¨�¢ �^ F¾8µ ¾��~����Ët¢£¡¥�+�?�L���J� Å � Å �h�$¦h¬£¬|¨�©¥� �~���h�c����¨�¢£�+��µ � ©¥�+�´����+��§L¬ ¨�� ¦����@§~���+¡ ¢£ �´�+²³¨�¢ �^ 0¾8µ ÃT¨´�/¡¥���2¢ Ëh�y²³�h���^¬£¬£¦��2¢ �+�|�^  ¨�©¥� �c�+©~¦�Ë8¢£�^§¥� � Å ¨�©~�y�h�~¨�¢£«$¦h¬¶¡~�+²�¢£��¢ �^ z�±§L¬ �+��µ

�� �� �������������������! ��"# %$&����'($*)

� ©¥��«T�8¡~�+¬¶­\��¡¥��Ëh�+¬ �h�c�+¡ Å �h�W¨�©¥��©¥�^«T�h¤h�+ ¥���^§~�;²�§~�´¨´�^«$���W�~��� �q¬ �+«-©~¦h��²³����¨�¦h¢£ ���¨´�±§~²³¨�§¥�±¦h¬H�~���h�c����¨�¢ �+�¨�©~¦�¨.­:���´¨�¦�¨´�W¢£  ¨�©~¢®�.�´�+²³¨�¢ �^ ¶µ � ©¥�+���:�±�+��§~¬ ¨���¦�����§~���+¡?¨´�y¡¥��Ëh�+¬ �h�/¦��~�L���+Æ¥¢£«$¦�¨�¢ �^ L�[� Å ¨�©¥�|Ë�¦h¬£§¥� Å §L ~²³¨�¢ �^ ¦h ~¡¦��L�~���+Æ¥¢£«$¦�¨´�B¨�©~�y�h�~¨�¢£«$¦h¬¶¡~�+²�¢£��¢ �^ z�±§L¬ �hµ

��� �_��¤^¢® z­W¢ ¨�© ¦ Å ��­ ¡¥� �L L¢ ¨�¢ �^ ~��µ

+ �-,$� � ��� ���/.�0 �%�21436587�$:9 + � � � � �� ���� <; � �=1>;@?*" ���#�8? �8?BADCFEHGJI �LK%��$ �6M�����15�NKJK>;,+PO � �8;� ��� �8?BADCFE#7� 1�1����=Q B :RQ � �S?BA �8? �UT B :VT � �6?WE#7

X ¸6Q B �YT B ½$# X ¸6Q B �YT � ½ � X ¸6Q � �YT B ½$# X ¸6Q � �YT � ½)$ ¸ºÁ^½

� ©¥���±�$¢£�y¦h ��+¦h�±¢ ���y­|¦+É�¨´�0²2©¥�+² � Å �h���±§ �L«T�t¡~§~¬£¦��2¢ ¨�É � Å ¦ Å §~ ~²³¨�¢£�^ p­W©~¢£²2©�¢£���~���+���+ N¨´�+¡�¢£  �1§~¨´���±«$¦h ¸ � ¼h¼��J½�¸��H�+«$«$¦ �¥µ ÄJµ ¹ �L�L¦�¤h� ��� »N½2ª

Z � � � \[�]^._0 � ��183`K�7Nab9 + � ��� � �� (���� <; � �1>;�? " ���#�8? ���^c�? � � ��?Dd#Cfe�7�g ���6M � � E�» � � F �8? � O � E�» � � �%$&$&$ F-Gh 1�183`K�7Nab9iK%�����SKSc �jK

X ¸ � �)� � � ½ # X ¸ � �)� ��»N½ � X ¸ � � � ½'# X ¸ � ��»N½ ¸º¼^½

1%��� �� � �K�7 ���/�NKLK>;,+PO � �8;� ��� �8?kd#Cfe�G

��§¥�±¢£ ~¤�¨�©¥� Å �^¬£¬ ��­W¢£ ¥¤�¡~¢£�±²�§~����¢ �^ �­:��§~�´� �c�h¨�©?� Å ¨�©¥��¦��c��Ëh��«T��¨�©¥�t¡~��¨´�B�~����Ëh�W��§ �L«$�8¡~§L¬£¦��±¢ ¨eÉhµ �� ~�h¨�©¥���¢£«T�c�h��¨�¦h J¨ �~���h�c����¨eÉU¢£��­W©~¦�¨ ­:�@²�¦h¬£¬ È Â�²³�^ JËh�³Æt¢£¨�É_ª

Page 10: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� Á

+ �-,$� � ��� ����[ � 1>;@?*" ���#�8?k; � �`c�? � �&��? O � E�» � � �±¾ �%$&$&$ F �SK���� ��� ������� � 1�1%��� � B � � �� ¸ � B � ��½$# � ¸ � � � ��½ : � ¸ � B ½ # � ¸ � � ½ ¸ � »N½

1%���LK���O � ���4K ��������� �6? � � � ����� G

�   �^§¥��­:�h� � � �>�±���~���+�´�+ J¨���¨�©¥�y²�¦��q¦h²�¢ ¨�É � Å ¨�©¥� �L¦�¨�²2©¶µ��� �´¨�¦���¨�­W¢£¨�©z¨�©¥�y�´¨´�±§~²³¨�§~�±¦h¬c�~���h�c����¨�¢ �+�+ª

� ������ � ����� � ����� � �

� � ��� � ���� .�� M�� ���� <; ��1>;�? " ���#�8? * +�¸ � ½ �SK ?*�8?*��� "%� � �8K �6?��&�6? � 1%��� � � » �%$&$&$&� G

� � ��� � ����W[�� M�� ���� <; ��1>;�? " ���#�8? * +�¸ � ½ �SK���� " �8?,��� 5 1���� � � » �%$&$&$&� G

��� ¡¥� Å ���W¨�©¥�B�~���8� Å ��� Å ¨�©¥�+���y�~���h�c����¨�¢ �+� §L N¨�¢£¬¶¬®¦�¨´���+µ�1���h�c����¨�¢ �+� � ¦h ~¡�¾z²2©~¦��±¦h²³¨´���±¢����U¦  ¥�^ ~¡¥�+²³���+¦h�±¢£ ¥¤zËǦh¬®§¥� Å §~ L²³¨�¢ �^ p­W¢ ¨�©k¨�©¥�T�³Æt¨´�±¦z�~���h�c����¨eÉ�� Å È Â

²³�^ JËh�³Æt¢ ¨eÉ0¸ �´��� �~¤^§¥��� � ½2Êt¨�©¥�+�����~���h�c����¨�¢ �+�1¦�����¢£«$�_�h�±¨�¦h N¨ �c�+²�¦h§~�´�W¨�©¥��ÉT¤^¢ Ëh��§L��¢£ ~��¢£¤^©N¨���¢® N¨´�y¨�©~����©~¦��c�� Å ¨�©¥�Ë�¦h¬£§¥� Å §~ ~²³¨�¢£�^ ¶µ È  ¥��­W¬ �+¡¥¤h� � Å ¨�©¥�k��©~¦��c� � Å ¨�©~¢£� Å §~ ~²³¨�¢£�^ #¤^¢ Ëh�+�/§~�U¢£¡¥�+¦h�T�^ F­W©~¦�¨(�t¢£ ~¡#� Ŧ��~�~�±�+Æ¥¢£«$¦�¨�¢ �^ ~�@¨´�0§~���hµ��¥¢£ ~²³�$¨�©¥�+���/�~���h�c����¨�¢£�+�y¦����z�´¨´�±§~²³¨�§¥�2¦h¬.���+��§~¬£¨�� �A¨�©¥��É�¦��±�/�L¦���¨�¢®²�§~¬£¦��±¬ Ép§~�´� Å §~¬¢£ �¡¥���±¢ Ët¢£ ¥¤ Å §~ ~²³¨�¢£�^ ~¦h¬1¦��~�L���+Æ¥¢£«$¦�¨�¢ �^ L��µ �t§~�~�c�^�´�$­:�*©~¦�Ëh� ¦��~�~����Æt¢£«/¦�¨´�+¡�¨�©~�/Ë�¦h¬£§¥� Å §~ L²³¨�¢ �^ ��* ��¦�¨¦��´�c�+²�¢ �q²*��¨�¦�¨´���� �:¨�©¥�+  «$�^ ¥�h¨´�^ ~¢£²�¢ ¨eÉ�¤^¢£Ëh�+�/§~�T¢£  Å �h�±«/¦�¨�¢ �^ �¦��c�^§¥¨T¨�©¥� Ë�¦h¬£§¥�¦�¨$�h¨�©¥���*�´¨�¦�¨´�+��µ � �h��³Æ¥¦h«T�L¬ � �t¢ Å ��� �� �8¨�©¥�+ ,*/¸ � ½ � �*U¸��� ½�¦h ~¡ Å �h� ��� �� �8­\��©~¦�Ëh� **¸ � ½ � �*/¸��� ½2µ È Â�²³�^ JËh�³Æt¢£¨�ÉU¢®�\¢£«$�_�h�±¨�¦h N¨­W©¥�+  ²³�^«$�L¦��±¢£ ¥¤$��¨�¦�¨´�+�W¨�©~¦�¨�¦���� �7§~ ~¢ ¨�� ¦��L¦��±¨�µ

���z¦h¬£�´�k¢£ N¨´�±�8¡~§L²³�T¨�©¥� Å �^¬£¬ ��­W¢£ ¥¤ ¨´�+²2©~ ~¢£²�¦h¬\�~���h�c����¨�¢ �+�y¨�©~¦�¨�¦����U�+¢ ¨�©¥���?¢£ J¨´���±«T�+¡~¢£¦�¨´�/���+��§~¬ ¨��y¨�©~¦�¨¦��±¢£����¢£ /¨�©¥�W�~�±�8²³�+���\� Å ¡¥���±¢£Ë8¢£ ~¤B¨�©¥��«$¦�´�h�:�~���h�c����¨�¢ �+�;¸ �~���h�c����¨�¢ �+�:Ã�Âe¹^½��h�|²³�h���^¬£¬®¦��±¢ �+�1¡~���±¢ Ëh�+¡ Å ���^«�¨�©¥�«$¦�´�h�����+��§L¬ ¨��@¸ �~���h�c����¨eÉ Äh½2µ� � � �D� � � � ����� � ���� � �! � � ��� � ����#"$� M�� " �8K���1:;@?*" ������? � +±¸ � � � ½5�SK ?*�8? � � " � � �4K �6?�� �6? � G� � ��� � ����&%'� M�� " �8K���1:;@?*" ������? � +±¸ � � � ½5�SKLK:; +PO � �8;� ��� �8? O C>E�» � � F-G

� � ��� � ����)( < +CBD��¸ � ( 3 � � � ½ �SK ?*�8?*��� "%� � �8K �6?��&�6? � 1%��� �� � � ( ,0Mf7 � , E�» � � F-G

Page 11: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� ¼

0 5 10 15 20 250

50

100

150

200

250

� ¢ ¤^§~��� � ª�� È Â�²³�^ NËh�³Æ Å §~ ~²³¨�¢£�^ z­W¢ ¨�© È � � »tµ

� � ��� � ���� � � + ¸ � B � � � � ½ # � + ¸ � � � � � � ½ � � + ¸ � B ��»N½$# � + ¸ � � ��»N½)$� � ��� � ������ � M���1>;�? " ���#�8? � �� G 2 +CBD��¸76 3 � � � ½+* +�¸76J½��NKLK>; +jO � �8;� ����1%��� �� � � � � �%$&$&$&� G

�� S� � ��"H"�� ) "����D��" %$&����'�$*)

� ©¥�B¦h¢£« � Å ¨�©~¢£�|�´�+²³¨�¢£�^ ¢£�|¨´�T�~����Ëh�y¨�©¥�B«$¦�´�h�W�´¨´�±§L²³¨�§¥�±¦h¬_���+��§~¬£¨�� ¦h ~¡z¨�©¥�B¨´�+²±©L ~¢£²�¦h¬c�~���h�c����¨�¢£�+�:��¨�¦�¨´�+¡¢£  �´�+²³¨�¢£�^ 0¾8µ � µ

���;�´¨�¦���¨1­�¢ ¨�©?¨�©¥���~�±�J� Å �.� Å �~���h�c����¨�¢ �+�\à � � �8À8µ � ©¥�+�´�W¦h¬ �^ ~¤y­W¢ ¨�©T¦y¬£�+«$«$¦ Å ���^« �\§¥¨´���±«$¦h �¸ � ¼h¼��J½¤^¢ Ëh�@§L� ¨�©¥�B���+��§L¬ ¨|� Å «$�^ ¥�h¨´�^ ~¢£²�¢ ¨eÉU� Å ¨�©¥�@ËǦh¬£§~� Å §~ ~²³¨�¢£�^ �¸ �~���h�c����¨�É � ½2µ

�1���h�c����¨�É ÃT¢£�|� �8Ë8¢£�^§~� Å ���^«w¨�©¥�@¡~� �L ~¢ ¨�¢ �^ *� Å � +´µ� � �9� � � � ��� ��� � ���� % ���y­:¦h J¨�¨´�*��©¥��­ ��§ �L«T�t¡~§~¬®¦��±¢ ¨�ÉT� Å � + �JÉ*�+�´¨�¦��L¬®¢£��©~¢£ ~¤¥ª

� +±¸ � B � � ½ # � +±¸ � B ��»N½ � � +±¸ � � � � ½ # � +±¸ � � ��»N½)$ ¸ ��� ½

Page 12: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� � »

�t§ �q�´¨�¢ ¨�§¥¨�¢£ ~¤�¨�©¥� Å §~ ~²³¨�¢£�^  � +1¢® N¨´��¸ ��� ½ �q­\�@« §~�´¨��±©¥��­R¨�©L¦�¨�ª� � A¸ � B # ��½ B # � B � � � A¸ � � # ��½ B #� � � $

� ©~¢£²2©��¢£«T�q¬£¢ �~�+�\¨´�¥ª

¸ � B # ��½ B # � B � ¸ � � # ��½ B # � � $ ¸ � ¾^½

�t¢£ L²³� � B : � � �[¸ � B # ��½ B � » � ¸ � � # ��½ B � »tµ � ©~¢£� ¢£«$�L¬£¢ �+�|¨�©¥���±�@¦����y¨�©¥�±���@�c�^����¢ �q¬ �;²�¦h���+� Å �h�� #8§~¦�¨�¢ �^ �¸ � ¾^½2ª�|¦h�´� � ª�¸ � B # ��½ B � »$¦h L¡�¸ � � # ��½ B � »tµ�   ¨�©~¢£� ²�¦h�´� ��¸ � ¾^½|���+¡L§~²³�+�|¨´� � � �4µ�|¦h�´� ¾8ª�¸ � B # ��½ B � »$¦h L¡�¸ � � # ��½ B � »tµ2������U¸ � ¾^½|�±�+¡~§~²³�+� ¨´� � � � � �¥­W©~¢£²2© Å �^¬®¬ ��­W�W��¢£ ~²³�$¸ � � # ��½ B � »$¢£«T�L¬®¢ �+�1¨�©L¦�¨ � � � �4µ�|¦h�´� Ã8ª�¸ � B # ��½ B � »$¦h L¡�¸ � � # ��½ B � »tµ-���­u¸ � ¾^½ ���+¡~§~²³�+��¨´� � � � � B µ � $ � $�� $

� � �9� � � � ��� ��� � ����#( �����h�c����¨eÉzÀT¢®�|� #J§~¢£ËǦh¬ �+ J¨ ¨´�/��©~��­W¢£ ~¤ ¨�©L¦�¨�ª<U¸ � ( 3 � �)� � � ½ : </¸ � ( 3 � � � ½

­W©~¢®²±© ¢£� ¨�©~�y��¦h«T�@¦h���±©¥��­W¢® ¥¤¥ª

< ,1./0 ! ¸ � �)� # � � ½ B � H : � ( % : <-,/./0 ! ¸ � # � � ½ B � H : � ( %�H��¨��4¸ � 3 � � � ½ � ¸ � � � # � � ½ B � H?¸ � ½�¦h ~¡��z¸ � 3 � � � ½ � ¸ � # � � ½ B � H?¸ � ½2µz� ¬ �+¦��±¬ É��4¸ � 3 � � � ½8:�z¸ � 3 � � � ½2µ ��� �q ¥��¨�©¥����Ëh�+ J¨���� � E � 3 �z¸ � 3 � � � ½�: � ( Fy¦h ~¡ �� � E � 3 �#¸ � 3 � � � ½�: � ( F^µ�� :���¢®«T�L¬£¢ �+�¨�©~¦�¨��� ?��� ­W©L¢£²±©�¢£«T�L¬£¢£�+��¨�©~¦�¨ <-,/./0 ! � : � ( % � < ,1./0 ! ���% : <-,/.10 ! ��$% � < ,1./0 ! � : � ( % ��­W©~¢®²±©�~����Ëh�+���^§¥�W���+��§~¬ ¨�µ � $ � $�� $

���?§~�´�y¨�©~� Å �^¬®¬ ��­W¢® ¥¤T���+��§~¬£¨ Å ���^« �1§¥¨´���2«$¦h �¸ � ¼h¼��J½ ¸��H�+«/«$¦ �¥µ ÄJµ à �_�L¦�¤h� � »^¹^½�¨´�/�L����Ëh� �~���h�c����¨eÉ� µ

Z � � � \[�] [ d&;)� ���8K%� �6M���1%�� � ��g �S?��

.&] � +±¸ � � � ½5�SKL?*�8?*��� "%� � �8K �6?��&�6? � 1���� �� � � , E�» � � F �8?*� � � » �%$&$&$&� # �

Page 13: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� ���

[ ] <D+CBD��¸ � ( 3 � � � ½5�NK ?*��?*� � " � � �4K �6?��'�S? � 1���� �� � � ( ,0M=7 � , E�» � � F-G"�] � ' ¸ � ½ �SKL?*�8?*� � "%� � �4K��S? �&�6? � G

� M��P? *�+±¸ � ½ �SK ?*�8?*� � "%� � �4K��S? �&�6? � 1%��� � � » �%$&$&$&� G

��� ¦����y ¥��­ ���+¦h¡¥É*¨´�$�~����Ëh�@«T�^ ¥�h¨´�^ L¢£²�¢ ¨�Éz� Å ¨�©¥�yË�¦h¬£§¥� Å §~ ~²³¨�¢ �^ Hª� � �9� � � � ��� ��� � ���� . � ©¥���~�±�´¨@¦h�±��§~«T�~¨�¢£�^ �� Å ¬ �+«$«/¦�¾8µ ¾�²³�^«$�+� Å ���^«>�~���h�c����¨eÉ�Ã8µ � ©¥�U���+²³�^ ~¡¦h����§L«T�~¨�¢ �^ *¢£�:�~���h�c����¨eÉUÀ8µ � ©¥��¨�©~¢ �2¡/¦h����§L«T�~¨�¢ �^ *­:��©L¦+Ëh� Å �h� Å �����B¢ Å ­:�B¦h����§~«$�;¦ �������T¨´���±«/¢£ ~¦h¬_²³�^�´¨Å §~ ~²³¨�¢ �^ ¶ª � ' ¸ � ½ � »tµ � ©8§~� ¨�©¥�BË�¦h¬£§¥� Å §~ ~²³¨�¢£�^  * + ¸ � ½|¢£�  ¥�^ L¡¥�+²³���+¦h��¢£ ~¤ Å �h�W¦h¬®¬ � � » �%$&$&$&� µ � $ � $�� $

� ©~�  ¥�³Æt¨B�´¨´����¢£��¨´� �~����Ëh� È Â�²³�^ NËh�³Æ¥¢ ¨�Ép� Å ¨�©~� Ë�¦h¬£§¥� Å §~ ~²³¨�¢ �^ �¸ �~���h�c����¨eÉ0¾^½2µ �  k�h�±¡¥����¨´�¡¥�*¨�©~¦�¨­:� �~�±�´¨B�L����Ëh�U�~���h�c����¨�É�¹ ¦h ~¡�¨�­:�k¢£ J¨´���±«T�+¡L¢£¦�¨´�$¬ �+«$«$¦h�+µ@�H�+«/«$¦¾8µ �0¢£�y�~����Ëh�+¡ �JÉ�¢£ L¡~§~²³¨�¢ �^ k�^  �

¦h ~¡¢£¨�§~�´�+��¬ �+«/«$¦$¾8µ ÃT¦h ~¡ �L���h�c����¨�Éz¹T¢£ z¨�©¥�@¢® ~¡~§~²³¨�¢ �^ z�´¨´���Aµ� � �9� �3� � ��� ��� � ���� � ���?��§ �L��¨�¢ ¨�§¥¨´� � +1¢£ �¸º¹^½�¦h ~¡�´�q¬£¢ ¨�¢ ¨�¢£ J¨´�$¨e­\�*� #J§L¦h¬£¢ ¨�¢ �+��µ � ©¥� �~�±��¨W²³�^ N¨�¦h¢® ~��^ ~¬ É*¨�©¥�@¢®«$«T�+¡~¢£¦�¨´�y²³�^��¨ Å §L ~²³¨�¢ �^ ~� � + ¦h ~¡¨�©¥�@�´�+²³�^ ~¡�²³�^ N¨�¦h¢£ L��¨�©¥�y�³Æt�c�+²³¨´�+¡�Ë�¦h¬£§¥� Å §~ ~²³¨�¢£�^ ~� Å �h��¨�©¥� ¥�³Æt¨�¨�¢£«T�B�c���±¢£�8¡¶µ �;¡~¡~¢£ ¥¤?¨�©¥�B¨e­\�U� #J§L¦h¬£¢ ¨�¢ �+�|¤^¢ Ëh�+�W¨�©¥�y¡¥�+��¢£���+¡z���+��§~¬£¨�µ

� ©8§~�|­:� �~�±��¨ �~����Ëh�hª

� +±¸ � B � � � � ½ # � +±¸ � � � � � � ½ � � +±¸ � B ��»N½ # � +±¸ � � ��»N½)$�t§ �q�´¨�¢ ¨�§¥¨�¢£ ~¤ � + �¥­\�@¤h��¨�ª

� � A¸ � B � � # ��½ B # � #�A¸ � � � � # ��½ B � � B # � �­W©~¢®²±© ¢£� � �8Ët¢ �^§~��¬ ÉU¨´�±§¥�y¦ Å ¨´���W�c��� Å �h�±«$¢£ ~¤�¨�©¥�y²�¦h L²³�+¬£¬£¦�¨�¢ �^ ~�+µ

��� ¦h¬£�´�$ ¥���+¡ ¨´�/�~����Ëh�@¨�©~¦�¨�ª

� �� G 2 +CBD��¸76 3 � B � � � � ½+* +CBD�h¸76J½ # � �� G 2 +CBD��¸76 3 � � � � � � ½+* +CBD�Ǹ768½ �� �� G 2 +CBD��¸76 3 � B ��»N½+*�+CBD�Ǹ76J½ # � �� G 2 +CBD��¸76 3 � � ��»N½+* +CBD��¸76J½ ¸ � Ã^½

�t§ �q�´¨�¢ ¨�§¥¨�¢£ ~¤�¨�©¥�B¨´�2¦h ~� Å �����~��� �L¦��q¢£¬£¢ ¨�¢ �+� Å ���^« ¸º¾^½|¦h ~¡�¸ºÃ^½|­:�@©~¦+Ëh�@¨�©¥� Å �^¬£¬ ��­�¢£ ¥¤?� #J§L¦h¬£¢ ¨�É_ª

� �� G ) +CBD��¸76 # � B ½+* +CBD��¸768½ # � �� G �

) +CBD��¸76�# � � ½+*�+CBD��¸76J½ �� �� G ) +CBD��¸76 # � B ½+*�+CBD��¸76J½$# � �� G �

) +CBD��¸76�# � � ½+*�+CBD�Ǹ768½ ¸ � �J½

Page 14: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� � ¾

¨�©~¦�¨W¢®�|� �JËt¢ �^§~��¬£É/¨´�±§¥�hµ � $ � $�� $

Z � � � \[�] " h 1 � �SK �!?*�8?*��� "%� � �8K �6?���1>;@?*" ���#�8? �8? O � E�» � � $&$&$ F �8?*� ���*�NK ��� " �8? ��� 5�7 �6M(�j? � �� G 2 +CBD��¸76 3 � � � ½ � ¸76J½�SK �FK>; +jO � �8;� ���1>;@?*" ���#�8? �8? O C E�» � � F-G

� � �9� � � � � � � � \[ ] " ���@§~�´�y¬ �+«/«$¦$¾8µ � ¨´�$�L����Ëh�@�±§ �L«T�t¡~§~¬£¦��2¢ ¨�Éhµ �8�T­:�@ ¥���+¡ ¨´�$�L����Ëh��¨�©~¦�¨�ª

��� G 2 +CBD��¸76 3 � � � � � ½ � ¸76J½ # ��� G 2 +CBD��¸76 3 � �)� ��»N½ � ¸76J½ � ¸ � À^½

��� G 2 +CBD��¸76 3 � � � ½ � ¸768½ # ��� G 2 +CBD��¸76 3 � ��»N½ � ¸76J½ ¸ � ¹^½

�|¦h�´� � ª � � � �)� � �4µ� ©¥�+ �¸ � ¹^½ �c�+²³�^«T�+��ª

��� G ) +CBD��¸76J½ � ¸76J½ # ��� G BD� ) +CBD��¸76�# � # � ½ � ¸768½ � ��� G ) +CBD��¸76J½ � ¸76J½ # ��� G ) +CBD��¸76�# � ½ � ¸76J½)�­W©~¢®²±© ��¢£«$�L¬£¢ �~�+�\¨´�¥ª

� �� G ) +CBD��¸76J½ � ¸76J½ # � �� G ) +CBD��¸76J½ � ¸76 � � � � ½ �� �� G ) +CBD��¸768½ � ¸76J½ # � �� G ) +CBD��¸768½ � ¸76 � � ½)$ ¸ � Äh½

�t¢£ L²³� � ¢£�  ¥�^ L¡¥�+²³���+¦h��¢£ ~¤¥ª � ¸76 � � � � ½�: � ¸76 � � ½2µ � ©8§~�y¸ � ¹^½ ¢®� ��¦�¨�¢£� �~�+¡Hµ�|¦h�´� ¾8ª � � � �)� : �4µ� ©¥�+ �¸ � ¹^½ �c�+²³�^«T�+��ª

��� G BD� � �

) +CBD� ¸76�# � # � � ��½ � ¸768½ # ��� G BD� ) +CBD� ¸76 # � # � ½ � ¸768½� ��� G � �

) +CBD� ¸76�# � � ��½ � ¸76J½$# ��� G ) +CBD� ¸76 # � ½ � ¸76J½)� ¸ � Á^½

­W©~¢®²±© ��¢£«$�L¬£¢ �~�+�\¨´�¥ª

��� G ) +CBD��¸76J½ ! � ¸76 � � �)� # ��½$# � ¸76 � � � � ½ % � ��� G ) +CBD��¸768½ ! � ¸76 � � # ��½'# � ¸76 � � ½ % $

Page 15: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� � Ã

%:É È Â�²³�^ NËh�³Æ¥¢ ¨eÉ �L­:�y¤h��¨W¨�©¥�y¡~�+��¢ ���+¡z���+�±§~¬ ¨�µ�8$ � $�� $

� ©~� Å �^¬£¬£��­W¢£ ~¤|¬£�+«$«$¦|§~�´�+�H¬£�+«$«$¦�¾8µ Ã|���+²�§~�±��¢ Ëh�+¬ É�¨´���+�´¨�¦��L¬®¢£��©B��§ � ²�¢ �+ N¨H²³�^ L¡~¢ ¨�¢ �^ ~� Å �h� È Â�²³�^ JËh�³Æ¥¢ ¨�Éhµ

Z � � � \[�] % d&;)� ���8K%�

.&] � +±¸ � � � ½ �SK ? �8?*� � "%� � �4K��S?�� �6? � 1%��� �� � � , E�» � � F�7 �8?*�UK:; +jO � ��; ��� �6?*M C E�» � � F%1%��� � � » �%$&$&$&� # �

[ ] <D+CBD��¸ � ( 3 � � � ½5�NK ?*��?*� � " � � �4K �6?��'�S? � 1���� �� � � ( ,0M=7 � , E�» � � F�7 �8?*� � � » �%$&$&$&� # �

"�] � ':¸ � ½ �SKL?*�8?*� � "%� � �4K��S? �&�6? � ��?*� ��� " �8?,����5�G� M(�j? * +±¸ � ½5�NK ��� " �8?,��� 5!1%��� � � » �%$&$&$&� G

� � �9� �3� � � � � �� [ ] % � ¢£�±�´¨� ¥�h¨´��¨�©~¦�¨ �8É ¬ �+«$«$¦/¾8µ ¾/¦h ~¡�¦h�±��§~«T�~¨�¢£�^ ~� � � ¾ � Ã/� Å ¬£�+«$«$¦/¾8µ � �_­\� ²�¦h ²³�^ ~²�¬£§L¡¥��¨�©~¦�¨ * +[¢£�1 ¥�^ L¡¥�+²³���+¦h��¢£ ~¤ Å �h�\¦h¬£¬ � µ ����§~�´�;¦h /¢£ ~¡L§~²³¨�¢ Ëh�W¦���¤^§L«T�+ N¨\¨´���L����Ëh�;¨�©¥�W�L���h�c�^��¢ ¨�¢ �^ ¶µ�t§¥�L�_�^��� * +CBD�?¢£� È Â�²³�^ JËh�³Æcµ � ©¥�+  �JÉ�¬£�+«$«$¦0¾8µ Ã�­:�*²�¦h 4²³�^ ~²�¬£§~¡~�$¨�©~¦�¨ � �� G 2 +CBD��¸76 3 � � � ½+* +CBD�h¸76J½@¢£���§ �q«T�8¡L§~¬£¦��1�^ /M C>E�» � � F^µ � ©¥�B��§L« � Å ¨e­\�/�±§ �L«T�t¡~§~¬£¦�� Å §L ~²³¨�¢ �^ ~�|¢£� ��§ �L«T�t¡~§~¬£¦��+ÊJ¨�©J§L� �JÉ*�~���h�c����¨eÉ� �,� +±¸ � � � ½|¢£� �±§ �L«T�t¡~§~¬£¦���ª

� + ¸ � B � � ½$# � + ¸ � � � � ½ � � + ¸ � B ��»N½$# � + ¸ � � ��»N½)$ ¸ � ¼^½

�;� �L ¥��� � + ¸ � ½ � � + ¸ � � � ½Y � + ¸ � ��»N½ Å �h� � ,0M�µ %\É$�±§ �L«T�t¡~§~¬£¦��2¢ ¨�Éy� Å � + �8­\��²³�^ ~²�¬£§~¡¥�W¨�©~¦�¨�� � + ¸ � ½¢£� ¦$ ~�^ ~¢£ ~²³���+¦h�±¢£ ¥¤T�´� #8§¥�+ ~²³�y¢£  � ª

� � +±¸ »N½�:�� � +±¸ � ½�: $&$&$ :�� � +�¸+*P½ : $&$&$

�A��¨�� �c�y��§~²±©¨�©~¦�¨�� � +�¸��[½|¢£�|¨�©~� �~�±�´¨  ¥��¤^¦�¨�¢£Ëh�@¨´���±«-¢® z¨�©¥�@�´� #8§¥�+ ~²³�� � +Pµ� �h� � :� ��� � +±¸ � ½ � » � � +±¸ � � � ½ � � +±¸ � ��»N½ � *�+�¸ � ½ � � +±¸ � � � ½2µ� �h� ��� � ��� � +±¸ � ½ :R» � � +±¸ � ��»N½ � � +±¸ � � � ½ � *�+±¸ � ½ � � +±¸ � ��»N½2µ6;��¢£ ¥¤�¸ � ¼^½|¦h L¡ �~���h�c����¨eÉz¹T¦h L¡z¨�©¥�@¦��c��Ëh��­\�B�~�±��Ëh�@¨�©~�B���+��§~¬ ¨ Å �h��¨�©¥� Å �^¬£¬ ��­W¢£ ¥¤ �~Ëh��²�¦h�´�+��ª�|¦h�´�*¸ ¦N½2ª � B � � � � � � � � � B � � � :��.µ

*�+±¸ � B � ��½'# * +±¸ � � � ��½ � � +±¸ � B � � � � ½ # � +±¸ � � � � � � ½� � +±¸ � B ��»N½$# � +±¸ � � ��»N½: � + ¸ � B � � ½$# � + ¸ � � � � ½

Page 16: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� � �

� *�+±¸ � B ½'# * +±¸ � � ½)$�|¦h�´�*¸ �_½2ª � � � � � � B � � � � � � � B � �4µ

*�+±¸ � B � ��½'# * +±¸ � � � ��½ � � +±¸ � B � � � � ½ # � +±¸ � � � � � � ½� � +±¸ � B ��»N½$# � +±¸ � � ��»N½� � +±¸ � B � � ½$# � +±¸ � � ��»N½� *�+±¸ � B ½'# * +±¸ � � ½)�

��¢£ L²³� � � +�¸ � B ½ � »tµ�|¦h�´�*¸ ²�½2ª � � � � B � � � � � � � � � B � �4µ

* + ¸ � B � ��½'# * + ¸ � � � ��½ � � + ¸ � B � � � � ½ # � + ¸ � � � � � � ½� � +±¸ � B ��»N½$# � +±¸ � � ��»N½� *�+±¸ � B ½'# * +±¸ � � ½)$

�|¦h�´�*¸ ¡q½2ª � � � � B � � � � � � � � � B � �4µ*�+±¸ � B � ��½'# * +±¸ � � � ��½ � � +±¸ � B � � � � ½ # � +±¸ � � � � ��»N½: � +±¸ � B � � � � ½ # � +±¸ � � � � � � ½

� � + ¸ � B ��»N½$# � + ¸ � � ��»N½� *�+±¸ � B ½'# * +±¸ � � ½)�

��¢£ L²³� � � +�¸ � � � ��½�: »tµ�|¦h�´�*¸ ��½2ª � � � � B � � � � � � � B � � � �.µ

*�+±¸ � B � ��½'# * +±¸ � � � ��½ � � +±¸ � B � � ��»N½ # � +±¸ � � � � ��»N½: � + ¸ � B � � � � ½ # � + ¸ � � � � � � ½� � +±¸ � B ��»N½$# � +±¸ � � ��»N½� *�+±¸ � B ½'# * +±¸ � � ½)�

��� ©~¦+Ëh�z�~�±��Ëh�+¡4¨�©~¦�¨?¢ Å * +CBD�?¢£� È Â�²³�^ NËh�³Æ4¨�©¥�+  * +�¢£� È Â�²³�^ NËh�³Æcµ �t¢£ ~²³�,* ' � � ' ¢£�@¦h���±§~«T�+¡�¨´� �c�È Â�²³�^ JËh�³Æ �q­\��©~¦+Ëh� �JÉz¢£ ~¡~§L²³¨�¢ �^ *¨�©~¦�¨ *�+±¸ � ½:¢£� È Â�²³�^ NËh�³Æ Å �h� � � » �%$&$&$&� µ � $ � $�� $

-���­ ­\�@¦��±�y���+¦h¡¥É*¨´�$�~����Ëh� È Â�²³�^ JËh�³Æt¢£¨�É � Å ¨�©¥�yË�¦h¬£§¥� Å §~ ~²³¨�¢ �^ ¶ª

Page 17: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� � 
� � �9� � � ��� � ��� � ��� � [ ����©~¦+Ëh� ¦h¬ ���+¦h¡¥ÉU�~����Ëh�+¡�¨�©~¦�¨W¢£ z�^§~��«T�8¡~�+¬c¨�©¥�y�^ ¥�B�c���±¢ �t¡z²³�^�´¨ Å §~ ~²³¨�¢ �^  � +¢£�� ¥�^ ~¡~�+²³���+¦h��¢£ ¥¤ ¦h ~¡��±§ �L«T�t¡~§~¬£¦��?¸ �~���h�c����¨�¢ �+�Bà � �J½;¦h L¡0¨�©L¦�¨=< +CBD�@¢£�� ¥�^ L¡¥�+²³���+¦h��¢£ ~¤*¢£  � ¸ �~���h�c����¨�ÉÀ^½2µ � ©8§~�y¦h����§L«T�~¨�¢ �^ ~� � ¦h ~¡�¾ � Å ¬ �+«$«$¦�¾8µ �k¦����U�±¦�¨�¢£� �~�+¡¶µ ���±��§~«T�/¦ ��������¨´���±«/¢£ ~¦h¬�²³�^�´¨ Å §L ~²³¨�¢ �^ ­W©~¢®²±©$¢£� �_�h¨�©U ¥�^ ~¡¥�+²³�±�+¦h��¢£ ¥¤y¦h ~¡ È Â�²³�^ JËh�³ÆU¦h ~¡/¢£¨���¦�¨�¢£� �~�+�\¦h����§~«$�~¨�¢ �^ /Ã�� Å ¬£�+«$«$¦@¾8µ �¥µ � ©8§~��¬£�+«$«$¦¾8µ �$¤^¢ Ëh�+��§~� È Â�²³�^ JËh�³Æ¥¢ ¨�É � Å ¨�©¥�yË�¦h¬£§¥� Å §~ ~²³¨�¢ �^  Å �h�W¦h¬£¬c¨�¢®«T�B�c���±¢ �t¡~��µ � $ � $�� $

�1���h�c����¨�ÉÄ?¢£� ¦T¨´�+²2©~ ~¢£²�¦h¬A²³�h���^¬£¬£¦���É*¨�©~¦�¨W¢®� � Å ¢® N¨´�����+��¨W¢£  �´�+²³¨�¢ �^ 0¾8µ Ã8µ� � �9� �N� � � ����� � ��� � �� %:É$�~���h�c����¨�¢£�+� � ¦h L¡*¾y­\��©~¦+Ëh��¨�©~¦�¨ * + ¢£� �_�h¨�© È Â�²³�^ JËh�³Æ*¦h ~¡* ¥�^ ~¡¥�+²³�±�+¦h��¢£ ¥¤Å �h��¦h¬®¬ � � » �%$&$&$&� µ � ©8§~� �JÉk¬ �+«$«$¦z¾8µ ÃU­\�$²³�^ ~²�¬®§~¡¥�@¨�©L¦�¨ � �� G 2 +CBD��¸76 3 � � � ½+* +�¸76J½;¢£����§ �q«T�8¡L§~¬£¦�� Å �h�¦h¬£¬ � � » �%$&$&$&� µ � $ � $�� $

�� �� � "��"���"���$��B ���'�� ��� � "#��' � '($*)

�;� � ' �8��� Å �����4¸ � ¼^Ä�Ã^½���©~��­:�+¡�¨�©~¦�¨?¢£ ���¨´�+¦h¡¥É��´¨�¦�¨´�*¨�©¥�/�h�L¨�¢£«$¦h¬��´���±Ë8¢£²³�*�´¨´�±¦�¨´��¤hÉ Å �^¬£¬ ��­W�@¦0²³�^ N¨´�±�^¬¬£¢£«/¢ ¨W�´¨´�±§L²³¨�§¥��� �c­W©~¢£²2©0¢®�W¨´� ��¦�É�¨�©~¦�¨�¨�©¥�T�´����Ët¢£²³�?��©¥�^§~¬®¡��c���c��� Å �h�±«T�+¡��^ ~¬ É­�©¥�+ �¨�©~�?¬£�+ ¥¤h¨�©�� Å ¨�©¥�#8§¥�+§¥�|�³Æt²³���+¡L��¦��L¦���¨�¢£²�§L¬£¦��.¬£¢£«$¢ ¨�µ �����~����Ëh� �h�~¨�¢£«$¦h¬£¢£¨�Éy� Å �c�^¬£¢£²�¢ �+�[­W¢ ¨�©?¦B ¥�^ ~��¨�¦�¨�¢ �^ ~¦���É ²³�^ N¨´���^¬L¬£¢£«$¢ ¨�µ�1§~¨´���±«$¦h �¸ � ¼h¼��J½��~����Ëh�+�W¦T��¢£«/¢£¬£¦��|¤h�+ ¥���±¦h¬A���+��§~¬ ¨|�^ «$�^ ¥�h¨´�^ ¥�y�c�^¬£¢£²�¢£�+��ª

�� � ��� � � . 3 ;(� ���>O �8? 7 �6M-O��*G� G��47 � G�������7��������89Dd&;)� �(�4K%�

.&] � +±¸ � � � ½5�SKJK:; +jO � ��; ��� �8? M C E�» � � F ��?*� ?*�8?*��� "%� � �8K �6?��&�6? � G[ ] < +CBD� ¸ � ( 3 � � � ½5�NK ?*��?*� � " � � �4K �6?��'�S? � 1���� �� � � ( ,0M �8? � � , E�» � � F-G"�] � �� G 2 +CBD��¸76 3 � � � ½ � ¸76J½ �SK � K>;,+PO � �8;� ���%1:;@?*" �����8?.�8? O C�E�» � � Fkg�M��j? �����%� � �NK � ?*��?*� � " � � �4K �6?��

1>;@?*" ���#�8? �8? O � E�» � � $&$&$ F-G% ] � ':¸ � ½ � * '\¸ � ½5�NK ?*��?*� � " � � �4K �6?��'�S? � G

� M��P?�6M(��� � � 5��SK��6K ��? � � ���6O �� � � "%�SK �#�8? �:;� � ���+ ¸ � ½ ,>E�» � � FBg�M ��"PM'�NKL?*��?*� � " � � �4K �6?��'�S? � 1%��� �� � � G� ©~�z©~�^«T�h¤h�+ ¥���^§~�$²�§L�´¨´�^«T��� �L¦�¨�²2© �´����Ët¢£²³�z�~��� �L¬£�+«>¡¥�8�+�T ¥�h¨T��¦�¨�¢£� Å É4²³�^ ~¡~¢ ¨�¢ �^ �Ã0� Å ¨�©~�¦��_��Ëh�

¨�©¥���h���+«µ � �^ ¥�h¨´�^ L¢£²�¢ ¨�É/� Å ¨�©¥��Ë�¦h¬£§¥� Å §L ~²³¨�¢ �^ /¢®�1 ¥�h¨:¦���§ � ²�¢ �+ J¨\²³�^ ~¡~¢£¨�¢ �^ $¨´� �~����Ëh�;��§ �L«T�t¡~§~¬£¦��±¢£¨�É@� Å� �� G 2 +CBD��¸76 3 � � � ½+*z¸768½2µ��;²�²³�h�±¡~¢£ ¥¤U¨´�*¬£�+«$«$¦/¾8µ Ã/­\�?¦h¬£�´�* ¥���+¡0¨�©¥�@�³Æt¨´�±¦*¦h����§~«T�L¨�¢ �^  � Å È Â�²³�^ JËh�³Æt¢£¨�É� Å ¨�©¥�UËǦh¬£§~� Å §~ ~²³¨�¢ �^ ¶µ'6;��¢£ ~¤¨�©¥�U��¨´�±§~²³¨�§¥�±¦h¬1�~���h�c����¨�¢ �+�y� Å �´�+²³¨�¢ �^ #¾8µ � ­:�*�~����Ëh�U�^§¥� Ëh���±��¢ �^ �� Å ¨�©¥����+��§L¬ ¨|­W©~¢£²2© ¢£ NËh�^¬£Ëh�+� È Â�²³�^ JËh�³Æt¢£¨�Éhµ

Page 18: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� � ¹

� � ��� � ��� ��� ��� . � ���5�6M�� M ��O �����j? � �8;�K " ;-K�� �8O ��� + ��� "jM K%�%� �)��"�� � � � +% �PO �6M��%� � ��5��SK��6K��8? � � ���6O �� � � " � �K �#�8?�>; � ���+ ¸ � ½ , E�» � � FBg�M �#"jM&�SK ?*�8?*� � "%� � �4K��S? �&�6? � 1%��� �� � � G

���B§~�´��¨�©¥� Å �^¬£¬£��­W¢£ ~¤@¬ �+«$«$¦ Å �±�^« �\§¥¨´���±«$¦h p¸ � ¼h¼��J½�¸��H�+«$«/¦��¥µ ÄJµ � �¥�L¦�¤h� � »��T¦h ~¡ 4.Æt���±²�¢£�´� �¥µ � µ Ä ��L¦�¤h� ��� À^½2µ

Z � � � \[�] ( d&;)� ���8K%�5� �SK � K>;,+PO � �8;� ���%1:;@?*" �����8?.�8? � C ���8?*� 1���� � � "jM Q�, �\7�«$¢£ �� � � �?¸6Q �YT~½� 5 �NK��6KjG � M��P?

X ¸6Q¶½ � «$¦ÇÆ � T ( ,¦���¤\«/¢£ � � � � ¸6QD�YT¥½�� ¸º¾�»N½

�SK O ��?*��� �8? � ?*�8?*� � "%� � �4K��S? �&�6? Q�G

� � �9� �N� � � ����� ����� ��� � ��. %:ÉT�~���h�c����¨�É � ­\��©~¦�Ëh��¨�©~¦�¨ � +±¸ � � � ½1¢£�\��§ �L«T�t¡~§~¬®¦�� Å �h�|¦h¬£¬ � µ %:É$�~���h�c����¨eÉÄ@­\�B©~¦�Ëh�B¨�©~¦�¨ � �� G 2 +CBD��¸76 3 � � � ½+*�+CBD��¸76J½:¢®�\��§ �L«T�t¡~§~¬£¦�� Å �h� ¦h¬£¬ � µ � ©J§~� � +�¢£�|��§ �L«$�8¡~§L¬£¦�� Å �h�|¦h¬®¬ � �¥¦h�¦@�±§~«�� Å ��§ �L«T�t¡~§~¬£¦�� Å §~ ~²³¨�¢ �^ ~�+µ 6;��¢£ ¥¤�¬ �+«$«$¦@¾8µ À@­:��²³�^ ~²�¬£§L¡¥��¨�©~¦�¨\¨�©¥�����W�³Æ¥¢£�´¨��\¦h /�h�L¨�¢£«$¦h¬~¡¥�+²�¢®��¢ �^ �±§~¬£� �t�±¦+É � �+ ¸ � ½ �~¨�©~¦�¨W¢£�  ~�^ ~¡¥�+²³���+¦h��¢® ¥¤$¢£  � Å �h�W¦h¬®¬ � µ � $ � $�� $

�1���h�c�^��¢ ¨�¢ �^  � ¤^¢£Ëh�+�\§~��¨�©¥�����+��§L¬ ¨.¨�©~¦�¨1�´���±Ë8¢£²³�+�\²�¦h  �c���c��� Å �h�±«T�+¡ �L¦h���+¡$�^ $¦@¨�¢£«T� Ë�¦���É8¢® ¥¤y²³�^ N¨´�±�^¬¬£¢£«/¢ ¨\�c�^¬£¢®²³Éhµ ��¬ ¨�©~�^§¥¤^©¢£ J¨�§~¢ ¨�¢ Ëh�+¬£É$� �JËt¢ �^§~��¨�©¥�y���+��§L¬ ¨�¢£� ¢£«T�c�h��¨�¦h J¨�¢£  ²�¬£¦h�±��¢ Å É8¢£ ~¤ �^§~���~��� �L¬£�+«w¢£ z¨�©¥�²�¬£¦h����� Å �L��� �L¬ �+«$�:­W¢£¨�© «T�^ ¥�h¨´�^ ¥���h�~¨�¢£«$¦h¬¶�´¨´�2¦�¨´��¤^¢ �+��µ

� � ���� �� � � ��� ������������ ©¥�B�h�L¨�¢£«$¦h¬£¢ ¨eÉ/� #8§~¦�¨�¢ �^ ~��� Å ¨�©~�y«T�8¡~�+¬¶­\�y¡~� �L ¥�y¢£  �´�+²³¨�¢£�^  � ¦����y¤^¢£Ëh�+ ��8Éqª

*�+±¸ O +e½ � «$¢£ - � E � +±¸ O +7� � +e½ � � ! *�+CBD�Ǹ O +CBD��½ 3 O +&%2F � ¸º¾ � ½

Å �h� � � » �%$&$&� # � µ � �h�@¨�©¥�$�L��� �L¬ �+« ¨�©~¦�¨@­\�$²³�^ ~�±¢£¡¥�����H­�©~¢£²±©p�³Æ¥©~¢ �q¢ ¨��B¦��²�¦h¬£¦����´¨�¦�¨´�/Ë�¦��±¢£¦��L¬£� �H¨�©¥��h�~¨�¢£«/¦h¬£¢ ¨�É@� #J§~¦�¨�¢£�^ ~��²�¦h  �_�|�´�^¬ Ëh�+¡T¡L¢ ���+²³¨�¬ Éy§~�±¢£ ¥¤�¦��L¦h² �J­|¦��±¡T¡¥Ét ~¦h«$¢®²1�~�±�h¤h�±¦h«$«$¢£ ~¤�¦h¬ ¤h�h�±¢£¨�©~«w¸ �´��� �Å �h�W�³Æ¥¦h«T�L¬ � � �1§¥¨´���±«/¦h �¸ � ¼h¼��J½´½2µ � ©~¦�¨�¢£���~¤^¢ Ëh�+  *�+CBD��¸ O +CBD��½ �q­\� ²�¦h  �L L¡ * +±¸ O +e½ �JÉ���^¬ Ë8¢® ¥¤T� #J§L¦�¨�¢ �^ ¸º¾ � ½ Å �h�y�+¦h²±©��c�^����¢ �L¬£� �´¨�¦�¨´� O +¿µ � ©¥�T�~��� �L¬£�+« ­�¢ ¨�©p¨�©~¢£��¦��~�~�±�^¦h²±©�¢®��¨�©~¦�¨y¢ ¨y¡¥�8�+�B ¥�h¨B¤h�+ ¥���±¦h¬®¢����T¨´�«T�h���*²³�^«T�L¬ �³Æ��~�±� �L¬ �+«$�B­�©¥����� O +�¢®�@¦Ëh�+²³¨´�h��µ � �h�?¨�©~¢£�B���+¦h���^ ���¨�©~¢£�y�´�+²³¨�¢£�^ ��~�±�h�_�^���+�y¦h �¦h¡L¦��~¨�¢ Ëh�¡¥Ét ~¦h«$¢£²;�~�±�h¤h�±¦h«$«$¢£ ~¤?¦h¬£¤h�h�±¢ ¨�©~« µ��B§¥� ¤h�^¦h¬H¢£� ¨´�U¢£ N¨´�±�8¡~§L²³��¦h �¦h¬ ¤h�h�±¢ ¨�©~« ­�©~¢£²±© ¢®� ��²�¦h¬£¦��L¬ �B¨´�U«T�h���²³�^«T�L¬£�³Æk�~�±� �L¬ �+«$� �H¦h L¡��´¨�§~¡¥É0¨�©¥�$�~�±�h�_���±¨�¢ �+�;� Å ¨�©L¢£�B¦h¬ ¤h�h�±¢ ¨�©L« ¢£ p¨�©¥�/²³�^ J¨´�³Æ8¨�� Å ¦ ��²�¦h¬£¦��@�L��� �L¬ �+«­W©~¢®²±© ²�¦h ��c�y�´�^¬£Ëh�+¡ ¨´�$�h�~¨�¢£«/¦h¬£¢ ¨�Éhµ

Page 19: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� � Ä

��§~�\�~�±�+�´�+ N¨�¦�¨�¢£�^  �_��¤^¢® ~� �8É/¢£ J¨´���t¡~§~²�¢£ ~¤y¨�©¥��²³�^ ~²³���~¨ � Å ¦h �6?*" ��O � � � �LK�� ��� �5����������+� �:­W©~¢£²2©z¬ �+¦h¡~�¨´�z¦/¡~¢"!c�����+ J¨W¡¥Ét ~¦h«$¢£²y�~���h¤h�±¦h«$«/¢£ ¥¤T���+²�§¥�2��¢ �^ ¶µ � ©¥�+ ­\� �~����Ë8¢£¡~�@¦ Å �h�±«$¦h¬[¡¥�+��²³�±¢ �L¨�¢ �^  � Å ¨�©¥� �L¦h��¢£²Å �h��­:¦��2¡$¡¥Ét ~¦h«$¢£²|�~�±�h¤h�±¦h«$«$¢£ ~¤�¦h¬£¤h�h�±¢ ¨�©~«�¦h L¡ �L ~¦h¬£¬£É@­:�W¢£ J¨´���8¡L§~²³�|�^§¥�\«T�^ ~�h¨´�^ ¥�W¦h¡~¦��~¨�¢£Ëh�W¡¥Ét ~¦h«$¢£²�~���h¤h�2¦h«$«$¢£ ¥¤T¦h¬ ¤h�h�2¢ ¨�©~«-¦h��¦$ËǦ��±¢®¦�¨�¢ �^ z� Å ¨�©¥� �L¦h��¢®²;¦h¬£¤h�h�±¢ ¨�©~« µ

�� �� ��� $R' ���@"��W ��($&�8$ ) ������$���� ��'���� �($

�;¬ ¨�©¥�^§¥¤^©k²³�^ ~��¢£¡~���±¦��L¬ � ¦�¨´¨´�+ N¨�¢ �^ �¢®�W¤^¢ Ëh�+ k¨´�z¨�©¥�?²³�^«T�L¬ �³Æ¥¢ ¨eÉ�¢£ J¨´���8¡L§~²³�+¡­W©~�+ �¨�©¥�?�´¨�¦�¨´�TË�¦��±¢£¦��L¬£� O +¢£�B¦zËh�+²³¨´�h�z¸ ¨�©¥�/�´��Â�²�¦h¬£¬£�+¡��´²�§¥�±�´�T� Å ¡~¢£«T�+ L��¢ �^ ~¦h¬£¢£¨�ÉN½ ­:�$©~¦+Ëh� Å �^§~ L¡p¨�©~¦�¨ �A¢£ p���+¦h¬.¦��L�L¬£¢£²�¦�¨�¢ �^ L� �¶¨�©¥��³Æt�_�+²³¨�¦�¨�¢£�^ �¢£  � #8§~¦�¨�¢ �^ �¸º¾ � ½�²�¦h  �_�B� #8§~¦h¬£¬£É/�~��� �q¬ �+«$¦�¨�¢£²�µ � ©¥�@�´¨�¦h ~¡L¦��±¡¦��~�~�±�^¦h²±©�¢®�|¨´�/�����L¬®¦h²³�y¨�©¥��³Æt�_�+²³¨�¦�¨�¢£�^ z­W¢ ¨�©*¦h *¦��L�~���+Æ¥¢£«$¦�¨�¢£�^  �L¦h���+¡U�^ z¦?��§ �q�´��¨1� Å ¨�©¥� Å §~¬£¬q��¦h«T�L¬£�;�´�q¦h²³� � �q§¥¨\¨�©L¢£�\²�¦h z�´¨�¢®¬£¬��c�²³�^«T�L§~¨�¦�¨�¢ �^ ~¦h¬£¬ É*¡~¢ � ²�§L¬ ¨ Å �h�W¬®¦���¤h�����~��� �q¬ �+«$��µ

���?­|¦h N¨�¨´�z²³�^ ~��¢£¡¥����¦h p¦��~�~����Æt¢®«$¦�¨�¢ �^ k§~��¢£ ¥¤ � �^ J¨´�/�|¦��±¬ �z��¦h«T�L¬®¢£ ¥¤¥µ � �h�B¦z��¦h«T�q¬ �@���+¦h¬®¢��+¦�¨�¢ �^ ��� ¸�� � �%$&$&$&��� ' ½\­\�@�~���h�c�^�´�B¨�©¥� Å �^¬£¬ ��­W¢£ ¥¤T¦��~�~�±�+Æ¥¢£«$¦�¨�¢ �^ ¶ª

�* +1¸ � +�½ � «$¢® - � � � +±¸ � +5� � +P½ � �*�+CBD�Ǹ � +CBD�2½"!¸�-��h¨´�$¨�©~¦�¨B­:� �±� Å ���B¨´� �^§¥�y¦��L�~���+Æ¥¢£«$¦�¨�¢£�^ k¦h� �*/Ê �* ¢£�B¦ Å §~ ~²³¨�¢ �^ k¨�©~¦�¨y�´���±Ëh�+�y¦h�B¦z�L¬£¦h²³�+©¥�^¬®¡¥�����¶ ¥�h¨¨´���_� ²³�^  Å §~�´�+¡0­W¢ ¨�© �*Tµ ½ 2W��­\��Ëh����� � +CBD� � ¸ � +'# � + ��½ B � �#+CBD�;¢£�;¦ Å §~ ~²³¨�¢£�^ � Å �#+CBD��¦h L¡�¨�©8§~��­W©¥�+ ²�¦h¬£²�§~¬®¦�¨�¢£ ¥¤@¨�©~����§ �c�h�~¨�¢®«$¦h¬ � +�¢£ /¨�©¥�;¦��c��Ëh�;«$¢® ~¢£«$¢��+¦�¨�¢£�^ T­\��¦�����§~��¢£ ~¤ Å §~¨�§¥���W¢£  Å �h�±«/¦�¨�¢ �^  Å ���^«�¨�¢£«T��$� � µ � �k²³�h�����+²³¨ ¨�©~¢£�y�L��� �L¬ �+« ­:�/�±��Ë8¢£�±¢ ¨B¨�©¥�/�L��� �L¬ �+«>¡¥� �L ~¢ ¨�¢£�^ �¦h ~¡�¢£ J¨´���t¡~§~²³�$­�©~¦�¨y­:�*²�¦h¬£¬1¨�©¥�¢£ ~²³�^«$�L¬ ��¨´�y�´¨�¦�¨´�yË�¦��±¢£¦��q¬ �hµ

� ©~�y¢£  Å �h�±«$¦�¨�¢ �^ z�~�±�8²³�+����� Å ¨�©¥��«T�8¡~�+¬¶¦h�W¡¥� �L ~�+¡ ¢£  �´�+²³¨�¢ �^  � ¢£� ¦h� Å �^¬£¬ ��­W��ª

E O � � ��H*��� O ��� � ����H �!� O � � � �!�%$&$&$&��HP'-� O '�F��� ­|¦h N¨?¨´��«T�+¦h��§¥�±�U¨�©¥� �´¨�¦�¨´�zË�¦��±¢£¦��L¬£�U¦�¨T¨�¢£«T� � �c� Å �h���U¨�©~�*¦����2¢ ËǦh¬®� H + µ ���²�¦h¬£¬1¨�©L¢£�? ¥��­w�´¨�¦�¨´�Ë�¦��±¢£¦��L¬ �0¨�©¥��¢£ ~²³�^«T�q¬ ��¨´�0�´¨�¦�¨´��Ë�¦��±¢£¦��q¬ �0¦h ~¡ ­:�p¡~�+ ¥�h¨´��¢ ¨��8É O �+ µ � ©8§~�U¨�©¥��¢£  Å �h�±«$¦�¨�¢ �^ F�~���t²³�+����c�+²³�^«T�+��ª

E O � ��H � � � O �� ��H=��� � ��� O �� ��HI�!� � �!�%$&$&$&� O �'��9� ��HP'��9�%� � '��9��� O �' F­W©¥���±� O � � O #�H �q¦h L¡ O �' � O ' #�H�'�µ -��h¨´��¨�©L¦�¨ O +1²³�^ J¨�¦h¢£ ~�W¨�©¥���±¦h«T��¦h«T�^§L N¨�� Å ¢£  Å �h�±«$¦�¨�¢ �^ ¦h� O �+ ¦h L¡/HP+.¨´�h¤h��¨�©~���+Êq¢£  Å ¦h²³¨�­:�@©~¦�Ëh� O + � O �+ � HI+´µ ���@�±§ �L�´¨�¢ ¨�§~¨´�B¨�©~¢£� ¢£ J¨´��¸º¾ � ½2ª

*�+±¸ O �+ � HI+e½ � «$¢® - � � � +±¸ O �+ � HP+7� � +e½ � � ! * +CBD��¸ O �+CBD�'� HI+CBD��½ 3 O �+ � HI+&% ! $ ¸º¾h¾^½

Page 20: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� � Á

� ©¥�y��É8�´¨´�+« ¡¥É8 L¦h«$¢£²�� ¦h¬£�´�U²±©~¦h ¥¤h�@¨´�¥ª

O �+CBD� �"! O �+ � HI+�# � � +&% B ¸º¾hÃ^½

-���­ ­:�y¨�¦��h���³Æ8�c�+²³¨�¦�¨�¢ �^ ~�W� Å �c�h¨�©�±¢£¡¥�+�|� Å ¸º¾h¾^½|²³�^ ~¡~¢£¨�¢ �^ ¥�+¡z�^  O �+ ª� � * +±¸ O �+ � HP+e½ 3 O �+�� � ��� «$¢£ - � � � +±¸ O �+ � HI+7� � +e½ � � ! *�+CBD�Ǹ O �+CBD�'� HP+CBD��½ 3 O �+ � HP++% ! 3 O �+��

�t¢£ L²³��¨�©¥�B�³Æt�c�+²³¨�¦�¨�¢ �^  � Å * +CBD� ¸ O �+CBD��� H +CBD� ½:²³�^ L¡~¢ ¨�¢ �^ ¥�+¡*�^  O �+ � H + ¢£� ¦h¬£���?¢® ~��¢£¡¥��¨�©¥�y«$¢£ L¢£«$¢��+¦�¨�¢ �^ ��Ëh��� � + �¥­:��²�¦h¬®²�§~¬£¦�¨´�B¨�©¥���³Æt�_�+²³¨�¦�¨�¢£�^ z� Å *�+CBD��¸ O �+CBD� � HI+CBD��½1¦h�±��§~«$¢£ ~¤�¨�©~¦�¨|­:� �8 ~��­ O �+ � HI+Y¦h L¡ � +´µ� ©¥�B¢£  Å �h�±«/¦�¨�¢ �^ z²³�^ N¨�¦h¢£ ~�+¡ ¢£  O �+ � HI+.¦h ~¡ � +�¢£�:¨�©¥�y��¦h«$�y¦h�|¨�©¥�y¢£  Å �h�±«$¦�¨�¢ �^ *¢£  O �+CBD� µ � ©8§~�|­:�y²�¦h �����L¬®¦h²³�B¨�©¥�@²³�^ ~¡~¢£¨�¢ �^ ~¦h¬c­W¢ ¨�© O �+CBD� ª

� � * +±¸ O �+ � HP+e½ 3 O �+ � � ��� «$¢£ - � � � +±¸ O �+ � HI+7� � +e½ � � ! *�+CBD�Ǹ O �+CBD� � HP+CBD��½ 3 O �+CBD� % ! 3 O �+ � ¸º¾��J½

-���­ ¬£��¨�§~��¡¥� �L ~�B¨�©¥� Å �^¬®¬ ��­W¢® ¥¤ Å §~ L²³¨�¢ �^ ~��ª

* �+ ¸ � ½ # � ! * +±¸ O �+ � HP+e½ 3 O �+ �5� % ¸º¾hÀ^½� �+ ¸ � �+ ���#+7� � +e½ # � +±¸ � �+ � �(+7� � +e½ � � � + � [¸ � �+ � �#+�# � � +e½ B ¸º¾h¹^½

� ©¥�+ �¸º¾��J½ �c�+²³�^«T�+��ª

* �+ ¸ O �+ ½ � ��� «$¢£ - � � � �+ ¸ O �+ ��HI+�� � +e½ � * �+CBD� ¸ O �+CBD� ½ !-3 O �+ � ¸º¾^Äh½

Å �h��¦h¬£¬ � � » � � �%$&$&$&� # � µ � ©¥�+�´�B¦����B¨�©¥���h�~¨�¢®«$¦h¬£¢ ¨eÉ/� #8§~¦�¨�¢ �^ ~�|²³�h�����+���_�^ L¡~¢£ ¥¤ ¨´�T¨�©¥�B¢£ ~²³�^«T�q¬ ��¨´�B�´¨�¦�¨´�Ë�¦��±¢£¦��L¬ �hµ -��h¨´� ¨�©~¦�¨?�^§¥�$ ¥��­ Ë�¦h¬£§¥� Å §L ~²³¨�¢ �^  * � ¢£� ¦h¬£�´�� ¥�^ ~¡~�+²³���+¦h��¢£ ¥¤p��¢£ L²³�*¢ ¨T¢£� ¨�©¥� ²³�^ ~¡L¢ ¨�¢ �^ ~¦h¬�³Æt�_�+²³¨�¦�¨�¢£�^  � Å ¦? ¥�^ ~¡~�+²³���+¦h��¢£ ¥¤ Å §~ ~²³¨�¢ �^ Hµ � ©¥�;¦��c��Ëh�B�³Æ8�c�+²³¨�¦�¨�¢ �^ ¢®�\¨�¦��h�+ ��Ëh���W¨�©~�;�2¦h ~¡¥�^«uËǦ��2¢£¦��L¬ �H + µ � ©J§~�;¢ Å ­:�T²³�^ ~��¢£¡¥���±�+¡0¦h p¦��~�~�±�+Æ¥¢£«$¦�¨�¢ �^ k§~��¢£ ~¤ � �^ N¨´�/�|¦��±¬£�*��¦h«$�L¬£¢£ ¥¤/¨�©¥� ¡~�+²�¢£��¢ �^ 0ËǦ��2¢£¦��L¬ � � +²³�^§~¬£¡ �c�y¡¥��¨´���2«$¢£ ¥�+¡ §~�±¢£ ¥¤?�^ ~¬ É*¨�©¥�y��¦h«T�q¬ �B���+¦h¬£¢��+¦�¨�¢£�^ z� Å ¦��±�±¢ Ë�¦h¬£� ¦�¨�¨�¢£«T� � ���#+´µ

�� S� ��� $ ���)8'�� � �� " ��'(� � ��  �¨�©L¢£���´�+²³¨�¢ �^  ­\� �L����Ët¢£¡¥�:¦ Å �h�±«$¦h¬t�´¨�¦�¨´�+«$�+ N¨�� Å ¨�©¥� �L¦h��¢£² Å �h��­:¦��2¡ ¡~É8 ~¦h«/¢£²��~�±�h¤h�±¦h«$«$¢£ ~¤;¦h¬ ¤h�h�2¢ ¨�©~« µ�  *���+²³¨�¢ �^ Ã8µ à ­\���~�±�8²³���+¡*¨´�$¢£ J¨´���t¡~§~²³�W�^§~�|«$�^ ¥�h¨´�^ ¥�y¦h¡~¦��~¨�¢£Ëh��¡~É8 ~¦h«/¢£²W�L���h¤h�±¦h«$«$¢® ¥¤�¦h¬ ¤h�h�2¢ ¨�©~« ¦h�¦TË�¦��±¢£¦�¨�¢ �^ z� Å ¨�©¥�y¦h¬£¤h�h�±¢ ¨�©~«-¡~�+��²³�±¢ �c�+¡z©¥�����hµ

Page 21: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� � ¼

���k�´¨�¦���¨$­W¢£¨�©#¨�©¥� ¥��­-�h�~¨�¢®«$¦h¬£¢ ¨eÉ�� #8§~¦�¨�¢ �^ ~�0¸º¾^Äh½2µ � �h�*¦���¦h«T�L¬£� ���+¦h¬£¢��+¦�¨�¢ �^  HP+�¸ � ½ � �#+ �\­:��~���h�c�^�´�B¨�©~� Å �^¬£¬£��­W¢£ ~¤?¦��L�~���+Æ¥¢£«$¦�¨�¢£�^ ¶ª

�*�+±¸ � �+ ½ � «$¢£ - � � � �+ ¸ � �+ ���(+7� � +P½ � �* +CBD��¸ � �+CBD� ½ ! ¸º¾hÁ^½

� ©¥� Å �h��­|¦��±¡�¡¥Ét ~¦h«$¢£²U�~���h¤h�±¦h«/«$¢£ ¥¤�¦h¬ ¤h�h�±¢£¨�©~«7��¢£« §~¬£¦�¨´�+� Å �h��­:¦��2¡#¨�©¥���^§¥¤^©#¨�¢£«$�z¦�¨/�+¦h²±© ¢ ¨´���2¦�¨�¢ �^ §~��¢® ¥¤U¦*��¦h«T�L¬£�����+¦h¬®¢��+¦�¨�¢ �^ k� Å ¨�©¥�T¦����±¢£ËǦh¬A�~���t²³�+����¦h ~¡k¨�©¥� ²�§~�����+ J¨��+�´¨�¢®«$¦�¨´�+��� Å ¨�©¥� Ë�¦h¬£§¥� Å §~ L²³¨�¢ �^ ~� �¨�©8§~��²³���+¦�¨�¢£ ¥¤ ¦z�´¨�¦�¨´�$¦h ~¡p¡¥�+²�¢£�±¢ �^ �¨´�2¦�¿�+²³¨´�h��É_ª �|¨;¨�¢®«T� � �¶�´¨�¦�¨´� � � � �+ �H§~��¢® ¥¤U¦z��¦h«T�q¬ �@���+¦h¬®¢��+¦�¨�¢ �^  � �+¦h ~¡p¨�©¥�$²�§~�����+ J¨;�+��¨�¢£«$¦�¨´�$� Å ¨�©¥�TËǦh¬®§¥� Å §~ L²³¨�¢ �^ ��* � �9�+CBD� �¶¨�©¥�$��§ �_�h�L¨�¢£«$¦h¬Y¡¥�+²�¢£��¢ �^  � �+ ¢£�B²�¦h¬£²�§L¬£¦�¨´�+¡ �JÉ�´�^¬ Ët¢£ ¥¤¥ª

� �+ � ¦��±¤\«$¢£ - � � �+ ¸ � � � �+ ��� �+ � � �+ ½ � �* � �9�+CBD� ¸ ! � � � �+ � � �+ # � � % B ½ ! ¸º¾h¼^½

� ©¥�+  ¨�©¥����¨�¦�¨´�W¦�¨.¨�©¥�� ¥�³Æt¨�¨�¢£«T�|�c���±¢ �t¡�¢£�.²�¦h¬®²�§~¬£¦�¨´�+¡T§~��¢® ¥¤�¨�©~� ¨´�±¦h ~� Å ��� Å §L ~²³¨�¢ �^  ¸º¾hÃ^½2µ � ©~¢£��²³�^ J¨�¢£ 8§¥�+�Å �h��­:¦��2¡ ¨�©¥���^§¥¤^©¨�¢£«T�y§~ J¨�¢£¬_¨�©¥�y�+ ~¡� Å ¨�©¥��©¥�h�±¢����^ ¶µ

� ©~���q¦h² �8­:¦��±¡��L¦h��� ¢£�|�c��� Å �h�±«T�+¡*¨´�/§~�c¡L¦�¨´�;¨�©~�BËǦh¬£§~� Å §~ ~²³¨�¢£�^ U�+��¨�¢£«$¦�¨´�+�W§L��¢£ ¥¤ ¨�©¥�y¡¥�+²�¢£�±¢ �^  ¦h ~¡�´¨�¦�¨´�B¨´�±¦�´�+²³¨´�h�±¢ �+�+ª E � �+ F '��9�+CG � E � � � �+ F

'+CG µ ���y§¥�¶¡~¦�¨´��¨�©~��Ë�¦h¬£§¥� Å §L ~²³¨�¢ �^ *�+�´¨�¢£«$¦�¨´� �* �+ ¦�¨ ¨�©¥���´¨�¦�¨´� � � � �+

§~��¢® ¥¤¥ª

�* � BD�+ ¸ � � � �+ ½ � ¸ � # � ½ �* �+ ¸ � � � �+ ½ � ��� � �+ ¸ � � � �+ ��� �+ � � �+ ½ � �* � BD�+CBD� ¸ � � � �+CBD� ½��

� ©¥�y¦��c��Ëh�@�´¨´���0¢®�|�_��� Å �h�±«$�+¡ �L¦h² �8­:¦��2¡�¢£ z¨�¢£«$� Å �h��¨ � �  � � µ µ µ � »tµ� ©~� Å �^¬£¬ ��­�¢£ ¥¤@¡¥�+�±²³�±¢ �~¨�¢ �^ $� Å ¨�©¥��¦h¬ ¤h�h�±¢ ¨�©~« �³Æ8�+²�§~¨´�+�|¨�©¥�;¦h¬£¤h�h�±¢ ¨�©~« Å �h��� ¢ ¨´���±¦�¨�¢£�^ ~�1¦h ~¡*¦h����§L«T�+�

¨�©~¦�¨�¨�©~�@�´¨�¦�¨´�yË�¦��±¢£¦��L¬£��¡~�J�+�� ~�h¨��³Æt²³���+¡ O�� � ­W©~¢®²±© ¢£� ²2©¥�^�´�+ ¨´� �_�y¦/��«$¦h¬£¬c« §~¬ ¨�¢ �L¬£�;� Å �4µ� ��� ���D��������� + ��� ���� � � � ��� � � � �+��� ��������!� � �

� ��� �/. � ¢ Ëh�+  � ª �8��¨ �* �+ ¸ � ½ � » Å �h�W¦h¬£¬ � � �z¦h ~¡ � , E�» �%$&$&$&� O���� F^µ �8��¨ � � � � � � Å �h�W¦h¬®¬��_µ �8��¨� � � µ

� ��� ��[ �|©¥�8�^�´�y¦?�±¦h L¡¥�^«w�^§¥¨�²³�^«T� � � � ¸�� � ��� � � �%$&$&$&��� �'��9� ½2µ� ��� �)" � �h� � � »?¨´� � � # � �c��� Å �h�±« ª

� �+ ª � ¦���¤ «/¢£ - ��� � ��� � � �+ ¸ � � � �+ ��� �+ � � ½ � �* �+CBD� ¸ ! � � � �+ � � �+ # � � % B ½"! ¸ºÃ�»N½¦h ~¡

� � � �+CBD� ª � ! � � � �+ � � �+ # � � �+ % B $ ¸ºÃ � ½

Page 22: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� ¾�»

� ��� �#% � �h� � � # � ¨´� � � »T�c��� Å �h�±« ª6;��¢£ ¥¤T¦$�´¨´���q��¢���� � �L§~�c¡L¦�¨´�;¨�©~�y�+�´¨�¢£«$¦�¨´��¦�¨W�´¨�¦�¨´� � � � �+ ª

�* � BD�+ ¸ � ½ ª �� ¸ � # � �* �+ ¸ � ½ � � � � �+ ¸ � � � �+ ��� �+ � � �+ ½ � �* � BD�+CBD� ¸ � � � �+CBD� ½ ! ¢ Å �3� � � � �+�* �+ ¸ � ½ �h¨�©~����­W¢£�´�

� ��� � ( � Å � � ���¥¨�©¥�+ ����¨ � � � �)� ¦h ~¡ ¤h�$¨´�/��¨´���0¾8Ê~�+¬£�´���´¨´�h��©¥�����hµ

�� �� ����� ��' �#$���� ��� ��'�� ��" �#��� � ��'�� � ��)8' � ��) �������������$

� ©¥��«T�^ ¥�h¨´�^ ¥��¦h¡~¦��~¨�¢ Ëh�;¡¥Ét ~¦h«$¢£²|�~�±�h¤h�±¦h«$«$¢£ ~¤@¦h¬ ¤h�h�±¢ ¨�©L« ¢£���±¢£«T�L¬ É?¦yË�¦��±¢£¦�¨�¢£�^ $� Å ¨�©¥� �q¦h��¢£² Å �h��­|¦��±¡¡¥Ét ~¦h«$¢£²@�~���h¤h�±¦h«/«$¢£ ¥¤*¦h¬ ¤h�h�±¢ ¨�©~«µ � ©¥� ¡L¢"!_���±�+ ~²³���t²�²�§¥�±��¢£ 0¨�©¥� ­:¦�É0­:�?§¥�¶¡~¦�¨´��¨�©~��Ë�¦h¬£§¥� Å §L ~²³¨�¢ �^ �+�´¨�¢£«/¦�¨´�+��µ ��§¥�;¦h¬ ¤h�h�±¢ ¨�©L« ²³�^ ~��¢®�´¨�� �L���+¦��8¢® ¥¤/�´¨´��� �*¢® N¨´�U¨�­:�*�´¨´���q��µ ��§¥������Ët¢£�´�+¡Ëh���2��¢ �^ � Å ��¨´��� �z¢£�¤^¢ Ëh�+  �8É_ª� �����)%� �h� � � # � ¨´� � � »T�c��� Å �h�±« ª

¸ ¦N½ 6��¶¡~¦�¨´�y¨�©¥�B�+��¨�¢£«$¦�¨´�@¦�¨W��¨�¦�¨´� � � � �+ ª

�* � BD�+ ¸ � ½ ª �� ¸ � # � �* �+ ¸ � ½ � � � � �+ ¸ � � � �+ ��� �+ � � �+ ½ � �* � BD�+CBD� ¸ � � � �+CBD� ½ ! ¢ Å � � � � � �+�* �+ ¸ � ½ �h¨�©¥����­�¢£�´�

¸ �_½ 4\  Å �h�±²³�@«T�^ ¥�h¨´�^ L¢£²�¢ ¨�É_ª¢°½ � �h��¦h¬£¬ * � � � � �+ # � �%$&$&$&��»*¦h ~¡ ­W©~¢®¬ ���* � BD�+ ¸+*P½ � �* � BD�+ ¸ � � � �+ ½2ª:�´��¨ �* � BD�+ ¸+*P½ ª � �* � BD�+ ¸ � � � �+ ½2µ¢£¢ ½ � �h��¦h¬£¬ * � � � � �+ � � �%$&$&$&� O�� � ¦h ~¡@­�©~¢£¬ � �* � BD�+ ¸+*P½ � �* � BD�+ ¸ � � � �+ ½2ªY�´��¨ �* � BD�+ ¸+*P½:ª � �* � BD�+ ¸ � � � �+ ½2µ

�8¨´���&�L¸ �c½:¢£�:¨´�T�~���+�´����Ëh�y«$�^ ¥�h¨´�^ ~¢£²�¢ ¨eÉ/� Å ¨�©¥��ËǦh¬®§¥� Å §~ L²³¨�¢ �^ *�+�´¨�¢£«$¦�¨´�+��µ �|¨ ¨�¢£«T� � ¢ ¨´���2¦�¨�¢ �^  �*¨�©¥��´¨�¦�¨´�?­W©¥�^����Ë�¦h¬£§¥� ©~¦h� �c���+ k§¥�¶¡~¦�¨´�+¡k¢£ ��´¨´����L¸ ¦N½W� Å ¨�©¥���L¦h��¢®²@¦h¬ ¤h�h�±¢£¨�©~« ¢®� � � � �+ ¦h ~¡�¨�©~�  ¥��­jË�¦h¬£§¥�¢£� �* � BD�+ ¸ � � � �+ ½2µ ���T¨�©¥�+ k²±©¥�+² �0¨�©¥� Ë�¦h¬£§¥� Å §~ L²³¨�¢ �^ ��+��¨�¢£«$¦�¨´�+��¦�¨�¨�¢£«T� � � Å  ¥�+¢ ¤^© �c�h�±¢£ ¥¤U�´¨�¦�¨´�+��� Å � � � �+¦h ~¡¢ Å ¨�©¥��ÉzËt¢ �^¬£¦�¨´�y«T�^ ¥�h¨´�^ L¢£²�¢ ¨�Éz­:�y�´��¨�¨�©¥�+« ¨´� �* �+ ¸ � � � �+ ½2µ

�8¨´����L¸ 0�½;¢£��­W©~¦�¨�«$¦��h�+��¨�©~¢£��¦h¬ ¤h�h�2¢ ¨�©~« ¦ Å §~ ~²³¨�¢£�^ ~¦h¬A¦��~�L���+Æ¥¢£«$¦�¨´�^ p¦h¬ ¤h�h�±¢£¨�©~« µ � ©~¢£�;¢£� �c�+²�¦h§~�´�¢ ¨W§L�´�+��¢®  Å �h�2«$¦�¨�¢ �^ ¦��c�^§¥¨�¨�©~���´¨´�±§L²³¨�§¥���y� Å ¨�©¥� Å §L ~²³¨�¢ �^ �¸ ¢£ �¨�©~¢£��²�¦h�´� «$�^ ¥�h¨´�^ ~¢£²�¢ ¨eÉ¥½ ¨´�U¢£«$�~����Ëh��¨�©¥��+�´¨�¢£«/¦�¨´�+�y� Å  ¥�+¢ ¤^© �c�h�±¢£ ¥¤z²³�+¬£¬£��µ � ©¥�����$¢£�B ¥� ~���+¡�¨´��²³Ét²�¬ �$¨�©¥�±�^§¥¤^©p¨�©¥�$­W©~�^¬ �T�´¨�¦�¨´�U�´�q¦h²³�$¨´��²±©~�+² �Å �h�;Ët¢ �^¬£¦�¨�¢£�^ ~�W� Å «T�^ ¥�h¨´�^ ~¢£²�¢£¨�Éhµ �� ~²³� ¨�©¥�?ËǦh¬®§¥� Å §~ ~²³¨�¢ �^ 0�+�´¨�¢£«/¦�¨´�?¦�¨B¦*�´�c�+²�¢ �L²��´¨�¦�¨´�$¢£�;§¥�¶¡~¦�¨´�+¡­:��^ ~¬ Éz ¥���+¡�¨´� �L ~¡�¦h ~¡§¥�¶¡~¦�¨´�@��¨�¦�¨´�+��¡~¢£���+²³¨�¬ Éz¦��_��Ëh� ¦h L¡¡~¢ ���+²³¨�¬£É��c�+¬ ��­R¨�©~¦�¨���¨�¦�¨´� §~ J¨�¢£¬c­:� �L L¡ �^ ¥�­W©¥�^���BËǦh¬£§~� Å §~ L²³¨�¢ �^ z�+�´¨�¢£«$¦�¨´��¡¥�J�+�W ¥�h¨�Ët¢ �^¬£¦�¨´�y«T�^ ¥�h¨´�^ L¢£²�¢ ¨�Éhµ

Page 23: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� ¾ �

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

500

1000Step 4(a)

new estimate new observationold estimate

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

500

1000Step 4(b)

new monotone estimatenew observation old estimates

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

500

1000New observation

old monotone estimate new observation smoothed new observation

� ¢ ¤^§¥����¾8ª �8¨´��� �#� Å ¨�©¥��«T�^ ~�h¨´�^ ¥� Å �h��­|¦��±¡ ¡¥Ét ~¦h«$¢£²��~���h¤h�±¦h«$«/¢£ ¥¤�¦h¬£¤h�h�±¢ ¨�©~« ª � ©¥�0���+²³�^ ~¡ �L¬ �h¨¢£¬£¬®§~�´¨´�±¦�¨´�+�|�´¨´��� �L¸ ¦N½ ¦h ~¡ ¨�©~�B¨�©~¢ �±¡z�L¬£�h¨�¢£¬£¬£§~��¨´�±¦�¨´�+�|�´¨´���'�L¸ �_½

� ¢ ¤^§¥�±��¾y��©~��­W�\¨�©¥�W�L���8²³�+�±��� Å §¥�¶¡~¦�¨�¢£ ~¤y ¥�+¢ ¤^© �_�h�2¢£ ¥¤B�´¨�¦�¨´�+�:­W©¥�+ /¨�©¥�+¢ �1Ë�¦h¬£§¥�WË8¢ �^¬®¦�¨´�+�\«T�^ ~�h¨´�^ ~¢£²�¢ ¨eÉhµ � ©~���L�±�´¨��L¬ �h¨B��©¥��­W��¨�©¥�T«T�^ ¥�h¨´�^ ~�$�+�´¨�¢£«$¦�¨´�$¦�¨y¢£¨´���±¦�¨�¢ �^  �0� Å ¨�©~� Ë�¦h¬£§¥� Å §~ ~²³¨�¢ �^  �* �+ �A¨�©¥�T ¥��­� �L�´���±ËǦ�¨�¢ �^ k¢£ 0¢ ¨´���±¦�¨�¢ �^  � � � � Å ¨�©¥��ËǦh¬£§~� Å §L ~²³¨�¢ �^ �¢® ��´¨�¦�¨´� � � � �+ � ÃU¦h�W­:�+¬£¬[¦h�W¨�©¥�? ¥��­j��«T�8�h¨�©¥�+¡�+�´¨�¢£«/¦�¨´�y� Å ¨�©¥�yË�¦h¬£§¥� Å §~ ~²³¨�¢ �^ ¢®  �´¨�¦�¨´� à � �* � BD�+ ¸ºÃ^½2µ � ©~�y�´�+²³�^ ~¡�q¬ �h¨ �~���+�´�+ J¨��W��¨´��� �L¸ ¦N½|� Å ¨�©¥� �L¦h��¢£²¦h¬ ¤h�h�±¢£¨�©~« ­W©¥���±��¨�©¥�W ¥��­ �+�´¨�¢£«$¦�¨´��� Å ¨�©¥�WË�¦h¬£§¥� Å §~ ~²³¨�¢ �^ /¢£����©¥��­W /¦�¨:¢ ¨´���±¦�¨�¢ �^  � � � �8É?¢® ~²³�h���c�h�±¦�¨�¢£ ¥¤¨�©¥�/ ~��­�ËǦh¬®§¥�$¦�¨��´¨�¦�¨´�zÃ8µ��|¨@¨�©L¢£�B�c�^¢£ J¨;­:�/¬ �8��� Å �h�� ¥�+¢ ¤^© �_�h�2¢£ ¥¤ �´¨�¦�¨´�+�y� Å �´¨�¦�¨´�zÃz¨´�0�����/¢ Å «T�^ ¥��¨´�^ ~¢£²�¢£¨�É ©~¦h� �_���+ Ë8¢ �^¬®¦�¨´�+¡¶µ ���  ¥�h¨´��¨�©~¦�¨�«$�^ ¥�h¨´�^ ~¢£²�¢ ¨eÉ©~¦h� �c���+ �Ë8¢ �^¬®¦�¨´�+¡��8Éz��¨�¦�¨´�?¾/¦h ~¡��´�/­\� �´��¨¨�©¥�y�+��¨�¢£«$¦�¨´�y� Å �´¨�¦�¨´� ¾T� #J§~¦h¬H¨´� �* � BD�+ ¸ºÃ^½ �q¦h�W��©¥��­� ¢£ z¨�©¥�y¨�©L¢ �±¡*�L¬ �h¨�µ

� ©~�U¦h¬ ¤h�h�±¢£¨�©~« ­�¢ ¨�©� ~�k§¥�¶¡~¦�¨´�+�@� Å  ¥�+¢ ¤^© �c�h�±¢£ ¥¤0²³�+¬£¬£� ©~¦h���_���+ 4§~�´�+¡4¦h ~¡4�´¨�§~¡~¢£�+¡ �8É %:����¨��´�&��¦h�¦h ~¡ � ��¢ ¨���¢$�8¬£¢®��µ � ©~¢®��¦h¬ ¤h�h�±¢ ¨�©L« ©~¦h�\�~����Ëh�+¡/¨´���c�W²³�^ NËh���±¤h�+ N¨ §~ ~¡~���1¨�©¥�;¦h����§~«T�L¨�¢ �^ T¨�©~¦�¨:�+¦h²±©z�´¨�¦�¨´��¢£�Ët¢£��¢ ¨´�+¡$¢®  �L ~¢ ¨´�+¬£É@«$¦h JÉ ¨�¢®«T�+��¸ %:����¨��´�&��¦h� ' � �±¢ ¨���¢��t¬£¢£� ¸ � ¼h¼h¹^½´½2µ 2���­:��Ëh�����¥¢£ $«/¦h NÉT�~�±� �L¬ �+«$�.¢£ L²�¬£§~¡~¢£ ~¤¨�©¥� �L¦�¨�²2©$�´����Ët¢£²³� �~�±� �L¬ �+«j¨�©¥������¢£�. ~��¦h�±��§¥�±¦h ~²³� ¨�©L¦�¨�¦h¬£¬¥�´¨�¦�¨´�+��­�¢£¬£¬ �_�|Ët¢£��¢ ¨´�+¡?¢£  �L ~¢ ¨´�+¬ ÉB� Å ¨´�+ ¶µ��;�.¨�©¥�

Page 24: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� ¾h¾

�~���t²³�+���\²³�^ NËh����¤h�+�|¨´� ¨�©¥���h�~¨�¢£«$¦h¬L�h�:��§ �¥Â��h�~¨�¢£«$¦h¬~�_�^¬®¢£²�¢ �+��²³����¨�¦h¢® /�´¨�¦�¨´�+�|¡¥��¨´���±«/¢£ ¥�+¡ �8ÉT¨�©¥�+�´���c�^¬£¢£²�¢ �+�¦�����Ët¢£��¢ ¨´�+¡ Å ¦�� «T�h�±� Å ��� #8§¥�+ N¨�¬£É$¨�©~¦h *�h¨�©¥���±��µ � ©¥������¢£�| ¥��¤^§L¦��±¦h N¨´���B� Å ²³�^ NËh����¤h�+ L²³�y� Å ¨�©~¢®�\¦h¬ ¤h�h�2¢ ¨�©~«¢£ z¨�©¥�@�~��� �L¬ �+«u­:�@¦����@��¨�§~¡¥Ét¢£ ¥¤¥µ

� ����� � � � � ��� � �

�  0¨�©L¢£�����+²³¨�¢ �^ k­\�?¨´�+�´¨�¨�©¥�?�c��� Å �h�±«$¦h ~²³� � Å �^§¥�B¦h¬ ¤h�h�±¢£¨�©~« µ ���?¨�¦��h�$¦h¡¥ËǦh J¨�¦�¤h�T� Å ¨�©~� Å ¦h²³¨�¨�©~¦�¨�¨�©¥�©¥�^«T�h¤h�+ ~���^§~� ²�§~�´¨´�^«T��� �L¦�¨�²2©z�´����Ët¢£²³���L��� �L¬ �+«�²�¦h ��c���´�^¬ Ëh�+¡U�h�~¨�¢£«$¦h¬®¬ É?¨´� ��Ë�¦h¬£§~¦�¨´��¨�©¥�;�c��� Å �h�±«$¦h ~²³�� Å ¨�©¥��¦h¬£¤h�h�±¢ ¨�©~«$�+µ����;¡¥�+��¢ ¤^ /¨�©¥���³Æt�_���2¢£«T�+ J¨���¢£ $�´�+²³¨�¢£�^  �¥µ � �¥²³�^ ~�´¨´�±§L²³¨�¨�©¥� Å �h�±«$¦�¨1� Å ¨�©¥�W¦h¬ ¤h�h�±¢£¨�©~«$�¢£  �´�+²³¨�¢£�^  �¥µ ¾ �y¦h ~¡ �L ~¦h¬®¬ ÉF¢£  �´�+²³¨�¢ �^  �¥µ Ã#­:��¡~¢£��²�§~�±�*¦h L¡ ²³�^«T�L¦�����¨�©~���_��� Å �h�±«/¦h ~²³�p� Å ¡~¢�!_�����+ J¨¦��~�~�±�+Æ¥¢£«$¦�¨�¢ �^  «T��¨�©~�8¡~�+µ

�� �� ���� %$&��' �W$ �������\$*)8' ���� ©~¢®�Y���+²³¨�¢ �^ $¡¥�+��²³�2¢ �c�+�[¨�©~�:�³Æt�c���±¢£«T�+ J¨���¨�©~¦�¨.¨´�+�´¨1¨�©¥� �c��� Å �h�2«$¦h ~²³�|� Å ¨�©~�|¦h¡L¦��~¨�¢ Ëh��¡¥Ét ~¦h«$¢£²:�~�±�h¤h�±¦h«?«$¢£ ~¤?¦h¬£¤h�h�±¢ ¨�©~« µ ��� ¡¥�+��¢ ¤^ z¨�©~�B�L¦��±¦h«T��¨´���;¡~¦�¨�¦$�´��¨���¦h ~¡¡¥�+��²³�2¢ �c�;¨�©~�y²³�^«T�c��¨�¢£ ¥¤$�´¨´�±¦�¨´��¤^¢£�+��µ+ �� �9� ���

�  k�h�±¡¥���;¨´�*¨´�+��¨;�^§~��¦h¬£¤h�h�±¢ ¨�©~« �_­:�?²³�±�+¦�¨´�$¡~¦�¨�¦z�´��¨����JÉË�¦���Ét¢£ ¥¤*¨�©¥�?¦����±¢ Ë�¦h¬[����¨��;¦h L¡k���^«T�?�h¨�©¥���«T�t¡¥�+¬c�L¦��±¦h«$��¨´���±��µ

���@¦h�±��§~«T�B¨�©~¦�¨ ¨�©~�y¦����±¢ Ë�¦h¬£�|¦�¨�¨�¢£«T� � ��H + � Å �^¬£¬ ��­R¦ �Y�^¢£���´�^  ¡L¢£�´¨´�±¢ �q§¥¨�¢ �^ $­�¢ ¨�© �8 ~��­W  «T�+¦h Hµ ���¤h�+ ¥���±¦�¨´�?¨´�+ k¦����±¢ Ë�¦h¬A�±²³�+ ~¦��±¢ �^� Å �h�;¨�©¥�?«T�+¦h ~��ª E��� + F '��9�+CG Å �h� * � � �%$&$&$&� � »z­�©¥�������� + ¢£��¨�©¥� «T�+¦h 0� Å ¨�©¥�¦����±¢£ËǦh¬£��¦�¨�¨�¢®«T� � ¢£ T�±²³�+ ~¦��±¢ � *�µ � �^§¥�\� Å ¨�©~�+�´��¦����±¢ Ë�¦h¬t��²³�+ ~¦��±¢£�^��¦���� �c���±¢ �t¡~¢£²\¦h ~¡T¨�©¥� ���+«/¦h¢£ ~¢£ ¥¤���¢ Æ?¦����¦��c���±¢ �t¡~¢£²�¤h�+ ¥���±¦�¨´�+¡k§~ ~¢ Å �h�±«$¬£É Å �±�^« E�» � � �±¾ �±Ã#FT�h�=E�» � � �%$&$&$&�±¹#F^µ � ©¥�B�c���±¢£�8¡~¢®²�¦����2¢ ËǦh¬¶����¨���¤h�/¨�©~���^§¥¤^©�c���±¢ �t¡~�?� Å ��������¦��±�±¢ Ë�¦h¬£� �1­�©~¢£²±©F¦����­W©~¦�¨/­\��²�¦h¬®¬�� ¡¥�+¦h¡F�_���2¢ �8¡L�%� µ � ©~¢£�T¢®�/¦� ~¦�¨�§~�±¦h¬ �L©¥�+ ~�^«T�+ ¥�^ ¢£ #¦h²³¨�§~¦h¬�¦��±�±¢ Ë�¦h¬:�±²³�+ ~¦��±¢ �^� Å �h�$�´�^«T�¦��~�q¬£¢£²�¦�¨�¢ �^ ~�+µ � ©8§~� ��­:�¦h¬£�´��¢£ J¨´���t¡~§~²³��� ¡¥�+¦h¡#�c���±¢ �t¡~�%��¢£ 4¨�©¥�¦��c���±¢ �t¡~¢£²z§~ ~¢ Å �h�±«/¬ É�¤h�+ ¥���±¦�¨´�+¡ ¦����±¢ Ë�¦h¬ �´��¨���µ � ¢ ¤^§¥���0Ã���©¥��­��T¨�©¥�¨�©¥�±���¨�É8�c�+�$� Å ¦����±¢ Ë�¦h¬ ��²³�+ ~¦��2¢ �^��ª�c���±¢ �t¡~¢£²��_§~ ~¢ Å �h�±«$¬ Éz¤h�+ ¥���2¦�¨´�+¡�¦��c���±¢ �t¡~¢£²y­�¢ ¨�©¥�^§¥¨�¡¥�+¦h¡p�c���±¢ �t¡~�;¦h ~¡k­W¢ ¨�©p¡¥�+¦h¡k�c���±¢ �t¡~��µ �  p¤h�+ ~���±¦h¬ �­:�:¨´�±Éy¨´�B¡¥�+��¢ ¤^ ��^§¥��¦����±¢ Ë�¦h¬J����¨��.�´��¨�©L¦�¨Y¨�©~��ÉyËǦ��±Éy­W¢ ¨�©����+�´�c�+²³¨Y¨´�;�c���±¢£�8¡~¢®²�¢ ¨�É ��¬ �+ ¥¤h¨�© � Å ¡¥�+¦h¡ �c���±¢ �t¡~� �«T�+¦h  8§~« �c��� � Å ¦��±�±¢ Ë�¦h¬£� ¦h ~¡«$¦ÇÆ¥¢£« §~«w 8§~« �c��� � Å ¦����±¢ Ë�¦h¬£��µ

��� �¥Æ$¨�©¥� Å �^¬£¬ ��­�¢£ ¥¤@�q¦��±¦h«T��¨´���±�:¨�©¥���^§~¤^©¥�^§¥¨1¨�©~�;¡~¦�¨�¦?�´��¨��+ª � ¾�»h» � � �3� � � �P� ¾�»h» � � » $ ¼h¼8µ2���­:��Ëh�����q­:�yË�¦���É*¨�©¥�y­|¦h¢ ¨�¢£ ¥¤T²³�^�´¨��_����²�§L�´¨´�^«T���W�c��� ¨�¢£«$�B�_���2¢ �8¡�¸# � ¾ �±Ã �±À �±Á � � »N½;¦h ~¡z¨�©~�@�´����Ët¢£²³�²�¦��L¦h²�¢ ¨eÉU¸ � � Á � � » � � À^½2µ�%:ÉBË�¦���Ét¢£ ¥¤�¨�©¥�+���\¨e­\�;�L¦��±¦h«T��¨´���2�[­:�\²³�±�+¦�¨´� � »;��¢ ¨�§L¦�¨�¢ �^ ~�¶­W©~�����1¨�©¥�:¦+Ëh���±¦�¤h�­|¦h¢ ¨�¢£ ¥¤?²³�^�´¨|�c���|²�§L�´¨´�^«T��� ¢£�:¤h���+¦�¨´����¨�©~¦h  �8¦��~�L���+Æ¥¢£«$¦�¨´�+¬ É/� #J§~¦h¬_¨´� �¥�h��¬ �+�±�1¨�©~¦h z¨�©~�;�´���±Ë8¢£²³��²³�^�´¨ �c���

Page 25: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� ¾hÃ

0 20 40 60 80 100 120 140 160 180 2000

2

4

6A

rriv

als−

set1

0 20 40 60 80 100 120 140 160 180 2000

2

4

6

Arr

ival

s−se

t2

0 20 40 60 80 100 120 140 160 180 2000

2

4

6

Time

Arr

ival

s−se

t3

� ¢ ¤^§¥���;Ã8ª �����±¢£ËǦh¬~����¨ � ¢£��¦��_���2¢ �8¡L¢£²|¦����±¢ Ë�¦h¬~�´��¨\­W¢ ¨�©$¡¥�+¦h¡U�c���±¢ �t¡~��Ê �����±¢£ËǦh¬~����¨��:¾@¦h ~¡UÃ@¦�����¦��c���±¢ �t¡~¢£²§~ ~¢ Å �h�±«/¬ ÉT¤h�+ ¥���±¦�¨´�+¡0­W¢ ¨�©¥�^§~¨|¡~�+¦h¡ �c���±¢ �t¡~�|¦h ~¡ ­W¢ ¨�© ¡~�+¦h¡ �c���±¢ �t¡~��µ

²�§~�´¨´�^«$���/¸ � � ��½2µ -W�h¨´�T¨�©~¦�¨B­:�T¦��~�~����Æt¢®«$¦�¨´�?¨�©¥�T¦+Ëh���±¦�¤h�/ 8§~« �c����� Å ¨�¢£«T�?�c���±¢ �t¡~��¨�©~¦�¨y¦z²�§~��¨´�^«T���­|¦h¢ ¨��T¦�¨T¨�©¥��´¨�¦�¨�¢ �^  �8É ¸ ����¾^½���� ��¦h����§~«$¢® ¥¤�¨�©~¦�¨T�^ �¦�Ëh���±¦�¤h��¨�©~���¢����z� Å �´���±Ë8¢£²³� �L¦�¨�²2© ¢£�T©~¦h¬ Å ¨�©¥��´����Ët¢£²³�/²�¦��q¦h²�¢ ¨�Éhµ � Å ­:�/¬ ��¨�� �c�T¨�©¥�U¦+Ëh���±¦�¤h�  8§~« �c���y� Å ¦����2¢ ËǦh¬®���_���@¨�¢£«T�$�c���±¢ �t¡k¨�©¥�+ �¨�©~�/¦+Ëh���±¦�¤h�­|¦h¢ ¨�¢£ ¥¤$²³�^�´¨W�_����²�§L�´¨´�^«T����¢®� A¸ ����¾^½����¶µ

� ©8§~�W­\� ©~¦+Ëh�?¦/¨´�h¨�¦h¬Y� Å � »h»*¡~¦�¨�¦/����¨��?¸ � »DC � »N½2µ �����±§L  ¨�©¥�@�³Æt�c���±¢£«T�+ J¨�� Å �h��¦h¬£¬ � »h»*¡~¦�¨�¦*�´��¨���L§¥¨ Å �h�W²³�^ JËh�+ ~¢ �+ ~²³� ¢£ *�����c�h��¨�¢£ ¥¤?���+��§L¬ ¨��|­\�@¤h���^§¥� ¨�©¥�+«-¢® N¨´�/�±¢ ÆU¤h���^§~�L� ¦h²�²³�h�±¡~¢£ ~¤T¨´�$¨�©¥�+¢ ��¡~¢�!_�����+ J¨Å �+¦�¨�§¥���+��µ �  �¨�¦��q¬ � � ­:�T�´���?¨�©~¦�¨�¨�©¥� ¡L¦�¨�¦ �´��¨���¦����?¤h���^§¥�c�+¡0¦h²�²³�h�2¡~¢£ ¥¤*¨´� �c���±¢ �t¡~¢£²�¢ ¨eÉ*¦h ~¡p¦h²�²³�h�±¡L¢£ ¥¤¨´�*¨�©¥���±�+¬£¦�¨�¢ �^ ~��©L¢ � �c��¨�­:���+ k¦�Ëh���±¦�¤h�T­|¦h¢ ¨�¢£ ¥¤*²³�^�´¨��c����²�§~�´¨´�^«T����¦h ~¡k�´����Ët¢£²³��²³�^��¨;�c���;²�§~�´¨´�^«T���+µ ��������c�h��¨ ¨�©¥�B�c��� Å �h�±«$¦h L²³�B���+��§~¬ ¨��|� Å ¨�©¥�+���@��¢ Æ*¤h���^§¥�L��µ� � � � � ���&��� � �� ��� � � � �

Page 26: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� ¾��

�t¢® ~²³�1�^§¥�.¡~¢®��²�§~����¢£�^ @¢£ �¨�©~¢£�[�L¦��c���Y¢®�A¨´���+«T�L©L¦h��¢����:¨�©¥�:¢®«T�c�h��¨�¦h ~²³�1� Å §~��¢£ ~¤��´¨´�±§L²³¨�§¥�±¦h¬8¢£  Å �h�±«$¦�¨�¢ �^ ­:��²³�^ ~��¢®¡¥���1¨�©¥�;¦h¡~¦��~¨�¢ Ëh�;¡¥Ét ~¦h«$¢£²|�~�±�h¤h�±¦h«$«$¢£ ~¤@¦h¬ ¤h�h�±¢ ¨�©L«�­W¢ ¨�©/ ¥���´¨´�2§~²³¨�§¥����¨´� �c�W�^§¥�:²³�^«T�c��¨�¢ ¨�¢ �^ ¦h ~¡ ­:�@²³�^«T�q¦����B�^§¥�����+�±§~¬ ¨�� ¨´�$¨�©L¢£� «T��¨�©¥�t¡¶µ ����²³�^«$�L¦���� �c�h¨�© «T��¨�©¥�t¡~� ¨´�/¨�©¥�B�h�~¨�¢£«$¦h¬A�´�^¬£§¥¨�¢ �^ Hµ

�� S� � ����' �������' � � � � $�� � �#" ��'`� � ��   ¨�©~¢£���´�+²³¨�¢ �^ ­:� �q ~¡z¨�©¥�� 8§~« �c��� � Å ¢ ¨´���±¦�¨�¢ �^ ~��¨´�/�±§~ ¨�©¥�@¦h¬ ¤h�h�±¢£¨�©~«$��¦h ~¡�­\� ¡¥�+²�¢£¡¥�B�^ 0¦/�´¨´���q��¢�����±§~¬£�hµ� � � � � � � ��� ��� � ��� �����

��§~�z��¨´�h�~�L¢£ ¥¤4�±§~¬£�+�U¦���� �q¦h�´�+¡ �^  ¡¥��¨´���±«$¢£ L¢£ ¥¤�¨�©~����� #J§L¢ ���+¡  J§~« �c���z� Å ¢ ¨´���±¦�¨�¢£�^ ~�*¨´���±§~  ¨�©¥�¦h¬ ¤h�h�±¢£¨�©~« �~¢£ z�h�±¡~����¨´�U¦h²2©~¢ ��Ëh�@¨�©~�@¦��~�~���h�L�±¢£¦�¨´���c��� Å �h�2«$¦h ~²³�y¬ ��Ëh�+¬£� Å �h���+¦h²±©k¦��~�~����Æt¢£«/¦�¨�¢ �^  «T��¨�©¥�t¡¶µ�   ¤h�+ ¥���±¦h¬ �¥�³Æt�c���±¢£«T�+ J¨���©~¦�Ëh�y¨�­:�/¨eÉJ�c�+� � Å ¢ ¨´���2¦�¨�¢ �^ ~��ª.¨´�±¦h¢® ~¢£ ¥¤T¦h ~¡z¨´�+�´¨�¢® ¥¤¥µ

� �±¦h¢£ ~¢® ¥¤4¢£¨´���±¦�¨�¢ �^ ~�¦��±���c��� Å �h�2«T�+¡ ¢£  �h�±¡¥���¨´�F�+�´¨�¢£«$¦�¨´��¨�©~��ËǦh¬£§~��¦��~�~����Æt¢£«/¦�¨�¢ �^ ~��µ �|¨�+¦h²2©¨´�±¦h¢£ L¢£ ¥¤y¢ ¨´���±¦�¨�¢£�^ /­:���2¦h ~¡¥�^«$¬ É?²±©~�J�^�´�B¦h *¢£ ~¢ ¨�¢®¦h¬¥�´¨�¦�¨´� Å ���^«�¨�©¥������¨PE�» � � �%$&$&$&� � # � F^µ � ©~¢£��¦h¬£¬£��­W�\¨�©¥�¦h¬ ¤h�h�±¢£¨�©~« ¨´�TËt¢£��¢£¨\«T�h�����´¨�¦�¨´�+��µ � �T¡¥��¨´���±«/¢£ ¥�;¨�©¥�;¦��L�~���h�~�±¢®¦�¨´�; 8§~« �_���:� Å ¨´�±¦h¢® ~¢£ ¥¤ ¢ ¨´���±¦�¨�¢ �^ ~���8­:�;�±¦h ¦������±¢ �+�y� Å ¨´�+��¨���µ ���*²³�^ L²�¬£§~¡¥�+¡�¨�©L¦�¨@¨�©¥�U�_��� Å �h�±«/¦h ~²³�$� Å �^§¥� «T�^ ¥�h¨´�^ ¥�z¦h¬ ¤h�h�±¢ ¨�©L« �´¨�¦��L¢£¬®¢����+�y¦ Å ¨´���� »h»h»/¢£¨´���±¦�¨�¢ �^ ~��¦h L¡ Å �h��²³�^«T�L¦��±¢£���^ z­\�@²2©¥�8�^�´�@¨´�$�2§~  ¢ ¨ Å �h��À�»h» � � »h»h»U¦h ~¡¾�»h»h»U¢£¨´���±¦�¨�¢ �^ ~��µ

� �+�´¨�¢£ ¥¤�¢£¨´���±¦�¨�¢ �^ ~�$¦��±��c��� Å �h�2«T�+¡4¨´��¨´�+�´¨/¨�©~� �_��� Å �h�±«/¦h ~²³� � Å ¨�©~��¦��~�~����Æt¢®«$¦�¨�¢ �^ �«T��¨�©¥�t¡���Ëh���¡~¢"!c�����+ J¨���¦h«$�L¬ �+��� Å ¦����±¢ Ë�¦h¬£��µ ���z�_��� Å �h�±« ¨�©¥�*¨´�+�´¨�¢£ ¥¤p¢ ¨´���±¦�¨�¢ �^ L� Å �h�T¡~¢"!c�����+ N¨�ËǦh¬®§¥�+� � Å ¨�©¥�z¢£ ~¢ ¨�¢£¦h¬�´¨�¦�¨´�¸ � � » �%$&$&$&� � # � ½W¦h ~¡0­\�?¦+Ëh���±¦�¤h�$²³�^��¨��B¦h²�²³�h�±¡~¢£ ¥¤^¬£Éhµ��;¬£¬H�³Æt�c���±¢£«T�+ J¨��W¦���� �c��� Å �h�±«T�+¡ Å �h� � »h»¨´�+�´¨�¢£ ~¤$¢ ¨´���±¦�¨�¢ �^ ~�W¦h ~¡ ¨�©¥�yË�¦h¬£§¥�+� ¦��±�@¦+Ëh���±¦�¤h�+¡k��Ëh����¢ ¨´���±¦�¨�¢£�^ ~��µ� ����� ��� � ��� �'� �

����§~�´�W¦��´¨´���L��¢$���W�±§~¬£� � Å ¨�©¥� Å �h�±« �t¸�� � �~½ �J­�©¥����� �/¢£�.¨�©¥�; J§~« �c���1� Å ¢ ¨´���±¦�¨�¢£�^ ~� � Å �h�|��«T�8�h¨�©~¢£ ¥¤¨�©¥�@§~�c¡L¦�¨´�+¡ ËǦh¬®§¥� Å §~ ~²³¨�¢ �^  ­�¢ ¨�© ¨�©¥�y�+��¨�¢£«$¦�¨´�@� Å ¨�©~�@¬£¦h�´¨W¢£¨´���±¦�¨�¢ �^ ¶µ ����¨´�+��¨´�+¡0¡~¢�!_�����+ J¨�Ë�¦h¬£§¥�+� � Å ¦h ~¡����^ ��´��Ëh���±¦h¬.¡L¦�¨�¦*����¨��y¦h ~¡ Å �^§L ~¡�¨�©L¦�¨y¦*��¨´���L��¢����?� Å � �t¸ºÀ � X ¸ �~½´½�­:�h� �h�+¡ �c�+�´¨ �c­W©¥���±� X ¸ �~½;¢£�¨�©¥�  J§~« �c���W� Å ¨�¢®«T�+�W¨�©L¦�¨;�´¨�¦�¨´�8* �c­W©¥�^���@Ë�¦h¬£§¥��¢®� �c�+¢£ ¥¤/�±«T�J�h¨�©~�+¡��_©L¦h� �c���+ Ët¢£��¢£¨´�+¡�§¥�0¨´�*¢ ¨´���2¦�¨�¢ �^ �_µ X ¸ �~½|¡¥�8�+�� ¥�h¨�²³�^§~ J¨�¨�©¥��§¥�¶¡~¦�¨´�+�|¨´�/�´¨�¦�¨´��* Å �±�^«- ¥�+¢ ¤^© �c�h�±¢£ ¥¤?�´¨�¦�¨´�+��µ � ©J§L� �¥¨�©¥�y�´¨�¦�¨´�+��¨�©~¦�¨W©~¦�Ëh� ¥�h¨��c���+ �Ët¢£��¢ ¨´�+¡4«$¦h JÉ�¨�¢£«T�+��­W¢£¬®¬1©~¦�Ëh�z©L¢ ¤^©��´¨´���q��¢����+�?¦h ~¡�¨�©¥�U�^ ~�+�?Ë8¢£�±¢ ¨´�+¡�«$¦h JÉ�¨�¢£«$�+�@­W¢®¬£¬1©~¦�Ëh�¬ ��­��´¨´���L��¢����+�+µ � ©~¢®�;��¨´���L��¢����?�±§~¬£��­:�+¢ ¤^©~�� ¥��­�¢£  Å �h�±«/¦�¨�¢ �^ k�^ p¨�©¥�?ËǦh¬£§~� � Å ¦ �´¨�¦�¨´�/¦h²�²³�h�±¡L¢£ ¥¤z¨´� ¨�©¥�¦h«T�^§~ J¨�� Å ¢£  Å �h�±«$¦�¨�¢ �^ z�±�+²³�+¢ Ëh�+¡�¢£ z�q¦h�´¨�¢ ¨´���±¦�¨�¢ �^ L� ����¤^¦��±¡~¢® ¥¤T¨�©~¦�¨W�´¨�¦�¨´�hµ

Page 27: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� ¾hÀ

� ��¨�©¥�t¡ «T�^ ~�h¨´�^ ¥� «T�^ ¥�h¨´�^ ~� «T�^ ¥�h¨´�^ ¥�  ¥�  ¥�  ~��´¨´�2§~²³¨�§¥��� �´¨´�±§~²³¨�§~��� �´¨´�±§~²³¨�§¥�±� ��¨´�±§~²³¨�§¥��� �´¨´�2§~²³¨�§¥��� �´¨´�±§L²³¨�§¥���

� ¨´���±¦�¨�¢£�^ ~� À�»h» � »h»h» ¾�»h»h» À�»h» � »h»h» ¾�»h»h»��¦�¨�¦��8��¨�� � � � � �_���2¢ �8¡L¢£² »tµ »^Ã�» � »tµ » � ¼�» »tµ » � ¾h¾ »tµ � ¼h¹h¹ »tµ ����� » »tµ »^¹h¾ �

¦��_���2¢ �8¡L¢£² »tµ »^¾^Ä�À »tµ » � ÁhÁ »tµ » � Ã^Ä »tµ � �N¹hÁ »tµ »^¼hÀhÀ »tµ »^¹hÀhÀ � � � � �_���2¢ �8¡L¢£² »tµ »^¾h¼hÀ »tµ » � �NÀ »tµ » � »^À »tµ ¾hÁh¼h¾ »tµ � Áh¾h¼ »tµ ��� Á �

¦��_���2¢ �8¡L¢£² »tµ »^¾hÁhà »tµ »^¾�»^À »tµ » � Ã�� »tµ Ãhà � Á »tµ � ÁhÀ^Ä »tµ � Ã^ÄhÄ � � � � �_���2¢ �8¡L¢£² »tµ »��NÀhà »tµ »^¾h¹ � »tµ »^¾�»^à »tµ �NÃhÁhÀ »tµ Ã�»^À�» »tµ ¾hÃ^Ä�Á

¦��_���2¢ �8¡L¢£² »tµ »����Nà »tµ »^¾h¹^Ä »tµ » � Á�� »tµ �NÀ��^» »tµ à � ¹�� »tµ ¾hÃ�» ��WËh���±¦�¤h� »tµ »^Ã��N¾ »tµ »^¾�»^¼ »tµ » � �JÄ »tµ Ã�»^¼hÀ »tµ � ¼h¼�� »tµ � � � ¼

� ¦��L¬ � � ª � �±¦h²³¨�¢ �^ ~¦h¬H²³�^��¨W�~���t¡~§~²³�+¡ �8É*�+¦h²±©�¦h¬£¤h�h�±¢ ¨�©~«w�±�+¬£¦�¨�¢ Ëh�B¨´�$�h�~¨�¢®«$¦h¬

�� �� ���� %$&��' �W$ ���������k$*)8���`��)

�   ¨�©~¢£�����+²³¨�¢ �^ �­:�y�����c�h��¨W�^ ¨�©¥�y�³Æt�c���±¢£«T�+ J¨�¦h¬¶���+�±§~¬ ¨���¦h ~¡�¡~¢£��²�§~�±�|¨�©¥�@�c��� Å �h�±«$¦h ~²³�B� Å �+¦h²2©k¦��L�~���+Æ8¢£«$¦�¨�¢£�^ k«$��¨�©¥�8¡Hµ � ©¥�?���+�±§~¬ ¨��;� Å �^§¥�B�³Æt�c���±¢£«T�+ J¨�� Å �h�B�+¦h²2©�«T��¨�©¥�t¡�¦h ~¡p�+¦h²2©�¡~¦�¨�¦z¤h���^§¥��¦��±�?¬®¢£�´¨´�+¡¢£ k¨�¦��L¬ � � µ � ©¥�?�+ N¨´�±¢£�+�;¢® 0¨�©~� ¨�¦��q¬ �T¦����?¨�©¥�?¨´�h¨�¦h¬�²³�^�´¨��B�³Æt�~���+�����+¡0�±�+¬£¦�¨�¢ Ëh�?¨´� ¨�©¥�?�h�~¨�¢£«$¦h¬��´�^¬®§¥¨�¢ �^ ��¦+Ëh���2¦�¤h�+¡0��Ëh���W¨�©¥��¡~¦�¨�¦$�´��¨��W� Å �+¦h²2©�¤h���^§~�Hµ� � �D� � � ��� � � � �� ��� � � � � �'� � � �� �����

� ©~��«T�^ ¥�h¨´�^ ~� Å §~ ~²³¨�¢ �^ L¦h¬L¦h¡~¦��~¨�¢£Ëh�;¡¥Ét ~¦h«$¢®² �~���h¤h�±¦h«/«$¢£ ¥¤@«$��¨�©¥�8¡U�c��� Å �h�±«$�1�±¢ ¤^ ~¢ �L²�¦h J¨�¬ É �c��¨´¨´���¨�©~¦h  ¨�©¥� ¦h¬ ¤h�h�±¢£¨�©~«j­W¢ ¨�©  ¥�B�´¨´�±§~²³¨�§¥�±�hµ �� ?¦�Ëh���±¦�¤h���^§¥�.¦h¬ ¤h�h�±¢£¨�©~«�©~¦h�Y�_���2²³�+ N¨�¦�¤h�����±���h�±� Å �±�^« �h�L¨�¢£«$¦h¬¨�©~¦�¨B¦���� ¨´�+ k¨�¢£«T�+����«$¦h¬£¬£���W¨�©~¦h k¨�©¥� �c���±²³�+ N¨�¦�¤h�T���±���h�±��� Å ¨�©¥� �L¦h��¢£² Å �h�±­:¦��±¡k¡¥Ét ~¦h«$¢£²@�~���h¤h�±¦h«/«$¢£ ¥¤¦h¬ ¤h�h�±¢£¨�©~« µ � ©¥� ��¦�¨�¢£� Å É8¢® ¥¤T�c��� Å �h�2«$¦h ~²³�y� Å �^§~��¦��~�~�±�+Æ¥¢£«$¦�¨�¢ �^ 0«T��¨�©¥�t¡�¢£��²³�^ ~��¢£�´¨´�+ J¨W¨�©~���^§¥¤^©¥�^§¥¨�¨�©¥�¡~¦�¨�¦*�´��¨���µ � ©~¢£��¢£�W ~�h¨�¨´�±§~� Å �h��¨�©~� ¦h¬ ¤h�h�2¢ ¨�©~«-­�¢ ¨�©0 ~�U�´¨´�±§L²³¨�§¥��� �_­W©¥�^�´���_���2²³�+ N¨�¦�¤h�?�������h�2� 0q§~²³¨�§~¦�¨´�©~¢ ¤^©L¬ É �c��¨�­:���+ �¡~¦�¨�¦/�´��¨���µ

��§~�:¦h¬ ¤h�h�2¢ ¨�©~«��±�+¦h²±©¥�+� ��¦�¨�¢®� Å ¦h²³¨´�h�±ÉU¬ ��Ëh�+¬£�\� Å �c��� Å �h�±«$¦h ~²³�;¦�¨:¦ ��«$¦h¬®¬ ���: 8§~« �c���1� Å ¢ ¨´���2¦�¨�¢ �^ ~�\¨�©~¦h ¨�©¥�T¦h¬ ¤h�h�±¢£¨�©~« ­�¢ ¨�©p ¥�*��¨´�±§~²³¨�§¥���hµ541Ëh�+ �¦�¨yÀ�»h» ¢ ¨´���±¦�¨�¢ �^ ~�;�^§¥�B¦h¬ ¤h�h�2¢ ¨�©~« �c��� Å �h�2«$�W­�¢ ¨�©~¢£ kÀ/�c���±²³�+ J¨� Å �h�~¨�¢£«/¦h¬1¦h ~¡�¦�¨?¾�»h»h»k¢ ¨´���±¦�¨�¢ �^ L��¢ ¨@�_��� Å �h�±«/�y²³�^ ~��¢£��¨´�+ N¨�¬ É�­�¢ ¨�©~¢£ �¾�_���2²³�+ N¨�� Å �h�~¨�¢£«$¦h¬�µ � ¨������+«$�¨�©~¦�¨�¨�©¥�T¦h¬ ¤h�h�±¢£¨�©~« ­W¢ ¨�©k ¥� �´¨´�±§~²³¨�§¥�±� ©~¦h��¦z��¬ ��­ �±¦�¨´�?� Å ²³�^ JËh����¤h�+ ~²³�hµ 4�Ëh�+ �¦�¨y¾�»h»h»¢ ¨´���2¦�¨�¢ �^ ~�;¨�©¥��c���±²³�+ J¨�¦�¤h�@�������h�2� Å �h���´�^«T�@¡~¦�¨�¦/�´��¨���¦����y�³Æt¨´���+«T�+¬£Éz©L¢ ¤^©¶µ

�; ¥�h¨�©¥����¤h�+ ~���±¦h¬¥� �L�´���±ËǦ�¨�¢ �^ /¢£��¨�©~¦�¨�¨�©¥��¦h¬£¤h�h�±¢ ¨�©~«�­W¢£¨�©T ¥�y�´¨´�±§~²³¨�§~��� ©~¦h�.¨�©¥��©~¢£¤^©¥�+�´¨��������h�±��­W©¥�+ ¨�©¥�B¦+Ëh���2¦�¤h�@­|¦h¢ ¨�¢£ ¥¤T²³�^�´¨ �c��� ²�§~�´¨´�^«T����¢£�|¬ ��­:��� ¨�©~¦h z¨�©¥�y�´���±Ë8¢£²³�B²³�^�´¨ �c��� ²�§~�´¨´�^«$���+µ � ©¥���c���±²³�+ J¨�¦�¤h�

Page 28: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� ¾h¹

�������h�2��ËǦ���É Å ���^«�¹��j¨´��¾hÃ��j­�©¥�+ T«T��Ët¢£ ¥¤ Å ���^«�©~¢£¤^©�­|¦h¢ ¨�¢£ ~¤�²³�^��¨1¡~¦�¨�¦y�´��¨���¨´�y¬ ��­ ­|¦h¢ ¨�¢£ ~¤�²³�^��¨�µ � ©~¢£���§¥¤h¤h�+��¨��|¨�©~¦�¨|¨�©~¢£�:¦��~�~����Æt¢£«/¦�¨�¢ �^ U«$��¨�©¥�8¡*¢£ ~²�§~�±�\¦ ¬ ��­\��� ¨´�h¨�¦h¬_­|¦h¢ ¨�¢£ ¥¤?²³�^�´¨|¨�©~¦h *¨�©¥�;�h�~¨�¢£«$¦h¬ �8­W©~¢®²±©«T�+¦h ~� ¨�©~¦�¨?¢ ¨�� �´����Ët¢£²³�z¡¥�+²�¢£��¢ �^ ��±§~¬ �+��©~¦+Ëh�z¨�©¥�z�´����Ëh��� �L§L�´É�¬ �^ ¥¤h��� ¨�©~¦h 4¨�©¥�*�h�~¨�¢£«$¦h¬:�´����Ët¢£²³�/�2§~¬ �+��µ� ©¥�+��� �q©¥�+ ¥�^«T�+ ~¦z¦����$ ~�h¨y¦��~�L¦����+ J¨y¢£ k¨�©¥�$²�¦h�´�T� Å �^§¥�y«T�^ ¥�h¨´�^ ~�$¦h¬ ¤h�h�±¢ ¨�©~« ­�©¥�^�´�?�c��� Å �h�2«$¦h ~²³�T¢£� ¥�h¨W¦ !c�+²³¨´�+¡��8Éz¨�©¥�y ~¦�¨�§¥�±�B� Å ¨�©~�@¡~¦�¨�¦$�´��¨��+µ� � � � � ����� �����

� ©~��«T�^ ¥�h¨´�^ ~� ¦h¡~¦��L¨�¢ Ëh� ¡~É8 ~¦h«/¢£²B�~���h¤h�2¦h«$«$¢£ ¥¤*¦h¬ ¤h�h�±¢ ¨�©L«-�+�´¨�¢£«/¦�¨´�+��¨�©~���h�~¨�¢®«$¦h¬A¡¥�+²�¢®��¢ �^ �2§~¬ �+�Å ¦h¢ �±¬ É$­:�+¬£¬_­W¢ ¨�©z��«/¦h¬£¬_�_���2²³�+ N¨�¦�¤h�y���±���h�±��µ � ©~��¦h¬£¤h�h�±¢ ¨�©~«w¢£�:��� �q§~�´¨ ¢£ z¢ ¨�� ²³�^ ~��¢®�´¨´�+ ~²³É*¨´�T�_��� Å �h�±«u­:�+¬£¬¨�©¥���^§~¤^©¥�^§¥¨\¡L¢"!_���±�+ N¨:¡~¦�¨�¦�����¨��:¦h L¡U¢ ¨:²³�^ NËh���±¤h�+� Å ¦h��¨´���\¨�©~¦h z¢ ¨��:²³�^«T�c��¨�¢ ¨�¢ �^ Hµ �|¨:¨�©¥����¦h«T�� J§L« �c���1� Å¢ ¨´���±¦�¨�¢£�^ ~�;�^§¥�B«T�^ ¥�h¨´�^ ¥�T¦h¬ ¤h�h�±¢£¨�©~« ¢£�;¨´�+ 0¨�¢®«T�+�B²�¬ �^�´����¨´�z�h�~¨�¢£«$¦h¬Y¨�©~¦h k¨�©¥� �L¦h�±¢£² Å �h��­|¦��±¡p¡¥Ét ~¦h«$¢£²�~���h¤h�2¦h«$«$¢£ ¥¤T¦h¬ ¤h�h�2¢ ¨�©~« µ � ¨�¢£�|Ëh����Éz�+ ~²³�^§¥�2¦�¤^¢£ ¥¤T¨�©~¦�¨ �^§¥��¦h¬ ¤h�h�±¢ ¨�©~« �c��� Å �h�±«$�|­:�+¬£¬c­W¢£¨�© §~ ~²³����¨�¦h¢£ J¨�É�c�+²�¦h§~�´�y¢£ z�L�±¦h²³¨�¢£²³�@¦����2¢ ËǦh¬®� ¦����@§~��§L¦h¬£¬ É/�´¨´�t²±©L¦h�´¨�¢£²�µ

� ¢£ ~¦h¬®¬ É �1¨�©¥�k¢£ J¨´���8¡L§~²³¨�¢ �^ #� Å �´¨´�±§~²³¨�§~����¢£ #¨�©¥�0¦h¬ ¤h�h�±¢ ¨�©~« ¢£�$¡¥� �q ~¢ ¨´�+¬ É4¦h F¢£«T�L����Ëh�+«T�+ J¨/��¢® ~²³�¢ ¨¦h¬£¬ ��­W�$¨�©~�0¦��~�~�±�+Æ¥¢£«$¦�¨�¢ �^ �¨´�+²2©~ ~¢$#8§¥�0¨´��«$¦��h�p§~�´�� Å ¢®«T�c�h��¨�¦h N¨U¢£  Å �h�±«$¦�¨�¢ �^ ��^ F¨�©¥�0�±©~¦��c�� Å ¨�©¥�Ë�¦h¬£§¥� Å §~ ~²³¨�¢£�^ ­W©¥�+ 0¦��~�~����Æt¢®«$¦�¨�¢£ ¥¤$¨�©~���h�~¨�¢®«$¦h¬A¡¥�+²�¢®��¢ �^ �c�^¬£¢®²�¢ �+��µ � ©¥�@§L�´�@� Å ��¨´�±§~²³¨�§¥��� ¢£  Å �h��­|¦��±¡¡¥Ét ~¦h«$¢£²y�~���h¤h�±¦h«$«/¢£ ¥¤/¦h¬ ¤h�h�2¢ ¨�©~«$��¢£��Ëh����É��~���^«$¢£�±¢£ ¥¤$¦h ~¡k²³�^§~¬£¡ �_���³Æ8¨´�+ L¡¥�+¡0¢® �©~¦��±¡~����¢£ ~��¨�¦h ~²³�+�W� Ũ�©¥�y�L��� �L¬ �+«u­:�����y¨�©¥�@��¨�¦�¨´�yËǦ��2¢£¦��L¬ �y¢£� « §~¬ ¨�¢®¡~¢£«T�+ ~�±¢ �^ ~¦h¬L¦h L¡ �h�~¨�¢£«$¦h¬¶���^¬£§¥¨�¢ �^ ~� ¦��±�@¢£ J¨´�±¦h²³¨�¦��L¬ �hµ

� ��� � �� � � ����� � � �

� ©~¢®�\�±�+�´�+¦��±²±©�­:¦h����§¥�~�c�h��¨´�+¡z¢£ z�q¦���¨ �JÉ*¤h�±¦h J¨�� � � ���� � �N¼h¹h¾�»ÇÂe¼hÃ� � Â�»h»^¼hÁ Å �±�^«w¨�©¥� �;¢ � � �h�±²³� � � ²³�� Å �t²�¢ �+ J¨�¢ �L²�W�+�´�+¦��±²±© �_¦h ~¡¤h�±¦h N¨ � � � »h»ÇÂeÁhÀhÃh¹hÁ Å �±�^«w¨�©¥� -;¦�¨�¢ �^ ~¦h¬ �t²�¢ �+ ~²³� � �^§~ ~¡~¦�¨�¢ �^ Hµ ���@­:�^§~¬£¡¦h¬£�´�$¬®¢��h�B¨´�/¦h² �t ¥��­W¬ �+¡¥¤h��¨�©¥�y²³�^«$«T�+ J¨���¦h L¡ ²�¦���� Å §L¬¶�~���8� Å �±�+¦h¡~¢£ ¥¤?� Å ¦$²³�^ ~��²�¢ �+ J¨�¢ �^§~������Ë8¢£��­\���+µ

Page 29: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� ¾^Ä

� ��� ������ �����

%|¦h¢£¬ ��É � -�µ � ¼hÀ��¥µ � ��  #8§¥�+§¥�+¢£ ¥¤?�~�±�8²³�+�����+� ­W¢ ¨�© �L§L¬��U�´���±Ë8¢£²³��� ��� G�� ��$��� G�d � ���(G�d*� " G � . � �cÁ�»��8Á^ÄJµ%|¦��± ¥��¨´¨ � �@µ � ¼^Ä�Ã8µ � ��  �h�c���±¦�¨�¢£ ~¤?¦/��©J§~¨´¨�¬ �y�´����Ët¢£²³��� � �%�^g/����� K " �_Ã�»^À �8à � �¥µ%:����¨�¦��&�+¢ � ��µ ' �8�c���±¦h  �+¦ � � µ � µ � ¼h¼h¼(��µ � �  JËh�+ N¨´�h�±Ék²³�^ J¨´���^¬.�^ ��´� #8§¥�+ ~²³�+�y� Å ¬£¢£ �8�;­W¢ ¨�©k¤^¢ Ëh�+ p¨´�2¦h ~�¿Â�c�h��¨�¦�¨�¢£�^  Å �±� #J§¥�+ L²�¢ �+�%� � h ? � �%�:?*� �����8? �� ��(�8;��>?*�� � 1 � � �8;," ���#�8?�� " �8?*�8O �#"jK (�� �q¾h¹ � �8¾^ÄÇ»tµ%:����¨�¦��&�+¢ � ��µ�' �8�c���±¦h  �+¦ � � µ � µ � ¼h¼h¼,+³µ � � ¢£ ~¢®«$¢��+¢£ ¥¤�¬ �h¤^¢£�´¨�¢£²|²³�^�´¨��1¢£ �« §~¬ ¨�¢®�´¨�¦�¤h����§¥�~�q¬ ÉB²±©~¦h¢£ L�%� ���������

� �>K%� ��� "jM 0 � ���SK����#"jK % � �qÃh¼h¼ � � � ÄJµ%:����¨�¦��&�+¢ � �1µ � %:����¨��´�&��¦h� �1� µ ' �8�c���±¦h  �+¦ � � µ � µ:¸º¾�»h»h»tµ��;�L¨�¢£«$¦h¬�¦h ~¡� ¥�+§¥����Â�¡~É8 ~¦h«/¢£²?�~���h¤h�±¦h«/«$¢£ ¥¤���^¬£§¥¨�¢ �^ ~� Å �h�|¦ �´¨´�8²2©~¦h�´¨�¢£²�¢£ JËh�+ J¨´�h��É$¨´�±¦h�´�c�h��¨�¦�¨�¢£�^ /�~�±� �L¬ �+« � 6� ¥�q§ �L¬£¢£�±©¥�+¡ ¨´�+²±©~ ~¢®²�¦h¬q�±���_�h�±¨ � 6; ~¢ ÂËh���2��¢ ¨�¦ �;��¤^¬£¢��t¨�§~¡~¢ ��¢ %:���+��²�¢£¦tµ%:����¨��´�&��¦h� ����µ ' � ��¢ ¨��±¢��8¬®¢£� ���¥µ � ¼h¼h¹8µ� �P; � � ����$4? �8O ��" � � � � ��O O �6?�� � �|¨�©¥�+ L¦ �t²�¢ �+ J¨�¢ �L²�� %:�+¬£«$�^ N¨ � � ��µ�|©~¦h§L¡~©¥��É � � µ�' � �+«T�L¬ ��¨´�^ ����~µ � ¼hÁhÃ8µ � � ����K���� ��; �:K%� �6?��=; ���� ; �j; �>K � � ¢£¬£��É �,-W��­! ��h� �qµ�;� � ���yµ � ¼^Ä�Á(�hµ � �;�~¨�¢£«$¦h¬L²³�^ N¨´���^¬_� Å �L¦�¨�²2©z�´����Ët¢£²³� #8§¥�+§¥�+�1­W¢ ¨�©/��­W¢ ¨�²±©L¢£ ¥¤@²³�^�´¨�� � � � �����8?*" �jK �6? � � �* !�#� �

� ��+ � +%�� !����$#"�� � ÄhÄ$� � ¼��¥µ�;� � � �yµ � ¼^Ä�Á,+�µ � �;�~¨�¢£«$¦h¬Y¡~¢£���L¦�¨�²±©~¢® ¥¤U� Å ¦ �L ~¢ ¨´� ²�¦��L¦h²�¢ ¨�Ép��©8§¥¨´¨�¬ ��� ��% �8?*� � �jO �j? �fd*"%� �j?*"�� [ % � � Ãh¹h¾ �

� Ã^Ä�¾8µ�;� � � �yµ ' �t²2©~«$¢£¡¥¨ �~�;µ � ¼hÁ^ÄJµ�� �;�~¨�¢®«$¦h¬q¦�Ëh���±¦�¤h��²³�^��¨:�c�^¬£¢®²�¢ �+� Å �h�|¨�©¥��¨�­:��Â�¨´���±«$¢£ L¦h¬¶��©J§~¨´¨�¬ ��� ��% �8?*� ��� �

O �P? �d "%� �j?*"�� "�" �q¹h¹h¾ �8¹h¹h¼8µ�;� � � �yµ' �8��� Å ����� � �yµ � ¼^Ä�Ã8µ�� ���~¨�¢£«$¦h¬_²³�^ N¨´�±�^¬¶� Å �L¦�¨�²2© �´����Ët¢£²³� #J§~�+§¥�+�%� � � �����8?*"��>K �6? � � �* !�#� � � ��+ � �

+%�� !����$ ( �qÃ��^»��8Ãh¹ � µ� ¤^ ~¦h¬£¬ � 4�µ ' È �^¬ �+�±¦���� �Yµ � ¼^Ä�¾8µ � �;�c���±¦�¨�¢£ ¥¤ ²±©~¦��±¦h²³¨´���2¢£�´¨�¢£²��:� Å ¦ ��¢£«T�L¬£����©J§~¨´¨�¬ ��§~ L¡¥���\¬ �t²�¦h¬L¡L¢£�´�L¦�¨�²2©~¢£ ¥¤�2§~¬ �+�%� �'& �(�%� �����#�8? K(� �>K � ��� "jM [�) � � »NÄhÄ$� � »^ÁhÁ8µ� ¤^ ~¦h¬£¬ � 4�µ�' È �^¬ �+�±¦����*�Yµ � ¼^Ä)�¥µ � ���_���2¦�¨�¢£ ¥¤/²2©~¦��±¦h²³¨´���±¢£��¨�¢£²��;� Å ¦h 0¢£  �L ~¢ ¨´�y²�¦��L¦h²�¢£¨�É0�±©J§¥¨´¨�¬£�hª|²³�^ J¨´���^¬Y¦�¨¦/��¢£ ¥¤^¬ ��¨´���±«/¢£ ~¦h¬#� �'& �(�%� �����#�8? K(� �>K � ��� "jM [2[ � � »h»^à � � »^¾��¥µÈ �^�´¨´�+  ����µ � ¼h¹^ÄJµ � ©¥�:²�§~�´¨´�t¡~¢£¦h ��~��� �L¬ �+«�� �6?�yµ^�:�±§¥�^  �Ç�+¡Hµ � � � §¥�+§¥�+¢£ ~¤ � ©¥���h�±É �����+²³�+ J¨1����Ëh�+¬£�h�L«T�+ J¨��¦h L¡ �W�L�L¬£¢£²�¦�¨�¢ �^ L�%� � 4\ ¥¤^¬£¢£�±© �1���+��� ���H¨�¡����A�^ ~¡¥�^ ��¥�~�Aµ¥¹hÀ �tÄÇ»tµÈ �^�´¨´�+  � ��µ � ¼^Ä�Ã8µHd � � "PM �4K����#" � M�� ����$ � 1Ud ����� �#" � d,$4K�� �jO K � �Y����¤^¦h«$�^  ���±�+��� �,-���­* ��h� �_µ� �+¡~©~¢ �+�~µ � ¼hÁ��¥µ,� � "��j? ��� ������ � �2O �j? �6K �6?-�f;� ��-� ;,�P; ���6?��.% � � �� <K � � ¢®¬ ��É 4\¦h�´¨´���±  �A¢®«$¢ ¨´�+¡���-���­ �;�+¬£©~¢ºµ-��+§¥¨�� � � µ � ¼h¹^ÄJµ � � ¤h�+ ¥���2¦h¬q²�¬£¦h�±��� Å �L§~¬$� #8§¥�+§¥�+��­�¢ ¨�© �1�^¢£���´�^ $¢® ¥�L§¥¨%� � � ?&?2G/% ���6M�G d � ���(G " " �¥Ä�Àh¼ �tÄhÄÇ»tµ�Y��­:�+¬£¬ � �jµ %;µ � ¼hÁhÀ8µ � �� L¦h¬ É8�±¢£�B� Å Ëh�+©~¢£²�¬£�/©¥�^¬£¡~¢® ¥¤¦h ~¡�²�¦h ~²³�+¬®¬£¦�¨�¢ �^ 4�´¨´�±¦�¨´��¤^¢ �+� ¢£  �L§L¬��k¦��±�±¢ Ë�¦h¬ � �L§L¬��������Ë8¢®²³� #J§¥�+§~�+�%� � � � �8?&K ������� �����#�8? d*"%� �j?*"�� .0� ¸��J½ �_ÃhÀh¾ �8Ã^ÄhÄJµ�Y��­:�+¬£¬ � �jµ %�µ '72;§~« �L¬ ��¨ �(�Yµ � ¼hÁh¹8µ � � ©¥� �L§~¬��$������Ë8¢®²³� #8§¥�+§¥�;­W¢ ¨�©z¦ ¤h�+ ¥���±¦h¬¶²³�^ J¨´���^¬¶�´¨´�2¦�¨´��¤hÉqª � ©¥����Â�±��¨�¢£²�¦h¬H¦h ~¦h¬ Ét��¢£� ¦h L¡¦T ¥��­ ²³�^«T�q§¥¨�¦�¨�¢ �^ ~¦h¬¶�L���8²³�+¡L§¥����� ��& �(�%� �����#�8? K(� �>K � ��� "jM " % ¸º¾^½ �c¾h¹^Ä$�8¾^Ä�À8µ�1§~¨´���±«$¦h �� � µ ��µ � ¼h¼��¥µ1% ���2�����3� � "%�SK �#�8? � � " �jK�K �>K �4�h�^©~ �� ¢£¬£��ÉU¦h ~¡ �t�^ ~� � �  ~²�µ � -���­5 ��h� �_µ�8�c���±¦h  �+¦ � � µ '76 �h��Ët¢£²±© � � µ � ¼h¼��¥µ � � ¢® ~¢£«$¢��+¢® ¥¤y¨´�±¦h�´�c�h��¨�¦�¨�¢£�^ *¦h L¡U¢£ JËh�+ J¨´�h��ÉU²³�^�´¨�� Å �h� �´��Ëh���±¦h¬_�~���t¡t§L²³¨�� �^ �¦T��¢® ¥¤^¬ �B¬£¢£  � � �'& �(��� �����#�8? K(� �>K � ��� "jM %�[ �qÁ^Ä�¼ �8Áh¼��¥µ�8�c���±¦h  �+¦ � � µ ' 6 �h��Ët¢£²2©����jµ � ¼h¼h¹8µ � �� �¦h¬£¤h�h�±¢ ¨�©~« Å �h�y�h�~¨�¢£«/¦h¬.��©~¢ �L«$�+ N¨��;­W¢ ¨�©p¤^¢£Ëh�+  Å �±� #J§¥�+ L²�¢ �+�%� �

������ � �jK%� ��� "PM 0 � � �NK�����"PK %�" �q¹hÀhÀ �8¹^Ä � µ

Page 30: Exploiting Structure in Adaptive Dynamic Programming - CiteSeer

��������������� ��������������� ¾hÁ

�t§¥¨´¨´�^  � �@µ ' %:¦��±¨´� � �@µ � ¼h¼hÁ8µ/� ���6? 1%��� "��jO �j?*� 0 � ���:? �S?�� � � ©¥� � � � �1���+��� �¥�|¦h« �~�±¢£¡¥¤h� � � ¦h����¦h²±©8§~�´��¨´¨��+µ���+¢£����� 2@µ � ¼hÁ � µ � � §¥��¨�©¥���y�±�+��§~¬ ¨��;�^ �¦h �¢£  �q ~¢ ¨´��²�¦��q¦h²�¢ ¨�Ép��©8§¥¨´¨�¬ �?­W¢ ¨�©p²³�^ J¨´���^¬.¦�¨y¦ ��¢£ ¥¤^¬£��¨´���±«/¢£ ~¦h¬#� �

& �,��� ��������? K � �jK%� ��� "PM [�� � � ¾ � ¾ � � ¾ � ÄJµ���+¢£����� 2@µ ' �1¬£¢£� �Ǧ ���_µ � ¼^Ä�¹8µ �;�~¨�¢£«$¦h¬¶²³�^ J¨´���^¬¶� Å �´�^«$� � ¦�� �h��Ë �~���t²³�+���´�+� ­�¢ ¨�© ¦��~�L¬£¢®²�¦�¨�¢ �^ ~�:¨´� �L¦�¨�²2©

#8§¥�+§¥�+¢® ¥¤$¦h ~¡�²³�^ J¨�¢£ 8§¥�^§~� �±��Ë8¢ ��­j¢£ JËh�+ N¨´�h��É �´Ét�´¨´�+«$���*��¢£��²�§~�±��¢ �^  �L¦��c���W ~�¥µq¾ � � � � ©¥� �:�+ J¨´��� Å �h�� ¦�¨�©¥�+«$¦�¨�¢®²�¦h¬ �8¨�§~¡L¢ �+� ¢£ &41²³�^ ~�^«$¢£²���¦h ~¡ � ¦h ~¦�¤h�+«$�+ N¨ �t²�¢ �+ ~²³� � -��h��¨�©J­\�+��¨´���±  6; ~¢ Ëh���±�±¢ ¨�Éhµ


Recommended