Robust Adaptive Markov Decision Processes in Multi-vehicle Applications
This paper presents a new robust and adaptive framework for Markov decision processes that accounts for errors in the transition probabilities. Robust policies are typically found off-line, but can be extremely conservative when implemented in the real system. Adaptive policies, on the other hand, a...
| Main Authors: | , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | en_US |
| Published: |
Institute of Electrical and Electronics Engineers
2010
|
| Online Access: | http://hdl.handle.net/1721.1/58906 https://orcid.org/0000-0001-8576-1930 |
| _version_ | 1826198490276954112 |
|---|---|
| author | How, Jonathan P. Bertuccelli, Luca F. Bethke, Brett M. |
| author2 | Massachusetts Institute of Technology. Aerospace Controls Laboratory |
| author_facet | Massachusetts Institute of Technology. Aerospace Controls Laboratory How, Jonathan P. Bertuccelli, Luca F. Bethke, Brett M. |
| author_sort | How, Jonathan P. |
| collection | MIT |
| description | This paper presents a new robust and adaptive framework for Markov decision processes that accounts for errors in the transition probabilities. Robust policies are typically found off-line, but can be extremely conservative when implemented in the real system. Adaptive policies, on the other hand, are specifically suited for on-line implementation, but may display undesirable transient performance as the model is updated though learning. A new method that exploits the individual strengths of the two approaches is presented in this paper. This robust and adaptive framework protects the adaptation process from exhibiting a worst-case performance during the model updating, and is shown to converge to the true, optimal value function in the limit of a large number of state transition observations. The proposed framework is investigated in simulation and actual flight experiments, and shown to improve transient behavior in the adaptation process and overall mission performance. |
| first_indexed | 2024-09-23T11:05:46Z |
| format | Article |
| id | mit-1721.1/58906 |
| institution | Massachusetts Institute of Technology |
| language | en_US |
| last_indexed | 2024-09-23T11:05:46Z |
| publishDate | 2010 |
| publisher | Institute of Electrical and Electronics Engineers |
| record_format | dspace |
| spelling | mit-1721.1/589062022-10-01T01:10:22Z Robust Adaptive Markov Decision Processes in Multi-vehicle Applications How, Jonathan P. Bertuccelli, Luca F. Bethke, Brett M. Massachusetts Institute of Technology. Aerospace Controls Laboratory Massachusetts Institute of Technology. Department of Aeronautics and Astronautics How, Jonathan P. How, Jonathan P. Bertuccelli, Luca F. Bethke, Brett M. This paper presents a new robust and adaptive framework for Markov decision processes that accounts for errors in the transition probabilities. Robust policies are typically found off-line, but can be extremely conservative when implemented in the real system. Adaptive policies, on the other hand, are specifically suited for on-line implementation, but may display undesirable transient performance as the model is updated though learning. A new method that exploits the individual strengths of the two approaches is presented in this paper. This robust and adaptive framework protects the adaptation process from exhibiting a worst-case performance during the model updating, and is shown to converge to the true, optimal value function in the limit of a large number of state transition observations. The proposed framework is investigated in simulation and actual flight experiments, and shown to improve transient behavior in the adaptation process and overall mission performance. United States. Air Force Office of Scientific Research (grant FA9550-08-1-0086) 2010-10-06T17:10:04Z 2010-10-06T17:10:04Z 2009-07 2009-06 Article http://purl.org/eprint/type/JournalArticle 978-1-4244-4523-3 0743-1619 INSPEC Accession Number: 10775888 http://hdl.handle.net/1721.1/58906 Bertuccelli, L.F., B. Bethke, and J.P. How. “Robust adaptive Markov Decision Processes in multi-vehicle applications.” American Control Conference, 2009. ACC '09. 2009. 1304-1309. ©2009 Institute of Electrical and Electronics Engineers. https://orcid.org/0000-0001-8576-1930 en_US http://dx.doi.org/10.1109/ACC.2009.5160511 American Control Conference, 2009. ACC '09 Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. application/pdf Institute of Electrical and Electronics Engineers IEEE |
| spellingShingle | How, Jonathan P. Bertuccelli, Luca F. Bethke, Brett M. Robust Adaptive Markov Decision Processes in Multi-vehicle Applications |
| title | Robust Adaptive Markov Decision Processes in Multi-vehicle Applications |
| title_full | Robust Adaptive Markov Decision Processes in Multi-vehicle Applications |
| title_fullStr | Robust Adaptive Markov Decision Processes in Multi-vehicle Applications |
| title_full_unstemmed | Robust Adaptive Markov Decision Processes in Multi-vehicle Applications |
| title_short | Robust Adaptive Markov Decision Processes in Multi-vehicle Applications |
| title_sort | robust adaptive markov decision processes in multi vehicle applications |
| url | http://hdl.handle.net/1721.1/58906 https://orcid.org/0000-0001-8576-1930 |
| work_keys_str_mv | AT howjonathanp robustadaptivemarkovdecisionprocessesinmultivehicleapplications AT bertuccellilucaf robustadaptivemarkovdecisionprocessesinmultivehicleapplications AT bethkebrettm robustadaptivemarkovdecisionprocessesinmultivehicleapplications |