Solving uncertain MDPs with objectives that are separable over instantiations of model uncertainty
Markov Decision Problems, MDPs offer an effective mechanism for planning under uncertainty. However, due to unavoidable uncertainty over models, it is difficult to obtain an exact specification of an MDP. We are interested in solving MDPs, where transition and reward functions are not exactly specif...
| Príomhchruthaitheoirí: | , , , |
|---|---|
| Rannpháirtithe: | |
| Formáid: | Alt |
| Teanga: | en_US |
| Foilsithe / Cruthaithe: |
AAAI Press
2018
|
| Rochtain ar líne: | http://hdl.handle.net/1721.1/116234 https://orcid.org/0000-0002-8585-6566 |
| _version_ | 1826193628183134208 |
|---|---|
| author | Adulyasak, Yossiri Varakantham, Pradeep Ahmed, Asrar Jaillet, Patrick |
| author2 | Massachusetts Institute of Technology. Department of Civil and Environmental Engineering |
| author_facet | Massachusetts Institute of Technology. Department of Civil and Environmental Engineering Adulyasak, Yossiri Varakantham, Pradeep Ahmed, Asrar Jaillet, Patrick |
| author_sort | Adulyasak, Yossiri |
| collection | MIT |
| description | Markov Decision Problems, MDPs offer an effective mechanism for planning under uncertainty. However, due to unavoidable uncertainty over models, it is difficult to obtain an exact specification of an MDP. We are interested in solving MDPs, where transition and reward functions are not exactly specified. Existing research has primarily focussed on computing infinite horizon stationary policies when optimizing robustness, regret and percentile based objectives. We focus specifically on finite horizon problems with a special emphasis on objectives that are separable over individual instantiations of model uncertainty (i.e., objectives that can be expressed as a sum over instantiations of model uncertainty): (a) First, we identify two separable objectives for uncertain MDPs: Average Value Maximization (AVM) and Confidence Probability Maximisation (CPM). (b) Second, we provide optimization based solutions to compute policies for uncertain MDPs with such objectives. In particular, we exploit the separability of AVM and CPM objectives by employing Lagrangian dual decomposition (LDD). (c) Finally, we demonstrate the utility of the LDD approach on a benchmark problem from the literature. |
| first_indexed | 2024-09-23T09:42:09Z |
| format | Article |
| id | mit-1721.1/116234 |
| institution | Massachusetts Institute of Technology |
| language | en_US |
| last_indexed | 2024-09-23T09:42:09Z |
| publishDate | 2018 |
| publisher | AAAI Press |
| record_format | dspace |
| spelling | mit-1721.1/1162342022-09-26T13:14:48Z Solving uncertain MDPs with objectives that are separable over instantiations of model uncertainty Adulyasak, Yossiri Varakantham, Pradeep Ahmed, Asrar Jaillet, Patrick Massachusetts Institute of Technology. Department of Civil and Environmental Engineering Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Adulyasak, Yossiri Jaillet, Patrick Markov Decision Problems, MDPs offer an effective mechanism for planning under uncertainty. However, due to unavoidable uncertainty over models, it is difficult to obtain an exact specification of an MDP. We are interested in solving MDPs, where transition and reward functions are not exactly specified. Existing research has primarily focussed on computing infinite horizon stationary policies when optimizing robustness, regret and percentile based objectives. We focus specifically on finite horizon problems with a special emphasis on objectives that are separable over individual instantiations of model uncertainty (i.e., objectives that can be expressed as a sum over instantiations of model uncertainty): (a) First, we identify two separable objectives for uncertain MDPs: Average Value Maximization (AVM) and Confidence Probability Maximisation (CPM). (b) Second, we provide optimization based solutions to compute policies for uncertain MDPs with such objectives. In particular, we exploit the separability of AVM and CPM objectives by employing Lagrangian dual decomposition (LDD). (c) Finally, we demonstrate the utility of the LDD approach on a benchmark problem from the literature. National Research Foundation of Singapore 2018-06-12T13:26:47Z 2018-06-12T13:26:47Z 2015-01 Article http://purl.org/eprint/type/ConferencePaper ISBN:0-262-51129-0 http://hdl.handle.net/1721.1/116234 Adulyasak, Yossiri, Pradeep Varakantham, Asrar Ahmed and Patrick Jaillet. "Solving uncertain MDPs with objectives that are separable over instantiations of model uncertainty." In Proceeding AAAI'15 Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, Austin, Texas, January 25-30 2015, AAAI Press, ©2015, pp. 3454-3460. https://orcid.org/0000-0002-8585-6566 en_US http://dl.acm.org/citation.cfm?id=2888196 Proceeding AAAI'15 Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence Creative Commons Attribution-Noncommercial-Share Alike http://creativecommons.org/licenses/by-nc-sa/4.0/ application/pdf AAAI Press MIT Web Domain |
| spellingShingle | Adulyasak, Yossiri Varakantham, Pradeep Ahmed, Asrar Jaillet, Patrick Solving uncertain MDPs with objectives that are separable over instantiations of model uncertainty |
| title | Solving uncertain MDPs with objectives that are separable over instantiations of model uncertainty |
| title_full | Solving uncertain MDPs with objectives that are separable over instantiations of model uncertainty |
| title_fullStr | Solving uncertain MDPs with objectives that are separable over instantiations of model uncertainty |
| title_full_unstemmed | Solving uncertain MDPs with objectives that are separable over instantiations of model uncertainty |
| title_short | Solving uncertain MDPs with objectives that are separable over instantiations of model uncertainty |
| title_sort | solving uncertain mdps with objectives that are separable over instantiations of model uncertainty |
| url | http://hdl.handle.net/1721.1/116234 https://orcid.org/0000-0002-8585-6566 |
| work_keys_str_mv | AT adulyasakyossiri solvinguncertainmdpswithobjectivesthatareseparableoverinstantiationsofmodeluncertainty AT varakanthampradeep solvinguncertainmdpswithobjectivesthatareseparableoverinstantiationsofmodeluncertainty AT ahmedasrar solvinguncertainmdpswithobjectivesthatareseparableoverinstantiationsofmodeluncertainty AT jailletpatrick solvinguncertainmdpswithobjectivesthatareseparableoverinstantiationsofmodeluncertainty |