Robust Markov decision processes: a place where AI and formal methods meet
Markov decision processes (MDPs) are a standard model for sequential decision-making problems and are widely used across many scientific areas, including formal methods and artificial intelligence (AI). MDPs do, however, come with the restrictive assumption that the transition probabilities need to...
| Main Authors: | , , , , |
|---|---|
| Format: | Book section |
| Language: | English |
| Published: |
Springer
2024
|
| _version_ | 1824458679590584320 |
|---|---|
| author | Suilen, M Badings, T Bovy, EM Parker, D Jansen, N |
| author_facet | Suilen, M Badings, T Bovy, EM Parker, D Jansen, N |
| author_sort | Suilen, M |
| collection | OXFORD |
| description | Markov decision processes (MDPs) are a standard model for sequential decision-making problems and are widely used across many scientific areas, including formal methods and artificial intelligence (AI). MDPs do, however, come with the restrictive assumption that the transition probabilities need to be precisely known. Robust MDPs (RMDPs) overcome this assumption by instead defining the transition probabilities to belong to some uncertainty set. We present a gentle survey on RMDPs, providing a tutorial covering their fundamentals. In particular, we discuss RMDP semantics and how to solve them by extending standard MDP methods such as value iteration and policy iteration. We also discuss how RMDPs relate to other models and how they are used in several contexts, including reinforcement learning and abstraction techniques. We conclude with some challenges for future work on RMDPs. |
| first_indexed | 2025-02-19T04:29:44Z |
| format | Book section |
| id | oxford-uuid:4be22ef5-3a3f-4344-b370-a1336a0a1dc3 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2025-02-19T04:29:44Z |
| publishDate | 2024 |
| publisher | Springer |
| record_format | dspace |
| spelling | oxford-uuid:4be22ef5-3a3f-4344-b370-a1336a0a1dc32024-12-16T12:36:49ZRobust Markov decision processes: a place where AI and formal methods meetBook sectionhttp://purl.org/coar/resource_type/c_3248uuid:4be22ef5-3a3f-4344-b370-a1336a0a1dc3EnglishSymplectic ElementsSpringer2024Suilen, MBadings, TBovy, EMParker, DJansen, NMarkov decision processes (MDPs) are a standard model for sequential decision-making problems and are widely used across many scientific areas, including formal methods and artificial intelligence (AI). MDPs do, however, come with the restrictive assumption that the transition probabilities need to be precisely known. Robust MDPs (RMDPs) overcome this assumption by instead defining the transition probabilities to belong to some uncertainty set. We present a gentle survey on RMDPs, providing a tutorial covering their fundamentals. In particular, we discuss RMDP semantics and how to solve them by extending standard MDP methods such as value iteration and policy iteration. We also discuss how RMDPs relate to other models and how they are used in several contexts, including reinforcement learning and abstraction techniques. We conclude with some challenges for future work on RMDPs. |
| spellingShingle | Suilen, M Badings, T Bovy, EM Parker, D Jansen, N Robust Markov decision processes: a place where AI and formal methods meet |
| title | Robust Markov decision processes: a place where AI and formal methods meet |
| title_full | Robust Markov decision processes: a place where AI and formal methods meet |
| title_fullStr | Robust Markov decision processes: a place where AI and formal methods meet |
| title_full_unstemmed | Robust Markov decision processes: a place where AI and formal methods meet |
| title_short | Robust Markov decision processes: a place where AI and formal methods meet |
| title_sort | robust markov decision processes a place where ai and formal methods meet |
| work_keys_str_mv | AT suilenm robustmarkovdecisionprocessesaplacewhereaiandformalmethodsmeet AT badingst robustmarkovdecisionprocessesaplacewhereaiandformalmethodsmeet AT bovyem robustmarkovdecisionprocessesaplacewhereaiandformalmethodsmeet AT parkerd robustmarkovdecisionprocessesaplacewhereaiandformalmethodsmeet AT jansenn robustmarkovdecisionprocessesaplacewhereaiandformalmethodsmeet |