Discovering blind spots in reinforcement learning
Agents trained in simulation may make errors in the real world due to mismatches between training and execution environments. These mistakes can be dangerous and difficult to discover because the agent cannot predict them a priori. We propose using oracle feedback to learn a predictive model of thes...
| Main Authors: | , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
2020
|
| Online Access: | https://hdl.handle.net/1721.1/125874 |
| _version_ | 1826202392268374016 |
|---|---|
| author | Ramakrishnan, Ramya Kamar, Ece Dey, Debadeepta Shah, Julie A Horvitz, Eric |
| author2 | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory |
| author_facet | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Ramakrishnan, Ramya Kamar, Ece Dey, Debadeepta Shah, Julie A Horvitz, Eric |
| author_sort | Ramakrishnan, Ramya |
| collection | MIT |
| description | Agents trained in simulation may make errors in the real world due to mismatches between training and execution environments. These mistakes can be dangerous and difficult to discover because the agent cannot predict them a priori. We propose using oracle feedback to learn a predictive model of these blind spots to reduce costly errors in real-world applications. We focus on blind spots in reinforcement learning (RL) that occur due to incomplete state representation: The agent does not have the appropriate features to represent the true state of the world and thus cannot distinguish among numerous states. We formalize the problem of discovering blind spots in RL as a noisy supervised learning problem with class imbalance. We learn models to predict blind spots in unseen regions of the state space by combining techniques for label aggregation, calibration, and supervised learning. The models take into consideration noise emerging from different forms of oracle feedback, including demonstrations and corrections. We evaluate our approach on two domains and show that it achieves higher predictive performance than baseline methods, and that the learned model can be used to selectively query an oracle at execution time to prevent errors. We also empirically analyze the biases of various feedback types and how they influence the discovery of blind spots. |
| first_indexed | 2024-09-23T12:06:44Z |
| format | Article |
| id | mit-1721.1/125874 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T12:06:44Z |
| publishDate | 2020 |
| record_format | dspace |
| spelling | mit-1721.1/1258742024-06-25T18:34:25Z Discovering blind spots in reinforcement learning Ramakrishnan, Ramya Kamar, Ece Dey, Debadeepta Shah, Julie A Horvitz, Eric Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Massachusetts Institute of Technology. Department of Aeronautics and Astronautics Agents trained in simulation may make errors in the real world due to mismatches between training and execution environments. These mistakes can be dangerous and difficult to discover because the agent cannot predict them a priori. We propose using oracle feedback to learn a predictive model of these blind spots to reduce costly errors in real-world applications. We focus on blind spots in reinforcement learning (RL) that occur due to incomplete state representation: The agent does not have the appropriate features to represent the true state of the world and thus cannot distinguish among numerous states. We formalize the problem of discovering blind spots in RL as a noisy supervised learning problem with class imbalance. We learn models to predict blind spots in unseen regions of the state space by combining techniques for label aggregation, calibration, and supervised learning. The models take into consideration noise emerging from different forms of oracle feedback, including demonstrations and corrections. We evaluate our approach on two domains and show that it achieves higher predictive performance than baseline methods, and that the learned model can be used to selectively query an oracle at execution time to prevent errors. We also empirically analyze the biases of various feedback types and how they influence the discovery of blind spots. 2020-06-18T21:32:43Z 2020-06-18T21:32:43Z 2018 2019-10-31T18:44:48Z Article http://purl.org/eprint/type/ConferencePaper 978-1-4503-5649-7 2523-5699 https://hdl.handle.net/1721.1/125874 Ramakrishnan, Ramya, et al., "Discovering blind spots in reinforcement learning." AAMAS '18: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, Stockholm, Sweden, July 2018 (New York, N.Y.: Association for Computing Machinery, 2018) url https://dl.acm.org/doi/10.5555/3237383.3237849 ©2018 Author(s) en https://dl.acm.org/doi/10.5555/3237383.3237849 AAMAS '18: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems Creative Commons Attribution-Noncommercial-Share Alike http://creativecommons.org/licenses/by-nc-sa/4.0/ application/pdf arXiv |
| spellingShingle | Ramakrishnan, Ramya Kamar, Ece Dey, Debadeepta Shah, Julie A Horvitz, Eric Discovering blind spots in reinforcement learning |
| title | Discovering blind spots in reinforcement learning |
| title_full | Discovering blind spots in reinforcement learning |
| title_fullStr | Discovering blind spots in reinforcement learning |
| title_full_unstemmed | Discovering blind spots in reinforcement learning |
| title_short | Discovering blind spots in reinforcement learning |
| title_sort | discovering blind spots in reinforcement learning |
| url | https://hdl.handle.net/1721.1/125874 |
| work_keys_str_mv | AT ramakrishnanramya discoveringblindspotsinreinforcementlearning AT kamarece discoveringblindspotsinreinforcementlearning AT deydebadeepta discoveringblindspotsinreinforcementlearning AT shahjuliea discoveringblindspotsinreinforcementlearning AT horvitzeric discoveringblindspotsinreinforcementlearning |