Non-Linear Monte-Carlo Search in Civilization II
This paper presents a new Monte-Carlo search algorithm for very large sequential decision-making problems. Our approach builds on the recent success of Monte-Carlo tree search algorithms, which estimate the value of states and actions from the mean outcome of random simulations. Instead of using a s...
| Main Authors: | , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | en_US |
| Published: |
AAAI Press/International Joint Conferences on Artificial Intelligence
2012
|
| Online Access: | http://hdl.handle.net/1721.1/74248 https://orcid.org/0000-0002-2921-8201 |
| _version_ | 1826214177217183744 |
|---|---|
| author | Branavan, Satchuthanan R. Silver, David Barzilay, Regina |
| author2 | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory |
| author_facet | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Branavan, Satchuthanan R. Silver, David Barzilay, Regina |
| author_sort | Branavan, Satchuthanan R. |
| collection | MIT |
| description | This paper presents a new Monte-Carlo search algorithm for very large sequential decision-making problems. Our approach builds on the recent success of Monte-Carlo tree search algorithms, which estimate the value of states and actions from the mean outcome of random simulations. Instead of using a search tree, we apply non-linear regression, online, to estimate a state-action value function from the outcomes of random simulations. This value function generalizes between related states and actions, and can therefore provide more accurate evaluations after fewer simulations. We apply our Monte-Carlo search algorithm to the game of Civilization II, a challenging multi-agent strategy game with an enormous state space and around $10^{21}$ joint actions. We approximate the value function by a neural network, augmented by linguistic knowledge that is extracted automatically from the official game manual. We show that this non-linear value function is significantly more efficient than a linear value function. Our non-linear Monte-Carlo search wins 80\% of games against the handcrafted, built-in AI for Civilization II. |
| first_indexed | 2024-09-23T16:01:19Z |
| format | Article |
| id | mit-1721.1/74248 |
| institution | Massachusetts Institute of Technology |
| language | en_US |
| last_indexed | 2024-09-23T16:01:19Z |
| publishDate | 2012 |
| publisher | AAAI Press/International Joint Conferences on Artificial Intelligence |
| record_format | dspace |
| spelling | mit-1721.1/742482022-09-29T17:42:09Z Non-Linear Monte-Carlo Search in Civilization II Branavan, Satchuthanan R. Silver, David Barzilay, Regina Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Barzilay, Regina Branavan, Satchuthanan R. Barzilay, Regina This paper presents a new Monte-Carlo search algorithm for very large sequential decision-making problems. Our approach builds on the recent success of Monte-Carlo tree search algorithms, which estimate the value of states and actions from the mean outcome of random simulations. Instead of using a search tree, we apply non-linear regression, online, to estimate a state-action value function from the outcomes of random simulations. This value function generalizes between related states and actions, and can therefore provide more accurate evaluations after fewer simulations. We apply our Monte-Carlo search algorithm to the game of Civilization II, a challenging multi-agent strategy game with an enormous state space and around $10^{21}$ joint actions. We approximate the value function by a neural network, augmented by linguistic knowledge that is extracted automatically from the official game manual. We show that this non-linear value function is significantly more efficient than a linear value function. Our non-linear Monte-Carlo search wins 80\% of games against the handcrafted, built-in AI for Civilization II. National Science Foundation (U.S.) (CAREER grant IIS-0448168) National Science Foundation (U.S.) (grant IIS-0835652) United States. Defense Advanced Research Projects Agency (DARPA Machine Reading Program (FA8750-09-C-0172)) Microsoft Research (New Faculty Fellowship) 2012-10-24T20:34:34Z 2012-10-24T20:34:34Z 2011-07 Article http://purl.org/eprint/type/ConferencePaper 978-1-57735-512-0 978-1-57735-516-8 http://hdl.handle.net/1721.1/74248 Branavan, S. R. K. David Silver, and Regina Barzilay. "Non-Linear Monte-Carlo Search in Civilization II." in Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence, Barcelona, Catalonia, Spain, 16–22 July 2011. p.2404. https://orcid.org/0000-0002-2921-8201 en_US http://ijcai-11.iiia.csic.es/program/paper/1252 Proceedings of the Twenty-second International Joint Conference on Artificial Intelligence Creative Commons Attribution-Noncommercial-Share Alike 3.0 http://creativecommons.org/licenses/by-nc-sa/3.0/ application/pdf AAAI Press/International Joint Conferences on Artificial Intelligence MIT web domain |
| spellingShingle | Branavan, Satchuthanan R. Silver, David Barzilay, Regina Non-Linear Monte-Carlo Search in Civilization II |
| title | Non-Linear Monte-Carlo Search in Civilization II |
| title_full | Non-Linear Monte-Carlo Search in Civilization II |
| title_fullStr | Non-Linear Monte-Carlo Search in Civilization II |
| title_full_unstemmed | Non-Linear Monte-Carlo Search in Civilization II |
| title_short | Non-Linear Monte-Carlo Search in Civilization II |
| title_sort | non linear monte carlo search in civilization ii |
| url | http://hdl.handle.net/1721.1/74248 https://orcid.org/0000-0002-2921-8201 |
| work_keys_str_mv | AT branavansatchuthananr nonlinearmontecarlosearchincivilizationii AT silverdavid nonlinearmontecarlosearchincivilizationii AT barzilayregina nonlinearmontecarlosearchincivilizationii |