Mitigating Social Dilemmas in Multi-Agent Reinforcement Learning with Formal Contracting
As society deploys more and more sophisticated artificial intelligence (AI) agents, it will be increasingly necessary for such agents, while pursuing their own objectives, to coexist in common environments in the physical or digital worlds. This may pose a challenge if the agents’ objectives conflic...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Published: |
Massachusetts Institute of Technology
2024
|
| Online Access: | https://hdl.handle.net/1721.1/153795 |
| _version_ | 1826211501167345664 |
|---|---|
| author | Christoffersen, Phillip Johannes Kerr |
| author2 | Hadfield-Menell, Dylan |
| author_facet | Hadfield-Menell, Dylan Christoffersen, Phillip Johannes Kerr |
| author_sort | Christoffersen, Phillip Johannes Kerr |
| collection | MIT |
| description | As society deploys more and more sophisticated artificial intelligence (AI) agents, it will be increasingly necessary for such agents, while pursuing their own objectives, to coexist in common environments in the physical or digital worlds. This may pose a challenge if the agents’ objectives conflict with each other– in the worst case, this can prevent any given agent from being able to fulfill their own objectives (e.g. self driving cars in a traffic jam). Situations such as these are termed social dilemmas.
In this thesis, it is demonstrated that providing RL agents with the software infrastructure to precommit to zero-sum incentive modifications
1. Induces maximal social welfare in theory; and
2. When implemented with deep multi-agent reinforcement learning (MARL), also avoids social dilemmas in practice.
Specifically, a novel algorithmic framework is proposed, termed formal contracting, which is formalized, studied game-theoretically, and investigated empirically. In formal contracting, before engaging in a given shared environment, agents are given the opportunity negotiate a binding modification to all agents’ objective functions, in order to provide incentives for the optimal use of shared resources. Within this framework, at all subgame-perfect equilibria (SPE), agents will in fact maximize social welfare, that is, the sum of all agent objectives in the original environment. Moreover, studies in simple domains, such as the classic prisoner’s dilemma, and more complex ones such as dynamic simulations of pollution management, show that this algorithmic framework can be implemented in MARL, and does indeed lead to outcomes with superior welfare in social dilemmas. This thesis concludes with discussions of related work, limitations of the approach, and future work, particularly involving scaling this methodology to larger problem instances containing more agents than studied. |
| first_indexed | 2024-09-23T15:06:55Z |
| format | Thesis |
| id | mit-1721.1/153795 |
| institution | Massachusetts Institute of Technology |
| last_indexed | 2024-09-23T15:06:55Z |
| publishDate | 2024 |
| publisher | Massachusetts Institute of Technology |
| record_format | dspace |
| spelling | mit-1721.1/1537952024-03-16T03:12:53Z Mitigating Social Dilemmas in Multi-Agent Reinforcement Learning with Formal Contracting Christoffersen, Phillip Johannes Kerr Hadfield-Menell, Dylan Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science As society deploys more and more sophisticated artificial intelligence (AI) agents, it will be increasingly necessary for such agents, while pursuing their own objectives, to coexist in common environments in the physical or digital worlds. This may pose a challenge if the agents’ objectives conflict with each other– in the worst case, this can prevent any given agent from being able to fulfill their own objectives (e.g. self driving cars in a traffic jam). Situations such as these are termed social dilemmas. In this thesis, it is demonstrated that providing RL agents with the software infrastructure to precommit to zero-sum incentive modifications 1. Induces maximal social welfare in theory; and 2. When implemented with deep multi-agent reinforcement learning (MARL), also avoids social dilemmas in practice. Specifically, a novel algorithmic framework is proposed, termed formal contracting, which is formalized, studied game-theoretically, and investigated empirically. In formal contracting, before engaging in a given shared environment, agents are given the opportunity negotiate a binding modification to all agents’ objective functions, in order to provide incentives for the optimal use of shared resources. Within this framework, at all subgame-perfect equilibria (SPE), agents will in fact maximize social welfare, that is, the sum of all agent objectives in the original environment. Moreover, studies in simple domains, such as the classic prisoner’s dilemma, and more complex ones such as dynamic simulations of pollution management, show that this algorithmic framework can be implemented in MARL, and does indeed lead to outcomes with superior welfare in social dilemmas. This thesis concludes with discussions of related work, limitations of the approach, and future work, particularly involving scaling this methodology to larger problem instances containing more agents than studied. S.M. 2024-03-15T19:24:34Z 2024-03-15T19:24:34Z 2024-02 2024-02-21T17:10:06.500Z Thesis https://hdl.handle.net/1721.1/153795 In Copyright - Educational Use Permitted Copyright retained by author(s) https://rightsstatements.org/page/InC-EDU/1.0/ application/pdf Massachusetts Institute of Technology |
| spellingShingle | Christoffersen, Phillip Johannes Kerr Mitigating Social Dilemmas in Multi-Agent Reinforcement Learning with Formal Contracting |
| title | Mitigating Social Dilemmas in Multi-Agent Reinforcement Learning with Formal Contracting |
| title_full | Mitigating Social Dilemmas in Multi-Agent Reinforcement Learning with Formal Contracting |
| title_fullStr | Mitigating Social Dilemmas in Multi-Agent Reinforcement Learning with Formal Contracting |
| title_full_unstemmed | Mitigating Social Dilemmas in Multi-Agent Reinforcement Learning with Formal Contracting |
| title_short | Mitigating Social Dilemmas in Multi-Agent Reinforcement Learning with Formal Contracting |
| title_sort | mitigating social dilemmas in multi agent reinforcement learning with formal contracting |
| url | https://hdl.handle.net/1721.1/153795 |
| work_keys_str_mv | AT christoffersenphillipjohanneskerr mitigatingsocialdilemmasinmultiagentreinforcementlearningwithformalcontracting |