Compositional RL Agents That Follow Language Commands in Temporal Logic
We demonstrate how a reinforcement learning agent can use compositional recurrent neural net- works to learn to carry out commands specified in linear temporal logic (LTL). Our approach takes as input an LTL formula, structures a deep network according to the parse of the formula, and determines sat...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Published: |
Center for Brains, Minds and Machines (CBMM), Frontiers in Robotics and AI
2022
|
| Online Access: | https://hdl.handle.net/1721.1/141357 |
| _version_ | 1826195559651737600 |
|---|---|
| author | Kuo, Yen-Ling Barbu, Andrei Katz, Boris |
| author_facet | Kuo, Yen-Ling Barbu, Andrei Katz, Boris |
| author_sort | Kuo, Yen-Ling |
| collection | MIT |
| description | We demonstrate how a reinforcement learning agent can use compositional recurrent neural net- works to learn to carry out commands specified in linear temporal logic (LTL). Our approach takes as input an LTL formula, structures a deep network according to the parse of the formula, and determines satisfying actions. This compositional structure of the network enables zero-shot generalization to sig- nificantly more complex unseen formulas. We demonstrate this ability in multiple problem domains with both discrete and continuous state-action spaces. In a symbolic domain, the agent finds a sequence of letters that satisfy a specification. In a Minecraft-like environment, the agent finds a sequence of actions that conform to a formula. In the Fetch environment, the robot finds a sequence of arm config- urations that move blocks on a table to fulfill the commands. While most prior work can learn to execute one formula reliably, we develop a novel form of multi-task learning for RL agents that allows them to learn from a diverse set of tasks and generalize to a new set of diverse tasks without any additional training. The compositional structures presented here are not specific to LTL, thus opening the path to RL agents that perform zero-shot generalization in other compositional domains. |
| first_indexed | 2024-09-23T10:14:38Z |
| format | Article |
| id | mit-1721.1/141357 |
| institution | Massachusetts Institute of Technology |
| last_indexed | 2024-09-23T10:14:38Z |
| publishDate | 2022 |
| publisher | Center for Brains, Minds and Machines (CBMM), Frontiers in Robotics and AI |
| record_format | dspace |
| spelling | mit-1721.1/1413572022-03-25T03:30:26Z Compositional RL Agents That Follow Language Commands in Temporal Logic Kuo, Yen-Ling Barbu, Andrei Katz, Boris We demonstrate how a reinforcement learning agent can use compositional recurrent neural net- works to learn to carry out commands specified in linear temporal logic (LTL). Our approach takes as input an LTL formula, structures a deep network according to the parse of the formula, and determines satisfying actions. This compositional structure of the network enables zero-shot generalization to sig- nificantly more complex unseen formulas. We demonstrate this ability in multiple problem domains with both discrete and continuous state-action spaces. In a symbolic domain, the agent finds a sequence of letters that satisfy a specification. In a Minecraft-like environment, the agent finds a sequence of actions that conform to a formula. In the Fetch environment, the robot finds a sequence of arm config- urations that move blocks on a table to fulfill the commands. While most prior work can learn to execute one formula reliably, we develop a novel form of multi-task learning for RL agents that allows them to learn from a diverse set of tasks and generalize to a new set of diverse tasks without any additional training. The compositional structures presented here are not specific to LTL, thus opening the path to RL agents that perform zero-shot generalization in other compositional domains. This material is based upon work supported by the Center for Brains, Minds and Machines (CBMM), funded by NSF STC award CCF-1231216. 2022-03-24T17:07:22Z 2022-03-24T17:07:22Z 2021-07-19 Article Technical Report Working Paper https://hdl.handle.net/1721.1/141357 CBMM Memo;127 application/pdf Center for Brains, Minds and Machines (CBMM), Frontiers in Robotics and AI |
| spellingShingle | Kuo, Yen-Ling Barbu, Andrei Katz, Boris Compositional RL Agents That Follow Language Commands in Temporal Logic |
| title | Compositional RL Agents That Follow Language Commands in Temporal Logic |
| title_full | Compositional RL Agents That Follow Language Commands in Temporal Logic |
| title_fullStr | Compositional RL Agents That Follow Language Commands in Temporal Logic |
| title_full_unstemmed | Compositional RL Agents That Follow Language Commands in Temporal Logic |
| title_short | Compositional RL Agents That Follow Language Commands in Temporal Logic |
| title_sort | compositional rl agents that follow language commands in temporal logic |
| url | https://hdl.handle.net/1721.1/141357 |
| work_keys_str_mv | AT kuoyenling compositionalrlagentsthatfollowlanguagecommandsintemporallogic AT barbuandrei compositionalrlagentsthatfollowlanguagecommandsintemporallogic AT katzboris compositionalrlagentsthatfollowlanguagecommandsintemporallogic |