A Framework for LLM-based Lifelong Learning in Robot Manipulation
While robotic agents have become increasingly adept at low-level manipulation skills, increasingly they are being guided by large language model planners that decompose complex tasks into subgoals. Recent works indicate that these language models may also be effective skill learners. We develop HaLP...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Published: |
Massachusetts Institute of Technology
2024
|
| Online Access: | https://hdl.handle.net/1721.1/153895 |
| _version_ | 1826197145005326336 |
|---|---|
| author | Mao, Jerry W. |
| author2 | Agrawal, Pulkit |
| author_facet | Agrawal, Pulkit Mao, Jerry W. |
| author_sort | Mao, Jerry W. |
| collection | MIT |
| description | While robotic agents have become increasingly adept at low-level manipulation skills, increasingly they are being guided by large language model planners that decompose complex tasks into subgoals. Recent works indicate that these language models may also be effective skill learners. We develop HaLP 2.0, a modular and extensible framework for lifelong learning in human-assisted language planning, using GPT-4 to propose a curriculum of skills that is learned, used, and intelligently reused. Our system is designed for large-scale experiments, is equipped with a user-friendly interface, and is extensible to new skill learning frameworks. We demonstrate extensibility by comparing alternative implementations of our abstractions and improving overall performance by incorporating novel frameworks. Moreover, we conduct a focused study of GPT-4, using crowd-sourced scene and task datasets, finding that language models are capable agents of skill reuse and adaptation. We observe that while performance is dependent on language context, supplying optimized prompts can yield exceptional skill reuse behaviors. We envision that as manipulation primitives and large language models become more powerful, our system will be ready to synthesize their capabilities to create an autonomous system for lifelong learning, that can one day be deployed in the real world. |
| first_indexed | 2024-09-23T10:43:29Z |
| format | Thesis |
| id | mit-1721.1/153895 |
| institution | Massachusetts Institute of Technology |
| last_indexed | 2024-09-23T10:43:29Z |
| publishDate | 2024 |
| publisher | Massachusetts Institute of Technology |
| record_format | dspace |
| spelling | mit-1721.1/1538952024-03-22T03:51:00Z A Framework for LLM-based Lifelong Learning in Robot Manipulation Mao, Jerry W. Agrawal, Pulkit Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science While robotic agents have become increasingly adept at low-level manipulation skills, increasingly they are being guided by large language model planners that decompose complex tasks into subgoals. Recent works indicate that these language models may also be effective skill learners. We develop HaLP 2.0, a modular and extensible framework for lifelong learning in human-assisted language planning, using GPT-4 to propose a curriculum of skills that is learned, used, and intelligently reused. Our system is designed for large-scale experiments, is equipped with a user-friendly interface, and is extensible to new skill learning frameworks. We demonstrate extensibility by comparing alternative implementations of our abstractions and improving overall performance by incorporating novel frameworks. Moreover, we conduct a focused study of GPT-4, using crowd-sourced scene and task datasets, finding that language models are capable agents of skill reuse and adaptation. We observe that while performance is dependent on language context, supplying optimized prompts can yield exceptional skill reuse behaviors. We envision that as manipulation primitives and large language models become more powerful, our system will be ready to synthesize their capabilities to create an autonomous system for lifelong learning, that can one day be deployed in the real world. M.Eng. 2024-03-21T19:14:18Z 2024-03-21T19:14:18Z 2024-02 2024-03-04T16:37:58.746Z Thesis https://hdl.handle.net/1721.1/153895 In Copyright - Educational Use Permitted Copyright retained by author(s) https://rightsstatements.org/page/InC-EDU/1.0/ application/pdf Massachusetts Institute of Technology |
| spellingShingle | Mao, Jerry W. A Framework for LLM-based Lifelong Learning in Robot Manipulation |
| title | A Framework for LLM-based Lifelong Learning in Robot Manipulation |
| title_full | A Framework for LLM-based Lifelong Learning in Robot Manipulation |
| title_fullStr | A Framework for LLM-based Lifelong Learning in Robot Manipulation |
| title_full_unstemmed | A Framework for LLM-based Lifelong Learning in Robot Manipulation |
| title_short | A Framework for LLM-based Lifelong Learning in Robot Manipulation |
| title_sort | framework for llm based lifelong learning in robot manipulation |
| url | https://hdl.handle.net/1721.1/153895 |
| work_keys_str_mv | AT maojerryw aframeworkforllmbasedlifelonglearninginrobotmanipulation AT maojerryw frameworkforllmbasedlifelonglearninginrobotmanipulation |