View Synthesis for Visuomotor Policy Learning
Visuomotor policy learning is the problem of teaching machines how to use visual information to determine how to interact with their environment. Recent approaches have harnessed deep learning models to demonstrate impressive results in multi-modal and multi-task generalization. However, these model...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Published: |
Massachusetts Institute of Technology
2023
|
| Online Access: | https://hdl.handle.net/1721.1/152632 |
| _version_ | 1811083308439699456 |
|---|---|
| author | Lin, Yen-Chen |
| author2 | Isola, Phillip |
| author_facet | Isola, Phillip Lin, Yen-Chen |
| author_sort | Lin, Yen-Chen |
| collection | MIT |
| description | Visuomotor policy learning is the problem of teaching machines how to use visual information to determine how to interact with their environment. Recent approaches have harnessed deep learning models to demonstrate impressive results in multi-modal and multi-task generalization. However, these models often lack a comprehensive understanding of the 3D world as they are primarily trained on large-scale RGB image datasets. In this thesis, we present a new framework that equips visuomotor policies with a view synthesizer. This generative model has the ability to envision novel viewpoints and perspectives of the 3D environment. Unlike training a visuomotor policy solely on real-world data, a view synthesizer can produce coherent views of a 3D scene in a controllable manner. This capability assists the policy in utilizing symmetries present in robotic tasks through learned and designed utilization. Learned utilization expands the training dataset of the visuomotor policy to implicitly encourage the emergence of symmetric properties through learning. On the other hand, designed utilization integrates symmetric properties into both the policy’s input representations and its model architectures to explicitly establish symmetric properties. We demonstrate that the proposed systems exhibit improved sample efficiency and generalization compared to visuomotor policies that lack the capability for view synthesis. |
| first_indexed | 2024-09-23T12:31:00Z |
| format | Thesis |
| id | mit-1721.1/152632 |
| institution | Massachusetts Institute of Technology |
| last_indexed | 2024-09-23T12:31:00Z |
| publishDate | 2023 |
| publisher | Massachusetts Institute of Technology |
| record_format | dspace |
| spelling | mit-1721.1/1526322023-11-03T03:35:25Z View Synthesis for Visuomotor Policy Learning Lin, Yen-Chen Isola, Phillip Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Visuomotor policy learning is the problem of teaching machines how to use visual information to determine how to interact with their environment. Recent approaches have harnessed deep learning models to demonstrate impressive results in multi-modal and multi-task generalization. However, these models often lack a comprehensive understanding of the 3D world as they are primarily trained on large-scale RGB image datasets. In this thesis, we present a new framework that equips visuomotor policies with a view synthesizer. This generative model has the ability to envision novel viewpoints and perspectives of the 3D environment. Unlike training a visuomotor policy solely on real-world data, a view synthesizer can produce coherent views of a 3D scene in a controllable manner. This capability assists the policy in utilizing symmetries present in robotic tasks through learned and designed utilization. Learned utilization expands the training dataset of the visuomotor policy to implicitly encourage the emergence of symmetric properties through learning. On the other hand, designed utilization integrates symmetric properties into both the policy’s input representations and its model architectures to explicitly establish symmetric properties. We demonstrate that the proposed systems exhibit improved sample efficiency and generalization compared to visuomotor policies that lack the capability for view synthesis. Ph.D. 2023-11-02T20:04:23Z 2023-11-02T20:04:23Z 2023-09 2023-09-21T14:25:48.244Z Thesis https://hdl.handle.net/1721.1/152632 In Copyright - Educational Use Permitted Copyright retained by author(s) https://rightsstatements.org/page/InC-EDU/1.0/ application/pdf Massachusetts Institute of Technology |
| spellingShingle | Lin, Yen-Chen View Synthesis for Visuomotor Policy Learning |
| title | View Synthesis for Visuomotor Policy Learning |
| title_full | View Synthesis for Visuomotor Policy Learning |
| title_fullStr | View Synthesis for Visuomotor Policy Learning |
| title_full_unstemmed | View Synthesis for Visuomotor Policy Learning |
| title_short | View Synthesis for Visuomotor Policy Learning |
| title_sort | view synthesis for visuomotor policy learning |
| url | https://hdl.handle.net/1721.1/152632 |
| work_keys_str_mv | AT linyenchen viewsynthesisforvisuomotorpolicylearning |