VirtualHome : learning to infer programs from synthetic videos of activities in the home
Thesis: M. Eng., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2017.
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Language: | eng |
| Published: |
Massachusetts Institute of Technology
2018
|
| Subjects: | |
| Online Access: | http://hdl.handle.net/1721.1/119527 |
| _version_ | 1826204873098526720 |
|---|---|
| author | Jain, Shantanu |
| author2 | Antonio Torralba. |
| author_facet | Antonio Torralba. Jain, Shantanu |
| author_sort | Jain, Shantanu |
| collection | MIT |
| description | Thesis: M. Eng., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2017. |
| first_indexed | 2024-09-23T13:02:26Z |
| format | Thesis |
| id | mit-1721.1/119527 |
| institution | Massachusetts Institute of Technology |
| language | eng |
| last_indexed | 2024-09-23T13:02:26Z |
| publishDate | 2018 |
| publisher | Massachusetts Institute of Technology |
| record_format | dspace |
| spelling | mit-1721.1/1195272019-04-11T10:58:04Z VirtualHome : learning to infer programs from synthetic videos of activities in the home Virtual Home Learning to infer programs from synthetic videos of activities in the home Jain, Shantanu Antonio Torralba. Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science. Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science. Electrical Engineering and Computer Science. Thesis: M. Eng., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2017. This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections. Cataloged from student-submitted PDF version of thesis. Includes bibliographical references (pages 53-57). This project models complex activities that occur in a typical household. Programs - sequences of atomic actions and interactions - are used as a high-level, unambiguous representation of complex activities executable by an agent. However, no dataset of household activity programs currently exists. This project builds such a dataset by crowdsourcing programs of typical household activities, via a game-like interface used for teaching kids how to code. The most common atomic actions are implemented in the Unity3D game engine, and videos are recorded of an agent executing the collected programs in a simulated household environment. The VirtualHome simulator allows the creation of a large activity video dataset with rich groundtruth, enabling training and testing of video understanding models. Using the collected dataset, a sequence-to-sequence neural encoder-decoder model with visual attention is built, and learns to infer programs directly from videos. It achieves 46.2% accuracy for action inference. by Shantanu Jain. M. Eng. 2018-12-11T20:38:50Z 2018-12-11T20:38:50Z 2017 2017 Thesis http://hdl.handle.net/1721.1/119527 1066694700 eng MIT theses are protected by copyright. They may be viewed, downloaded, or printed from this source but further reproduction or distribution in any format is prohibited without written permission. http://dspace.mit.edu/handle/1721.1/7582 57 pages application/pdf Massachusetts Institute of Technology |
| spellingShingle | Electrical Engineering and Computer Science. Jain, Shantanu VirtualHome : learning to infer programs from synthetic videos of activities in the home |
| title | VirtualHome : learning to infer programs from synthetic videos of activities in the home |
| title_full | VirtualHome : learning to infer programs from synthetic videos of activities in the home |
| title_fullStr | VirtualHome : learning to infer programs from synthetic videos of activities in the home |
| title_full_unstemmed | VirtualHome : learning to infer programs from synthetic videos of activities in the home |
| title_short | VirtualHome : learning to infer programs from synthetic videos of activities in the home |
| title_sort | virtualhome learning to infer programs from synthetic videos of activities in the home |
| topic | Electrical Engineering and Computer Science. |
| url | http://hdl.handle.net/1721.1/119527 |
| work_keys_str_mv | AT jainshantanu virtualhomelearningtoinferprogramsfromsyntheticvideosofactivitiesinthehome AT jainshantanu virtualhome AT jainshantanu learningtoinferprogramsfromsyntheticvideosofactivitiesinthehome |