Learning models for following natural language directions in unknown environments
Natural language offers an intuitive and flexible means for humans to communicate with the robots that we will increasingly work alongside in our homes and workplaces. Recent advancements have given rise to robots that are able to interpret natural language manipulation and navigation commands, but...
| Main Authors: | , , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | en_US |
| Published: |
Institute of Electrical and Electronics Engineers (IEEE)
2017
|
| Online Access: | http://hdl.handle.net/1721.1/109133 https://orcid.org/0000-0002-8293-0492 |
| _version_ | 1826202784527024128 |
|---|---|
| author | Duvallet, Felix Howard, Thomas M. Stentz, Anthony Walter, Matthew R. Hemachandra, Sachithra Madhawa Roy, Nicholas |
| author2 | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory |
| author_facet | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Duvallet, Felix Howard, Thomas M. Stentz, Anthony Walter, Matthew R. Hemachandra, Sachithra Madhawa Roy, Nicholas |
| author_sort | Duvallet, Felix |
| collection | MIT |
| description | Natural language offers an intuitive and flexible means for humans to communicate with the robots that we will increasingly work alongside in our homes and workplaces. Recent advancements have given rise to robots that are able to interpret natural language manipulation and navigation
commands, but these methods require a prior map of the robot’s environment. In this paper, we propose a novel learning framework that enables robots to successfully follow natural language route directions without any previous knowledge of the environment. The algorithm utilizes spatial and semantic information that the human conveys through the command to learn a distribution over the metric and semantic properties
of spatially extended environments. Our method uses this distribution in place of the latent world model and interprets the natural language instruction as a distribution over the intended behavior. A novel belief space planner reasons directly over the map and behavior distributions to solve for a policy using imitation learning. We evaluate our framework on a voice-commandable wheelchair. The results demonstrate that by learning and performing inference over a latent environment model, the algorithm is able to successfully follow natural language route directions within novel, extended environments |
| first_indexed | 2024-09-23T12:18:53Z |
| format | Article |
| id | mit-1721.1/109133 |
| institution | Massachusetts Institute of Technology |
| language | en_US |
| last_indexed | 2024-09-23T12:18:53Z |
| publishDate | 2017 |
| publisher | Institute of Electrical and Electronics Engineers (IEEE) |
| record_format | dspace |
| spelling | mit-1721.1/1091332022-10-01T08:57:28Z Learning models for following natural language directions in unknown environments Duvallet, Felix Howard, Thomas M. Stentz, Anthony Walter, Matthew R. Hemachandra, Sachithra Madhawa Roy, Nicholas Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology. Department of Aeronautics and Astronautics Hemachandra, Sachithra Madhawa Roy, Nicholas Natural language offers an intuitive and flexible means for humans to communicate with the robots that we will increasingly work alongside in our homes and workplaces. Recent advancements have given rise to robots that are able to interpret natural language manipulation and navigation commands, but these methods require a prior map of the robot’s environment. In this paper, we propose a novel learning framework that enables robots to successfully follow natural language route directions without any previous knowledge of the environment. The algorithm utilizes spatial and semantic information that the human conveys through the command to learn a distribution over the metric and semantic properties of spatially extended environments. Our method uses this distribution in place of the latent world model and interprets the natural language instruction as a distribution over the intended behavior. A novel belief space planner reasons directly over the map and behavior distributions to solve for a policy using imitation learning. We evaluate our framework on a voice-commandable wheelchair. The results demonstrate that by learning and performing inference over a latent environment model, the algorithm is able to successfully follow natural language route directions within novel, extended environments United States. Army Research Laboratory. Collaborative Technology Alliance Program (W911NF-10-2-0016) United States. Office of Naval Research. Multidisciplinary University Research Initiative (N00014-09-1-1052) 2017-05-16T20:42:57Z 2017-05-16T20:42:57Z 2015-07 2015-05 Article http://purl.org/eprint/type/ConferencePaper 978-1-4799-6923-4 http://hdl.handle.net/1721.1/109133 Hemachandra, Sachithra et al. “Learning Models for Following Natural Language Directions in Unknown Environments.” 2015 IEEE International Conference on Robotics and Automation (ICRA), 26-30 May, 2015, Seattle WA, USA, IEEE, 2015. 5608–5615. https://orcid.org/0000-0002-8293-0492 en_US http://dx.doi.org/10.1109/ICRA.2015.7139984 Proceedings of the 2015 IEEE International Conference on Robotics and Automation (ICRA) Creative Commons Attribution-Noncommercial-Share Alike http://creativecommons.org/licenses/by-nc-sa/4.0/ application/pdf Institute of Electrical and Electronics Engineers (IEEE) arXiv |
| spellingShingle | Duvallet, Felix Howard, Thomas M. Stentz, Anthony Walter, Matthew R. Hemachandra, Sachithra Madhawa Roy, Nicholas Learning models for following natural language directions in unknown environments |
| title | Learning models for following natural language directions in unknown environments |
| title_full | Learning models for following natural language directions in unknown environments |
| title_fullStr | Learning models for following natural language directions in unknown environments |
| title_full_unstemmed | Learning models for following natural language directions in unknown environments |
| title_short | Learning models for following natural language directions in unknown environments |
| title_sort | learning models for following natural language directions in unknown environments |
| url | http://hdl.handle.net/1721.1/109133 https://orcid.org/0000-0002-8293-0492 |
| work_keys_str_mv | AT duvalletfelix learningmodelsforfollowingnaturallanguagedirectionsinunknownenvironments AT howardthomasm learningmodelsforfollowingnaturallanguagedirectionsinunknownenvironments AT stentzanthony learningmodelsforfollowingnaturallanguagedirectionsinunknownenvironments AT waltermatthewr learningmodelsforfollowingnaturallanguagedirectionsinunknownenvironments AT hemachandrasachithramadhawa learningmodelsforfollowingnaturallanguagedirectionsinunknownenvironments AT roynicholas learningmodelsforfollowingnaturallanguagedirectionsinunknownenvironments |