An Intelligence Architecture for Grounded Language Communication with Field Robots
<jats:p>For humans and robots to collaborate effectively as teammates in unstructured environments, robots must be able to construct semantically rich models of the environment, communicate efficiently with teammates, and perform sequences of tasks robustly with minimal human intervention, as...
| Main Authors: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Field Robotics Publication Society
2022
|
| Online Access: | https://hdl.handle.net/1721.1/145529 |
| _version_ | 1826195518219354112 |
|---|---|
| author | Howard, Thomas Stump, Ethan Fink, Jonathan Arkin, Jacob Paul, Rohan Park, Daehyung Roy, Subhro Barber, Daniel Bendell, Rhyse Schmeckpeper, Karl Tian, Junjiao Oh, Jean Wigness, Maggie Quang, Long Rothrock, Brandon Nash, Jeremy Walter, Matthew Jentsch, Florian Roy, Nicholas |
| author2 | Massachusetts Institute of Technology. Department of Aeronautics and Astronautics |
| author_facet | Massachusetts Institute of Technology. Department of Aeronautics and Astronautics Howard, Thomas Stump, Ethan Fink, Jonathan Arkin, Jacob Paul, Rohan Park, Daehyung Roy, Subhro Barber, Daniel Bendell, Rhyse Schmeckpeper, Karl Tian, Junjiao Oh, Jean Wigness, Maggie Quang, Long Rothrock, Brandon Nash, Jeremy Walter, Matthew Jentsch, Florian Roy, Nicholas |
| author_sort | Howard, Thomas |
| collection | MIT |
| description | <jats:p>For humans and robots to collaborate effectively as teammates in unstructured environments, robots must be able to construct semantically rich models of the environment, communicate efficiently with teammates, and perform sequences of tasks robustly with minimal human intervention, as direct human guidance may be infrequent and/or intermittent. Contemporary architectures for human-robot interaction often rely on engineered human-interface devices or structured languages that require extensive prior training and inherently limit the kinds of information that humans and robots can communicate. Natural language, particularly when situated with a visual representation of the robot’s environment, allows humans and robots to exchange information about abstract goals, specific actions, and/or properties of the environment quickly and effectively. In addition, it serves as a mechanism to resolve inconsistencies in the mental models of the environment across the human-robot team. This article details a novel intelligence architecture that exploits a centralized representation of the environment to perform complex tasks in unstructured environments. The centralized environment model is informed by a visual perception pipeline, declarative knowledge, deliberate interactive estimation, and a multimodal interface. The language pipeline also exploits proactive symbol grounding to resolve uncertainty in ambiguous statements through inverse semantics. A series of experiments on three different, unmanned ground vehicles demonstrates the utility of this architecture through its robust ability to perform language-guided spatial navigation, mobile manipulation, and bidirectional communication with human operators. Experimental results give examples of component-level behaviors and overall system performance that guide a discussion on observed performance and opportunities for future innovation.</jats:p> |
| first_indexed | 2024-09-23T10:14:00Z |
| format | Article |
| id | mit-1721.1/145529 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T10:14:00Z |
| publishDate | 2022 |
| publisher | Field Robotics Publication Society |
| record_format | dspace |
| spelling | mit-1721.1/1455292022-09-30T19:47:38Z An Intelligence Architecture for Grounded Language Communication with Field Robots Howard, Thomas Stump, Ethan Fink, Jonathan Arkin, Jacob Paul, Rohan Park, Daehyung Roy, Subhro Barber, Daniel Bendell, Rhyse Schmeckpeper, Karl Tian, Junjiao Oh, Jean Wigness, Maggie Quang, Long Rothrock, Brandon Nash, Jeremy Walter, Matthew Jentsch, Florian Roy, Nicholas Massachusetts Institute of Technology. Department of Aeronautics and Astronautics <jats:p>For humans and robots to collaborate effectively as teammates in unstructured environments, robots must be able to construct semantically rich models of the environment, communicate efficiently with teammates, and perform sequences of tasks robustly with minimal human intervention, as direct human guidance may be infrequent and/or intermittent. Contemporary architectures for human-robot interaction often rely on engineered human-interface devices or structured languages that require extensive prior training and inherently limit the kinds of information that humans and robots can communicate. Natural language, particularly when situated with a visual representation of the robot’s environment, allows humans and robots to exchange information about abstract goals, specific actions, and/or properties of the environment quickly and effectively. In addition, it serves as a mechanism to resolve inconsistencies in the mental models of the environment across the human-robot team. This article details a novel intelligence architecture that exploits a centralized representation of the environment to perform complex tasks in unstructured environments. The centralized environment model is informed by a visual perception pipeline, declarative knowledge, deliberate interactive estimation, and a multimodal interface. The language pipeline also exploits proactive symbol grounding to resolve uncertainty in ambiguous statements through inverse semantics. A series of experiments on three different, unmanned ground vehicles demonstrates the utility of this architecture through its robust ability to perform language-guided spatial navigation, mobile manipulation, and bidirectional communication with human operators. Experimental results give examples of component-level behaviors and overall system performance that guide a discussion on observed performance and opportunities for future innovation.</jats:p> 2022-09-20T16:56:23Z 2022-09-20T16:56:23Z 2022 2022-09-20T16:49:56Z Article http://purl.org/eprint/type/JournalArticle https://hdl.handle.net/1721.1/145529 Howard, Thomas, Stump, Ethan, Fink, Jonathan, Arkin, Jacob, Paul, Rohan et al. 2022. "An Intelligence Architecture for Grounded Language Communication with Field Robots." Field Robotics, 2 (1). en 10.55417/FR.2022017 Field Robotics Creative Commons Attribution 4.0 International license https://creativecommons.org/licenses/by/4.0/ application/pdf Field Robotics Publication Society Field Robotics |
| spellingShingle | Howard, Thomas Stump, Ethan Fink, Jonathan Arkin, Jacob Paul, Rohan Park, Daehyung Roy, Subhro Barber, Daniel Bendell, Rhyse Schmeckpeper, Karl Tian, Junjiao Oh, Jean Wigness, Maggie Quang, Long Rothrock, Brandon Nash, Jeremy Walter, Matthew Jentsch, Florian Roy, Nicholas An Intelligence Architecture for Grounded Language Communication with Field Robots |
| title | An Intelligence Architecture for Grounded Language Communication with Field Robots |
| title_full | An Intelligence Architecture for Grounded Language Communication with Field Robots |
| title_fullStr | An Intelligence Architecture for Grounded Language Communication with Field Robots |
| title_full_unstemmed | An Intelligence Architecture for Grounded Language Communication with Field Robots |
| title_short | An Intelligence Architecture for Grounded Language Communication with Field Robots |
| title_sort | intelligence architecture for grounded language communication with field robots |
| url | https://hdl.handle.net/1721.1/145529 |
| work_keys_str_mv | AT howardthomas anintelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT stumpethan anintelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT finkjonathan anintelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT arkinjacob anintelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT paulrohan anintelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT parkdaehyung anintelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT roysubhro anintelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT barberdaniel anintelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT bendellrhyse anintelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT schmeckpeperkarl anintelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT tianjunjiao anintelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT ohjean anintelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT wignessmaggie anintelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT quanglong anintelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT rothrockbrandon anintelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT nashjeremy anintelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT waltermatthew anintelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT jentschflorian anintelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT roynicholas anintelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT howardthomas intelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT stumpethan intelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT finkjonathan intelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT arkinjacob intelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT paulrohan intelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT parkdaehyung intelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT roysubhro intelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT barberdaniel intelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT bendellrhyse intelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT schmeckpeperkarl intelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT tianjunjiao intelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT ohjean intelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT wignessmaggie intelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT quanglong intelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT rothrockbrandon intelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT nashjeremy intelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT waltermatthew intelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT jentschflorian intelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots AT roynicholas intelligencearchitectureforgroundedlanguagecommunicationwithfieldrobots |