Leveraging Past References for Robust Language Grounding
© 2019 Association for Computational Linguistics. Grounding referring expressions to objects in an environment has traditionally been considered a one-off, ahistorical task. However, in realistic applications of grounding, multiple users will repeatedly refer to the same set of objects. As a result,...
| Main Authors: | , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Association for Computational Linguistics (ACL)
2021
|
| Online Access: | https://hdl.handle.net/1721.1/137308 |
| _version_ | 1826192937609854976 |
|---|---|
| author | Roy, Subhro Noseworthy, Michael Paul, Rohan Park, Daehyung Roy, Nicholas |
| author2 | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory |
| author_facet | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Roy, Subhro Noseworthy, Michael Paul, Rohan Park, Daehyung Roy, Nicholas |
| author_sort | Roy, Subhro |
| collection | MIT |
| description | © 2019 Association for Computational Linguistics. Grounding referring expressions to objects in an environment has traditionally been considered a one-off, ahistorical task. However, in realistic applications of grounding, multiple users will repeatedly refer to the same set of objects. As a result, past referring expressions for objects can provide strong signals for grounding subsequent referring expressions. We therefore reframe the grounding problem from the perspective of coreference detection and propose a neural network that detects when two expressions are referring to the same object. The network combines information from vision and past referring expressions to resolve which object is being referred to. Our experiments show that detecting referring expression coreference is an effective way to ground objects described by subtle visual properties, which standard visual grounding models have difficulty capturing. We also show the ability to detect object coreference allows the grounding model to perform well even when it encounters object categories not seen in the training data. |
| first_indexed | 2024-09-23T09:31:27Z |
| format | Article |
| id | mit-1721.1/137308 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T09:31:27Z |
| publishDate | 2021 |
| publisher | Association for Computational Linguistics (ACL) |
| record_format | dspace |
| spelling | mit-1721.1/1373082022-09-26T12:01:25Z Leveraging Past References for Robust Language Grounding Roy, Subhro Noseworthy, Michael Paul, Rohan Park, Daehyung Roy, Nicholas Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory © 2019 Association for Computational Linguistics. Grounding referring expressions to objects in an environment has traditionally been considered a one-off, ahistorical task. However, in realistic applications of grounding, multiple users will repeatedly refer to the same set of objects. As a result, past referring expressions for objects can provide strong signals for grounding subsequent referring expressions. We therefore reframe the grounding problem from the perspective of coreference detection and propose a neural network that detects when two expressions are referring to the same object. The network combines information from vision and past referring expressions to resolve which object is being referred to. Our experiments show that detecting referring expression coreference is an effective way to ground objects described by subtle visual properties, which standard visual grounding models have difficulty capturing. We also show the ability to detect object coreference allows the grounding model to perform well even when it encounters object categories not seen in the training data. 2021-11-03T19:18:05Z 2021-11-03T19:18:05Z 2019-11 2021-05-03T18:28:16Z Article http://purl.org/eprint/type/ConferencePaper https://hdl.handle.net/1721.1/137308 Roy, Subhro, Noseworthy, Michael, Paul, Rohan, Park, Daehyung and Roy, Nicholas. 2019. "Leveraging Past References for Robust Language Grounding." CoNLL 2019 - 23rd Conference on Computational Natural Language Learning, Proceedings of the Conference. en http://dx.doi.org/10.18653/V1/K19-1040 CoNLL 2019 - 23rd Conference on Computational Natural Language Learning, Proceedings of the Conference Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. application/pdf Association for Computational Linguistics (ACL) Association for Computational Linguistics |
| spellingShingle | Roy, Subhro Noseworthy, Michael Paul, Rohan Park, Daehyung Roy, Nicholas Leveraging Past References for Robust Language Grounding |
| title | Leveraging Past References for Robust Language Grounding |
| title_full | Leveraging Past References for Robust Language Grounding |
| title_fullStr | Leveraging Past References for Robust Language Grounding |
| title_full_unstemmed | Leveraging Past References for Robust Language Grounding |
| title_short | Leveraging Past References for Robust Language Grounding |
| title_sort | leveraging past references for robust language grounding |
| url | https://hdl.handle.net/1721.1/137308 |
| work_keys_str_mv | AT roysubhro leveragingpastreferencesforrobustlanguagegrounding AT noseworthymichael leveragingpastreferencesforrobustlanguagegrounding AT paulrohan leveragingpastreferencesforrobustlanguagegrounding AT parkdaehyung leveragingpastreferencesforrobustlanguagegrounding AT roynicholas leveragingpastreferencesforrobustlanguagegrounding |