Crossmodal Language Comprehension—Psycholinguistic Insights and Computational Approaches
Crossmodal interaction in situated language comprehension is important for effective and efficient communication. The relationship between linguistic and visual stimuli provides mutual benefit: While vision contributes, for instance, information to improve language understanding, language in turn pl...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2020-01-01
|
| Series: | Frontiers in Neurorobotics |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/article/10.3389/fnbot.2020.00002/full |
| _version_ | 1818557620955381760 |
|---|---|
| author | Özge Alaçam Xingshan Li Wolfgang Menzel Tobias Staron |
| author_facet | Özge Alaçam Xingshan Li Wolfgang Menzel Tobias Staron |
| author_sort | Özge Alaçam |
| collection | DOAJ |
| description | Crossmodal interaction in situated language comprehension is important for effective and efficient communication. The relationship between linguistic and visual stimuli provides mutual benefit: While vision contributes, for instance, information to improve language understanding, language in turn plays a role in driving the focus of attention in the visual environment. However, language and vision are two different representational modalities, which accommodate different aspects and granularities of conceptualizations. To integrate them into a single, coherent system solution is still a challenge, which could profit from inspiration by human crossmodal processing. Based on fundamental psycholinguistic insights into the nature of situated language comprehension, we derive a set of performance characteristics facilitating the robustness of language understanding, such as crossmodal reference resolution, attention guidance, or predictive processing. Artificial systems for language comprehension should meet these characteristics in order to be able to perform in a natural and smooth manner. We discuss how empirical findings on the crossmodal support of language comprehension in humans can be applied in computational solutions for situated language comprehension and how they can help to mitigate the shortcomings of current approaches. |
| first_indexed | 2024-12-14T00:01:55Z |
| format | Article |
| id | doaj.art-05e2f22b08f740a58e677404e2dfa87c |
| institution | Directory Open Access Journal |
| issn | 1662-5218 |
| language | English |
| last_indexed | 2024-12-14T00:01:55Z |
| publishDate | 2020-01-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Neurorobotics |
| spelling | doaj.art-05e2f22b08f740a58e677404e2dfa87c2022-12-21T23:26:17ZengFrontiers Media S.A.Frontiers in Neurorobotics1662-52182020-01-011410.3389/fnbot.2020.00002475864Crossmodal Language Comprehension—Psycholinguistic Insights and Computational ApproachesÖzge Alaçam0Xingshan Li1Wolfgang Menzel2Tobias Staron3Natural Language Systems Group, Department of Informatics, University of Hamburg, Hamburg, GermanyReading and Visual Cognition Lab, Institute of Psychology, Chinese Academy of Science, Beijing, ChinaNatural Language Systems Group, Department of Informatics, University of Hamburg, Hamburg, GermanyNatural Language Systems Group, Department of Informatics, University of Hamburg, Hamburg, GermanyCrossmodal interaction in situated language comprehension is important for effective and efficient communication. The relationship between linguistic and visual stimuli provides mutual benefit: While vision contributes, for instance, information to improve language understanding, language in turn plays a role in driving the focus of attention in the visual environment. However, language and vision are two different representational modalities, which accommodate different aspects and granularities of conceptualizations. To integrate them into a single, coherent system solution is still a challenge, which could profit from inspiration by human crossmodal processing. Based on fundamental psycholinguistic insights into the nature of situated language comprehension, we derive a set of performance characteristics facilitating the robustness of language understanding, such as crossmodal reference resolution, attention guidance, or predictive processing. Artificial systems for language comprehension should meet these characteristics in order to be able to perform in a natural and smooth manner. We discuss how empirical findings on the crossmodal support of language comprehension in humans can be applied in computational solutions for situated language comprehension and how they can help to mitigate the shortcomings of current approaches.https://www.frontiersin.org/article/10.3389/fnbot.2020.00002/fulllanguage comprehensioncrossmodalitypsycholinguisticsincrementalitypredictionspeaker intention |
| spellingShingle | Özge Alaçam Xingshan Li Wolfgang Menzel Tobias Staron Crossmodal Language Comprehension—Psycholinguistic Insights and Computational Approaches Frontiers in Neurorobotics language comprehension crossmodality psycholinguistics incrementality prediction speaker intention |
| title | Crossmodal Language Comprehension—Psycholinguistic Insights and Computational Approaches |
| title_full | Crossmodal Language Comprehension—Psycholinguistic Insights and Computational Approaches |
| title_fullStr | Crossmodal Language Comprehension—Psycholinguistic Insights and Computational Approaches |
| title_full_unstemmed | Crossmodal Language Comprehension—Psycholinguistic Insights and Computational Approaches |
| title_short | Crossmodal Language Comprehension—Psycholinguistic Insights and Computational Approaches |
| title_sort | crossmodal language comprehension psycholinguistic insights and computational approaches |
| topic | language comprehension crossmodality psycholinguistics incrementality prediction speaker intention |
| url | https://www.frontiersin.org/article/10.3389/fnbot.2020.00002/full |
| work_keys_str_mv | AT ozgealacam crossmodallanguagecomprehensionpsycholinguisticinsightsandcomputationalapproaches AT xingshanli crossmodallanguagecomprehensionpsycholinguisticinsightsandcomputationalapproaches AT wolfgangmenzel crossmodallanguagecomprehensionpsycholinguisticinsightsandcomputationalapproaches AT tobiasstaron crossmodallanguagecomprehensionpsycholinguisticinsightsandcomputationalapproaches |