The role of multimodal cues in second language comprehension
Abstract In face-to-face communication, multimodal cues such as prosody, gestures, and mouth movements can play a crucial role in language processing. While several studies have addressed how these cues contribute to native (L1) language processing, their impact on non-native (L2) comprehension is l...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2023-11-01
|
| Series: | Scientific Reports |
| Online Access: | https://doi.org/10.1038/s41598-023-47643-2 |
| _version_ | 1797415456526565376 |
|---|---|
| author | Ye Zhang Rong Ding Diego Frassinelli Jyrki Tuomainen Sebastian Klavinskis-Whiting Gabriella Vigliocco |
| author_facet | Ye Zhang Rong Ding Diego Frassinelli Jyrki Tuomainen Sebastian Klavinskis-Whiting Gabriella Vigliocco |
| author_sort | Ye Zhang |
| collection | DOAJ |
| description | Abstract In face-to-face communication, multimodal cues such as prosody, gestures, and mouth movements can play a crucial role in language processing. While several studies have addressed how these cues contribute to native (L1) language processing, their impact on non-native (L2) comprehension is largely unknown. Comprehension of naturalistic language by L2 comprehenders may be supported by the presence of (at least some) multimodal cues, as these provide correlated and convergent information that may aid linguistic processing. However, it is also the case that multimodal cues may be less used by L2 comprehenders because linguistic processing is more demanding than for L1 comprehenders, leaving more limited resources for the processing of multimodal cues. In this study, we investigated how L2 comprehenders use multimodal cues in naturalistic stimuli (while participants watched videos of a speaker), as measured by electrophysiological responses (N400) to words, and whether there are differences between L1 and L2 comprehenders. We found that prosody, gestures, and informative mouth movements each reduced the N400 in L2, indexing easier comprehension. Nevertheless, L2 participants showed weaker effects for each cue compared to L1 comprehenders, with the exception of meaningful gestures and informative mouth movements. These results show that L2 comprehenders focus on specific multimodal cues – meaningful gestures that support meaningful interpretation and mouth movements that enhance the acoustic signal – while using multimodal cues to a lesser extent than L1 comprehenders overall. |
| first_indexed | 2024-03-09T05:47:53Z |
| format | Article |
| id | doaj.art-239cd1a3acd84e689800f4c1b6cd7a8f |
| institution | Directory Open Access Journal |
| issn | 2045-2322 |
| language | English |
| last_indexed | 2024-03-09T05:47:53Z |
| publishDate | 2023-11-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj.art-239cd1a3acd84e689800f4c1b6cd7a8f2023-12-03T12:19:03ZengNature PortfolioScientific Reports2045-23222023-11-0113111110.1038/s41598-023-47643-2The role of multimodal cues in second language comprehensionYe Zhang0Rong Ding1Diego Frassinelli2Jyrki Tuomainen3Sebastian Klavinskis-Whiting4Gabriella Vigliocco5Experimental Psychology, University College LondonLanguage and Computation in Neural Systems, Max Planck Institute for PsycholinguisticsDepartment of Linguistics, University of KonstanzSpeech, Hearing and Phonetic Sciences, University College LondonChrist Church CollegeExperimental Psychology, University College LondonAbstract In face-to-face communication, multimodal cues such as prosody, gestures, and mouth movements can play a crucial role in language processing. While several studies have addressed how these cues contribute to native (L1) language processing, their impact on non-native (L2) comprehension is largely unknown. Comprehension of naturalistic language by L2 comprehenders may be supported by the presence of (at least some) multimodal cues, as these provide correlated and convergent information that may aid linguistic processing. However, it is also the case that multimodal cues may be less used by L2 comprehenders because linguistic processing is more demanding than for L1 comprehenders, leaving more limited resources for the processing of multimodal cues. In this study, we investigated how L2 comprehenders use multimodal cues in naturalistic stimuli (while participants watched videos of a speaker), as measured by electrophysiological responses (N400) to words, and whether there are differences between L1 and L2 comprehenders. We found that prosody, gestures, and informative mouth movements each reduced the N400 in L2, indexing easier comprehension. Nevertheless, L2 participants showed weaker effects for each cue compared to L1 comprehenders, with the exception of meaningful gestures and informative mouth movements. These results show that L2 comprehenders focus on specific multimodal cues – meaningful gestures that support meaningful interpretation and mouth movements that enhance the acoustic signal – while using multimodal cues to a lesser extent than L1 comprehenders overall.https://doi.org/10.1038/s41598-023-47643-2 |
| spellingShingle | Ye Zhang Rong Ding Diego Frassinelli Jyrki Tuomainen Sebastian Klavinskis-Whiting Gabriella Vigliocco The role of multimodal cues in second language comprehension Scientific Reports |
| title | The role of multimodal cues in second language comprehension |
| title_full | The role of multimodal cues in second language comprehension |
| title_fullStr | The role of multimodal cues in second language comprehension |
| title_full_unstemmed | The role of multimodal cues in second language comprehension |
| title_short | The role of multimodal cues in second language comprehension |
| title_sort | role of multimodal cues in second language comprehension |
| url | https://doi.org/10.1038/s41598-023-47643-2 |
| work_keys_str_mv | AT yezhang theroleofmultimodalcuesinsecondlanguagecomprehension AT rongding theroleofmultimodalcuesinsecondlanguagecomprehension AT diegofrassinelli theroleofmultimodalcuesinsecondlanguagecomprehension AT jyrkituomainen theroleofmultimodalcuesinsecondlanguagecomprehension AT sebastianklavinskiswhiting theroleofmultimodalcuesinsecondlanguagecomprehension AT gabriellavigliocco theroleofmultimodalcuesinsecondlanguagecomprehension AT yezhang roleofmultimodalcuesinsecondlanguagecomprehension AT rongding roleofmultimodalcuesinsecondlanguagecomprehension AT diegofrassinelli roleofmultimodalcuesinsecondlanguagecomprehension AT jyrkituomainen roleofmultimodalcuesinsecondlanguagecomprehension AT sebastianklavinskiswhiting roleofmultimodalcuesinsecondlanguagecomprehension AT gabriellavigliocco roleofmultimodalcuesinsecondlanguagecomprehension |