Multimodal prosody: gestures and speech in the perception of prominence in Spanish
Multimodal communication cannot be properly understood without analyzing the natural interweaving of speech and gestures as it typically occurs in everyday spoken language, thus moving beyond studies that elicit gestures in the laboratory, most of which are also conducted for English. Therefore, thi...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2024-03-01
|
| Series: | Frontiers in Communication |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fcomm.2024.1287363/full |
| _version_ | 1797242875503706112 |
|---|---|
| author | Miguel Jiménez-Bravo Victoria Marrero-Aguiar |
| author_facet | Miguel Jiménez-Bravo Victoria Marrero-Aguiar |
| author_sort | Miguel Jiménez-Bravo |
| collection | DOAJ |
| description | Multimodal communication cannot be properly understood without analyzing the natural interweaving of speech and gestures as it typically occurs in everyday spoken language, thus moving beyond studies that elicit gestures in the laboratory, most of which are also conducted for English. Therefore, this study addresses the effect of both visual and acoustic cues in the perception of prominence in Castilian Spanish using spontaneous speech from a TV talent-show. Four between-subjects experiments in each modality—audio-only and audiovisual—were conducted online, each including a different combination of manipulated cues: Exp1 (flat F0), Exp2 (flat intensity), and Exp3 (flat F0 + flat intensity), while all cues remained intact in the control experiment Exp0. Additionally, the capability of the different gesture phases to convey prominence was analyzed in their interaction with the acoustic cues. The results showed that, when prominence was perceived in manipulated stimuli, the effect of the visual information depended on the acoustic cues available in the signal and was also reduced when compared to non-manipulated stimuli, pointing to a strong integration of both modalities in prominence perception. In non-manipulated stimuli, all acoustic cues—except for spectral balance—played a role in the perception of prominence; however, when the visual information was added, it reduced the perceptual effect of the acoustic cues, and the main role played by duration was combined with that of the stroke phase of gestures. |
| first_indexed | 2024-04-24T18:46:10Z |
| format | Article |
| id | doaj.art-457533fd078a4b808404e9fea585a713 |
| institution | Directory Open Access Journal |
| issn | 2297-900X |
| language | English |
| last_indexed | 2024-04-24T18:46:10Z |
| publishDate | 2024-03-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Communication |
| spelling | doaj.art-457533fd078a4b808404e9fea585a7132024-03-27T05:11:38ZengFrontiers Media S.A.Frontiers in Communication2297-900X2024-03-01910.3389/fcomm.2024.12873631287363Multimodal prosody: gestures and speech in the perception of prominence in SpanishMiguel Jiménez-Bravo0Victoria Marrero-Aguiar1Department of Linguistics and Eastern Studies, Complutense University of Madrid, Madrid, SpainDepartment of General Linguistics, National University of Distance Education (UNED), Madrid, SpainMultimodal communication cannot be properly understood without analyzing the natural interweaving of speech and gestures as it typically occurs in everyday spoken language, thus moving beyond studies that elicit gestures in the laboratory, most of which are also conducted for English. Therefore, this study addresses the effect of both visual and acoustic cues in the perception of prominence in Castilian Spanish using spontaneous speech from a TV talent-show. Four between-subjects experiments in each modality—audio-only and audiovisual—were conducted online, each including a different combination of manipulated cues: Exp1 (flat F0), Exp2 (flat intensity), and Exp3 (flat F0 + flat intensity), while all cues remained intact in the control experiment Exp0. Additionally, the capability of the different gesture phases to convey prominence was analyzed in their interaction with the acoustic cues. The results showed that, when prominence was perceived in manipulated stimuli, the effect of the visual information depended on the acoustic cues available in the signal and was also reduced when compared to non-manipulated stimuli, pointing to a strong integration of both modalities in prominence perception. In non-manipulated stimuli, all acoustic cues—except for spectral balance—played a role in the perception of prominence; however, when the visual information was added, it reduced the perceptual effect of the acoustic cues, and the main role played by duration was combined with that of the stroke phase of gestures.https://www.frontiersin.org/articles/10.3389/fcomm.2024.1287363/fullaudiovisual prosodymultimodalityspeech perceptionacoustic cuesgestureprominence |
| spellingShingle | Miguel Jiménez-Bravo Victoria Marrero-Aguiar Multimodal prosody: gestures and speech in the perception of prominence in Spanish Frontiers in Communication audiovisual prosody multimodality speech perception acoustic cues gesture prominence |
| title | Multimodal prosody: gestures and speech in the perception of prominence in Spanish |
| title_full | Multimodal prosody: gestures and speech in the perception of prominence in Spanish |
| title_fullStr | Multimodal prosody: gestures and speech in the perception of prominence in Spanish |
| title_full_unstemmed | Multimodal prosody: gestures and speech in the perception of prominence in Spanish |
| title_short | Multimodal prosody: gestures and speech in the perception of prominence in Spanish |
| title_sort | multimodal prosody gestures and speech in the perception of prominence in spanish |
| topic | audiovisual prosody multimodality speech perception acoustic cues gesture prominence |
| url | https://www.frontiersin.org/articles/10.3389/fcomm.2024.1287363/full |
| work_keys_str_mv | AT migueljimenezbravo multimodalprosodygesturesandspeechintheperceptionofprominenceinspanish AT victoriamarreroaguiar multimodalprosodygesturesandspeechintheperceptionofprominenceinspanish |