Cross-modal Association between Auditory and Visuospatial Information in Mandarin Tone Perception in Noise by Native and Non-native Perceivers
Speech perception involves multiple input modalities. Research has indicated that perceivers establish cross-modal associations between auditory and visuospatial events to aid perception. Such intermodal relations can be particularly beneficial for speech development and learning, where infants and...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2017-12-01
|
| Series: | Frontiers in Psychology |
| Subjects: | |
| Online Access: | http://journal.frontiersin.org/article/10.3389/fpsyg.2017.02051/full |
| _version_ | 1819073070340505600 |
|---|---|
| author | Beverly Hannah Yue Wang Allard Jongman Joan A. Sereno Jiguo Cao Yunlong Nie |
| author_facet | Beverly Hannah Yue Wang Allard Jongman Joan A. Sereno Jiguo Cao Yunlong Nie |
| author_sort | Beverly Hannah |
| collection | DOAJ |
| description | Speech perception involves multiple input modalities. Research has indicated that perceivers establish cross-modal associations between auditory and visuospatial events to aid perception. Such intermodal relations can be particularly beneficial for speech development and learning, where infants and non-native perceivers need additional resources to acquire and process new sounds. This study examines how facial articulatory cues and co-speech hand gestures mimicking pitch contours in space affect non-native Mandarin tone perception. Native English as well as Mandarin perceivers identified tones embedded in noise with either congruent or incongruent Auditory-Facial (AF) and Auditory-FacialGestural (AFG) inputs. Native Mandarin results showed the expected ceiling-level performance in the congruent AF and AFG conditions. In the incongruent conditions, while AF identification was primarily auditory-based, AFG identification was partially based on gestures, demonstrating the use of gestures as valid cues in tone identification. The English perceivers’ performance was poor in the congruent AF condition, but improved significantly in AFG. While the incongruent AF identification showed some reliance on facial information, incongruent AFG identification relied more on gestural than auditory-facial information. These results indicate positive effects of facial and especially gestural input on non-native tone perception, suggesting that cross-modal (visuospatial) resources can be recruited to aid auditory perception when phonetic demands are high. The current findings may inform patterns of tone acquisition and development, suggesting how multi-modal speech enhancement principles may be applied to facilitate speech learning. |
| first_indexed | 2024-12-21T17:47:46Z |
| format | Article |
| id | doaj.art-258349ff6d6945359b0f56348b7e9067 |
| institution | Directory Open Access Journal |
| issn | 1664-1078 |
| language | English |
| last_indexed | 2024-12-21T17:47:46Z |
| publishDate | 2017-12-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Psychology |
| spelling | doaj.art-258349ff6d6945359b0f56348b7e90672022-12-21T18:55:26ZengFrontiers Media S.A.Frontiers in Psychology1664-10782017-12-01810.3389/fpsyg.2017.02051289653Cross-modal Association between Auditory and Visuospatial Information in Mandarin Tone Perception in Noise by Native and Non-native PerceiversBeverly Hannah0Yue Wang1Allard Jongman2Joan A. Sereno3Jiguo Cao4Yunlong Nie5Language and Brain Lab, Department of Linguistics, Simon Fraser University, Burnaby, BC, CanadaLanguage and Brain Lab, Department of Linguistics, Simon Fraser University, Burnaby, BC, CanadaPhonetics and Psycholinguistics Laboratory, Department of Linguistics, University of Kansas, Lawrence, KS, United StatesPhonetics and Psycholinguistics Laboratory, Department of Linguistics, University of Kansas, Lawrence, KS, United StatesDepartment of Statistics and Actuarial Science, Simon Fraser University, Burnaby, BC, CanadaDepartment of Statistics and Actuarial Science, Simon Fraser University, Burnaby, BC, CanadaSpeech perception involves multiple input modalities. Research has indicated that perceivers establish cross-modal associations between auditory and visuospatial events to aid perception. Such intermodal relations can be particularly beneficial for speech development and learning, where infants and non-native perceivers need additional resources to acquire and process new sounds. This study examines how facial articulatory cues and co-speech hand gestures mimicking pitch contours in space affect non-native Mandarin tone perception. Native English as well as Mandarin perceivers identified tones embedded in noise with either congruent or incongruent Auditory-Facial (AF) and Auditory-FacialGestural (AFG) inputs. Native Mandarin results showed the expected ceiling-level performance in the congruent AF and AFG conditions. In the incongruent conditions, while AF identification was primarily auditory-based, AFG identification was partially based on gestures, demonstrating the use of gestures as valid cues in tone identification. The English perceivers’ performance was poor in the congruent AF condition, but improved significantly in AFG. While the incongruent AF identification showed some reliance on facial information, incongruent AFG identification relied more on gestural than auditory-facial information. These results indicate positive effects of facial and especially gestural input on non-native tone perception, suggesting that cross-modal (visuospatial) resources can be recruited to aid auditory perception when phonetic demands are high. The current findings may inform patterns of tone acquisition and development, suggesting how multi-modal speech enhancement principles may be applied to facilitate speech learning.http://journal.frontiersin.org/article/10.3389/fpsyg.2017.02051/fullcross-modal associationgestureaudio-visualtone perceptionMandarinEnglish |
| spellingShingle | Beverly Hannah Yue Wang Allard Jongman Joan A. Sereno Jiguo Cao Yunlong Nie Cross-modal Association between Auditory and Visuospatial Information in Mandarin Tone Perception in Noise by Native and Non-native Perceivers Frontiers in Psychology cross-modal association gesture audio-visual tone perception Mandarin English |
| title | Cross-modal Association between Auditory and Visuospatial Information in Mandarin Tone Perception in Noise by Native and Non-native Perceivers |
| title_full | Cross-modal Association between Auditory and Visuospatial Information in Mandarin Tone Perception in Noise by Native and Non-native Perceivers |
| title_fullStr | Cross-modal Association between Auditory and Visuospatial Information in Mandarin Tone Perception in Noise by Native and Non-native Perceivers |
| title_full_unstemmed | Cross-modal Association between Auditory and Visuospatial Information in Mandarin Tone Perception in Noise by Native and Non-native Perceivers |
| title_short | Cross-modal Association between Auditory and Visuospatial Information in Mandarin Tone Perception in Noise by Native and Non-native Perceivers |
| title_sort | cross modal association between auditory and visuospatial information in mandarin tone perception in noise by native and non native perceivers |
| topic | cross-modal association gesture audio-visual tone perception Mandarin English |
| url | http://journal.frontiersin.org/article/10.3389/fpsyg.2017.02051/full |
| work_keys_str_mv | AT beverlyhannah crossmodalassociationbetweenauditoryandvisuospatialinformationinmandarintoneperceptioninnoisebynativeandnonnativeperceivers AT yuewang crossmodalassociationbetweenauditoryandvisuospatialinformationinmandarintoneperceptioninnoisebynativeandnonnativeperceivers AT allardjongman crossmodalassociationbetweenauditoryandvisuospatialinformationinmandarintoneperceptioninnoisebynativeandnonnativeperceivers AT joanasereno crossmodalassociationbetweenauditoryandvisuospatialinformationinmandarintoneperceptioninnoisebynativeandnonnativeperceivers AT jiguocao crossmodalassociationbetweenauditoryandvisuospatialinformationinmandarintoneperceptioninnoisebynativeandnonnativeperceivers AT yunlongnie crossmodalassociationbetweenauditoryandvisuospatialinformationinmandarintoneperceptioninnoisebynativeandnonnativeperceivers |