The effect of gaze on EEG measures of multisensory integration in a cocktail party scenario
Seeing the speaker’s face greatly improves our speech comprehension in noisy environments. This is due to the brain’s ability to combine the auditory and the visual information around us, a process known as multisensory integration. Selective attention also strongly influences what we comprehend in...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2023-12-01
|
| Series: | Frontiers in Human Neuroscience |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fnhum.2023.1283206/full |
| _version_ | 1797390316691521536 |
|---|---|
| author | Farhin Ahmed Aaron R. Nidiffer Edmund C. Lalor |
| author_facet | Farhin Ahmed Aaron R. Nidiffer Edmund C. Lalor |
| author_sort | Farhin Ahmed |
| collection | DOAJ |
| description | Seeing the speaker’s face greatly improves our speech comprehension in noisy environments. This is due to the brain’s ability to combine the auditory and the visual information around us, a process known as multisensory integration. Selective attention also strongly influences what we comprehend in scenarios with multiple speakers–an effect known as the cocktail-party phenomenon. However, the interaction between attention and multisensory integration is not fully understood, especially when it comes to natural, continuous speech. In a recent electroencephalography (EEG) study, we explored this issue and showed that multisensory integration is enhanced when an audiovisual speaker is attended compared to when that speaker is unattended. Here, we extend that work to investigate how this interaction varies depending on a person’s gaze behavior, which affects the quality of the visual information they have access to. To do so, we recorded EEG from 31 healthy adults as they performed selective attention tasks in several paradigms involving two concurrently presented audiovisual speakers. We then modeled how the recorded EEG related to the audio speech (envelope) of the presented speakers. Crucially, we compared two classes of model – one that assumed underlying multisensory integration (AV) versus another that assumed two independent unisensory audio and visual processes (A+V). This comparison revealed evidence of strong attentional effects on multisensory integration when participants were looking directly at the face of an audiovisual speaker. This effect was not apparent when the speaker’s face was in the peripheral vision of the participants. Overall, our findings suggest a strong influence of attention on multisensory integration when high fidelity visual (articulatory) speech information is available. More generally, this suggests that the interplay between attention and multisensory integration during natural audiovisual speech is dynamic and is adaptable based on the specific task and environment. |
| first_indexed | 2024-03-08T23:09:00Z |
| format | Article |
| id | doaj.art-e94d424718ed4cf5aa03c5e2ed5c96f8 |
| institution | Directory Open Access Journal |
| issn | 1662-5161 |
| language | English |
| last_indexed | 2024-03-08T23:09:00Z |
| publishDate | 2023-12-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Human Neuroscience |
| spelling | doaj.art-e94d424718ed4cf5aa03c5e2ed5c96f82023-12-15T09:01:38ZengFrontiers Media S.A.Frontiers in Human Neuroscience1662-51612023-12-011710.3389/fnhum.2023.12832061283206The effect of gaze on EEG measures of multisensory integration in a cocktail party scenarioFarhin AhmedAaron R. NidifferEdmund C. LalorSeeing the speaker’s face greatly improves our speech comprehension in noisy environments. This is due to the brain’s ability to combine the auditory and the visual information around us, a process known as multisensory integration. Selective attention also strongly influences what we comprehend in scenarios with multiple speakers–an effect known as the cocktail-party phenomenon. However, the interaction between attention and multisensory integration is not fully understood, especially when it comes to natural, continuous speech. In a recent electroencephalography (EEG) study, we explored this issue and showed that multisensory integration is enhanced when an audiovisual speaker is attended compared to when that speaker is unattended. Here, we extend that work to investigate how this interaction varies depending on a person’s gaze behavior, which affects the quality of the visual information they have access to. To do so, we recorded EEG from 31 healthy adults as they performed selective attention tasks in several paradigms involving two concurrently presented audiovisual speakers. We then modeled how the recorded EEG related to the audio speech (envelope) of the presented speakers. Crucially, we compared two classes of model – one that assumed underlying multisensory integration (AV) versus another that assumed two independent unisensory audio and visual processes (A+V). This comparison revealed evidence of strong attentional effects on multisensory integration when participants were looking directly at the face of an audiovisual speaker. This effect was not apparent when the speaker’s face was in the peripheral vision of the participants. Overall, our findings suggest a strong influence of attention on multisensory integration when high fidelity visual (articulatory) speech information is available. More generally, this suggests that the interplay between attention and multisensory integration during natural audiovisual speech is dynamic and is adaptable based on the specific task and environment.https://www.frontiersin.org/articles/10.3389/fnhum.2023.1283206/fullmultisensory integrationselective attentioncocktail party problemaudiovisual speechEEGspeech envelope |
| spellingShingle | Farhin Ahmed Aaron R. Nidiffer Edmund C. Lalor The effect of gaze on EEG measures of multisensory integration in a cocktail party scenario Frontiers in Human Neuroscience multisensory integration selective attention cocktail party problem audiovisual speech EEG speech envelope |
| title | The effect of gaze on EEG measures of multisensory integration in a cocktail party scenario |
| title_full | The effect of gaze on EEG measures of multisensory integration in a cocktail party scenario |
| title_fullStr | The effect of gaze on EEG measures of multisensory integration in a cocktail party scenario |
| title_full_unstemmed | The effect of gaze on EEG measures of multisensory integration in a cocktail party scenario |
| title_short | The effect of gaze on EEG measures of multisensory integration in a cocktail party scenario |
| title_sort | effect of gaze on eeg measures of multisensory integration in a cocktail party scenario |
| topic | multisensory integration selective attention cocktail party problem audiovisual speech EEG speech envelope |
| url | https://www.frontiersin.org/articles/10.3389/fnhum.2023.1283206/full |
| work_keys_str_mv | AT farhinahmed theeffectofgazeoneegmeasuresofmultisensoryintegrationinacocktailpartyscenario AT aaronrnidiffer theeffectofgazeoneegmeasuresofmultisensoryintegrationinacocktailpartyscenario AT edmundclalor theeffectofgazeoneegmeasuresofmultisensoryintegrationinacocktailpartyscenario AT farhinahmed effectofgazeoneegmeasuresofmultisensoryintegrationinacocktailpartyscenario AT aaronrnidiffer effectofgazeoneegmeasuresofmultisensoryintegrationinacocktailpartyscenario AT edmundclalor effectofgazeoneegmeasuresofmultisensoryintegrationinacocktailpartyscenario |