Neural correlates of multisensory enhancement in audiovisual narrative speech perception: A fMRI investigation
This fMRI study investigated the effect of seeing articulatory movements of a speaker while listening to a naturalistic narrative stimulus. It had the goal to identify regions of the language network showing multisensory enhancement under synchronous audiovisual conditions. We expected this enhancem...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2022-11-01
|
| Series: | NeuroImage |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S1053811922007133 |
| _version_ | 1811325889764392960 |
|---|---|
| author | Lars A. Ross Sophie Molholm John S. Butler Victor A. Del Bene John J. Foxe |
| author_facet | Lars A. Ross Sophie Molholm John S. Butler Victor A. Del Bene John J. Foxe |
| author_sort | Lars A. Ross |
| collection | DOAJ |
| description | This fMRI study investigated the effect of seeing articulatory movements of a speaker while listening to a naturalistic narrative stimulus. It had the goal to identify regions of the language network showing multisensory enhancement under synchronous audiovisual conditions. We expected this enhancement to emerge in regions known to underlie the integration of auditory and visual information such as the posterior superior temporal gyrus as well as parts of the broader language network, including the semantic system. To this end we presented 53 participants with a continuous narration of a story in auditory alone, visual alone, and both synchronous and asynchronous audiovisual speech conditions while recording brain activity using BOLD fMRI. We found multisensory enhancement in an extensive network of regions underlying multisensory integration and parts of the semantic network as well as extralinguistic regions not usually associated with multisensory integration, namely the primary visual cortex and the bilateral amygdala. Analysis also revealed involvement of thalamic brain regions along the visual and auditory pathways more commonly associated with early sensory processing. We conclude that under natural listening conditions, multisensory enhancement not only involves sites of multisensory integration but many regions of the wider semantic network and includes regions associated with extralinguistic sensory, perceptual and cognitive processing. |
| first_indexed | 2024-04-13T14:41:22Z |
| format | Article |
| id | doaj.art-48740a63daa348918871e9b792865f3c |
| institution | Directory Open Access Journal |
| issn | 1095-9572 |
| language | English |
| last_indexed | 2024-04-13T14:41:22Z |
| publishDate | 2022-11-01 |
| publisher | Elsevier |
| record_format | Article |
| series | NeuroImage |
| spelling | doaj.art-48740a63daa348918871e9b792865f3c2022-12-22T02:42:53ZengElsevierNeuroImage1095-95722022-11-01263119598Neural correlates of multisensory enhancement in audiovisual narrative speech perception: A fMRI investigationLars A. Ross0Sophie Molholm1John S. Butler2Victor A. Del Bene3John J. Foxe4The Frederick J. and Marion A. Schindler Cognitive Neurophysiology Laboratory, The Ernest J. Del Monte Institute for Neuroscience, Department of Neuroscience, University of Rochester School of Medicine and Dentistry, Rochester, New York, 14642, USA; Department of Imaging Sciences, University of Rochester Medical Center, University of Rochester School of Medicine and Dentistry, Rochester, New York, 14642, USA; The Cognitive Neurophysiology Laboratory, Departments of Pediatrics and Neuroscience, Albert Einstein College of Medicine & Montefiore Medical Center, Bronx, New York, 10461, USA; Corresponding authors.The Frederick J. and Marion A. Schindler Cognitive Neurophysiology Laboratory, The Ernest J. Del Monte Institute for Neuroscience, Department of Neuroscience, University of Rochester School of Medicine and Dentistry, Rochester, New York, 14642, USA; The Cognitive Neurophysiology Laboratory, Departments of Pediatrics and Neuroscience, Albert Einstein College of Medicine & Montefiore Medical Center, Bronx, New York, 10461, USAThe Cognitive Neurophysiology Laboratory, Departments of Pediatrics and Neuroscience, Albert Einstein College of Medicine & Montefiore Medical Center, Bronx, New York, 10461, USA; School of Mathematical Sciences, Technological University Dublin, Kevin Street Campus, Dublin, IrelandThe Cognitive Neurophysiology Laboratory, Departments of Pediatrics and Neuroscience, Albert Einstein College of Medicine & Montefiore Medical Center, Bronx, New York, 10461, USA; University of Alabama at Birmingham, Heersink School of Medicine, Department of Neurology, Birmingham, Alabama, 35233, USAThe Frederick J. and Marion A. Schindler Cognitive Neurophysiology Laboratory, The Ernest J. Del Monte Institute for Neuroscience, Department of Neuroscience, University of Rochester School of Medicine and Dentistry, Rochester, New York, 14642, USA; The Cognitive Neurophysiology Laboratory, Departments of Pediatrics and Neuroscience, Albert Einstein College of Medicine & Montefiore Medical Center, Bronx, New York, 10461, USA; Corresponding authors.This fMRI study investigated the effect of seeing articulatory movements of a speaker while listening to a naturalistic narrative stimulus. It had the goal to identify regions of the language network showing multisensory enhancement under synchronous audiovisual conditions. We expected this enhancement to emerge in regions known to underlie the integration of auditory and visual information such as the posterior superior temporal gyrus as well as parts of the broader language network, including the semantic system. To this end we presented 53 participants with a continuous narration of a story in auditory alone, visual alone, and both synchronous and asynchronous audiovisual speech conditions while recording brain activity using BOLD fMRI. We found multisensory enhancement in an extensive network of regions underlying multisensory integration and parts of the semantic network as well as extralinguistic regions not usually associated with multisensory integration, namely the primary visual cortex and the bilateral amygdala. Analysis also revealed involvement of thalamic brain regions along the visual and auditory pathways more commonly associated with early sensory processing. We conclude that under natural listening conditions, multisensory enhancement not only involves sites of multisensory integration but many regions of the wider semantic network and includes regions associated with extralinguistic sensory, perceptual and cognitive processing.http://www.sciencedirect.com/science/article/pii/S1053811922007133SubcorticalNaturalistic speechCrossmodalSemanticSuperior temporal gyrusNeuroimaging |
| spellingShingle | Lars A. Ross Sophie Molholm John S. Butler Victor A. Del Bene John J. Foxe Neural correlates of multisensory enhancement in audiovisual narrative speech perception: A fMRI investigation NeuroImage Subcortical Naturalistic speech Crossmodal Semantic Superior temporal gyrus Neuroimaging |
| title | Neural correlates of multisensory enhancement in audiovisual narrative speech perception: A fMRI investigation |
| title_full | Neural correlates of multisensory enhancement in audiovisual narrative speech perception: A fMRI investigation |
| title_fullStr | Neural correlates of multisensory enhancement in audiovisual narrative speech perception: A fMRI investigation |
| title_full_unstemmed | Neural correlates of multisensory enhancement in audiovisual narrative speech perception: A fMRI investigation |
| title_short | Neural correlates of multisensory enhancement in audiovisual narrative speech perception: A fMRI investigation |
| title_sort | neural correlates of multisensory enhancement in audiovisual narrative speech perception a fmri investigation |
| topic | Subcortical Naturalistic speech Crossmodal Semantic Superior temporal gyrus Neuroimaging |
| url | http://www.sciencedirect.com/science/article/pii/S1053811922007133 |
| work_keys_str_mv | AT larsaross neuralcorrelatesofmultisensoryenhancementinaudiovisualnarrativespeechperceptionafmriinvestigation AT sophiemolholm neuralcorrelatesofmultisensoryenhancementinaudiovisualnarrativespeechperceptionafmriinvestigation AT johnsbutler neuralcorrelatesofmultisensoryenhancementinaudiovisualnarrativespeechperceptionafmriinvestigation AT victoradelbene neuralcorrelatesofmultisensoryenhancementinaudiovisualnarrativespeechperceptionafmriinvestigation AT johnjfoxe neuralcorrelatesofmultisensoryenhancementinaudiovisualnarrativespeechperceptionafmriinvestigation |