A representation of abstract linguistic categories in the visual system underlies successful lipreading
There is considerable debate over how visual speech is processed in the absence of sound and whether neural activity supporting lipreading occurs in visual brain areas. Much of the ambiguity stems from a lack of behavioral grounding and neurophysiological analyses that cannot disentangle high-level...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2023-11-01
|
| Series: | NeuroImage |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S1053811923005426 |
| _version_ | 1797647898757824512 |
|---|---|
| author | Aaron R Nidiffer Cody Zhewei Cao Aisling O'Sullivan Edmund C Lalor |
| author_facet | Aaron R Nidiffer Cody Zhewei Cao Aisling O'Sullivan Edmund C Lalor |
| author_sort | Aaron R Nidiffer |
| collection | DOAJ |
| description | There is considerable debate over how visual speech is processed in the absence of sound and whether neural activity supporting lipreading occurs in visual brain areas. Much of the ambiguity stems from a lack of behavioral grounding and neurophysiological analyses that cannot disentangle high-level linguistic and phonetic/energetic contributions from visual speech. To address this, we recorded EEG from human observers as they watched silent videos, half of which were novel and half of which were previously rehearsed with the accompanying audio. We modeled how the EEG responses to novel and rehearsed silent speech reflected the processing of low-level visual features (motion, lip movements) and a higher-level categorical representation of linguistic units, known as visemes. The ability of these visemes to account for the EEG – beyond the motion and lip movements – was significantly enhanced for rehearsed videos in a way that correlated with participants’ trial-by-trial ability to lipread that speech. Source localization of viseme processing showed clear contributions from visual cortex, with no strong evidence for the involvement of auditory areas. We interpret this as support for the idea that the visual system produces its own specialized representation of speech that is (1) well-described by categorical linguistic features, (2) dissociable from lip movements, and (3) predictive of lipreading ability. We also suggest a reinterpretation of previous findings of auditory cortical activation during silent speech that is consistent with hierarchical accounts of visual and audiovisual speech perception. |
| first_indexed | 2024-03-11T15:24:22Z |
| format | Article |
| id | doaj.art-88a422e2bc9f4b6788b1de7815107748 |
| institution | Directory Open Access Journal |
| issn | 1095-9572 |
| language | English |
| last_indexed | 2024-03-11T15:24:22Z |
| publishDate | 2023-11-01 |
| publisher | Elsevier |
| record_format | Article |
| series | NeuroImage |
| spelling | doaj.art-88a422e2bc9f4b6788b1de78151077482023-10-28T05:06:46ZengElsevierNeuroImage1095-95722023-11-01282120391A representation of abstract linguistic categories in the visual system underlies successful lipreadingAaron R Nidiffer0Cody Zhewei Cao1Aisling O'Sullivan2Edmund C Lalor3Department of Biomedical Engineering, Department of Neuroscience, Del Monte Institute for Neuroscience, University of Rochester, Rochester, NY, USADepartment of Psychology, University of Michigan, Ann Arbor, MI, USASchool of Engineering, Trinity College Institute of Neuroscience, Trinity Centre for Biomedical Engineering, Trinity College, Dublin, IrelandDepartment of Biomedical Engineering, Department of Neuroscience, Del Monte Institute for Neuroscience, University of Rochester, Rochester, NY, USA; School of Engineering, Trinity College Institute of Neuroscience, Trinity Centre for Biomedical Engineering, Trinity College, Dublin, Ireland; Corresponding author at: Department of Biomedical Engineering, University of Rochester, 204 Robert B. Goergen Hall, P.O. Box 270168, Rochester, NY 14627, USA.There is considerable debate over how visual speech is processed in the absence of sound and whether neural activity supporting lipreading occurs in visual brain areas. Much of the ambiguity stems from a lack of behavioral grounding and neurophysiological analyses that cannot disentangle high-level linguistic and phonetic/energetic contributions from visual speech. To address this, we recorded EEG from human observers as they watched silent videos, half of which were novel and half of which were previously rehearsed with the accompanying audio. We modeled how the EEG responses to novel and rehearsed silent speech reflected the processing of low-level visual features (motion, lip movements) and a higher-level categorical representation of linguistic units, known as visemes. The ability of these visemes to account for the EEG – beyond the motion and lip movements – was significantly enhanced for rehearsed videos in a way that correlated with participants’ trial-by-trial ability to lipread that speech. Source localization of viseme processing showed clear contributions from visual cortex, with no strong evidence for the involvement of auditory areas. We interpret this as support for the idea that the visual system produces its own specialized representation of speech that is (1) well-described by categorical linguistic features, (2) dissociable from lip movements, and (3) predictive of lipreading ability. We also suggest a reinterpretation of previous findings of auditory cortical activation during silent speech that is consistent with hierarchical accounts of visual and audiovisual speech perception.http://www.sciencedirect.com/science/article/pii/S1053811923005426Visual speechEEGLipreadingLinear modeling |
| spellingShingle | Aaron R Nidiffer Cody Zhewei Cao Aisling O'Sullivan Edmund C Lalor A representation of abstract linguistic categories in the visual system underlies successful lipreading NeuroImage Visual speech EEG Lipreading Linear modeling |
| title | A representation of abstract linguistic categories in the visual system underlies successful lipreading |
| title_full | A representation of abstract linguistic categories in the visual system underlies successful lipreading |
| title_fullStr | A representation of abstract linguistic categories in the visual system underlies successful lipreading |
| title_full_unstemmed | A representation of abstract linguistic categories in the visual system underlies successful lipreading |
| title_short | A representation of abstract linguistic categories in the visual system underlies successful lipreading |
| title_sort | representation of abstract linguistic categories in the visual system underlies successful lipreading |
| topic | Visual speech EEG Lipreading Linear modeling |
| url | http://www.sciencedirect.com/science/article/pii/S1053811923005426 |
| work_keys_str_mv | AT aaronrnidiffer arepresentationofabstractlinguisticcategoriesinthevisualsystemunderliessuccessfullipreading AT codyzheweicao arepresentationofabstractlinguisticcategoriesinthevisualsystemunderliessuccessfullipreading AT aislingosullivan arepresentationofabstractlinguisticcategoriesinthevisualsystemunderliessuccessfullipreading AT edmundclalor arepresentationofabstractlinguisticcategoriesinthevisualsystemunderliessuccessfullipreading AT aaronrnidiffer representationofabstractlinguisticcategoriesinthevisualsystemunderliessuccessfullipreading AT codyzheweicao representationofabstractlinguisticcategoriesinthevisualsystemunderliessuccessfullipreading AT aislingosullivan representationofabstractlinguisticcategoriesinthevisualsystemunderliessuccessfullipreading AT edmundclalor representationofabstractlinguisticcategoriesinthevisualsystemunderliessuccessfullipreading |