Explaining neural activity in human listeners with deep learning via natural language processing of narrative text
Abstract Deep learning (DL) approaches may also inform the analysis of human brain activity. Here, a state-of-art DL tool for natural language processing, the Generative Pre-trained Transformer version 2 (GPT-2), is shown to generate meaningful neural encodings in functional MRI during narrative lis...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2022-10-01
|
| Series: | Scientific Reports |
| Online Access: | https://doi.org/10.1038/s41598-022-21782-4 |
| _version_ | 1811257070874263552 |
|---|---|
| author | Andrea G. Russo Assunta Ciarlo Sara Ponticorvo Francesco Di Salle Gioacchino Tedeschi Fabrizio Esposito |
| author_facet | Andrea G. Russo Assunta Ciarlo Sara Ponticorvo Francesco Di Salle Gioacchino Tedeschi Fabrizio Esposito |
| author_sort | Andrea G. Russo |
| collection | DOAJ |
| description | Abstract Deep learning (DL) approaches may also inform the analysis of human brain activity. Here, a state-of-art DL tool for natural language processing, the Generative Pre-trained Transformer version 2 (GPT-2), is shown to generate meaningful neural encodings in functional MRI during narrative listening. Linguistic features of word unpredictability (surprisal) and contextual importance (saliency) were derived from the GPT-2 applied to the text of a 12-min narrative. Segments of variable duration (from 15 to 90 s) defined the context for the next word, resulting in different sets of neural predictors for functional MRI signals recorded in 27 healthy listeners of the narrative. GPT-2 surprisal, estimating word prediction errors from the artificial network, significantly explained the neural data in superior and middle temporal gyri (bilaterally), in anterior and posterior cingulate cortices, and in the left prefrontal cortex. GPT-2 saliency, weighing the importance of context words, significantly explained the neural data for longer segments in left superior and middle temporal gyri. These results add novel support to the use of DL tools in the search for neural encodings in functional MRI. A DL language model like the GPT-2 may feature useful data about neural processes subserving language comprehension in humans, including next-word context-related prediction. |
| first_indexed | 2024-04-12T17:50:59Z |
| format | Article |
| id | doaj.art-6f1741c4939c4ddd85e14e9d5e077e0d |
| institution | Directory Open Access Journal |
| issn | 2045-2322 |
| language | English |
| last_indexed | 2024-04-12T17:50:59Z |
| publishDate | 2022-10-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj.art-6f1741c4939c4ddd85e14e9d5e077e0d2022-12-22T03:22:30ZengNature PortfolioScientific Reports2045-23222022-10-011211910.1038/s41598-022-21782-4Explaining neural activity in human listeners with deep learning via natural language processing of narrative textAndrea G. Russo0Assunta Ciarlo1Sara Ponticorvo2Francesco Di Salle3Gioacchino Tedeschi4Fabrizio Esposito5Department of Advanced Medical and Surgical Sciences, School of Medicine and Surgery, University of Campania “Luigi Vanvitelli”Department of Medicine, Surgery and Dentistry, “Scuola Medica Salernitana”, University of SalernoDepartment of Medicine, Surgery and Dentistry, “Scuola Medica Salernitana”, University of SalernoDepartment of Medicine, Surgery and Dentistry, “Scuola Medica Salernitana”, University of SalernoDepartment of Advanced Medical and Surgical Sciences, School of Medicine and Surgery, University of Campania “Luigi Vanvitelli”Department of Advanced Medical and Surgical Sciences, School of Medicine and Surgery, University of Campania “Luigi Vanvitelli”Abstract Deep learning (DL) approaches may also inform the analysis of human brain activity. Here, a state-of-art DL tool for natural language processing, the Generative Pre-trained Transformer version 2 (GPT-2), is shown to generate meaningful neural encodings in functional MRI during narrative listening. Linguistic features of word unpredictability (surprisal) and contextual importance (saliency) were derived from the GPT-2 applied to the text of a 12-min narrative. Segments of variable duration (from 15 to 90 s) defined the context for the next word, resulting in different sets of neural predictors for functional MRI signals recorded in 27 healthy listeners of the narrative. GPT-2 surprisal, estimating word prediction errors from the artificial network, significantly explained the neural data in superior and middle temporal gyri (bilaterally), in anterior and posterior cingulate cortices, and in the left prefrontal cortex. GPT-2 saliency, weighing the importance of context words, significantly explained the neural data for longer segments in left superior and middle temporal gyri. These results add novel support to the use of DL tools in the search for neural encodings in functional MRI. A DL language model like the GPT-2 may feature useful data about neural processes subserving language comprehension in humans, including next-word context-related prediction.https://doi.org/10.1038/s41598-022-21782-4 |
| spellingShingle | Andrea G. Russo Assunta Ciarlo Sara Ponticorvo Francesco Di Salle Gioacchino Tedeschi Fabrizio Esposito Explaining neural activity in human listeners with deep learning via natural language processing of narrative text Scientific Reports |
| title | Explaining neural activity in human listeners with deep learning via natural language processing of narrative text |
| title_full | Explaining neural activity in human listeners with deep learning via natural language processing of narrative text |
| title_fullStr | Explaining neural activity in human listeners with deep learning via natural language processing of narrative text |
| title_full_unstemmed | Explaining neural activity in human listeners with deep learning via natural language processing of narrative text |
| title_short | Explaining neural activity in human listeners with deep learning via natural language processing of narrative text |
| title_sort | explaining neural activity in human listeners with deep learning via natural language processing of narrative text |
| url | https://doi.org/10.1038/s41598-022-21782-4 |
| work_keys_str_mv | AT andreagrusso explainingneuralactivityinhumanlistenerswithdeeplearningvianaturallanguageprocessingofnarrativetext AT assuntaciarlo explainingneuralactivityinhumanlistenerswithdeeplearningvianaturallanguageprocessingofnarrativetext AT saraponticorvo explainingneuralactivityinhumanlistenerswithdeeplearningvianaturallanguageprocessingofnarrativetext AT francescodisalle explainingneuralactivityinhumanlistenerswithdeeplearningvianaturallanguageprocessingofnarrativetext AT gioacchinotedeschi explainingneuralactivityinhumanlistenerswithdeeplearningvianaturallanguageprocessingofnarrativetext AT fabrizioesposito explainingneuralactivityinhumanlistenerswithdeeplearningvianaturallanguageprocessingofnarrativetext |