Prediction as a basis for skilled reading: insights from modern language models
Reading is not an inborn human capability, and yet, English-speaking adults read with impressive speed. This study considered how predictions of upcoming words impact on this skilled behaviour. We used a powerful language model (GPT-2) to derive predictions of upcoming words in text passages. These...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
The Royal Society
2022-06-01
|
| Series: | Royal Society Open Science |
| Subjects: | |
| Online Access: | https://royalsocietypublishing.org/doi/10.1098/rsos.211837 |
| _version_ | 1797837621996552192 |
|---|---|
| author | Benedetta Cevoli Chris Watkins Kathleen Rastle |
| author_facet | Benedetta Cevoli Chris Watkins Kathleen Rastle |
| author_sort | Benedetta Cevoli |
| collection | DOAJ |
| description | Reading is not an inborn human capability, and yet, English-speaking adults read with impressive speed. This study considered how predictions of upcoming words impact on this skilled behaviour. We used a powerful language model (GPT-2) to derive predictions of upcoming words in text passages. These predictions were highly accurate and showed a tight relationship to fine-grained aspects of eye-movement behaviour when adults read those same passages, including whether to skip the next word and how long to spend on it. Strong predictions that were incorrect resulted in a prediction error cost on fixation durations. Our findings suggest that predictions for upcoming words can be made based on the analysis of text statistics and that these predictions guide how our eyes interrogate text at very short timescales. These findings open new perspectives on reading and language comprehension and illustrate the capability of modern language models to inform understanding of human language processing. |
| first_indexed | 2024-04-09T15:27:40Z |
| format | Article |
| id | doaj.art-24478bdf3c92450db0db1957ca142617 |
| institution | Directory Open Access Journal |
| issn | 2054-5703 |
| language | English |
| last_indexed | 2024-04-09T15:27:40Z |
| publishDate | 2022-06-01 |
| publisher | The Royal Society |
| record_format | Article |
| series | Royal Society Open Science |
| spelling | doaj.art-24478bdf3c92450db0db1957ca1426172023-04-28T10:52:23ZengThe Royal SocietyRoyal Society Open Science2054-57032022-06-019610.1098/rsos.211837Prediction as a basis for skilled reading: insights from modern language modelsBenedetta Cevoli0Chris Watkins1Kathleen Rastle2Department of Psychology, Royal Holloway, University of London, Egham, UKDepartment of Computer Science, Royal Holloway, University of London, Egham, UKDepartment of Psychology, Royal Holloway, University of London, Egham, UKReading is not an inborn human capability, and yet, English-speaking adults read with impressive speed. This study considered how predictions of upcoming words impact on this skilled behaviour. We used a powerful language model (GPT-2) to derive predictions of upcoming words in text passages. These predictions were highly accurate and showed a tight relationship to fine-grained aspects of eye-movement behaviour when adults read those same passages, including whether to skip the next word and how long to spend on it. Strong predictions that were incorrect resulted in a prediction error cost on fixation durations. Our findings suggest that predictions for upcoming words can be made based on the analysis of text statistics and that these predictions guide how our eyes interrogate text at very short timescales. These findings open new perspectives on reading and language comprehension and illustrate the capability of modern language models to inform understanding of human language processing.https://royalsocietypublishing.org/doi/10.1098/rsos.211837predictioneye movementsGPT-2language modelsreading |
| spellingShingle | Benedetta Cevoli Chris Watkins Kathleen Rastle Prediction as a basis for skilled reading: insights from modern language models Royal Society Open Science prediction eye movements GPT-2 language models reading |
| title | Prediction as a basis for skilled reading: insights from modern language models |
| title_full | Prediction as a basis for skilled reading: insights from modern language models |
| title_fullStr | Prediction as a basis for skilled reading: insights from modern language models |
| title_full_unstemmed | Prediction as a basis for skilled reading: insights from modern language models |
| title_short | Prediction as a basis for skilled reading: insights from modern language models |
| title_sort | prediction as a basis for skilled reading insights from modern language models |
| topic | prediction eye movements GPT-2 language models reading |
| url | https://royalsocietypublishing.org/doi/10.1098/rsos.211837 |
| work_keys_str_mv | AT benedettacevoli predictionasabasisforskilledreadinginsightsfrommodernlanguagemodels AT chriswatkins predictionasabasisforskilledreadinginsightsfrommodernlanguagemodels AT kathleenrastle predictionasabasisforskilledreadinginsightsfrommodernlanguagemodels |