Asking Questions about Scientific Articles—Identifying Large N Studies with LLMs
The exponential growth of scientific publications increases the effort required to identify relevant articles. Moreover, the scale of studies is a frequent barrier to research as the majority of studies are low or medium-scaled and do not generalize well while lacking statistical power. As such, we...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-09-01
|
| Series: | Electronics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2079-9292/12/19/3996 |
| _version_ | 1797576017452204032 |
|---|---|
| author | Razvan Paroiu Stefan Ruseti Mihai Dascalu Stefan Trausan-Matu Danielle S. McNamara |
| author_facet | Razvan Paroiu Stefan Ruseti Mihai Dascalu Stefan Trausan-Matu Danielle S. McNamara |
| author_sort | Razvan Paroiu |
| collection | DOAJ |
| description | The exponential growth of scientific publications increases the effort required to identify relevant articles. Moreover, the scale of studies is a frequent barrier to research as the majority of studies are low or medium-scaled and do not generalize well while lacking statistical power. As such, we introduce an automated method that supports the identification of large-scale studies in terms of population. First, we introduce a training corpus of 1229 manually annotated paragraphs extracted from 20 articles with different structures and considered populations. Our method considers prompting a FLAN-T5 language model with targeted questions and paragraphs from the previous corpus so that the model returns the number of participants from the study. We adopt a dialogic extensible approach in which the model is asked a sequence of questions that are gradual in terms of focus. Second, we use a validation corpus with 200 articles labeled for having <i>N</i> larger than 1000 to assess the performance of our language model. Our model, without any preliminary filtering with heuristics, achieves an F1 score of 0.52, surpassing previous analyses performed that obtained an F1 score of 0.51. Moreover, we achieved an F1 score of 0.69 when combined with previous extraction heuristics, thus arguing for the robustness and extensibility of our approach. Finally, we apply our model to a newly introduced dataset of ERIC publications to observe trends across the years in the Education domain. A spike was observed in 2019, followed by a decrease in 2020 and, afterward, a positive trend; nevertheless, the overall percentage is lower than 3%, suggesting a major problem in terms of scale and the need for a change in perspective. |
| first_indexed | 2024-03-10T21:46:32Z |
| format | Article |
| id | doaj.art-5359ec55e22846fd8c30ca8c92d6d7ab |
| institution | Directory Open Access Journal |
| issn | 2079-9292 |
| language | English |
| last_indexed | 2024-03-10T21:46:32Z |
| publishDate | 2023-09-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Electronics |
| spelling | doaj.art-5359ec55e22846fd8c30ca8c92d6d7ab2023-11-19T14:15:45ZengMDPI AGElectronics2079-92922023-09-011219399610.3390/electronics12193996Asking Questions about Scientific Articles—Identifying Large N Studies with LLMsRazvan Paroiu0Stefan Ruseti1Mihai Dascalu2Stefan Trausan-Matu3Danielle S. McNamara4Computer Science and Engineering Department, National University of Science and Technology Politehnica of Bucharest, 313 Splaiul Independentei, 060042 Bucharest, RomaniaComputer Science and Engineering Department, National University of Science and Technology Politehnica of Bucharest, 313 Splaiul Independentei, 060042 Bucharest, RomaniaComputer Science and Engineering Department, National University of Science and Technology Politehnica of Bucharest, 313 Splaiul Independentei, 060042 Bucharest, RomaniaComputer Science and Engineering Department, National University of Science and Technology Politehnica of Bucharest, 313 Splaiul Independentei, 060042 Bucharest, RomaniaDepartment of Psychology, Arizona State University, Tempe, AZ 85287, USAThe exponential growth of scientific publications increases the effort required to identify relevant articles. Moreover, the scale of studies is a frequent barrier to research as the majority of studies are low or medium-scaled and do not generalize well while lacking statistical power. As such, we introduce an automated method that supports the identification of large-scale studies in terms of population. First, we introduce a training corpus of 1229 manually annotated paragraphs extracted from 20 articles with different structures and considered populations. Our method considers prompting a FLAN-T5 language model with targeted questions and paragraphs from the previous corpus so that the model returns the number of participants from the study. We adopt a dialogic extensible approach in which the model is asked a sequence of questions that are gradual in terms of focus. Second, we use a validation corpus with 200 articles labeled for having <i>N</i> larger than 1000 to assess the performance of our language model. Our model, without any preliminary filtering with heuristics, achieves an F1 score of 0.52, surpassing previous analyses performed that obtained an F1 score of 0.51. Moreover, we achieved an F1 score of 0.69 when combined with previous extraction heuristics, thus arguing for the robustness and extensibility of our approach. Finally, we apply our model to a newly introduced dataset of ERIC publications to observe trends across the years in the Education domain. A spike was observed in 2019, followed by a decrease in 2020 and, afterward, a positive trend; nevertheless, the overall percentage is lower than 3%, suggesting a major problem in terms of scale and the need for a change in perspective.https://www.mdpi.com/2079-9292/12/19/3996large language modelsquestion answeringscientific article processing |
| spellingShingle | Razvan Paroiu Stefan Ruseti Mihai Dascalu Stefan Trausan-Matu Danielle S. McNamara Asking Questions about Scientific Articles—Identifying Large N Studies with LLMs Electronics large language models question answering scientific article processing |
| title | Asking Questions about Scientific Articles—Identifying Large N Studies with LLMs |
| title_full | Asking Questions about Scientific Articles—Identifying Large N Studies with LLMs |
| title_fullStr | Asking Questions about Scientific Articles—Identifying Large N Studies with LLMs |
| title_full_unstemmed | Asking Questions about Scientific Articles—Identifying Large N Studies with LLMs |
| title_short | Asking Questions about Scientific Articles—Identifying Large N Studies with LLMs |
| title_sort | asking questions about scientific articles identifying large n studies with llms |
| topic | large language models question answering scientific article processing |
| url | https://www.mdpi.com/2079-9292/12/19/3996 |
| work_keys_str_mv | AT razvanparoiu askingquestionsaboutscientificarticlesidentifyinglargenstudieswithllms AT stefanruseti askingquestionsaboutscientificarticlesidentifyinglargenstudieswithllms AT mihaidascalu askingquestionsaboutscientificarticlesidentifyinglargenstudieswithllms AT stefantrausanmatu askingquestionsaboutscientificarticlesidentifyinglargenstudieswithllms AT daniellesmcnamara askingquestionsaboutscientificarticlesidentifyinglargenstudieswithllms |