#293 : Trends of Statistical Significance Reporting in Abstracts of Reproductive Medicine Studies from 1990 to 2022
Background and Aims: Despite efforts to minimize publication reporting bias in medical research, it remains unclear whether the magnitude and patterns of the bias have changed over time. We aim to assess the frequency in which the statistical information is presented in forms other than P-values and...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
World Scientific Publishing
2023-12-01
|
| Series: | Fertility & Reproduction |
| Online Access: | https://www.worldscientific.com/doi/10.1142/S2661318223742728 |
| _version_ | 1831878007996809216 |
|---|---|
| author | Qian Feng Wentao Li Ben Mol |
| author_facet | Qian Feng Wentao Li Ben Mol |
| author_sort | Qian Feng |
| collection | DOAJ |
| description | Background and Aims: Despite efforts to minimize publication reporting bias in medical research, it remains unclear whether the magnitude and patterns of the bias have changed over time. We aim to assess the frequency in which the statistical information is presented in forms other than P-values and evaluate how statistical significance was reported in the abstracts of reproductive medicine studies. Method: We studied reproductive medicine studies published in 23 Q1 journals in reproductive medicine, and five top medical journals, including JAMA, Lancet, BMJ, NEJM, and PLOS Medicine. We text-mined abstracts from 1990 to 2022 to extract P-value, confidence interval (CI), and text description in statistical significance; the presence of effect size metrics and Bayes factors were searched. One thousand abstracts were randomly selected and manually checked. The extracted statistical significance information was then analysed for temporal trends and distribution. Results: We identified 24,550 reproductive medicine abstracts. The proportion of abstracts reporting isolated P-values was 13.9% in 1990 and 18.8% in 2021 (Figure 1); reporting isolated P-values was 3.0% (95% CI, 0.5%-11.3%) in meta-analyses, 20.8% (95% CI, 11.3%-34.5%) in randomized controlled trials, and 33.3% (95% CI, 23.8%-44.3%) in basic research in 2021. By contrast, the proportion of abstracts in reporting effect measures without P-value surged from 4.1% to 23.3% from 1990 to 2021. The reported P-values (n = 18,132) mainly clustered at 0.001 (20.0%) and 0.05 (16.2%). Of the 14,689 abstracts containing at least one statement describing statistical significance, 74.9% of abstracts made at least one statistically significant statement; this proportion had merely changed over time (Figure 2). Conclusion: Despite the increasing adoption of effect measures to convey statistical significance, reporting isolated P-values remains common. Publication reporting bias is pervasive and persisting in reproductive medicine abstracts; the inflated treatment could mislead all types of patient care and policy decisions. |
| first_indexed | 2024-04-24T17:13:55Z |
| format | Article |
| id | doaj.art-7d1b6087ce06443abbc3b1602d5c13e8 |
| institution | Directory Open Access Journal |
| issn | 2661-3182 2661-3174 |
| language | English |
| last_indexed | 2025-03-21T21:24:40Z |
| publishDate | 2023-12-01 |
| publisher | World Scientific Publishing |
| record_format | Article |
| series | Fertility & Reproduction |
| spelling | doaj.art-7d1b6087ce06443abbc3b1602d5c13e82024-05-28T17:42:15ZengWorld Scientific PublishingFertility & Reproduction2661-31822661-31742023-12-01050450650710.1142/S2661318223742728#293 : Trends of Statistical Significance Reporting in Abstracts of Reproductive Medicine Studies from 1990 to 2022Qian Feng0Wentao Li1Ben Mol2Department of Obstetrics and Gynaecology, Monash University, Clayton, Melbourne, AustraliaDepartment of Obstetrics and Gynaecology, Monash University, Clayton, Melbourne, AustraliaDepartment of Obstetrics and Gynaecology, Monash University, Clayton, Melbourne, AustraliaBackground and Aims: Despite efforts to minimize publication reporting bias in medical research, it remains unclear whether the magnitude and patterns of the bias have changed over time. We aim to assess the frequency in which the statistical information is presented in forms other than P-values and evaluate how statistical significance was reported in the abstracts of reproductive medicine studies. Method: We studied reproductive medicine studies published in 23 Q1 journals in reproductive medicine, and five top medical journals, including JAMA, Lancet, BMJ, NEJM, and PLOS Medicine. We text-mined abstracts from 1990 to 2022 to extract P-value, confidence interval (CI), and text description in statistical significance; the presence of effect size metrics and Bayes factors were searched. One thousand abstracts were randomly selected and manually checked. The extracted statistical significance information was then analysed for temporal trends and distribution. Results: We identified 24,550 reproductive medicine abstracts. The proportion of abstracts reporting isolated P-values was 13.9% in 1990 and 18.8% in 2021 (Figure 1); reporting isolated P-values was 3.0% (95% CI, 0.5%-11.3%) in meta-analyses, 20.8% (95% CI, 11.3%-34.5%) in randomized controlled trials, and 33.3% (95% CI, 23.8%-44.3%) in basic research in 2021. By contrast, the proportion of abstracts in reporting effect measures without P-value surged from 4.1% to 23.3% from 1990 to 2021. The reported P-values (n = 18,132) mainly clustered at 0.001 (20.0%) and 0.05 (16.2%). Of the 14,689 abstracts containing at least one statement describing statistical significance, 74.9% of abstracts made at least one statistically significant statement; this proportion had merely changed over time (Figure 2). Conclusion: Despite the increasing adoption of effect measures to convey statistical significance, reporting isolated P-values remains common. Publication reporting bias is pervasive and persisting in reproductive medicine abstracts; the inflated treatment could mislead all types of patient care and policy decisions.https://www.worldscientific.com/doi/10.1142/S2661318223742728 |
| spellingShingle | Qian Feng Wentao Li Ben Mol #293 : Trends of Statistical Significance Reporting in Abstracts of Reproductive Medicine Studies from 1990 to 2022 Fertility & Reproduction |
| title | #293 : Trends of Statistical Significance Reporting in Abstracts of Reproductive Medicine Studies from 1990 to 2022 |
| title_full | #293 : Trends of Statistical Significance Reporting in Abstracts of Reproductive Medicine Studies from 1990 to 2022 |
| title_fullStr | #293 : Trends of Statistical Significance Reporting in Abstracts of Reproductive Medicine Studies from 1990 to 2022 |
| title_full_unstemmed | #293 : Trends of Statistical Significance Reporting in Abstracts of Reproductive Medicine Studies from 1990 to 2022 |
| title_short | #293 : Trends of Statistical Significance Reporting in Abstracts of Reproductive Medicine Studies from 1990 to 2022 |
| title_sort | 293 trends of statistical significance reporting in abstracts of reproductive medicine studies from 1990 to 2022 |
| url | https://www.worldscientific.com/doi/10.1142/S2661318223742728 |
| work_keys_str_mv | AT qianfeng 293trendsofstatisticalsignificancereportinginabstractsofreproductivemedicinestudiesfrom1990to2022 AT wentaoli 293trendsofstatisticalsignificancereportinginabstractsofreproductivemedicinestudiesfrom1990to2022 AT benmol 293trendsofstatisticalsignificancereportinginabstractsofreproductivemedicinestudiesfrom1990to2022 |