Following the science? Comparison of methodological and reporting quality of covid-19 and other research from the first wave of the pandemic
Abstract Background Following the initial identification of the 2019 coronavirus disease (covid-19), the subsequent months saw substantial increases in published biomedical research. Concerns have been raised in both scientific and lay press around the quality of some of this research. We assessed c...
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2021-02-01
|
| Series: | BMC Medicine |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12916-021-01920-x |
| _version_ | 1818409498384007168 |
|---|---|
| author | Terence J. Quinn Jennifer K. Burton Ben Carter Nicola Cooper Kerry Dwan Ryan Field Suzanne C. Freeman Claudia Geue Ping-Hsuan Hsieh Kris McGill Clareece R. Nevill Dikshyanta Rana Alex Sutton Martin Taylor Rowan Yiqiao Xin |
| author_facet | Terence J. Quinn Jennifer K. Burton Ben Carter Nicola Cooper Kerry Dwan Ryan Field Suzanne C. Freeman Claudia Geue Ping-Hsuan Hsieh Kris McGill Clareece R. Nevill Dikshyanta Rana Alex Sutton Martin Taylor Rowan Yiqiao Xin |
| author_sort | Terence J. Quinn |
| collection | DOAJ |
| description | Abstract Background Following the initial identification of the 2019 coronavirus disease (covid-19), the subsequent months saw substantial increases in published biomedical research. Concerns have been raised in both scientific and lay press around the quality of some of this research. We assessed clinical research from major clinical journals, comparing methodological and reporting quality of covid-19 papers published in the first wave (here defined as December 2019 to May 2020 inclusive) of the viral pandemic with non-covid papers published at the same time. Methods We reviewed research publications (print and online) from The BMJ, Journal of the American Medical Association (JAMA), The Lancet, and New England Journal of Medicine, from first publication of a covid-19 research paper (February 2020) to May 2020 inclusive. Paired reviewers were randomly allocated to extract data on methodological quality (risk of bias) and reporting quality (adherence to reporting guidance) from each paper using validated assessment tools. A random 10% of papers were assessed by a third, independent rater. Overall methodological quality for each paper was rated high, low or unclear. Reporting quality was described as percentage of total items reported. Results From 168 research papers, 165 were eligible, including 54 (33%) papers with a covid-19 focus. For methodological quality, 18 (33%) covid-19 papers and 83 (73%) non-covid papers were rated as low risk of bias, OR 6.32 (95%CI 2.85 to 14.00). The difference in quality was maintained after adjusting for publication date, results, funding, study design, journal and raters (OR 6.09 (95%CI 2.09 to 17.72)). For reporting quality, adherence to reporting guidelines was poorer for covid-19 papers, mean percentage of total items reported 72% (95%CI:66 to 77) for covid-19 papers and 84% (95%CI:81 to 87) for non-covid. Conclusions Across various measures, we have demonstrated that covid-19 research from the first wave of the pandemic was potentially of lower quality than contemporaneous non-covid research. While some differences may be an inevitable consequence of conducting research during a viral pandemic, poor reporting should not be accepted. |
| first_indexed | 2024-12-14T10:00:35Z |
| format | Article |
| id | doaj.art-b1161b4c9d0444a4b78a9b73a4b6d7e4 |
| institution | Directory Open Access Journal |
| issn | 1741-7015 |
| language | English |
| last_indexed | 2024-12-14T10:00:35Z |
| publishDate | 2021-02-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medicine |
| spelling | doaj.art-b1161b4c9d0444a4b78a9b73a4b6d7e42022-12-21T23:07:17ZengBMCBMC Medicine1741-70152021-02-0119111010.1186/s12916-021-01920-xFollowing the science? Comparison of methodological and reporting quality of covid-19 and other research from the first wave of the pandemicTerence J. Quinn0Jennifer K. Burton1Ben Carter2Nicola Cooper3Kerry Dwan4Ryan Field5Suzanne C. Freeman6Claudia Geue7Ping-Hsuan Hsieh8Kris McGill9Clareece R. Nevill10Dikshyanta Rana11Alex Sutton12Martin Taylor Rowan13Yiqiao Xin14Institute of Cardiovascular and Medical Sciences, University of GlasgowInstitute of Cardiovascular and Medical Sciences, University of GlasgowInstitute of Psychiatry, Psychology and Neuroscience Kings College LondonDepartment of Health Sciences, University of LeicesterCochrane Methods Support Unit, Cochrane, UKHealth Economics and Health Technology Assessment, University of GlasgowDepartment of Health Sciences, University of LeicesterHealth Economics and Health Technology Assessment, University of GlasgowHealth Economics and Health Technology Assessment, University of GlasgowNMAHP Research Unit, Glasgow Caledonian UniversityDepartment of Health Sciences, University of LeicesterDepartment of Health Sciences, University of LeicesterInstitute of Cardiovascular and Medical Sciences, University of GlasgowInstitute of Cardiovascular and Medical Sciences, University of GlasgowHealth Economics and Health Technology Assessment, University of GlasgowAbstract Background Following the initial identification of the 2019 coronavirus disease (covid-19), the subsequent months saw substantial increases in published biomedical research. Concerns have been raised in both scientific and lay press around the quality of some of this research. We assessed clinical research from major clinical journals, comparing methodological and reporting quality of covid-19 papers published in the first wave (here defined as December 2019 to May 2020 inclusive) of the viral pandemic with non-covid papers published at the same time. Methods We reviewed research publications (print and online) from The BMJ, Journal of the American Medical Association (JAMA), The Lancet, and New England Journal of Medicine, from first publication of a covid-19 research paper (February 2020) to May 2020 inclusive. Paired reviewers were randomly allocated to extract data on methodological quality (risk of bias) and reporting quality (adherence to reporting guidance) from each paper using validated assessment tools. A random 10% of papers were assessed by a third, independent rater. Overall methodological quality for each paper was rated high, low or unclear. Reporting quality was described as percentage of total items reported. Results From 168 research papers, 165 were eligible, including 54 (33%) papers with a covid-19 focus. For methodological quality, 18 (33%) covid-19 papers and 83 (73%) non-covid papers were rated as low risk of bias, OR 6.32 (95%CI 2.85 to 14.00). The difference in quality was maintained after adjusting for publication date, results, funding, study design, journal and raters (OR 6.09 (95%CI 2.09 to 17.72)). For reporting quality, adherence to reporting guidelines was poorer for covid-19 papers, mean percentage of total items reported 72% (95%CI:66 to 77) for covid-19 papers and 84% (95%CI:81 to 87) for non-covid. Conclusions Across various measures, we have demonstrated that covid-19 research from the first wave of the pandemic was potentially of lower quality than contemporaneous non-covid research. While some differences may be an inevitable consequence of conducting research during a viral pandemic, poor reporting should not be accepted.https://doi.org/10.1186/s12916-021-01920-xClinical trialsCOVID-19MethodologyObservational researchPublishingReporting |
| spellingShingle | Terence J. Quinn Jennifer K. Burton Ben Carter Nicola Cooper Kerry Dwan Ryan Field Suzanne C. Freeman Claudia Geue Ping-Hsuan Hsieh Kris McGill Clareece R. Nevill Dikshyanta Rana Alex Sutton Martin Taylor Rowan Yiqiao Xin Following the science? Comparison of methodological and reporting quality of covid-19 and other research from the first wave of the pandemic BMC Medicine Clinical trials COVID-19 Methodology Observational research Publishing Reporting |
| title | Following the science? Comparison of methodological and reporting quality of covid-19 and other research from the first wave of the pandemic |
| title_full | Following the science? Comparison of methodological and reporting quality of covid-19 and other research from the first wave of the pandemic |
| title_fullStr | Following the science? Comparison of methodological and reporting quality of covid-19 and other research from the first wave of the pandemic |
| title_full_unstemmed | Following the science? Comparison of methodological and reporting quality of covid-19 and other research from the first wave of the pandemic |
| title_short | Following the science? Comparison of methodological and reporting quality of covid-19 and other research from the first wave of the pandemic |
| title_sort | following the science comparison of methodological and reporting quality of covid 19 and other research from the first wave of the pandemic |
| topic | Clinical trials COVID-19 Methodology Observational research Publishing Reporting |
| url | https://doi.org/10.1186/s12916-021-01920-x |
| work_keys_str_mv | AT terencejquinn followingthesciencecomparisonofmethodologicalandreportingqualityofcovid19andotherresearchfromthefirstwaveofthepandemic AT jenniferkburton followingthesciencecomparisonofmethodologicalandreportingqualityofcovid19andotherresearchfromthefirstwaveofthepandemic AT bencarter followingthesciencecomparisonofmethodologicalandreportingqualityofcovid19andotherresearchfromthefirstwaveofthepandemic AT nicolacooper followingthesciencecomparisonofmethodologicalandreportingqualityofcovid19andotherresearchfromthefirstwaveofthepandemic AT kerrydwan followingthesciencecomparisonofmethodologicalandreportingqualityofcovid19andotherresearchfromthefirstwaveofthepandemic AT ryanfield followingthesciencecomparisonofmethodologicalandreportingqualityofcovid19andotherresearchfromthefirstwaveofthepandemic AT suzannecfreeman followingthesciencecomparisonofmethodologicalandreportingqualityofcovid19andotherresearchfromthefirstwaveofthepandemic AT claudiageue followingthesciencecomparisonofmethodologicalandreportingqualityofcovid19andotherresearchfromthefirstwaveofthepandemic AT pinghsuanhsieh followingthesciencecomparisonofmethodologicalandreportingqualityofcovid19andotherresearchfromthefirstwaveofthepandemic AT krismcgill followingthesciencecomparisonofmethodologicalandreportingqualityofcovid19andotherresearchfromthefirstwaveofthepandemic AT clareecernevill followingthesciencecomparisonofmethodologicalandreportingqualityofcovid19andotherresearchfromthefirstwaveofthepandemic AT dikshyantarana followingthesciencecomparisonofmethodologicalandreportingqualityofcovid19andotherresearchfromthefirstwaveofthepandemic AT alexsutton followingthesciencecomparisonofmethodologicalandreportingqualityofcovid19andotherresearchfromthefirstwaveofthepandemic AT martintaylorrowan followingthesciencecomparisonofmethodologicalandreportingqualityofcovid19andotherresearchfromthefirstwaveofthepandemic AT yiqiaoxin followingthesciencecomparisonofmethodologicalandreportingqualityofcovid19andotherresearchfromthefirstwaveofthepandemic |