Accuracy in detecting inadequate research reporting by early career peer reviewers using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process: a cross-sectional diagnostic study
Abstract Background The peer review process has been questioned as it may fail to allow the publication of high-quality articles. This study aimed to evaluate the accuracy in identifying inadequate reporting in RCT reports by early career researchers (ECRs) using an online CONSORT-based peer-review...
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2019-11-01
|
| Series: | BMC Medicine |
| Subjects: | |
| Online Access: | http://link.springer.com/article/10.1186/s12916-019-1436-0 |
| _version_ | 1818229815498506240 |
|---|---|
| author | Anthony Chauvin Philippe Ravaud David Moher David Schriger Sally Hopewell Daniel Shanahan Sabina Alam Gabriel Baron Jean-Philippe Regnaux Perrine Crequit Valeria Martinez Carolina Riveros Laurence Le Cleach Alessandro Recchioni Douglas G. Altman Isabelle Boutron |
| author_facet | Anthony Chauvin Philippe Ravaud David Moher David Schriger Sally Hopewell Daniel Shanahan Sabina Alam Gabriel Baron Jean-Philippe Regnaux Perrine Crequit Valeria Martinez Carolina Riveros Laurence Le Cleach Alessandro Recchioni Douglas G. Altman Isabelle Boutron |
| author_sort | Anthony Chauvin |
| collection | DOAJ |
| description | Abstract Background The peer review process has been questioned as it may fail to allow the publication of high-quality articles. This study aimed to evaluate the accuracy in identifying inadequate reporting in RCT reports by early career researchers (ECRs) using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process. Methods We performed a cross-sectional diagnostic study of 119 manuscripts, from BMC series medical journals, BMJ, BMJ Open, and Annals of Emergency Medicine reporting the results of two-arm parallel-group RCTs. One hundred and nineteen ECRs who had never reviewed an RCT manuscript were recruited from December 2017 to January 2018. Each ECR assessed one manuscript. To assess accuracy in identifying inadequate reporting, we used two tests: (1) ECRs assessing a manuscript using the COBPeer tool (after completing an online training module) and (2) the usual peer-review process. The reference standard was the assessment of the manuscript by two systematic reviewers. Inadequate reporting was defined as incomplete reporting or a switch in primary outcome and considered nine domains: the eight most important CONSORT domains and a switch in primary outcome(s). The primary outcome was the mean number of domains accurately classified (scale from 0 to 9). Results The mean (SD) number of domains (0 to 9) accurately classified per manuscript was 6.39 (1.49) for ECRs using COBPeer versus 5.03 (1.84) for the journal’s usual peer-review process, with a mean difference [95% CI] of 1.36 [0.88–1.84] (p < 0.001). Concerning secondary outcomes, the sensitivity of ECRs using COBPeer versus the usual peer-review process in detecting incompletely reported CONSORT items was 86% [95% CI 82–89] versus 20% [16–24] and in identifying a switch in primary outcome 61% [44–77] versus 11% [3–26]. The specificity of ECRs using COBPeer versus the usual process to detect incompletely reported CONSORT domains was 61% [57–65] versus 77% [74–81] and to identify a switch in primary outcome 77% [67–86] versus 98% [92–100]. Conclusions Trained ECRs using the COBPeer tool were more likely to detect inadequate reporting in RCTs than the usual peer review processes used by journals. Implementing a two-step peer-review process could help improve the quality of reporting. Trial registration Clinical.Trials.gov NCT03119376 (Registered April, 18, 2017). |
| first_indexed | 2024-12-12T10:24:36Z |
| format | Article |
| id | doaj.art-d7405a1c4d0b4b408de5d34f630d9f32 |
| institution | Directory Open Access Journal |
| issn | 1741-7015 |
| language | English |
| last_indexed | 2024-12-12T10:24:36Z |
| publishDate | 2019-11-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medicine |
| spelling | doaj.art-d7405a1c4d0b4b408de5d34f630d9f322022-12-22T00:27:29ZengBMCBMC Medicine1741-70152019-11-0117111210.1186/s12916-019-1436-0Accuracy in detecting inadequate research reporting by early career peer reviewers using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process: a cross-sectional diagnostic studyAnthony Chauvin0Philippe Ravaud1David Moher2David Schriger3Sally Hopewell4Daniel Shanahan5Sabina Alam6Gabriel Baron7Jean-Philippe Regnaux8Perrine Crequit9Valeria Martinez10Carolina Riveros11Laurence Le Cleach12Alessandro Recchioni13Douglas G. Altman14Isabelle Boutron15Université de Paris, CRESS, Inserm, INRAUniversité de Paris, CRESS, Inserm, INRACentre for Journalology, Ottawa Hospital Research Institute, School of Epidemiology and Public Health, University of OttawaDepartment of Emergency Medicine, School of Medicine, University of CaliforniaCentre for Statistics in Medicine, University of OxfordCochrane Central ExecutiveTaylor and Francis groupUniversité de Paris, CRESS, Inserm, INRAUniversité de Paris, CRESS, Inserm, INRAUniversité de Paris, CRESS, Inserm, INRADepartment of Anesthesiology, Assistance Publique Hôpitaux de Paris, Hôpital Raymond PoincaréUniversité de Paris, CRESS, Inserm, INRAService de Dermatologie, Hopital Mondor, Assistance Publique des Hôpitaux de Paris, Université Paris Est Creteil, EpidermEBMC MedicineCentre for Statistics in Medicine, University of OxfordUniversité de Paris, CRESS, Inserm, INRAAbstract Background The peer review process has been questioned as it may fail to allow the publication of high-quality articles. This study aimed to evaluate the accuracy in identifying inadequate reporting in RCT reports by early career researchers (ECRs) using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process. Methods We performed a cross-sectional diagnostic study of 119 manuscripts, from BMC series medical journals, BMJ, BMJ Open, and Annals of Emergency Medicine reporting the results of two-arm parallel-group RCTs. One hundred and nineteen ECRs who had never reviewed an RCT manuscript were recruited from December 2017 to January 2018. Each ECR assessed one manuscript. To assess accuracy in identifying inadequate reporting, we used two tests: (1) ECRs assessing a manuscript using the COBPeer tool (after completing an online training module) and (2) the usual peer-review process. The reference standard was the assessment of the manuscript by two systematic reviewers. Inadequate reporting was defined as incomplete reporting or a switch in primary outcome and considered nine domains: the eight most important CONSORT domains and a switch in primary outcome(s). The primary outcome was the mean number of domains accurately classified (scale from 0 to 9). Results The mean (SD) number of domains (0 to 9) accurately classified per manuscript was 6.39 (1.49) for ECRs using COBPeer versus 5.03 (1.84) for the journal’s usual peer-review process, with a mean difference [95% CI] of 1.36 [0.88–1.84] (p < 0.001). Concerning secondary outcomes, the sensitivity of ECRs using COBPeer versus the usual peer-review process in detecting incompletely reported CONSORT items was 86% [95% CI 82–89] versus 20% [16–24] and in identifying a switch in primary outcome 61% [44–77] versus 11% [3–26]. The specificity of ECRs using COBPeer versus the usual process to detect incompletely reported CONSORT domains was 61% [57–65] versus 77% [74–81] and to identify a switch in primary outcome 77% [67–86] versus 98% [92–100]. Conclusions Trained ECRs using the COBPeer tool were more likely to detect inadequate reporting in RCTs than the usual peer review processes used by journals. Implementing a two-step peer-review process could help improve the quality of reporting. Trial registration Clinical.Trials.gov NCT03119376 (Registered April, 18, 2017).http://link.springer.com/article/10.1186/s12916-019-1436-0Peer reviewersRandomized controlled trialsReportingCONSORT statement |
| spellingShingle | Anthony Chauvin Philippe Ravaud David Moher David Schriger Sally Hopewell Daniel Shanahan Sabina Alam Gabriel Baron Jean-Philippe Regnaux Perrine Crequit Valeria Martinez Carolina Riveros Laurence Le Cleach Alessandro Recchioni Douglas G. Altman Isabelle Boutron Accuracy in detecting inadequate research reporting by early career peer reviewers using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process: a cross-sectional diagnostic study BMC Medicine Peer reviewers Randomized controlled trials Reporting CONSORT statement |
| title | Accuracy in detecting inadequate research reporting by early career peer reviewers using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process: a cross-sectional diagnostic study |
| title_full | Accuracy in detecting inadequate research reporting by early career peer reviewers using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process: a cross-sectional diagnostic study |
| title_fullStr | Accuracy in detecting inadequate research reporting by early career peer reviewers using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process: a cross-sectional diagnostic study |
| title_full_unstemmed | Accuracy in detecting inadequate research reporting by early career peer reviewers using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process: a cross-sectional diagnostic study |
| title_short | Accuracy in detecting inadequate research reporting by early career peer reviewers using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process: a cross-sectional diagnostic study |
| title_sort | accuracy in detecting inadequate research reporting by early career peer reviewers using an online consort based peer review tool cobpeer versus the usual peer review process a cross sectional diagnostic study |
| topic | Peer reviewers Randomized controlled trials Reporting CONSORT statement |
| url | http://link.springer.com/article/10.1186/s12916-019-1436-0 |
| work_keys_str_mv | AT anthonychauvin accuracyindetectinginadequateresearchreportingbyearlycareerpeerreviewersusinganonlineconsortbasedpeerreviewtoolcobpeerversustheusualpeerreviewprocessacrosssectionaldiagnosticstudy AT philipperavaud accuracyindetectinginadequateresearchreportingbyearlycareerpeerreviewersusinganonlineconsortbasedpeerreviewtoolcobpeerversustheusualpeerreviewprocessacrosssectionaldiagnosticstudy AT davidmoher accuracyindetectinginadequateresearchreportingbyearlycareerpeerreviewersusinganonlineconsortbasedpeerreviewtoolcobpeerversustheusualpeerreviewprocessacrosssectionaldiagnosticstudy AT davidschriger accuracyindetectinginadequateresearchreportingbyearlycareerpeerreviewersusinganonlineconsortbasedpeerreviewtoolcobpeerversustheusualpeerreviewprocessacrosssectionaldiagnosticstudy AT sallyhopewell accuracyindetectinginadequateresearchreportingbyearlycareerpeerreviewersusinganonlineconsortbasedpeerreviewtoolcobpeerversustheusualpeerreviewprocessacrosssectionaldiagnosticstudy AT danielshanahan accuracyindetectinginadequateresearchreportingbyearlycareerpeerreviewersusinganonlineconsortbasedpeerreviewtoolcobpeerversustheusualpeerreviewprocessacrosssectionaldiagnosticstudy AT sabinaalam accuracyindetectinginadequateresearchreportingbyearlycareerpeerreviewersusinganonlineconsortbasedpeerreviewtoolcobpeerversustheusualpeerreviewprocessacrosssectionaldiagnosticstudy AT gabrielbaron accuracyindetectinginadequateresearchreportingbyearlycareerpeerreviewersusinganonlineconsortbasedpeerreviewtoolcobpeerversustheusualpeerreviewprocessacrosssectionaldiagnosticstudy AT jeanphilipperegnaux accuracyindetectinginadequateresearchreportingbyearlycareerpeerreviewersusinganonlineconsortbasedpeerreviewtoolcobpeerversustheusualpeerreviewprocessacrosssectionaldiagnosticstudy AT perrinecrequit accuracyindetectinginadequateresearchreportingbyearlycareerpeerreviewersusinganonlineconsortbasedpeerreviewtoolcobpeerversustheusualpeerreviewprocessacrosssectionaldiagnosticstudy AT valeriamartinez accuracyindetectinginadequateresearchreportingbyearlycareerpeerreviewersusinganonlineconsortbasedpeerreviewtoolcobpeerversustheusualpeerreviewprocessacrosssectionaldiagnosticstudy AT carolinariveros accuracyindetectinginadequateresearchreportingbyearlycareerpeerreviewersusinganonlineconsortbasedpeerreviewtoolcobpeerversustheusualpeerreviewprocessacrosssectionaldiagnosticstudy AT laurencelecleach accuracyindetectinginadequateresearchreportingbyearlycareerpeerreviewersusinganonlineconsortbasedpeerreviewtoolcobpeerversustheusualpeerreviewprocessacrosssectionaldiagnosticstudy AT alessandrorecchioni accuracyindetectinginadequateresearchreportingbyearlycareerpeerreviewersusinganonlineconsortbasedpeerreviewtoolcobpeerversustheusualpeerreviewprocessacrosssectionaldiagnosticstudy AT douglasgaltman accuracyindetectinginadequateresearchreportingbyearlycareerpeerreviewersusinganonlineconsortbasedpeerreviewtoolcobpeerversustheusualpeerreviewprocessacrosssectionaldiagnosticstudy AT isabelleboutron accuracyindetectinginadequateresearchreportingbyearlycareerpeerreviewersusinganonlineconsortbasedpeerreviewtoolcobpeerversustheusualpeerreviewprocessacrosssectionaldiagnosticstudy |