Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition
Access to data is a critical feature of an efficient, progressive and ultimately self-correcting scientific ecosystem. But the extent to which in-principle benefits of data sharing are realized in practice is unclear. Crucially, it is largely unknown whether published findings can be reproduced by r...
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
The Royal Society
2018-01-01
|
| Series: | Royal Society Open Science |
| Subjects: | |
| Online Access: | https://royalsocietypublishing.org/doi/pdf/10.1098/rsos.180448 |
| _version_ | 1818991810425389056 |
|---|---|
| author | Tom E. Hardwicke Maya B. Mathur Kyle MacDonald Gustav Nilsonne George C. Banks Mallory C. Kidwell Alicia Hofelich Mohr Elizabeth Clayton Erica J. Yoon Michael Henry Tessler Richie L. Lenne Sara Altman Bria Long Michael C. Frank |
| author_facet | Tom E. Hardwicke Maya B. Mathur Kyle MacDonald Gustav Nilsonne George C. Banks Mallory C. Kidwell Alicia Hofelich Mohr Elizabeth Clayton Erica J. Yoon Michael Henry Tessler Richie L. Lenne Sara Altman Bria Long Michael C. Frank |
| author_sort | Tom E. Hardwicke |
| collection | DOAJ |
| description | Access to data is a critical feature of an efficient, progressive and ultimately self-correcting scientific ecosystem. But the extent to which in-principle benefits of data sharing are realized in practice is unclear. Crucially, it is largely unknown whether published findings can be reproduced by repeating reported analyses upon shared data (‘analytic reproducibility’). To investigate this, we conducted an observational evaluation of a mandatory open data policy introduced at the journal Cognition. Interrupted time-series analyses indicated a substantial post-policy increase in data available statements (104/417, 25% pre-policy to 136/174, 78% post-policy), although not all data appeared reusable (23/104, 22% pre-policy to 85/136, 62%, post-policy). For 35 of the articles determined to have reusable data, we attempted to reproduce 1324 target values. Ultimately, 64 values could not be reproduced within a 10% margin of error. For 22 articles all target values were reproduced, but 11 of these required author assistance. For 13 articles at least one value could not be reproduced despite author assistance. Importantly, there were no clear indications that original conclusions were seriously impacted. Mandatory open data policies can increase the frequency and quality of data sharing. However, suboptimal data curation, unclear analysis specification and reporting errors can impede analytic reproducibility, undermining the utility of data sharing and the credibility of scientific findings. |
| first_indexed | 2024-12-20T20:16:11Z |
| format | Article |
| id | doaj.art-9159a399ecf14bc3a5b4b2656e011149 |
| institution | Directory Open Access Journal |
| issn | 2054-5703 |
| language | English |
| last_indexed | 2024-12-20T20:16:11Z |
| publishDate | 2018-01-01 |
| publisher | The Royal Society |
| record_format | Article |
| series | Royal Society Open Science |
| spelling | doaj.art-9159a399ecf14bc3a5b4b2656e0111492022-12-21T19:27:43ZengThe Royal SocietyRoyal Society Open Science2054-57032018-01-015810.1098/rsos.180448180448Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal CognitionTom E. HardwickeMaya B. MathurKyle MacDonaldGustav NilsonneGeorge C. BanksMallory C. KidwellAlicia Hofelich MohrElizabeth ClaytonErica J. YoonMichael Henry TesslerRichie L. LenneSara AltmanBria LongMichael C. FrankAccess to data is a critical feature of an efficient, progressive and ultimately self-correcting scientific ecosystem. But the extent to which in-principle benefits of data sharing are realized in practice is unclear. Crucially, it is largely unknown whether published findings can be reproduced by repeating reported analyses upon shared data (‘analytic reproducibility’). To investigate this, we conducted an observational evaluation of a mandatory open data policy introduced at the journal Cognition. Interrupted time-series analyses indicated a substantial post-policy increase in data available statements (104/417, 25% pre-policy to 136/174, 78% post-policy), although not all data appeared reusable (23/104, 22% pre-policy to 85/136, 62%, post-policy). For 35 of the articles determined to have reusable data, we attempted to reproduce 1324 target values. Ultimately, 64 values could not be reproduced within a 10% margin of error. For 22 articles all target values were reproduced, but 11 of these required author assistance. For 13 articles at least one value could not be reproduced despite author assistance. Importantly, there were no clear indications that original conclusions were seriously impacted. Mandatory open data policies can increase the frequency and quality of data sharing. However, suboptimal data curation, unclear analysis specification and reporting errors can impede analytic reproducibility, undermining the utility of data sharing and the credibility of scientific findings.https://royalsocietypublishing.org/doi/pdf/10.1098/rsos.180448open datareproducibilityopen sciencemeta-scienceinterrupted time seriesjournal policy |
| spellingShingle | Tom E. Hardwicke Maya B. Mathur Kyle MacDonald Gustav Nilsonne George C. Banks Mallory C. Kidwell Alicia Hofelich Mohr Elizabeth Clayton Erica J. Yoon Michael Henry Tessler Richie L. Lenne Sara Altman Bria Long Michael C. Frank Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition Royal Society Open Science open data reproducibility open science meta-science interrupted time series journal policy |
| title | Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition |
| title_full | Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition |
| title_fullStr | Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition |
| title_full_unstemmed | Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition |
| title_short | Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition |
| title_sort | data availability reusability and analytic reproducibility evaluating the impact of a mandatory open data policy at the journal cognition |
| topic | open data reproducibility open science meta-science interrupted time series journal policy |
| url | https://royalsocietypublishing.org/doi/pdf/10.1098/rsos.180448 |
| work_keys_str_mv | AT tomehardwicke dataavailabilityreusabilityandanalyticreproducibilityevaluatingtheimpactofamandatoryopendatapolicyatthejournalcognition AT mayabmathur dataavailabilityreusabilityandanalyticreproducibilityevaluatingtheimpactofamandatoryopendatapolicyatthejournalcognition AT kylemacdonald dataavailabilityreusabilityandanalyticreproducibilityevaluatingtheimpactofamandatoryopendatapolicyatthejournalcognition AT gustavnilsonne dataavailabilityreusabilityandanalyticreproducibilityevaluatingtheimpactofamandatoryopendatapolicyatthejournalcognition AT georgecbanks dataavailabilityreusabilityandanalyticreproducibilityevaluatingtheimpactofamandatoryopendatapolicyatthejournalcognition AT malloryckidwell dataavailabilityreusabilityandanalyticreproducibilityevaluatingtheimpactofamandatoryopendatapolicyatthejournalcognition AT aliciahofelichmohr dataavailabilityreusabilityandanalyticreproducibilityevaluatingtheimpactofamandatoryopendatapolicyatthejournalcognition AT elizabethclayton dataavailabilityreusabilityandanalyticreproducibilityevaluatingtheimpactofamandatoryopendatapolicyatthejournalcognition AT ericajyoon dataavailabilityreusabilityandanalyticreproducibilityevaluatingtheimpactofamandatoryopendatapolicyatthejournalcognition AT michaelhenrytessler dataavailabilityreusabilityandanalyticreproducibilityevaluatingtheimpactofamandatoryopendatapolicyatthejournalcognition AT richiellenne dataavailabilityreusabilityandanalyticreproducibilityevaluatingtheimpactofamandatoryopendatapolicyatthejournalcognition AT saraaltman dataavailabilityreusabilityandanalyticreproducibilityevaluatingtheimpactofamandatoryopendatapolicyatthejournalcognition AT brialong dataavailabilityreusabilityandanalyticreproducibilityevaluatingtheimpactofamandatoryopendatapolicyatthejournalcognition AT michaelcfrank dataavailabilityreusabilityandanalyticreproducibilityevaluatingtheimpactofamandatoryopendatapolicyatthejournalcognition |