Analytic reproducibility in articles receiving open data badges at the journal Psychological Science: an observational study
For any scientific report, repeating the original analyses upon the original data should yield the original outcomes. We evaluated analytic reproducibility in 25 Psychological Science articles awarded open data badges between 2014 and 2015. Initially, 16 (64%, 95% confidence interval [43,81]) articl...
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
The Royal Society
2021-01-01
|
| Series: | Royal Society Open Science |
| Subjects: | |
| Online Access: | https://royalsocietypublishing.org/doi/pdf/10.1098/rsos.201494 |
| _version_ | 1818728551650689024 |
|---|---|
| author | Tom E. Hardwicke Manuel Bohn Kyle MacDonald Emily Hembacher Michèle B. Nuijten Benjamin N. Peloquin Benjamin E. deMayo Bria Long Erica J. Yoon Michael C. Frank |
| author_facet | Tom E. Hardwicke Manuel Bohn Kyle MacDonald Emily Hembacher Michèle B. Nuijten Benjamin N. Peloquin Benjamin E. deMayo Bria Long Erica J. Yoon Michael C. Frank |
| author_sort | Tom E. Hardwicke |
| collection | DOAJ |
| description | For any scientific report, repeating the original analyses upon the original data should yield the original outcomes. We evaluated analytic reproducibility in 25 Psychological Science articles awarded open data badges between 2014 and 2015. Initially, 16 (64%, 95% confidence interval [43,81]) articles contained at least one ‘major numerical discrepancy' (>10% difference) prompting us to request input from original authors. Ultimately, target values were reproducible without author involvement for 9 (36% [20,59]) articles; reproducible with author involvement for 6 (24% [8,47]) articles; not fully reproducible with no substantive author response for 3 (12% [0,35]) articles; and not fully reproducible despite author involvement for 7 (28% [12,51]) articles. Overall, 37 major numerical discrepancies remained out of 789 checked values (5% [3,6]), but original conclusions did not appear affected. Non-reproducibility was primarily caused by unclear reporting of analytic procedures. These results highlight that open data alone is not sufficient to ensure analytic reproducibility. |
| first_indexed | 2024-12-17T22:31:48Z |
| format | Article |
| id | doaj.art-a335bf47676946b08708f32f6cfd059d |
| institution | Directory Open Access Journal |
| issn | 2054-5703 |
| language | English |
| last_indexed | 2024-12-17T22:31:48Z |
| publishDate | 2021-01-01 |
| publisher | The Royal Society |
| record_format | Article |
| series | Royal Society Open Science |
| spelling | doaj.art-a335bf47676946b08708f32f6cfd059d2022-12-21T21:30:11ZengThe Royal SocietyRoyal Society Open Science2054-57032021-01-018110.1098/rsos.201494201494Analytic reproducibility in articles receiving open data badges at the journal Psychological Science: an observational studyTom E. HardwickeManuel BohnKyle MacDonaldEmily HembacherMichèle B. NuijtenBenjamin N. PeloquinBenjamin E. deMayoBria LongErica J. YoonMichael C. FrankFor any scientific report, repeating the original analyses upon the original data should yield the original outcomes. We evaluated analytic reproducibility in 25 Psychological Science articles awarded open data badges between 2014 and 2015. Initially, 16 (64%, 95% confidence interval [43,81]) articles contained at least one ‘major numerical discrepancy' (>10% difference) prompting us to request input from original authors. Ultimately, target values were reproducible without author involvement for 9 (36% [20,59]) articles; reproducible with author involvement for 6 (24% [8,47]) articles; not fully reproducible with no substantive author response for 3 (12% [0,35]) articles; and not fully reproducible despite author involvement for 7 (28% [12,51]) articles. Overall, 37 major numerical discrepancies remained out of 789 checked values (5% [3,6]), but original conclusions did not appear affected. Non-reproducibility was primarily caused by unclear reporting of analytic procedures. These results highlight that open data alone is not sufficient to ensure analytic reproducibility.https://royalsocietypublishing.org/doi/pdf/10.1098/rsos.201494open dataopen badgesreproducibilityopen sciencemeta-researchjournal policy |
| spellingShingle | Tom E. Hardwicke Manuel Bohn Kyle MacDonald Emily Hembacher Michèle B. Nuijten Benjamin N. Peloquin Benjamin E. deMayo Bria Long Erica J. Yoon Michael C. Frank Analytic reproducibility in articles receiving open data badges at the journal Psychological Science: an observational study Royal Society Open Science open data open badges reproducibility open science meta-research journal policy |
| title | Analytic reproducibility in articles receiving open data badges at the journal Psychological Science: an observational study |
| title_full | Analytic reproducibility in articles receiving open data badges at the journal Psychological Science: an observational study |
| title_fullStr | Analytic reproducibility in articles receiving open data badges at the journal Psychological Science: an observational study |
| title_full_unstemmed | Analytic reproducibility in articles receiving open data badges at the journal Psychological Science: an observational study |
| title_short | Analytic reproducibility in articles receiving open data badges at the journal Psychological Science: an observational study |
| title_sort | analytic reproducibility in articles receiving open data badges at the journal psychological science an observational study |
| topic | open data open badges reproducibility open science meta-research journal policy |
| url | https://royalsocietypublishing.org/doi/pdf/10.1098/rsos.201494 |
| work_keys_str_mv | AT tomehardwicke analyticreproducibilityinarticlesreceivingopendatabadgesatthejournalpsychologicalscienceanobservationalstudy AT manuelbohn analyticreproducibilityinarticlesreceivingopendatabadgesatthejournalpsychologicalscienceanobservationalstudy AT kylemacdonald analyticreproducibilityinarticlesreceivingopendatabadgesatthejournalpsychologicalscienceanobservationalstudy AT emilyhembacher analyticreproducibilityinarticlesreceivingopendatabadgesatthejournalpsychologicalscienceanobservationalstudy AT michelebnuijten analyticreproducibilityinarticlesreceivingopendatabadgesatthejournalpsychologicalscienceanobservationalstudy AT benjaminnpeloquin analyticreproducibilityinarticlesreceivingopendatabadgesatthejournalpsychologicalscienceanobservationalstudy AT benjaminedemayo analyticreproducibilityinarticlesreceivingopendatabadgesatthejournalpsychologicalscienceanobservationalstudy AT brialong analyticreproducibilityinarticlesreceivingopendatabadgesatthejournalpsychologicalscienceanobservationalstudy AT ericajyoon analyticreproducibilityinarticlesreceivingopendatabadgesatthejournalpsychologicalscienceanobservationalstudy AT michaelcfrank analyticreproducibilityinarticlesreceivingopendatabadgesatthejournalpsychologicalscienceanobservationalstudy |