Eleven years of student replication projects provide evidence on the correlates of replicability in psychology
Cumulative scientific progress requires empirical results that are robust enough to support theory construction and extension. Yet in psychology, some prominent findings have failed to replicate, and large-scale studies suggest replicability issues are widespread. The identification of predictors of...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
The Royal Society
2023-11-01
|
| Series: | Royal Society Open Science |
| Subjects: | |
| Online Access: | https://royalsocietypublishing.org/doi/10.1098/rsos.231240 |
| _version_ | 1827386608312123392 |
|---|---|
| author | Veronica Boyce Maya Mathur Michael C. Frank |
| author_facet | Veronica Boyce Maya Mathur Michael C. Frank |
| author_sort | Veronica Boyce |
| collection | DOAJ |
| description | Cumulative scientific progress requires empirical results that are robust enough to support theory construction and extension. Yet in psychology, some prominent findings have failed to replicate, and large-scale studies suggest replicability issues are widespread. The identification of predictors of replication success is limited by the difficulty of conducting large samples of independent replication experiments, however: most investigations reanalyse the same set of [Formula: see text]. We introduce a new dataset of 176 replications from students in a graduate-level methods course. Replication results were judged to be successful in 49% of replications; of the 136 where effect sizes could be numerically compared, 46% had point estimates within the prediction interval of the original outcome (versus the expected 95%). Larger original effect sizes and within-participants designs were especially related to replication success. Our results indicate that, consistent with prior reports, the robustness of the psychology literature is low enough to limit cumulative progress by student investigators. |
| first_indexed | 2024-03-08T15:47:53Z |
| format | Article |
| id | doaj.art-96b51a940c244f3e8d2d0794a90d5897 |
| institution | Directory Open Access Journal |
| issn | 2054-5703 |
| language | English |
| last_indexed | 2024-03-08T15:47:53Z |
| publishDate | 2023-11-01 |
| publisher | The Royal Society |
| record_format | Article |
| series | Royal Society Open Science |
| spelling | doaj.art-96b51a940c244f3e8d2d0794a90d58972024-01-09T09:26:53ZengThe Royal SocietyRoyal Society Open Science2054-57032023-11-01101110.1098/rsos.231240Eleven years of student replication projects provide evidence on the correlates of replicability in psychologyVeronica Boyce0Maya Mathur1Michael C. Frank2Department of Psychology, Department of Medicine, Stanford University, Stanford, CA, USAQuantitative Sciences Unit, Department of Medicine, Stanford University, Stanford, CA, USADepartment of Psychology, Department of Medicine, Stanford University, Stanford, CA, USACumulative scientific progress requires empirical results that are robust enough to support theory construction and extension. Yet in psychology, some prominent findings have failed to replicate, and large-scale studies suggest replicability issues are widespread. The identification of predictors of replication success is limited by the difficulty of conducting large samples of independent replication experiments, however: most investigations reanalyse the same set of [Formula: see text]. We introduce a new dataset of 176 replications from students in a graduate-level methods course. Replication results were judged to be successful in 49% of replications; of the 136 where effect sizes could be numerically compared, 46% had point estimates within the prediction interval of the original outcome (versus the expected 95%). Larger original effect sizes and within-participants designs were especially related to replication success. Our results indicate that, consistent with prior reports, the robustness of the psychology literature is low enough to limit cumulative progress by student investigators.https://royalsocietypublishing.org/doi/10.1098/rsos.231240metasciencereplicationlarge-scale replication projectpedagogical replicationsocial psychologycognitive psychology |
| spellingShingle | Veronica Boyce Maya Mathur Michael C. Frank Eleven years of student replication projects provide evidence on the correlates of replicability in psychology Royal Society Open Science metascience replication large-scale replication project pedagogical replication social psychology cognitive psychology |
| title | Eleven years of student replication projects provide evidence on the correlates of replicability in psychology |
| title_full | Eleven years of student replication projects provide evidence on the correlates of replicability in psychology |
| title_fullStr | Eleven years of student replication projects provide evidence on the correlates of replicability in psychology |
| title_full_unstemmed | Eleven years of student replication projects provide evidence on the correlates of replicability in psychology |
| title_short | Eleven years of student replication projects provide evidence on the correlates of replicability in psychology |
| title_sort | eleven years of student replication projects provide evidence on the correlates of replicability in psychology |
| topic | metascience replication large-scale replication project pedagogical replication social psychology cognitive psychology |
| url | https://royalsocietypublishing.org/doi/10.1098/rsos.231240 |
| work_keys_str_mv | AT veronicaboyce elevenyearsofstudentreplicationprojectsprovideevidenceonthecorrelatesofreplicabilityinpsychology AT mayamathur elevenyearsofstudentreplicationprojectsprovideevidenceonthecorrelatesofreplicabilityinpsychology AT michaelcfrank elevenyearsofstudentreplicationprojectsprovideevidenceonthecorrelatesofreplicabilityinpsychology |