A Bayesian bird's eye view of ‘Replications of important results in social psychology’
We applied three Bayesian methods to reanalyse the preregistered contributions to the Social Psychology special issue ‘Replications of Important Results in Social Psychology’ (Nosek & Lakens. 2014 Registered reports: a method to increase the credibility of published results. Soc. Psychol. 45, 13...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
The Royal Society
2017-01-01
|
| Series: | Royal Society Open Science |
| Subjects: | |
| Online Access: | https://royalsocietypublishing.org/doi/pdf/10.1098/rsos.160426 |
| _version_ | 1828762241666646016 |
|---|---|
| author | Maarten Marsman Felix D. Schönbrodt Richard D. Morey Yuling Yao Andrew Gelman Eric-Jan Wagenmakers |
| author_facet | Maarten Marsman Felix D. Schönbrodt Richard D. Morey Yuling Yao Andrew Gelman Eric-Jan Wagenmakers |
| author_sort | Maarten Marsman |
| collection | DOAJ |
| description | We applied three Bayesian methods to reanalyse the preregistered contributions to the Social Psychology special issue ‘Replications of Important Results in Social Psychology’ (Nosek & Lakens. 2014 Registered reports: a method to increase the credibility of published results. Soc. Psychol. 45, 137–141. (doi:10.1027/1864-9335/a000192)). First, individual-experiment Bayesian parameter estimation revealed that for directed effect size measures, only three out of 44 central 95% credible intervals did not overlap with zero and fell in the expected direction. For undirected effect size measures, only four out of 59 credible intervals contained values greater than 0.10 (10% of variance explained) and only 19 intervals contained values larger than 0.05. Second, a Bayesian random-effects meta-analysis for all 38 t-tests showed that only one out of the 38 hierarchically estimated credible intervals did not overlap with zero and fell in the expected direction. Third, a Bayes factor hypothesis test was used to quantify the evidence for the null hypothesis against a default one-sided alternative. Only seven out of 60 Bayes factors indicated non-anecdotal support in favour of the alternative hypothesis (BF10>3), whereas 51 Bayes factors indicated at least some support for the null hypothesis. We hope that future analyses of replication success will embrace a more inclusive statistical approach by adopting a wider range of complementary techniques. |
| first_indexed | 2024-12-11T01:42:28Z |
| format | Article |
| id | doaj.art-8b5ce70fd30440968b919df8b630fd05 |
| institution | Directory Open Access Journal |
| issn | 2054-5703 |
| language | English |
| last_indexed | 2024-12-11T01:42:28Z |
| publishDate | 2017-01-01 |
| publisher | The Royal Society |
| record_format | Article |
| series | Royal Society Open Science |
| spelling | doaj.art-8b5ce70fd30440968b919df8b630fd052022-12-22T01:25:00ZengThe Royal SocietyRoyal Society Open Science2054-57032017-01-014110.1098/rsos.160426160426A Bayesian bird's eye view of ‘Replications of important results in social psychology’Maarten MarsmanFelix D. SchönbrodtRichard D. MoreyYuling YaoAndrew GelmanEric-Jan WagenmakersWe applied three Bayesian methods to reanalyse the preregistered contributions to the Social Psychology special issue ‘Replications of Important Results in Social Psychology’ (Nosek & Lakens. 2014 Registered reports: a method to increase the credibility of published results. Soc. Psychol. 45, 137–141. (doi:10.1027/1864-9335/a000192)). First, individual-experiment Bayesian parameter estimation revealed that for directed effect size measures, only three out of 44 central 95% credible intervals did not overlap with zero and fell in the expected direction. For undirected effect size measures, only four out of 59 credible intervals contained values greater than 0.10 (10% of variance explained) and only 19 intervals contained values larger than 0.05. Second, a Bayesian random-effects meta-analysis for all 38 t-tests showed that only one out of the 38 hierarchically estimated credible intervals did not overlap with zero and fell in the expected direction. Third, a Bayes factor hypothesis test was used to quantify the evidence for the null hypothesis against a default one-sided alternative. Only seven out of 60 Bayes factors indicated non-anecdotal support in favour of the alternative hypothesis (BF10>3), whereas 51 Bayes factors indicated at least some support for the null hypothesis. We hope that future analyses of replication success will embrace a more inclusive statistical approach by adopting a wider range of complementary techniques.https://royalsocietypublishing.org/doi/pdf/10.1098/rsos.160426preregistrationevidencereproducibilitycredible intervalbayes factor |
| spellingShingle | Maarten Marsman Felix D. Schönbrodt Richard D. Morey Yuling Yao Andrew Gelman Eric-Jan Wagenmakers A Bayesian bird's eye view of ‘Replications of important results in social psychology’ Royal Society Open Science preregistration evidence reproducibility credible interval bayes factor |
| title | A Bayesian bird's eye view of ‘Replications of important results in social psychology’ |
| title_full | A Bayesian bird's eye view of ‘Replications of important results in social psychology’ |
| title_fullStr | A Bayesian bird's eye view of ‘Replications of important results in social psychology’ |
| title_full_unstemmed | A Bayesian bird's eye view of ‘Replications of important results in social psychology’ |
| title_short | A Bayesian bird's eye view of ‘Replications of important results in social psychology’ |
| title_sort | bayesian bird s eye view of replications of important results in social psychology |
| topic | preregistration evidence reproducibility credible interval bayes factor |
| url | https://royalsocietypublishing.org/doi/pdf/10.1098/rsos.160426 |
| work_keys_str_mv | AT maartenmarsman abayesianbirdseyeviewofreplicationsofimportantresultsinsocialpsychology AT felixdschonbrodt abayesianbirdseyeviewofreplicationsofimportantresultsinsocialpsychology AT richarddmorey abayesianbirdseyeviewofreplicationsofimportantresultsinsocialpsychology AT yulingyao abayesianbirdseyeviewofreplicationsofimportantresultsinsocialpsychology AT andrewgelman abayesianbirdseyeviewofreplicationsofimportantresultsinsocialpsychology AT ericjanwagenmakers abayesianbirdseyeviewofreplicationsofimportantresultsinsocialpsychology AT maartenmarsman bayesianbirdseyeviewofreplicationsofimportantresultsinsocialpsychology AT felixdschonbrodt bayesianbirdseyeviewofreplicationsofimportantresultsinsocialpsychology AT richarddmorey bayesianbirdseyeviewofreplicationsofimportantresultsinsocialpsychology AT yulingyao bayesianbirdseyeviewofreplicationsofimportantresultsinsocialpsychology AT andrewgelman bayesianbirdseyeviewofreplicationsofimportantresultsinsocialpsychology AT ericjanwagenmakers bayesianbirdseyeviewofreplicationsofimportantresultsinsocialpsychology |