Ensuring the quality and specificity of preregistrations.
Researchers face many, often seemingly arbitrary, choices in formulating hypotheses, designing protocols, collecting data, analyzing data, and reporting results. Opportunistic use of "researcher degrees of freedom" aimed at obtaining statistical significance increases the likelihood of obt...
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science (PLoS)
2020-12-01
|
| Series: | PLoS Biology |
| Online Access: | https://doi.org/10.1371/journal.pbio.3000937 |
| _version_ | 1819011309312671744 |
|---|---|
| author | Marjan Bakker Coosje L S Veldkamp Marcel A L M van Assen Elise A V Crompvoets How Hwee Ong Brian A Nosek Courtney K Soderberg David Mellor Jelte M Wicherts |
| author_facet | Marjan Bakker Coosje L S Veldkamp Marcel A L M van Assen Elise A V Crompvoets How Hwee Ong Brian A Nosek Courtney K Soderberg David Mellor Jelte M Wicherts |
| author_sort | Marjan Bakker |
| collection | DOAJ |
| description | Researchers face many, often seemingly arbitrary, choices in formulating hypotheses, designing protocols, collecting data, analyzing data, and reporting results. Opportunistic use of "researcher degrees of freedom" aimed at obtaining statistical significance increases the likelihood of obtaining and publishing false-positive results and overestimated effect sizes. Preregistration is a mechanism for reducing such degrees of freedom by specifying designs and analysis plans before observing the research outcomes. The effectiveness of preregistration may depend, in part, on whether the process facilitates sufficiently specific articulation of such plans. In this preregistered study, we compared 2 formats of preregistration available on the OSF: Standard Pre-Data Collection Registration and Prereg Challenge Registration (now called "OSF Preregistration," http://osf.io/prereg/). The Prereg Challenge format was a "structured" workflow with detailed instructions and an independent review to confirm completeness; the "Standard" format was "unstructured" with minimal direct guidance to give researchers flexibility for what to prespecify. Results of comparing random samples of 53 preregistrations from each format indicate that the "structured" format restricted the opportunistic use of researcher degrees of freedom better (Cliff's Delta = 0.49) than the "unstructured" format, but neither eliminated all researcher degrees of freedom. We also observed very low concordance among coders about the number of hypotheses (14%), indicating that they are often not clearly stated. We conclude that effective preregistration is challenging, and registration formats that provide effective guidance may improve the quality of research. |
| first_indexed | 2024-12-21T01:26:06Z |
| format | Article |
| id | doaj.art-3217826dd20f48eb893d886d6934e2fa |
| institution | Directory Open Access Journal |
| issn | 1544-9173 1545-7885 |
| language | English |
| last_indexed | 2024-12-21T01:26:06Z |
| publishDate | 2020-12-01 |
| publisher | Public Library of Science (PLoS) |
| record_format | Article |
| series | PLoS Biology |
| spelling | doaj.art-3217826dd20f48eb893d886d6934e2fa2022-12-21T19:20:30ZengPublic Library of Science (PLoS)PLoS Biology1544-91731545-78852020-12-011812e300093710.1371/journal.pbio.3000937Ensuring the quality and specificity of preregistrations.Marjan BakkerCoosje L S VeldkampMarcel A L M van AssenElise A V CrompvoetsHow Hwee OngBrian A NosekCourtney K SoderbergDavid MellorJelte M WichertsResearchers face many, often seemingly arbitrary, choices in formulating hypotheses, designing protocols, collecting data, analyzing data, and reporting results. Opportunistic use of "researcher degrees of freedom" aimed at obtaining statistical significance increases the likelihood of obtaining and publishing false-positive results and overestimated effect sizes. Preregistration is a mechanism for reducing such degrees of freedom by specifying designs and analysis plans before observing the research outcomes. The effectiveness of preregistration may depend, in part, on whether the process facilitates sufficiently specific articulation of such plans. In this preregistered study, we compared 2 formats of preregistration available on the OSF: Standard Pre-Data Collection Registration and Prereg Challenge Registration (now called "OSF Preregistration," http://osf.io/prereg/). The Prereg Challenge format was a "structured" workflow with detailed instructions and an independent review to confirm completeness; the "Standard" format was "unstructured" with minimal direct guidance to give researchers flexibility for what to prespecify. Results of comparing random samples of 53 preregistrations from each format indicate that the "structured" format restricted the opportunistic use of researcher degrees of freedom better (Cliff's Delta = 0.49) than the "unstructured" format, but neither eliminated all researcher degrees of freedom. We also observed very low concordance among coders about the number of hypotheses (14%), indicating that they are often not clearly stated. We conclude that effective preregistration is challenging, and registration formats that provide effective guidance may improve the quality of research.https://doi.org/10.1371/journal.pbio.3000937 |
| spellingShingle | Marjan Bakker Coosje L S Veldkamp Marcel A L M van Assen Elise A V Crompvoets How Hwee Ong Brian A Nosek Courtney K Soderberg David Mellor Jelte M Wicherts Ensuring the quality and specificity of preregistrations. PLoS Biology |
| title | Ensuring the quality and specificity of preregistrations. |
| title_full | Ensuring the quality and specificity of preregistrations. |
| title_fullStr | Ensuring the quality and specificity of preregistrations. |
| title_full_unstemmed | Ensuring the quality and specificity of preregistrations. |
| title_short | Ensuring the quality and specificity of preregistrations. |
| title_sort | ensuring the quality and specificity of preregistrations |
| url | https://doi.org/10.1371/journal.pbio.3000937 |
| work_keys_str_mv | AT marjanbakker ensuringthequalityandspecificityofpreregistrations AT coosjelsveldkamp ensuringthequalityandspecificityofpreregistrations AT marcelalmvanassen ensuringthequalityandspecificityofpreregistrations AT eliseavcrompvoets ensuringthequalityandspecificityofpreregistrations AT howhweeong ensuringthequalityandspecificityofpreregistrations AT briananosek ensuringthequalityandspecificityofpreregistrations AT courtneyksoderberg ensuringthequalityandspecificityofpreregistrations AT davidmellor ensuringthequalityandspecificityofpreregistrations AT jeltemwicherts ensuringthequalityandspecificityofpreregistrations |