Recommendations in pre-registrations and internal review board proposals promote formal power analyses but do not increase sample size.
In this preregistered study, we investigated whether the statistical power of a study is higher when researchers are asked to make a formal power analysis before collecting data. We compared the sample size descriptions from two sources: (i) a sample of pre-registrations created according to the gui...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science (PLoS)
2020-01-01
|
| Series: | PLoS ONE |
| Online Access: | https://doi.org/10.1371/journal.pone.0236079 |
| _version_ | 1818916742444875776 |
|---|---|
| author | Marjan Bakker Coosje L S Veldkamp Olmo R van den Akker Marcel A L M van Assen Elise Crompvoets How Hwee Ong Jelte M Wicherts |
| author_facet | Marjan Bakker Coosje L S Veldkamp Olmo R van den Akker Marcel A L M van Assen Elise Crompvoets How Hwee Ong Jelte M Wicherts |
| author_sort | Marjan Bakker |
| collection | DOAJ |
| description | In this preregistered study, we investigated whether the statistical power of a study is higher when researchers are asked to make a formal power analysis before collecting data. We compared the sample size descriptions from two sources: (i) a sample of pre-registrations created according to the guidelines for the Center for Open Science Preregistration Challenge (PCRs) and a sample of institutional review board (IRB) proposals from Tilburg School of Behavior and Social Sciences, which both include a recommendation to do a formal power analysis, and (ii) a sample of pre-registrations created according to the guidelines for Open Science Framework Standard Pre-Data Collection Registrations (SPRs) in which no guidance on sample size planning is given. We found that PCRs and IRBs (72%) more often included sample size decisions based on power analyses than the SPRs (45%). However, this did not result in larger planned sample sizes. The determined sample size of the PCRs and IRB proposals (Md = 90.50) was not higher than the determined sample size of the SPRs (Md = 126.00; W = 3389.5, p = 0.936). Typically, power analyses in the registrations were conducted with G*power, assuming a medium effect size, α = .05 and a power of .80. Only 20% of the power analyses contained enough information to fully reproduce the results and only 62% of these power analyses pertained to the main hypothesis test in the pre-registration. Therefore, we see ample room for improvements in the quality of the registrations and we offer several recommendations to do so. |
| first_indexed | 2024-12-20T00:23:00Z |
| format | Article |
| id | doaj.art-5bf4d66aa7b0471ea03ffce5bdfcb54c |
| institution | Directory Open Access Journal |
| issn | 1932-6203 |
| language | English |
| last_indexed | 2024-12-20T00:23:00Z |
| publishDate | 2020-01-01 |
| publisher | Public Library of Science (PLoS) |
| record_format | Article |
| series | PLoS ONE |
| spelling | doaj.art-5bf4d66aa7b0471ea03ffce5bdfcb54c2022-12-21T20:00:08ZengPublic Library of Science (PLoS)PLoS ONE1932-62032020-01-01157e023607910.1371/journal.pone.0236079Recommendations in pre-registrations and internal review board proposals promote formal power analyses but do not increase sample size.Marjan BakkerCoosje L S VeldkampOlmo R van den AkkerMarcel A L M van AssenElise CrompvoetsHow Hwee OngJelte M WichertsIn this preregistered study, we investigated whether the statistical power of a study is higher when researchers are asked to make a formal power analysis before collecting data. We compared the sample size descriptions from two sources: (i) a sample of pre-registrations created according to the guidelines for the Center for Open Science Preregistration Challenge (PCRs) and a sample of institutional review board (IRB) proposals from Tilburg School of Behavior and Social Sciences, which both include a recommendation to do a formal power analysis, and (ii) a sample of pre-registrations created according to the guidelines for Open Science Framework Standard Pre-Data Collection Registrations (SPRs) in which no guidance on sample size planning is given. We found that PCRs and IRBs (72%) more often included sample size decisions based on power analyses than the SPRs (45%). However, this did not result in larger planned sample sizes. The determined sample size of the PCRs and IRB proposals (Md = 90.50) was not higher than the determined sample size of the SPRs (Md = 126.00; W = 3389.5, p = 0.936). Typically, power analyses in the registrations were conducted with G*power, assuming a medium effect size, α = .05 and a power of .80. Only 20% of the power analyses contained enough information to fully reproduce the results and only 62% of these power analyses pertained to the main hypothesis test in the pre-registration. Therefore, we see ample room for improvements in the quality of the registrations and we offer several recommendations to do so.https://doi.org/10.1371/journal.pone.0236079 |
| spellingShingle | Marjan Bakker Coosje L S Veldkamp Olmo R van den Akker Marcel A L M van Assen Elise Crompvoets How Hwee Ong Jelte M Wicherts Recommendations in pre-registrations and internal review board proposals promote formal power analyses but do not increase sample size. PLoS ONE |
| title | Recommendations in pre-registrations and internal review board proposals promote formal power analyses but do not increase sample size. |
| title_full | Recommendations in pre-registrations and internal review board proposals promote formal power analyses but do not increase sample size. |
| title_fullStr | Recommendations in pre-registrations and internal review board proposals promote formal power analyses but do not increase sample size. |
| title_full_unstemmed | Recommendations in pre-registrations and internal review board proposals promote formal power analyses but do not increase sample size. |
| title_short | Recommendations in pre-registrations and internal review board proposals promote formal power analyses but do not increase sample size. |
| title_sort | recommendations in pre registrations and internal review board proposals promote formal power analyses but do not increase sample size |
| url | https://doi.org/10.1371/journal.pone.0236079 |
| work_keys_str_mv | AT marjanbakker recommendationsinpreregistrationsandinternalreviewboardproposalspromoteformalpoweranalysesbutdonotincreasesamplesize AT coosjelsveldkamp recommendationsinpreregistrationsandinternalreviewboardproposalspromoteformalpoweranalysesbutdonotincreasesamplesize AT olmorvandenakker recommendationsinpreregistrationsandinternalreviewboardproposalspromoteformalpoweranalysesbutdonotincreasesamplesize AT marcelalmvanassen recommendationsinpreregistrationsandinternalreviewboardproposalspromoteformalpoweranalysesbutdonotincreasesamplesize AT elisecrompvoets recommendationsinpreregistrationsandinternalreviewboardproposalspromoteformalpoweranalysesbutdonotincreasesamplesize AT howhweeong recommendationsinpreregistrationsandinternalreviewboardproposalspromoteformalpoweranalysesbutdonotincreasesamplesize AT jeltemwicherts recommendationsinpreregistrationsandinternalreviewboardproposalspromoteformalpoweranalysesbutdonotincreasesamplesize |