Limited Usefulness of Capture Procedure and Capture Percentage for Evaluating Reproducibility in Psychological Science
In psychological science, there is an increasing concern regarding the reproducibility of scientific findings. For instance, Replication Project: Psychology (Open Science Collaboration, 2015) found that the proportion of successful replication in psychology was 41%. This proportion was calculated ba...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2018-09-01
|
| Series: | Frontiers in Psychology |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/article/10.3389/fpsyg.2018.01657/full |
| _version_ | 1811328115717177344 |
|---|---|
| author | Yongtian Cheng Johnson Ching-Hong Li Xiyao Liu |
| author_facet | Yongtian Cheng Johnson Ching-Hong Li Xiyao Liu |
| author_sort | Yongtian Cheng |
| collection | DOAJ |
| description | In psychological science, there is an increasing concern regarding the reproducibility of scientific findings. For instance, Replication Project: Psychology (Open Science Collaboration, 2015) found that the proportion of successful replication in psychology was 41%. This proportion was calculated based on Cumming and Maillardet (2006) widely employed capture procedure (CPro) and capture percentage (CPer). Despite the popularity of CPro and CPer, we believe that using them may lead to an incorrect conclusion of (a) successful replication when the population effect sizes in the original and replicated studies are different; and (b) unsuccessful replication when the population effect sizes in the original and replicated studies are identical but their sample sizes are different. Our simulation results show that the performances of CPro and CPer become biased, such that researchers can easily make a wrong conclusion of successful/unsuccessful replication. Implications of these findings are considered in the conclusion. |
| first_indexed | 2024-04-13T15:19:29Z |
| format | Article |
| id | doaj.art-eb12cf77c7364bd8a46692cdbac1ff2b |
| institution | Directory Open Access Journal |
| issn | 1664-1078 |
| language | English |
| last_indexed | 2024-04-13T15:19:29Z |
| publishDate | 2018-09-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Psychology |
| spelling | doaj.art-eb12cf77c7364bd8a46692cdbac1ff2b2022-12-22T02:41:43ZengFrontiers Media S.A.Frontiers in Psychology1664-10782018-09-01910.3389/fpsyg.2018.01657385273Limited Usefulness of Capture Procedure and Capture Percentage for Evaluating Reproducibility in Psychological ScienceYongtian Cheng0Johnson Ching-Hong Li1Xiyao Liu2Department of Psychology, University of Manitoba, Winnipeg, MB, CanadaDepartment of Psychology, University of Manitoba, Winnipeg, MB, CanadaDepartment of Psychology, University of Oregon, Eugene, OR, United StatesIn psychological science, there is an increasing concern regarding the reproducibility of scientific findings. For instance, Replication Project: Psychology (Open Science Collaboration, 2015) found that the proportion of successful replication in psychology was 41%. This proportion was calculated based on Cumming and Maillardet (2006) widely employed capture procedure (CPro) and capture percentage (CPer). Despite the popularity of CPro and CPer, we believe that using them may lead to an incorrect conclusion of (a) successful replication when the population effect sizes in the original and replicated studies are different; and (b) unsuccessful replication when the population effect sizes in the original and replicated studies are identical but their sample sizes are different. Our simulation results show that the performances of CPro and CPer become biased, such that researchers can easily make a wrong conclusion of successful/unsuccessful replication. Implications of these findings are considered in the conclusion.https://www.frontiersin.org/article/10.3389/fpsyg.2018.01657/fullreproducibilityeffect sizescapture percentagecapture proceduresimulation |
| spellingShingle | Yongtian Cheng Johnson Ching-Hong Li Xiyao Liu Limited Usefulness of Capture Procedure and Capture Percentage for Evaluating Reproducibility in Psychological Science Frontiers in Psychology reproducibility effect sizes capture percentage capture procedure simulation |
| title | Limited Usefulness of Capture Procedure and Capture Percentage for Evaluating Reproducibility in Psychological Science |
| title_full | Limited Usefulness of Capture Procedure and Capture Percentage for Evaluating Reproducibility in Psychological Science |
| title_fullStr | Limited Usefulness of Capture Procedure and Capture Percentage for Evaluating Reproducibility in Psychological Science |
| title_full_unstemmed | Limited Usefulness of Capture Procedure and Capture Percentage for Evaluating Reproducibility in Psychological Science |
| title_short | Limited Usefulness of Capture Procedure and Capture Percentage for Evaluating Reproducibility in Psychological Science |
| title_sort | limited usefulness of capture procedure and capture percentage for evaluating reproducibility in psychological science |
| topic | reproducibility effect sizes capture percentage capture procedure simulation |
| url | https://www.frontiersin.org/article/10.3389/fpsyg.2018.01657/full |
| work_keys_str_mv | AT yongtiancheng limitedusefulnessofcaptureprocedureandcapturepercentageforevaluatingreproducibilityinpsychologicalscience AT johnsonchinghongli limitedusefulnessofcaptureprocedureandcapturepercentageforevaluatingreproducibilityinpsychologicalscience AT xiyaoliu limitedusefulnessofcaptureprocedureandcapturepercentageforevaluatingreproducibilityinpsychologicalscience |