Challenges for assessing replicability in preclinical cancer biology
We conducted the Reproducibility Project: Cancer Biology to investigate the replicability of preclinical research in cancer biology. The initial aim of the project was to repeat 193 experiments from 53 high-impact papers, using an approach in which the experimental protocols and plans for data analy...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
eLife Sciences Publications Ltd
2021-12-01
|
| Series: | eLife |
| Subjects: | |
| Online Access: | https://elifesciences.org/articles/67995 |
| _version_ | 1797804216932106240 |
|---|---|
| author | Timothy M Errington Alexandria Denis Nicole Perfito Elizabeth Iorns Brian A Nosek |
| author_facet | Timothy M Errington Alexandria Denis Nicole Perfito Elizabeth Iorns Brian A Nosek |
| author_sort | Timothy M Errington |
| collection | DOAJ |
| description | We conducted the Reproducibility Project: Cancer Biology to investigate the replicability of preclinical research in cancer biology. The initial aim of the project was to repeat 193 experiments from 53 high-impact papers, using an approach in which the experimental protocols and plans for data analysis had to be peer reviewed and accepted for publication before experimental work could begin. However, the various barriers and challenges we encountered while designing and conducting the experiments meant that we were only able to repeat 50 experiments from 23 papers. Here we report these barriers and challenges. First, many original papers failed to report key descriptive and inferential statistics: the data needed to compute effect sizes and conduct power analyses was publicly accessible for just 4 of 193 experiments. Moreover, despite contacting the authors of the original papers, we were unable to obtain these data for 68% of the experiments. Second, none of the 193 experiments were described in sufficient detail in the original paper to enable us to design protocols to repeat the experiments, so we had to seek clarifications from the original authors. While authors were extremely or very helpful for 41% of experiments, they were minimally helpful for 9% of experiments, and not at all helpful (or did not respond to us) for 32% of experiments. Third, once experimental work started, 67% of the peer-reviewed protocols required modifications to complete the research and just 41% of those modifications could be implemented. Cumulatively, these three factors limited the number of experiments that could be repeated. This experience draws attention to a basic and fundamental concern about replication – it is hard to assess whether reported findings are credible. |
| first_indexed | 2024-03-13T05:33:49Z |
| format | Article |
| id | doaj.art-c140748641af48698583cbe0c0ef2860 |
| institution | Directory Open Access Journal |
| issn | 2050-084X |
| language | English |
| last_indexed | 2024-03-13T05:33:49Z |
| publishDate | 2021-12-01 |
| publisher | eLife Sciences Publications Ltd |
| record_format | Article |
| series | eLife |
| spelling | doaj.art-c140748641af48698583cbe0c0ef28602023-06-14T14:50:29ZengeLife Sciences Publications LtdeLife2050-084X2021-12-011010.7554/eLife.67995Challenges for assessing replicability in preclinical cancer biologyTimothy M Errington0https://orcid.org/0000-0002-4959-5143Alexandria Denis1Nicole Perfito2https://orcid.org/0000-0001-9546-215XElizabeth Iorns3https://orcid.org/0000-0002-5515-1258Brian A Nosek4https://orcid.org/0000-0001-6797-5476Center for Open Science, Charlottesville, United StatesCenter for Open Science, Charlottesville, United StatesScience Exchange, Palo Alto, United StatesScience Exchange, Palo Alto, United StatesCenter for Open Science, Charlottesville, United States; University of Virginia, Charlottesville, United StatesWe conducted the Reproducibility Project: Cancer Biology to investigate the replicability of preclinical research in cancer biology. The initial aim of the project was to repeat 193 experiments from 53 high-impact papers, using an approach in which the experimental protocols and plans for data analysis had to be peer reviewed and accepted for publication before experimental work could begin. However, the various barriers and challenges we encountered while designing and conducting the experiments meant that we were only able to repeat 50 experiments from 23 papers. Here we report these barriers and challenges. First, many original papers failed to report key descriptive and inferential statistics: the data needed to compute effect sizes and conduct power analyses was publicly accessible for just 4 of 193 experiments. Moreover, despite contacting the authors of the original papers, we were unable to obtain these data for 68% of the experiments. Second, none of the 193 experiments were described in sufficient detail in the original paper to enable us to design protocols to repeat the experiments, so we had to seek clarifications from the original authors. While authors were extremely or very helpful for 41% of experiments, they were minimally helpful for 9% of experiments, and not at all helpful (or did not respond to us) for 32% of experiments. Third, once experimental work started, 67% of the peer-reviewed protocols required modifications to complete the research and just 41% of those modifications could be implemented. Cumulatively, these three factors limited the number of experiments that could be repeated. This experience draws attention to a basic and fundamental concern about replication – it is hard to assess whether reported findings are credible.https://elifesciences.org/articles/67995Reproducibility Project: Cancer Biologyreplicationreproducibilityopen dataopen sciencepreregistration |
| spellingShingle | Timothy M Errington Alexandria Denis Nicole Perfito Elizabeth Iorns Brian A Nosek Challenges for assessing replicability in preclinical cancer biology eLife Reproducibility Project: Cancer Biology replication reproducibility open data open science preregistration |
| title | Challenges for assessing replicability in preclinical cancer biology |
| title_full | Challenges for assessing replicability in preclinical cancer biology |
| title_fullStr | Challenges for assessing replicability in preclinical cancer biology |
| title_full_unstemmed | Challenges for assessing replicability in preclinical cancer biology |
| title_short | Challenges for assessing replicability in preclinical cancer biology |
| title_sort | challenges for assessing replicability in preclinical cancer biology |
| topic | Reproducibility Project: Cancer Biology replication reproducibility open data open science preregistration |
| url | https://elifesciences.org/articles/67995 |
| work_keys_str_mv | AT timothymerrington challengesforassessingreplicabilityinpreclinicalcancerbiology AT alexandriadenis challengesforassessingreplicabilityinpreclinicalcancerbiology AT nicoleperfito challengesforassessingreplicabilityinpreclinicalcancerbiology AT elizabethiorns challengesforassessingreplicabilityinpreclinicalcancerbiology AT briananosek challengesforassessingreplicabilityinpreclinicalcancerbiology |