The mysterious case of the disappearing pilot study: a review of publication bias in preliminary behavioral interventions presented at health behavior conferences
Abstract Background The number of preliminary studies conducted and published has increased in recent years. However, there are likely many preliminary studies that go unpublished because preliminary studies are typically small and may not be perceived as methodologically rigorous. The extent of pub...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2023-07-01
|
| Series: | Pilot and Feasibility Studies |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s40814-023-01345-8 |
| _version_ | 1797784737041874944 |
|---|---|
| author | Lauren von Klinggraeff Kaitlyn Ramey Christopher D. Pfledderer Sarah Burkart Bridget Armstrong R. Glenn Weaver Michael W. Beets |
| author_facet | Lauren von Klinggraeff Kaitlyn Ramey Christopher D. Pfledderer Sarah Burkart Bridget Armstrong R. Glenn Weaver Michael W. Beets |
| author_sort | Lauren von Klinggraeff |
| collection | DOAJ |
| description | Abstract Background The number of preliminary studies conducted and published has increased in recent years. However, there are likely many preliminary studies that go unpublished because preliminary studies are typically small and may not be perceived as methodologically rigorous. The extent of publication bias within preliminary studies is unknown but can prove useful to determine whether preliminary studies appearing in peer-reviewed journals are fundamentally different than those that are unpublished. The purpose of this study was to identify characteristics associated with publication in a sample of abstracts of preliminary studies of behavioral interventions presented at conferences. Methods Abstract supplements from two primary outlets for behavioral intervention research (Society of Behavioral Medicine and International Society of Behavioral Nutrition and Physical Activity) were searched to identify all abstracts reporting findings of behavioral interventions from preliminary studies. Study characteristics were extracted from the abstracts including year presented, sample size, design, and statistical significance. To determine if abstracts had a matching peer-reviewed publication, a search of authors’ curriculum vitae and research databases was conducted. Iterative logistic regression models were used to estimate odds of abstract publication. Authors with unpublished preliminary studies were surveyed to identify reasons for nonpublication. Results Across conferences, a total of 18,961 abstracts were presented. Of these, 791 were preliminary behavioral interventions, of which 49% (388) were published in a peer-reviewed journal. For models with main effects only, preliminary studies with sample sizes greater than n = 24 were more likely to be published (range of odds ratios, 1.82 to 2.01). For models including interactions among study characteristics, no significant associations were found. Authors of unpublished preliminary studies indicated small sample sizes and being underpowered to detect effects as barriers to attempting publication. Conclusions Half of preliminary studies presented at conferences go unpublished, but published preliminary studies appearing in peer-reviewed literature are not systematically different from those that remain unpublished. Without publication, it is difficult to assess the quality of information regarding the early-stage development of interventions. This inaccessibility inhibits our ability to learn from the progression of preliminary studies. |
| first_indexed | 2024-03-13T00:44:09Z |
| format | Article |
| id | doaj.art-02eae3e1094346519d357895b041045e |
| institution | Directory Open Access Journal |
| issn | 2055-5784 |
| language | English |
| last_indexed | 2024-03-13T00:44:09Z |
| publishDate | 2023-07-01 |
| publisher | BMC |
| record_format | Article |
| series | Pilot and Feasibility Studies |
| spelling | doaj.art-02eae3e1094346519d357895b041045e2023-07-09T11:06:47ZengBMCPilot and Feasibility Studies2055-57842023-07-019111010.1186/s40814-023-01345-8The mysterious case of the disappearing pilot study: a review of publication bias in preliminary behavioral interventions presented at health behavior conferencesLauren von Klinggraeff0Kaitlyn Ramey1Christopher D. Pfledderer2Sarah Burkart3Bridget Armstrong4R. Glenn Weaver5Michael W. Beets6Department of Exercise Science, Arnold School of Public Health, University of South CarolinaDepartment of Exercise Science, Arnold School of Public Health, University of South CarolinaDepartment of Exercise Science, Arnold School of Public Health, University of South CarolinaDepartment of Exercise Science, Arnold School of Public Health, University of South CarolinaDepartment of Exercise Science, Arnold School of Public Health, University of South CarolinaDepartment of Exercise Science, Arnold School of Public Health, University of South CarolinaDepartment of Exercise Science, Arnold School of Public Health, University of South CarolinaAbstract Background The number of preliminary studies conducted and published has increased in recent years. However, there are likely many preliminary studies that go unpublished because preliminary studies are typically small and may not be perceived as methodologically rigorous. The extent of publication bias within preliminary studies is unknown but can prove useful to determine whether preliminary studies appearing in peer-reviewed journals are fundamentally different than those that are unpublished. The purpose of this study was to identify characteristics associated with publication in a sample of abstracts of preliminary studies of behavioral interventions presented at conferences. Methods Abstract supplements from two primary outlets for behavioral intervention research (Society of Behavioral Medicine and International Society of Behavioral Nutrition and Physical Activity) were searched to identify all abstracts reporting findings of behavioral interventions from preliminary studies. Study characteristics were extracted from the abstracts including year presented, sample size, design, and statistical significance. To determine if abstracts had a matching peer-reviewed publication, a search of authors’ curriculum vitae and research databases was conducted. Iterative logistic regression models were used to estimate odds of abstract publication. Authors with unpublished preliminary studies were surveyed to identify reasons for nonpublication. Results Across conferences, a total of 18,961 abstracts were presented. Of these, 791 were preliminary behavioral interventions, of which 49% (388) were published in a peer-reviewed journal. For models with main effects only, preliminary studies with sample sizes greater than n = 24 were more likely to be published (range of odds ratios, 1.82 to 2.01). For models including interactions among study characteristics, no significant associations were found. Authors of unpublished preliminary studies indicated small sample sizes and being underpowered to detect effects as barriers to attempting publication. Conclusions Half of preliminary studies presented at conferences go unpublished, but published preliminary studies appearing in peer-reviewed literature are not systematically different from those that remain unpublished. Without publication, it is difficult to assess the quality of information regarding the early-stage development of interventions. This inaccessibility inhibits our ability to learn from the progression of preliminary studies.https://doi.org/10.1186/s40814-023-01345-8Pilot projectsFeasibility studiesPublication biasGray literaturePeer reviewResearch |
| spellingShingle | Lauren von Klinggraeff Kaitlyn Ramey Christopher D. Pfledderer Sarah Burkart Bridget Armstrong R. Glenn Weaver Michael W. Beets The mysterious case of the disappearing pilot study: a review of publication bias in preliminary behavioral interventions presented at health behavior conferences Pilot and Feasibility Studies Pilot projects Feasibility studies Publication bias Gray literature Peer review Research |
| title | The mysterious case of the disappearing pilot study: a review of publication bias in preliminary behavioral interventions presented at health behavior conferences |
| title_full | The mysterious case of the disappearing pilot study: a review of publication bias in preliminary behavioral interventions presented at health behavior conferences |
| title_fullStr | The mysterious case of the disappearing pilot study: a review of publication bias in preliminary behavioral interventions presented at health behavior conferences |
| title_full_unstemmed | The mysterious case of the disappearing pilot study: a review of publication bias in preliminary behavioral interventions presented at health behavior conferences |
| title_short | The mysterious case of the disappearing pilot study: a review of publication bias in preliminary behavioral interventions presented at health behavior conferences |
| title_sort | mysterious case of the disappearing pilot study a review of publication bias in preliminary behavioral interventions presented at health behavior conferences |
| topic | Pilot projects Feasibility studies Publication bias Gray literature Peer review Research |
| url | https://doi.org/10.1186/s40814-023-01345-8 |
| work_keys_str_mv | AT laurenvonklinggraeff themysteriouscaseofthedisappearingpilotstudyareviewofpublicationbiasinpreliminarybehavioralinterventionspresentedathealthbehaviorconferences AT kaitlynramey themysteriouscaseofthedisappearingpilotstudyareviewofpublicationbiasinpreliminarybehavioralinterventionspresentedathealthbehaviorconferences AT christopherdpfledderer themysteriouscaseofthedisappearingpilotstudyareviewofpublicationbiasinpreliminarybehavioralinterventionspresentedathealthbehaviorconferences AT sarahburkart themysteriouscaseofthedisappearingpilotstudyareviewofpublicationbiasinpreliminarybehavioralinterventionspresentedathealthbehaviorconferences AT bridgetarmstrong themysteriouscaseofthedisappearingpilotstudyareviewofpublicationbiasinpreliminarybehavioralinterventionspresentedathealthbehaviorconferences AT rglennweaver themysteriouscaseofthedisappearingpilotstudyareviewofpublicationbiasinpreliminarybehavioralinterventionspresentedathealthbehaviorconferences AT michaelwbeets themysteriouscaseofthedisappearingpilotstudyareviewofpublicationbiasinpreliminarybehavioralinterventionspresentedathealthbehaviorconferences AT laurenvonklinggraeff mysteriouscaseofthedisappearingpilotstudyareviewofpublicationbiasinpreliminarybehavioralinterventionspresentedathealthbehaviorconferences AT kaitlynramey mysteriouscaseofthedisappearingpilotstudyareviewofpublicationbiasinpreliminarybehavioralinterventionspresentedathealthbehaviorconferences AT christopherdpfledderer mysteriouscaseofthedisappearingpilotstudyareviewofpublicationbiasinpreliminarybehavioralinterventionspresentedathealthbehaviorconferences AT sarahburkart mysteriouscaseofthedisappearingpilotstudyareviewofpublicationbiasinpreliminarybehavioralinterventionspresentedathealthbehaviorconferences AT bridgetarmstrong mysteriouscaseofthedisappearingpilotstudyareviewofpublicationbiasinpreliminarybehavioralinterventionspresentedathealthbehaviorconferences AT rglennweaver mysteriouscaseofthedisappearingpilotstudyareviewofpublicationbiasinpreliminarybehavioralinterventionspresentedathealthbehaviorconferences AT michaelwbeets mysteriouscaseofthedisappearingpilotstudyareviewofpublicationbiasinpreliminarybehavioralinterventionspresentedathealthbehaviorconferences |