Examining publication bias—a simulation-based evaluation of statistical tests on publication bias
Background Publication bias is a form of scientific misconduct. It threatens the validity of research results and the credibility of science. Although several tests on publication bias exist, no in-depth evaluations are available that examine which test performs best for different research settings....
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
PeerJ Inc.
2017-11-01
|
| Series: | PeerJ |
| Subjects: | |
| Online Access: | https://peerj.com/articles/4115.pdf |
| _version_ | 1797420394893803520 |
|---|---|
| author | Andreas Schneck |
| author_facet | Andreas Schneck |
| author_sort | Andreas Schneck |
| collection | DOAJ |
| description | Background Publication bias is a form of scientific misconduct. It threatens the validity of research results and the credibility of science. Although several tests on publication bias exist, no in-depth evaluations are available that examine which test performs best for different research settings. Methods Four tests on publication bias, Egger’s test (FAT), p-uniform, the test of excess significance (TES), as well as the caliper test, were evaluated in a Monte Carlo simulation. Two different types of publication bias and its degree (0%, 50%, 100%) were simulated. The type of publication bias was defined either as file-drawer, meaning the repeated analysis of new datasets, or p-hacking, meaning the inclusion of covariates in order to obtain a significant result. In addition, the underlying effect (β = 0, 0.5, 1, 1.5), effect heterogeneity, the number of observations in the simulated primary studies (N = 100, 500), and the number of observations for the publication bias tests (K = 100, 1,000) were varied. Results All tests evaluated were able to identify publication bias both in the file-drawer and p-hacking condition. The false positive rates were, with the exception of the 15%- and 20%-caliper test, unbiased. The FAT had the largest statistical power in the file-drawer conditions, whereas under p-hacking the TES was, except under effect heterogeneity, slightly better. The CTs were, however, inferior to the other tests under effect homogeneity and had a decent statistical power only in conditions with 1,000 primary studies. Discussion The FAT is recommended as a test for publication bias in standard meta-analyses with no or only small effect heterogeneity. If two-sided publication bias is suspected as well as under p-hacking the TES is the first alternative to the FAT. The 5%-caliper test is recommended under conditions of effect heterogeneity and a large number of primary studies, which may be found if publication bias is examined in a discipline-wide setting when primary studies cover different research problems. |
| first_indexed | 2024-03-09T07:00:49Z |
| format | Article |
| id | doaj.art-265912b7bd2046c798df83c0a623bdae |
| institution | Directory Open Access Journal |
| issn | 2167-8359 |
| language | English |
| last_indexed | 2024-03-09T07:00:49Z |
| publishDate | 2017-11-01 |
| publisher | PeerJ Inc. |
| record_format | Article |
| series | PeerJ |
| spelling | doaj.art-265912b7bd2046c798df83c0a623bdae2023-12-03T09:51:25ZengPeerJ Inc.PeerJ2167-83592017-11-015e411510.7717/peerj.4115Examining publication bias—a simulation-based evaluation of statistical tests on publication biasAndreas Schneck0Department of Sociology, Ludwig-Maximilians-Universität München, Munich, GermanyBackground Publication bias is a form of scientific misconduct. It threatens the validity of research results and the credibility of science. Although several tests on publication bias exist, no in-depth evaluations are available that examine which test performs best for different research settings. Methods Four tests on publication bias, Egger’s test (FAT), p-uniform, the test of excess significance (TES), as well as the caliper test, were evaluated in a Monte Carlo simulation. Two different types of publication bias and its degree (0%, 50%, 100%) were simulated. The type of publication bias was defined either as file-drawer, meaning the repeated analysis of new datasets, or p-hacking, meaning the inclusion of covariates in order to obtain a significant result. In addition, the underlying effect (β = 0, 0.5, 1, 1.5), effect heterogeneity, the number of observations in the simulated primary studies (N = 100, 500), and the number of observations for the publication bias tests (K = 100, 1,000) were varied. Results All tests evaluated were able to identify publication bias both in the file-drawer and p-hacking condition. The false positive rates were, with the exception of the 15%- and 20%-caliper test, unbiased. The FAT had the largest statistical power in the file-drawer conditions, whereas under p-hacking the TES was, except under effect heterogeneity, slightly better. The CTs were, however, inferior to the other tests under effect homogeneity and had a decent statistical power only in conditions with 1,000 primary studies. Discussion The FAT is recommended as a test for publication bias in standard meta-analyses with no or only small effect heterogeneity. If two-sided publication bias is suspected as well as under p-hacking the TES is the first alternative to the FAT. The 5%-caliper test is recommended under conditions of effect heterogeneity and a large number of primary studies, which may be found if publication bias is examined in a discipline-wide setting when primary studies cover different research problems.https://peerj.com/articles/4115.pdfStatisticsPublication biasTest for excess significanceCaliper testMonte carlo simulationp-uniform |
| spellingShingle | Andreas Schneck Examining publication bias—a simulation-based evaluation of statistical tests on publication bias PeerJ Statistics Publication bias Test for excess significance Caliper test Monte carlo simulation p-uniform |
| title | Examining publication bias—a simulation-based evaluation of statistical tests on publication bias |
| title_full | Examining publication bias—a simulation-based evaluation of statistical tests on publication bias |
| title_fullStr | Examining publication bias—a simulation-based evaluation of statistical tests on publication bias |
| title_full_unstemmed | Examining publication bias—a simulation-based evaluation of statistical tests on publication bias |
| title_short | Examining publication bias—a simulation-based evaluation of statistical tests on publication bias |
| title_sort | examining publication bias a simulation based evaluation of statistical tests on publication bias |
| topic | Statistics Publication bias Test for excess significance Caliper test Monte carlo simulation p-uniform |
| url | https://peerj.com/articles/4115.pdf |
| work_keys_str_mv | AT andreasschneck examiningpublicationbiasasimulationbasedevaluationofstatisticaltestsonpublicationbias |