Agreement test of P value versus Bayes factor for sample means comparison: analysis of articles from the Angle Orthodontist journal
Abstract Background Researchers are cautioned against misinterpreting the conventional P value, especially while implementing the popular t test. Therefore, this study evaluated the agreement between the P value and Bayes factor (BF01) results obtained from a comparison of sample means in published...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2023-02-01
|
| Series: | BMC Medical Research Methodology |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12874-023-01858-z |
| _version_ | 1797864335779823616 |
|---|---|
| author | Natchalee Srimaneekarn Pattamon Leelachaikul Sasipa Thiradilok Somchai Manopatanakul |
| author_facet | Natchalee Srimaneekarn Pattamon Leelachaikul Sasipa Thiradilok Somchai Manopatanakul |
| author_sort | Natchalee Srimaneekarn |
| collection | DOAJ |
| description | Abstract Background Researchers are cautioned against misinterpreting the conventional P value, especially while implementing the popular t test. Therefore, this study evaluated the agreement between the P value and Bayes factor (BF01) results obtained from a comparison of sample means in published orthodontic articles. Methods Data pooling was undertaken using the modified PRISMA flow diagram. Per the inclusion criteria applied to The Angle Orthodontist journal for a two-year period (November 2016 to September 2018), all articles that utilised the t test for statistical analysis were selected. The agreement was evaluated between the P value and Bayes factor set at 0.05 and 1, respectively. The percentage of agreement and Kappa coefficient were calculated. Plotting of effect size against P value and BF01 was analysed. Results From 265 articles, 82 utilised the t test. Of these, only 37 articles met the inclusion criteria. The study identified 793 justifiable t tests (438 independent-sample and 355 dependent-sample t tests) for which the agreement percentage and Kappa coefficient were found to be 93.57% and 0.87, respectively. However, when anecdotal evidence (1/3 < BF01 < 3) was considered, almost half of the studies missed statistical significance. Furthermore, two-thirds of the significantly reported P values (0.01 < P < 0.05; 30 independent-sample and 20 dependent-sample t tests) showed only anecdotal evidence (1/3 < BF01 < 1). Moreover, BF01 indicated moderate evidence (BF01 > 3) for approximately one-third of the total studies, with nonsignificant P values (P > 0.05). Furthermore, accompanying the P values, the effect sizes, especially for studies with independent-sample t tests, were very high with a strong potential to show substantive significance. Although it is best to extend the statistical calculation of a doubted P value (just below 0.05), especially for orthodontic innovation, orthodontists may reach a balanced decision relying on cephalometric measurements. Conclusions The Kappa coefficient indicated perfect agreement between the two methods. BF01 restricted this judgement to approximately half of them, with two-thirds of these studies showing nonsignificant P values. Simple extensions of statistical calculations, especially effect size and BF01, can be useful and should be considered when finalising statistical analyses, especially for orthodontic studies without cephalometric analysis. |
| first_indexed | 2024-04-09T22:51:16Z |
| format | Article |
| id | doaj.art-1ac13ecc057a4e9ba2ec5004b84d96cd |
| institution | Directory Open Access Journal |
| issn | 1471-2288 |
| language | English |
| last_indexed | 2024-04-09T22:51:16Z |
| publishDate | 2023-02-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Research Methodology |
| spelling | doaj.art-1ac13ecc057a4e9ba2ec5004b84d96cd2023-03-22T11:38:42ZengBMCBMC Medical Research Methodology1471-22882023-02-0123111110.1186/s12874-023-01858-zAgreement test of P value versus Bayes factor for sample means comparison: analysis of articles from the Angle Orthodontist journalNatchalee Srimaneekarn0Pattamon Leelachaikul1Sasipa Thiradilok2Somchai Manopatanakul3Department of Anatomy, Faculty of Dentistry, Mahidol UniversityPu-Kradueng HospitalDepartment of Advanced General Dentistry, Faculty of Dentistry, Mahidol UniversityDepartment of Advanced General Dentistry, Faculty of Dentistry, Mahidol UniversityAbstract Background Researchers are cautioned against misinterpreting the conventional P value, especially while implementing the popular t test. Therefore, this study evaluated the agreement between the P value and Bayes factor (BF01) results obtained from a comparison of sample means in published orthodontic articles. Methods Data pooling was undertaken using the modified PRISMA flow diagram. Per the inclusion criteria applied to The Angle Orthodontist journal for a two-year period (November 2016 to September 2018), all articles that utilised the t test for statistical analysis were selected. The agreement was evaluated between the P value and Bayes factor set at 0.05 and 1, respectively. The percentage of agreement and Kappa coefficient were calculated. Plotting of effect size against P value and BF01 was analysed. Results From 265 articles, 82 utilised the t test. Of these, only 37 articles met the inclusion criteria. The study identified 793 justifiable t tests (438 independent-sample and 355 dependent-sample t tests) for which the agreement percentage and Kappa coefficient were found to be 93.57% and 0.87, respectively. However, when anecdotal evidence (1/3 < BF01 < 3) was considered, almost half of the studies missed statistical significance. Furthermore, two-thirds of the significantly reported P values (0.01 < P < 0.05; 30 independent-sample and 20 dependent-sample t tests) showed only anecdotal evidence (1/3 < BF01 < 1). Moreover, BF01 indicated moderate evidence (BF01 > 3) for approximately one-third of the total studies, with nonsignificant P values (P > 0.05). Furthermore, accompanying the P values, the effect sizes, especially for studies with independent-sample t tests, were very high with a strong potential to show substantive significance. Although it is best to extend the statistical calculation of a doubted P value (just below 0.05), especially for orthodontic innovation, orthodontists may reach a balanced decision relying on cephalometric measurements. Conclusions The Kappa coefficient indicated perfect agreement between the two methods. BF01 restricted this judgement to approximately half of them, with two-thirds of these studies showing nonsignificant P values. Simple extensions of statistical calculations, especially effect size and BF01, can be useful and should be considered when finalising statistical analyses, especially for orthodontic studies without cephalometric analysis.https://doi.org/10.1186/s12874-023-01858-zAgreement testBayes factorP valueEffect sideOrthodontics |
| spellingShingle | Natchalee Srimaneekarn Pattamon Leelachaikul Sasipa Thiradilok Somchai Manopatanakul Agreement test of P value versus Bayes factor for sample means comparison: analysis of articles from the Angle Orthodontist journal BMC Medical Research Methodology Agreement test Bayes factor P value Effect side Orthodontics |
| title | Agreement test of P value versus Bayes factor for sample means comparison: analysis of articles from the Angle Orthodontist journal |
| title_full | Agreement test of P value versus Bayes factor for sample means comparison: analysis of articles from the Angle Orthodontist journal |
| title_fullStr | Agreement test of P value versus Bayes factor for sample means comparison: analysis of articles from the Angle Orthodontist journal |
| title_full_unstemmed | Agreement test of P value versus Bayes factor for sample means comparison: analysis of articles from the Angle Orthodontist journal |
| title_short | Agreement test of P value versus Bayes factor for sample means comparison: analysis of articles from the Angle Orthodontist journal |
| title_sort | agreement test of p value versus bayes factor for sample means comparison analysis of articles from the angle orthodontist journal |
| topic | Agreement test Bayes factor P value Effect side Orthodontics |
| url | https://doi.org/10.1186/s12874-023-01858-z |
| work_keys_str_mv | AT natchaleesrimaneekarn agreementtestofpvalueversusbayesfactorforsamplemeanscomparisonanalysisofarticlesfromtheangleorthodontistjournal AT pattamonleelachaikul agreementtestofpvalueversusbayesfactorforsamplemeanscomparisonanalysisofarticlesfromtheangleorthodontistjournal AT sasipathiradilok agreementtestofpvalueversusbayesfactorforsamplemeanscomparisonanalysisofarticlesfromtheangleorthodontistjournal AT somchaimanopatanakul agreementtestofpvalueversusbayesfactorforsamplemeanscomparisonanalysisofarticlesfromtheangleorthodontistjournal |