Inter-rater reliability of the extended Composite Quality Score (CQS-2)
AimTo establish the inter-rater reliability of the Composite Quality Score (CQS-2) and to test the null hypothesis that it did not differ significantly from that of the first CQS version (CQS-1).Materials and methodsFour independent raters were selected to rate 45 clinical trial reports using CQS-1...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2023-08-01
|
| Series: | Frontiers in Medicine |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fmed.2023.1201517/full |
| _version_ | 1797741405150380032 |
|---|---|
| author | Steffen Mickenautsch Steffen Mickenautsch Steffen Mickenautsch Stefan Rupf Ivana Miletić Ulf Tilman Strähle Richard Sturm Faheema Kimmie-Dhansay Kata Vidosusić Veerasamy Yengopal |
| author_facet | Steffen Mickenautsch Steffen Mickenautsch Steffen Mickenautsch Stefan Rupf Ivana Miletić Ulf Tilman Strähle Richard Sturm Faheema Kimmie-Dhansay Kata Vidosusić Veerasamy Yengopal |
| author_sort | Steffen Mickenautsch |
| collection | DOAJ |
| description | AimTo establish the inter-rater reliability of the Composite Quality Score (CQS-2) and to test the null hypothesis that it did not differ significantly from that of the first CQS version (CQS-1).Materials and methodsFour independent raters were selected to rate 45 clinical trial reports using CQS-1 and CQS-2. The raters remained unaware of each other’s participation in this study until all rating had been completed. Each rater received only one rating template at a time in a random sequence for CQS-1 and CQS-2 rating. Raters completed each template and sent these back to the principal investigator. Each rater received their next template 2 weeks after submission of the completed previous template. The inter-rater reliabilities for the overall appraisal score of the CQS-1 and the CQS-2 were established by using the Brennan-Prediger coefficient (BPC). The coefficients of both CQS versions were compared by using the two-sample z-test. During secondary analysis, the BPCs for every criterion and each corroboration level for both CQS versions were established.ResultsThe BPC for the CQS-1 was 0.85 (95% CI: 0.64–1.00) and for the CQS-2 it was 1.00 (95% CI: 0.94–1.00), suggesting a very high inter-rater reliability for both. The difference between the two CQS versions was statistically not significant (p = 0.17). The null hypothesis was accepted.ConclusionThe CQS-2 is still under development, This study shows that it is associated with a very high inter-rater reliability, which did not statistically significantly differ from that of the CQS-1. The promising results of this study warrant further investigation in the applicability of the CQS-2 as an appraisal tool for prospective controlled clinical therapy trials. |
| first_indexed | 2024-03-12T14:26:13Z |
| format | Article |
| id | doaj.art-5546c0c204bc42bc942885770bbf0ce4 |
| institution | Directory Open Access Journal |
| issn | 2296-858X |
| language | English |
| last_indexed | 2024-03-12T14:26:13Z |
| publishDate | 2023-08-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Medicine |
| spelling | doaj.art-5546c0c204bc42bc942885770bbf0ce42023-08-18T05:23:43ZengFrontiers Media S.A.Frontiers in Medicine2296-858X2023-08-011010.3389/fmed.2023.12015171201517Inter-rater reliability of the extended Composite Quality Score (CQS-2)Steffen Mickenautsch0Steffen Mickenautsch1Steffen Mickenautsch2Stefan Rupf3Ivana Miletić4Ulf Tilman Strähle5Richard Sturm6Faheema Kimmie-Dhansay7Kata Vidosusić8Veerasamy Yengopal9Faculty of Dentistry, University of the Western Cape, Bellville, South AfricaDepartment of Community Dentistry, Faculty of Health Sciences, School of Oral Health Sciences, University of the Witwatersrand, Johannesburg, South AfricaReview Centre for Health Science Research, Johannesburg, South AfricaSynoptic Dentistry, Saarland University, Homburg, GermanyDepartment of Endodontics and Restorative Dentistry, School of Dental Medicine, University of Zagreb, Zagreb, CroatiaSynoptic Dentistry, Saarland University, Homburg, GermanyDepartment of Operative, Preventive and Paediatric Dentistry, Charité – Universitätsmedizin Berlin, Berlin, GermanyDepartment of Community Oral Health, Faculty of Dentistry, University of the Western Cape, Bellville, South AfricaDepartment of Endodontics and Restorative Dentistry, School of Dental Medicine, University of Zagreb, Zagreb, CroatiaFaculty of Dentistry, University of the Western Cape, Bellville, South AfricaAimTo establish the inter-rater reliability of the Composite Quality Score (CQS-2) and to test the null hypothesis that it did not differ significantly from that of the first CQS version (CQS-1).Materials and methodsFour independent raters were selected to rate 45 clinical trial reports using CQS-1 and CQS-2. The raters remained unaware of each other’s participation in this study until all rating had been completed. Each rater received only one rating template at a time in a random sequence for CQS-1 and CQS-2 rating. Raters completed each template and sent these back to the principal investigator. Each rater received their next template 2 weeks after submission of the completed previous template. The inter-rater reliabilities for the overall appraisal score of the CQS-1 and the CQS-2 were established by using the Brennan-Prediger coefficient (BPC). The coefficients of both CQS versions were compared by using the two-sample z-test. During secondary analysis, the BPCs for every criterion and each corroboration level for both CQS versions were established.ResultsThe BPC for the CQS-1 was 0.85 (95% CI: 0.64–1.00) and for the CQS-2 it was 1.00 (95% CI: 0.94–1.00), suggesting a very high inter-rater reliability for both. The difference between the two CQS versions was statistically not significant (p = 0.17). The null hypothesis was accepted.ConclusionThe CQS-2 is still under development, This study shows that it is associated with a very high inter-rater reliability, which did not statistically significantly differ from that of the CQS-1. The promising results of this study warrant further investigation in the applicability of the CQS-2 as an appraisal tool for prospective controlled clinical therapy trials.https://www.frontiersin.org/articles/10.3389/fmed.2023.1201517/fullComposite Quality Scoreclinical rialtrial appraisalinter-rater reliabilitysystematic review |
| spellingShingle | Steffen Mickenautsch Steffen Mickenautsch Steffen Mickenautsch Stefan Rupf Ivana Miletić Ulf Tilman Strähle Richard Sturm Faheema Kimmie-Dhansay Kata Vidosusić Veerasamy Yengopal Inter-rater reliability of the extended Composite Quality Score (CQS-2) Frontiers in Medicine Composite Quality Score clinical rial trial appraisal inter-rater reliability systematic review |
| title | Inter-rater reliability of the extended Composite Quality Score (CQS-2) |
| title_full | Inter-rater reliability of the extended Composite Quality Score (CQS-2) |
| title_fullStr | Inter-rater reliability of the extended Composite Quality Score (CQS-2) |
| title_full_unstemmed | Inter-rater reliability of the extended Composite Quality Score (CQS-2) |
| title_short | Inter-rater reliability of the extended Composite Quality Score (CQS-2) |
| title_sort | inter rater reliability of the extended composite quality score cqs 2 |
| topic | Composite Quality Score clinical rial trial appraisal inter-rater reliability systematic review |
| url | https://www.frontiersin.org/articles/10.3389/fmed.2023.1201517/full |
| work_keys_str_mv | AT steffenmickenautsch interraterreliabilityoftheextendedcompositequalityscorecqs2 AT steffenmickenautsch interraterreliabilityoftheextendedcompositequalityscorecqs2 AT steffenmickenautsch interraterreliabilityoftheextendedcompositequalityscorecqs2 AT stefanrupf interraterreliabilityoftheextendedcompositequalityscorecqs2 AT ivanamiletic interraterreliabilityoftheextendedcompositequalityscorecqs2 AT ulftilmanstrahle interraterreliabilityoftheextendedcompositequalityscorecqs2 AT richardsturm interraterreliabilityoftheextendedcompositequalityscorecqs2 AT faheemakimmiedhansay interraterreliabilityoftheextendedcompositequalityscorecqs2 AT katavidosusic interraterreliabilityoftheextendedcompositequalityscorecqs2 AT veerasamyyengopal interraterreliabilityoftheextendedcompositequalityscorecqs2 |