Identification of tools used to assess the external validity of randomized controlled trials in reviews: a systematic review of measurement properties
Abstract Background Internal and external validity are the most relevant components when critically appraising randomized controlled trials (RCTs) for systematic reviews. However, there is no gold standard to assess external validity. This might be related to the heterogeneity of the terminology as...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2022-04-01
|
| Series: | BMC Medical Research Methodology |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12874-022-01561-5 |
| _version_ | 1819059673708363776 |
|---|---|
| author | Andres Jung Julia Balzer Tobias Braun Kerstin Luedtke |
| author_facet | Andres Jung Julia Balzer Tobias Braun Kerstin Luedtke |
| author_sort | Andres Jung |
| collection | DOAJ |
| description | Abstract Background Internal and external validity are the most relevant components when critically appraising randomized controlled trials (RCTs) for systematic reviews. However, there is no gold standard to assess external validity. This might be related to the heterogeneity of the terminology as well as to unclear evidence of the measurement properties of available tools. The aim of this review was to identify tools to assess the external validity of RCTs. It was further, to evaluate the quality of identified tools and to recommend the use of individual tools to assess the external validity of RCTs in future systematic reviews. Methods A two-phase systematic literature search was performed in four databases: PubMed, Scopus, PsycINFO via OVID, and CINAHL via EBSCO. First, tools to assess the external validity of RCTs were identified. Second, studies investigating the measurement properties of these tools were selected. The measurement properties of each included tool were appraised using an adapted version of the COnsensus based Standards for the selection of health Measurement INstruments (COSMIN) guidelines. Results 38 publications reporting on the development or validation of 28 included tools were included. For 61% (17/28) of the included tools, there was no evidence for measurement properties. For the remaining tools, reliability was the most frequently assessed property. Reliability was judged as “sufficient” for three tools (very low certainty of evidence). Content validity was rated as “sufficient” for one tool (moderate certainty of evidence). Conclusions Based on these results, no available tool can be fully recommended to assess the external validity of RCTs in systematic reviews. Several steps are required to overcome the identified difficulties to either adapt and validate available tools or to develop a better suitable tool. Trial registration Prospective registration at Open Science Framework (OSF): https://doi.org/10.17605/OSF.IO/PTG4D . |
| first_indexed | 2024-12-21T14:14:50Z |
| format | Article |
| id | doaj.art-c56cd1e3653541b9aac1437dabc1ea1c |
| institution | Directory Open Access Journal |
| issn | 1471-2288 |
| language | English |
| last_indexed | 2024-12-21T14:14:50Z |
| publishDate | 2022-04-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Research Methodology |
| spelling | doaj.art-c56cd1e3653541b9aac1437dabc1ea1c2022-12-21T19:00:56ZengBMCBMC Medical Research Methodology1471-22882022-04-0122112310.1186/s12874-022-01561-5Identification of tools used to assess the external validity of randomized controlled trials in reviews: a systematic review of measurement propertiesAndres Jung0Julia Balzer1Tobias Braun2Kerstin Luedtke3Institute of Health Sciences, Department of Physiotherapy, Pain and Exercise Research Luebeck (P.E.R.L), Universität zu LübeckFaculty of Applied Public Health, European University of Applied SciencesDivision of Physiotherapy, Department of Applied Health Sciences, Hochschule für Gesundheit (University of Applied Sciences)Institute of Health Sciences, Department of Physiotherapy, Pain and Exercise Research Luebeck (P.E.R.L), Universität zu LübeckAbstract Background Internal and external validity are the most relevant components when critically appraising randomized controlled trials (RCTs) for systematic reviews. However, there is no gold standard to assess external validity. This might be related to the heterogeneity of the terminology as well as to unclear evidence of the measurement properties of available tools. The aim of this review was to identify tools to assess the external validity of RCTs. It was further, to evaluate the quality of identified tools and to recommend the use of individual tools to assess the external validity of RCTs in future systematic reviews. Methods A two-phase systematic literature search was performed in four databases: PubMed, Scopus, PsycINFO via OVID, and CINAHL via EBSCO. First, tools to assess the external validity of RCTs were identified. Second, studies investigating the measurement properties of these tools were selected. The measurement properties of each included tool were appraised using an adapted version of the COnsensus based Standards for the selection of health Measurement INstruments (COSMIN) guidelines. Results 38 publications reporting on the development or validation of 28 included tools were included. For 61% (17/28) of the included tools, there was no evidence for measurement properties. For the remaining tools, reliability was the most frequently assessed property. Reliability was judged as “sufficient” for three tools (very low certainty of evidence). Content validity was rated as “sufficient” for one tool (moderate certainty of evidence). Conclusions Based on these results, no available tool can be fully recommended to assess the external validity of RCTs in systematic reviews. Several steps are required to overcome the identified difficulties to either adapt and validate available tools or to develop a better suitable tool. Trial registration Prospective registration at Open Science Framework (OSF): https://doi.org/10.17605/OSF.IO/PTG4D .https://doi.org/10.1186/s12874-022-01561-5External validityGeneralizabilityApplicabilityMeasurement propertiesToolsRandomized controlled trial |
| spellingShingle | Andres Jung Julia Balzer Tobias Braun Kerstin Luedtke Identification of tools used to assess the external validity of randomized controlled trials in reviews: a systematic review of measurement properties BMC Medical Research Methodology External validity Generalizability Applicability Measurement properties Tools Randomized controlled trial |
| title | Identification of tools used to assess the external validity of randomized controlled trials in reviews: a systematic review of measurement properties |
| title_full | Identification of tools used to assess the external validity of randomized controlled trials in reviews: a systematic review of measurement properties |
| title_fullStr | Identification of tools used to assess the external validity of randomized controlled trials in reviews: a systematic review of measurement properties |
| title_full_unstemmed | Identification of tools used to assess the external validity of randomized controlled trials in reviews: a systematic review of measurement properties |
| title_short | Identification of tools used to assess the external validity of randomized controlled trials in reviews: a systematic review of measurement properties |
| title_sort | identification of tools used to assess the external validity of randomized controlled trials in reviews a systematic review of measurement properties |
| topic | External validity Generalizability Applicability Measurement properties Tools Randomized controlled trial |
| url | https://doi.org/10.1186/s12874-022-01561-5 |
| work_keys_str_mv | AT andresjung identificationoftoolsusedtoassesstheexternalvalidityofrandomizedcontrolledtrialsinreviewsasystematicreviewofmeasurementproperties AT juliabalzer identificationoftoolsusedtoassesstheexternalvalidityofrandomizedcontrolledtrialsinreviewsasystematicreviewofmeasurementproperties AT tobiasbraun identificationoftoolsusedtoassesstheexternalvalidityofrandomizedcontrolledtrialsinreviewsasystematicreviewofmeasurementproperties AT kerstinluedtke identificationoftoolsusedtoassesstheexternalvalidityofrandomizedcontrolledtrialsinreviewsasystematicreviewofmeasurementproperties |