A cautionary tale: an evaluation of the performance of treatment switching adjustment methods in a real world case study
Abstract Background Treatment switching in randomised controlled trials (RCTs) is a problem for health technology assessment when substantial proportions of patients switch onto effective treatments that would not be available in standard clinical practice. Often statistical methods are used to adju...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2024-01-01
|
| Series: | BMC Medical Research Methodology |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12874-024-02140-6 |
| _version_ | 1797276447997427712 |
|---|---|
| author | Nicholas R Latimer Alice Dewdney Marco Campioni |
| author_facet | Nicholas R Latimer Alice Dewdney Marco Campioni |
| author_sort | Nicholas R Latimer |
| collection | DOAJ |
| description | Abstract Background Treatment switching in randomised controlled trials (RCTs) is a problem for health technology assessment when substantial proportions of patients switch onto effective treatments that would not be available in standard clinical practice. Often statistical methods are used to adjust for switching: these can be applied in different ways, and performance has been assessed in simulation studies, but not in real-world case studies. We assessed the performance of adjustment methods described in National Institute for Health and Care Excellence Decision Support Unit Technical Support Document 16, applying them to an RCT comparing panitumumab to best supportive care (BSC) in colorectal cancer, in which 76% of patients randomised to BSC switched onto panitumumab. The RCT resulted in intention-to-treat hazard ratios (HR) for overall survival (OS) of 1.00 (95% confidence interval [CI] 0.82–1.22) for all patients, and 0.99 (95% CI 0.75–1.29) for patients with wild-type KRAS (Kirsten rat sarcoma virus). Methods We tested several applications of inverse probability of censoring weights (IPCW), rank preserving structural failure time models (RPSFTM) and simple and complex two-stage estimation (TSE) to estimate treatment effects that would have been observed if BSC patients had not switched onto panitumumab. To assess the performance of these analyses we ascertained the true effectiveness of panitumumab based on: (i) subsequent RCTs of panitumumab that disallowed treatment switching; (ii) studies of cetuximab that disallowed treatment switching, (iii) analyses demonstrating that only patients with wild-type KRAS benefit from panitumumab. These sources suggest the true OS HR for panitumumab is 0.76–0.77 (95% CI 0.60–0.98) for all patients, and 0.55–0.73 (95% CI 0.41–0.93) for patients with wild-type KRAS. Results Some applications of IPCW and TSE provided treatment effect estimates that closely matched the point-estimates and CIs of the expected truths. However, other applications produced estimates towards the boundaries of the expected truths, with some TSE applications producing estimates that lay outside the expected true confidence intervals. The RPSFTM performed relatively poorly, with all applications providing treatment effect estimates close to 1, often with extremely wide confidence intervals. Conclusions Adjustment analyses may provide unreliable results. How each method is applied must be scrutinised to assess reliability. |
| first_indexed | 2024-03-07T15:28:22Z |
| format | Article |
| id | doaj.art-08464f6ea30a43ceb22ea46f3c857e49 |
| institution | Directory Open Access Journal |
| issn | 1471-2288 |
| language | English |
| last_indexed | 2024-03-07T15:28:22Z |
| publishDate | 2024-01-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Research Methodology |
| spelling | doaj.art-08464f6ea30a43ceb22ea46f3c857e492024-03-05T16:33:32ZengBMCBMC Medical Research Methodology1471-22882024-01-0124111310.1186/s12874-024-02140-6A cautionary tale: an evaluation of the performance of treatment switching adjustment methods in a real world case studyNicholas R Latimer0Alice Dewdney1Marco Campioni2Sheffield Centre for Health and Related Research (SCHARR), University of SheffieldWeston Park Cancer Centre, Sheffield Teaching HospitalAmgen (Europe) GmbHAbstract Background Treatment switching in randomised controlled trials (RCTs) is a problem for health technology assessment when substantial proportions of patients switch onto effective treatments that would not be available in standard clinical practice. Often statistical methods are used to adjust for switching: these can be applied in different ways, and performance has been assessed in simulation studies, but not in real-world case studies. We assessed the performance of adjustment methods described in National Institute for Health and Care Excellence Decision Support Unit Technical Support Document 16, applying them to an RCT comparing panitumumab to best supportive care (BSC) in colorectal cancer, in which 76% of patients randomised to BSC switched onto panitumumab. The RCT resulted in intention-to-treat hazard ratios (HR) for overall survival (OS) of 1.00 (95% confidence interval [CI] 0.82–1.22) for all patients, and 0.99 (95% CI 0.75–1.29) for patients with wild-type KRAS (Kirsten rat sarcoma virus). Methods We tested several applications of inverse probability of censoring weights (IPCW), rank preserving structural failure time models (RPSFTM) and simple and complex two-stage estimation (TSE) to estimate treatment effects that would have been observed if BSC patients had not switched onto panitumumab. To assess the performance of these analyses we ascertained the true effectiveness of panitumumab based on: (i) subsequent RCTs of panitumumab that disallowed treatment switching; (ii) studies of cetuximab that disallowed treatment switching, (iii) analyses demonstrating that only patients with wild-type KRAS benefit from panitumumab. These sources suggest the true OS HR for panitumumab is 0.76–0.77 (95% CI 0.60–0.98) for all patients, and 0.55–0.73 (95% CI 0.41–0.93) for patients with wild-type KRAS. Results Some applications of IPCW and TSE provided treatment effect estimates that closely matched the point-estimates and CIs of the expected truths. However, other applications produced estimates towards the boundaries of the expected truths, with some TSE applications producing estimates that lay outside the expected true confidence intervals. The RPSFTM performed relatively poorly, with all applications providing treatment effect estimates close to 1, often with extremely wide confidence intervals. Conclusions Adjustment analyses may provide unreliable results. How each method is applied must be scrutinised to assess reliability.https://doi.org/10.1186/s12874-024-02140-6Treatment switchingTreatment crossoverSurvival analysisCancerInverse probability weightingRank preserving structural failure time model |
| spellingShingle | Nicholas R Latimer Alice Dewdney Marco Campioni A cautionary tale: an evaluation of the performance of treatment switching adjustment methods in a real world case study BMC Medical Research Methodology Treatment switching Treatment crossover Survival analysis Cancer Inverse probability weighting Rank preserving structural failure time model |
| title | A cautionary tale: an evaluation of the performance of treatment switching adjustment methods in a real world case study |
| title_full | A cautionary tale: an evaluation of the performance of treatment switching adjustment methods in a real world case study |
| title_fullStr | A cautionary tale: an evaluation of the performance of treatment switching adjustment methods in a real world case study |
| title_full_unstemmed | A cautionary tale: an evaluation of the performance of treatment switching adjustment methods in a real world case study |
| title_short | A cautionary tale: an evaluation of the performance of treatment switching adjustment methods in a real world case study |
| title_sort | cautionary tale an evaluation of the performance of treatment switching adjustment methods in a real world case study |
| topic | Treatment switching Treatment crossover Survival analysis Cancer Inverse probability weighting Rank preserving structural failure time model |
| url | https://doi.org/10.1186/s12874-024-02140-6 |
| work_keys_str_mv | AT nicholasrlatimer acautionarytaleanevaluationoftheperformanceoftreatmentswitchingadjustmentmethodsinarealworldcasestudy AT alicedewdney acautionarytaleanevaluationoftheperformanceoftreatmentswitchingadjustmentmethodsinarealworldcasestudy AT marcocampioni acautionarytaleanevaluationoftheperformanceoftreatmentswitchingadjustmentmethodsinarealworldcasestudy AT nicholasrlatimer cautionarytaleanevaluationoftheperformanceoftreatmentswitchingadjustmentmethodsinarealworldcasestudy AT alicedewdney cautionarytaleanevaluationoftheperformanceoftreatmentswitchingadjustmentmethodsinarealworldcasestudy AT marcocampioni cautionarytaleanevaluationoftheperformanceoftreatmentswitchingadjustmentmethodsinarealworldcasestudy |