Assessing technical skill in ankle fracture surgery from the postoperative radiograph: pilot development and validation of a final product analysis core outcome set
Aims: To identify a core outcome set of postoperative radiographic measurements to assess technical skill in ankle fracture open reduction internal fixation (ORIF), and to validate these against Van der Vleuten’s criteria for effective assessment. Methods: An e-Delphi exercise was undertaken at a ma...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
The British Editorial Society of Bone & Joint Surgery
2022-06-01
|
| Series: | Bone & Joint Open |
| Subjects: | |
| Online Access: | https://online.boneandjoint.org.uk/doi/epdf/10.1302/2633-1462.36.BJO-2021-0212.R1 |
| _version_ | 1818530992269295616 |
|---|---|
| author | Hannah K. James James Griffin Giles T. R. Pattison |
| author_facet | Hannah K. James James Griffin Giles T. R. Pattison |
| author_sort | Hannah K. James |
| collection | DOAJ |
| description | Aims: To identify a core outcome set of postoperative radiographic measurements to assess technical skill in ankle fracture open reduction internal fixation (ORIF), and to validate these against Van der Vleuten’s criteria for effective assessment. Methods: An e-Delphi exercise was undertaken at a major trauma centre (n = 39) to identify relevant parameters. Feasibility was tested by two authors. Reliability and validity was tested using postoperative radiographs of ankle fracture operations performed by trainees enrolled in an educational trial (IRCTN 20431944). To determine construct validity, trainees were divided into novice (performed < ten cases at baseline) and intermediate groups (performed ≥ ten cases at baseline). To assess concurrent validity, the procedure-based assessment (PBA) was considered the gold standard. The inter-rater and intrarater reliability was tested using a randomly selected subset of 25 cases. Results: Overall, 235 ankle ORIFs were performed by 24 postgraduate year three to five trainees during ten months at nine NHS hospitals in England, UK. Overall, 42 PBAs were completed. The e-Delphi panel identified five ‘final product analysis’ parameters and defined acceptability thresholds: medial clear space (MCS); medial malleolar displacement (MMD); lateral malleolar displacement (LMD); tibiofibular clear space (TFCS) (all in mm); and talocrural angle (TCA) in degrees. Face validity, content validity, and feasibility were excellent. PBA global rating scale scores in this population showed excellent construct validity as continuous (p < 0.001) and categorical (p = 0.001) variables. Concurrent validity of all metrics was poor against PBA score. Intrarater reliability was substantial for all parameters (intraclass correlation coefficient (ICC) > 0.8), and inter-rater reliability was substantial for LMD, MMD, TCA, and moderate (ICC 0.61 to 0.80) for MCS and TFCS. Assessment was time efficient compared to PBA. Conclusion: Assessment of technical skill in ankle fracture surgery using the first postoperative radiograph satisfies the tested Van der Vleuten’s utility criteria for effective assessment. 'Final product analysis' assessment may be useful to assess skill transfer in the simulation-based research setting. Cite this article: Bone Jt Open 2022;3(6):502–509. |
| first_indexed | 2024-12-11T17:26:42Z |
| format | Article |
| id | doaj.art-cc61f807ba7a47d69a968d31d67ba2a7 |
| institution | Directory Open Access Journal |
| issn | 2633-1462 |
| language | English |
| last_indexed | 2024-12-11T17:26:42Z |
| publishDate | 2022-06-01 |
| publisher | The British Editorial Society of Bone & Joint Surgery |
| record_format | Article |
| series | Bone & Joint Open |
| spelling | doaj.art-cc61f807ba7a47d69a968d31d67ba2a72022-12-22T00:56:57ZengThe British Editorial Society of Bone & Joint SurgeryBone & Joint Open2633-14622022-06-013650250910.1302/2633-1462.36.BJO-2021-0212.R1Assessing technical skill in ankle fracture surgery from the postoperative radiograph: pilot development and validation of a final product analysis core outcome setHannah K. James0James Griffin1Giles T. R. Pattison2Warwick Clinical Trials Unit, Warwick Medical School, Coventry, UKWarwick Clinical Trials Unit, Warwick Medical School, Coventry, UKDepartment of Trauma & Orthopaedics, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, UKAims: To identify a core outcome set of postoperative radiographic measurements to assess technical skill in ankle fracture open reduction internal fixation (ORIF), and to validate these against Van der Vleuten’s criteria for effective assessment. Methods: An e-Delphi exercise was undertaken at a major trauma centre (n = 39) to identify relevant parameters. Feasibility was tested by two authors. Reliability and validity was tested using postoperative radiographs of ankle fracture operations performed by trainees enrolled in an educational trial (IRCTN 20431944). To determine construct validity, trainees were divided into novice (performed < ten cases at baseline) and intermediate groups (performed ≥ ten cases at baseline). To assess concurrent validity, the procedure-based assessment (PBA) was considered the gold standard. The inter-rater and intrarater reliability was tested using a randomly selected subset of 25 cases. Results: Overall, 235 ankle ORIFs were performed by 24 postgraduate year three to five trainees during ten months at nine NHS hospitals in England, UK. Overall, 42 PBAs were completed. The e-Delphi panel identified five ‘final product analysis’ parameters and defined acceptability thresholds: medial clear space (MCS); medial malleolar displacement (MMD); lateral malleolar displacement (LMD); tibiofibular clear space (TFCS) (all in mm); and talocrural angle (TCA) in degrees. Face validity, content validity, and feasibility were excellent. PBA global rating scale scores in this population showed excellent construct validity as continuous (p < 0.001) and categorical (p = 0.001) variables. Concurrent validity of all metrics was poor against PBA score. Intrarater reliability was substantial for all parameters (intraclass correlation coefficient (ICC) > 0.8), and inter-rater reliability was substantial for LMD, MMD, TCA, and moderate (ICC 0.61 to 0.80) for MCS and TFCS. Assessment was time efficient compared to PBA. Conclusion: Assessment of technical skill in ankle fracture surgery using the first postoperative radiograph satisfies the tested Van der Vleuten’s utility criteria for effective assessment. 'Final product analysis' assessment may be useful to assess skill transfer in the simulation-based research setting. Cite this article: Bone Jt Open 2022;3(6):502–509.https://online.boneandjoint.org.uk/doi/epdf/10.1302/2633-1462.36.BJO-2021-0212.R1surgical educationtrainingassessmentoutcome validationpostoperative radiographsankle fracture surgeryankle fracturesankleopen reduction internal fixation (orif)mcsintraclass correlation coefficient (icc)traumaradiographic measurementsglobal rating scale |
| spellingShingle | Hannah K. James James Griffin Giles T. R. Pattison Assessing technical skill in ankle fracture surgery from the postoperative radiograph: pilot development and validation of a final product analysis core outcome set Bone & Joint Open surgical education training assessment outcome validation postoperative radiographs ankle fracture surgery ankle fractures ankle open reduction internal fixation (orif) mcs intraclass correlation coefficient (icc) trauma radiographic measurements global rating scale |
| title | Assessing technical skill in ankle fracture surgery from the postoperative radiograph: pilot development and validation of a final product analysis core outcome set |
| title_full | Assessing technical skill in ankle fracture surgery from the postoperative radiograph: pilot development and validation of a final product analysis core outcome set |
| title_fullStr | Assessing technical skill in ankle fracture surgery from the postoperative radiograph: pilot development and validation of a final product analysis core outcome set |
| title_full_unstemmed | Assessing technical skill in ankle fracture surgery from the postoperative radiograph: pilot development and validation of a final product analysis core outcome set |
| title_short | Assessing technical skill in ankle fracture surgery from the postoperative radiograph: pilot development and validation of a final product analysis core outcome set |
| title_sort | assessing technical skill in ankle fracture surgery from the postoperative radiograph pilot development and validation of a final product analysis core outcome set |
| topic | surgical education training assessment outcome validation postoperative radiographs ankle fracture surgery ankle fractures ankle open reduction internal fixation (orif) mcs intraclass correlation coefficient (icc) trauma radiographic measurements global rating scale |
| url | https://online.boneandjoint.org.uk/doi/epdf/10.1302/2633-1462.36.BJO-2021-0212.R1 |
| work_keys_str_mv | AT hannahkjames assessingtechnicalskillinanklefracturesurgeryfromthepostoperativeradiographpilotdevelopmentandvalidationofafinalproductanalysiscoreoutcomeset AT jamesgriffin assessingtechnicalskillinanklefracturesurgeryfromthepostoperativeradiographpilotdevelopmentandvalidationofafinalproductanalysiscoreoutcomeset AT gilestrpattison assessingtechnicalskillinanklefracturesurgeryfromthepostoperativeradiographpilotdevelopmentandvalidationofafinalproductanalysiscoreoutcomeset |