Discrimination, Reliability, Sensitivity, and Specificity of Robotic Surgical Proficiency Assessment With Global Evaluative Assessment of Robotic Skills and Binary Scoring Metrics: Results From a Randomized Controlled Trial
Objective:. To compare binary metrics and Global Evaluative Assessment of Robotic Skills (GEARS) evaluations of training outcome assessments for reliability, sensitivity, and specificity. Background:. GEARS–Likert-scale skills assessment are a widely accepted tool for robotic surgical training outco...
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Wolters Kluwer Health
2023-09-01
|
| Series: | Annals of Surgery Open |
| Online Access: | http://journals.lww.com/10.1097/AS9.0000000000000307 |
| _version_ | 1827780727237771264 |
|---|---|
| author | Ruben De Groote, MD Stefano Puliatti, MD Marco Amato, MD Elio Mazzone, MD Alessandro Larcher, MD Rui Farinha, MD Artur Paludo, MD Liesbeth Desender, MD, PhD Nicolas Hubert, MD Ben Van Cleynenbreugel, MD, PhD Brendan P. Bunting, PhD Alexandre Mottrie, MD, PhD Anthony G. Gallagher, PhD, DSc, MAE On behalf of the Junior ERUS/ YAU working group on robot-assisted surgery of the European Association of Urology and the ERUS Education Working Group. Collaborators: Giuseppe Rosiello, MD Pieter Uvin, MD, PhD Jasper Decoene, MD Tom Tuyten, MD Mathieu D’Hondt, MD Charles Chatzopoulos, MD Bart De Troyer, MD Filippo Turri, MD Paolo Dell’Oglio, MD Nikolaos Liakos, MD Carlo Andrea Bravi, MD Edward Lambert, MD Iulia Andras, MD Fabrizio Di Maida, MD Wouter Everaerts, MD, PhD |
| author_facet | Ruben De Groote, MD Stefano Puliatti, MD Marco Amato, MD Elio Mazzone, MD Alessandro Larcher, MD Rui Farinha, MD Artur Paludo, MD Liesbeth Desender, MD, PhD Nicolas Hubert, MD Ben Van Cleynenbreugel, MD, PhD Brendan P. Bunting, PhD Alexandre Mottrie, MD, PhD Anthony G. Gallagher, PhD, DSc, MAE On behalf of the Junior ERUS/ YAU working group on robot-assisted surgery of the European Association of Urology and the ERUS Education Working Group. Collaborators: Giuseppe Rosiello, MD Pieter Uvin, MD, PhD Jasper Decoene, MD Tom Tuyten, MD Mathieu D’Hondt, MD Charles Chatzopoulos, MD Bart De Troyer, MD Filippo Turri, MD Paolo Dell’Oglio, MD Nikolaos Liakos, MD Carlo Andrea Bravi, MD Edward Lambert, MD Iulia Andras, MD Fabrizio Di Maida, MD Wouter Everaerts, MD, PhD |
| author_sort | Ruben De Groote, MD |
| collection | DOAJ |
| description | Objective:. To compare binary metrics and Global Evaluative Assessment of Robotic Skills (GEARS) evaluations of training outcome assessments for reliability, sensitivity, and specificity.
Background:. GEARS–Likert-scale skills assessment are a widely accepted tool for robotic surgical training outcome evaluations. Proficiency-based progression (PBP) training is another methodology but uses binary performance metrics for evaluations.
Methods:. In a prospective, randomized, and blinded study, we compared conventional with PBP training for a robotic suturing, knot-tying anastomosis task. Thirty-six surgical residents from 16 Belgium residency programs were randomized. In the skills laboratory, the PBP group trained until they demonstrated a quantitatively defined proficiency benchmark. The conventional group were yoked to the same training time but without the proficiency requirement. The final trial was video recorded and assessed with binary metrics and GEARS by robotic surgeons blinded to individual, group, and residency program. Sensitivity and specificity of the two assessment methods were evaluated with area under the curve (AUC) and receiver operating characteristics (ROC) curves.
Results:. The PBP group made 42% fewer objectively assessed performance errors than the conventional group (P < 0.001) and scored 15% better on the GEARS assessment (P = 0.033). The mean interrater reliability for binary metrics and GEARS was 0.87 and 0.38, respectively. Binary total error metrics AUC was 97% and for GEARS 85%. With a sensitivity threshold of 0.8, false positives rates were 3% and 25% for, respectively, the binary and GEARS assessments.
Conclusions:. Binary metrics for scoring a robotic VUA task demonstrated better psychometric properties than the GEARS assessment. |
| first_indexed | 2024-03-11T15:09:10Z |
| format | Article |
| id | doaj.art-ba77ce581bbe4fd8ac19b27f821d7365 |
| institution | Directory Open Access Journal |
| issn | 2691-3593 |
| language | English |
| last_indexed | 2024-03-11T15:09:10Z |
| publishDate | 2023-09-01 |
| publisher | Wolters Kluwer Health |
| record_format | Article |
| series | Annals of Surgery Open |
| spelling | doaj.art-ba77ce581bbe4fd8ac19b27f821d73652023-10-30T03:37:09ZengWolters Kluwer HealthAnnals of Surgery Open2691-35932023-09-0143e30710.1097/AS9.0000000000000307202309000-00017Discrimination, Reliability, Sensitivity, and Specificity of Robotic Surgical Proficiency Assessment With Global Evaluative Assessment of Robotic Skills and Binary Scoring Metrics: Results From a Randomized Controlled TrialRuben De Groote, MD0Stefano Puliatti, MD1Marco Amato, MD2Elio Mazzone, MD3Alessandro Larcher, MD4Rui Farinha, MD5Artur Paludo, MD6Liesbeth Desender, MD, PhD7Nicolas Hubert, MD8Ben Van Cleynenbreugel, MD, PhD9Brendan P. Bunting, PhD10Alexandre Mottrie, MD, PhD11Anthony G. Gallagher, PhD, DSc, MAE12On behalf of the Junior ERUS/ YAU working group on robot-assisted surgery of the European Association of Urology and the ERUS Education Working Group. Collaborators:Giuseppe Rosiello, MD13Pieter Uvin, MD, PhD14Jasper Decoene, MD15Tom Tuyten, MD16Mathieu D’Hondt, MD17Charles Chatzopoulos, MD18Bart De Troyer, MD19Filippo Turri, MD20Paolo Dell’Oglio, MD21Nikolaos Liakos, MD22Carlo Andrea Bravi, MD23Edward Lambert, MD24Iulia Andras, MD25Fabrizio Di Maida, MD26Wouter Everaerts, MD, PhD27From the * ORSI Academy, Ghent, BelgiumFrom the * ORSI Academy, Ghent, BelgiumFrom the * ORSI Academy, Ghent, Belgium∥ Division of Oncology/Unit of Urology, URI, IRCCS Ospedale San Raffaele, Milan, Italy∥ Division of Oncology/Unit of Urology, URI, IRCCS Ospedale San Raffaele, Milan, ItalyFrom the * ORSI Academy, Ghent, BelgiumFrom the * ORSI Academy, Ghent, Belgium** Department of Thoracovascular Surgery, University Hospital Ghent, Ghent, Belgium†† Department of Urology, CHR de la Citadelle, Liège, Belgium‡ Department of Development and Regeneration, KU Leuven, Leuven, Belgium‡‡ School of Psychology, Ulster University, Coleraine, Northern Ireland, United KingdomFrom the * ORSI Academy, Ghent, BelgiumFrom the * ORSI Academy, Ghent, BelgiumFrom the * ORSI Academy, Ghent, Belgium∥∥ Department of Urology, AZ Sint-Jan, Bruges, Belgium¶¶ Department of Urology, OLV van Lourdes Hospital, Waregem, Belgium## Department of Urology, Jessa Hospital, Hasselt, Belgium*** Department of Surgery, AZ Groeninge, Kortrijk, Belgium††† Department of Urology, Chirec Hospital, Brussels, Belgium‡‡‡ Department of Urology, AZ Nikolaas, Sint-Niklaas, Belgium§ Department of Urology, University of Modena and Reggio Emilia, Modena, Italy§§§ Department of Urology, Niguarda Hospital, Milan, Italy∥∥∥ Prostate Center Northwest, Department of Urology, Pediatric Urology and Uro-Oncology, St. Antonius-Hospital, Gronau, GermanyFrom the * ORSI Academy, Ghent, Belgium† Department of Urology, OLV, Aalst, Belgium¶¶¶ Department of Urology, Iuliu Hatieganu University of Medicine and Pharmacy, Cluj-Napoca, Romania### Department of Urology, University of Florence, Florence, Italy**** Department of Cellular and Molecular Medicine, KU Leuven, Leuven, Belgium.Objective:. To compare binary metrics and Global Evaluative Assessment of Robotic Skills (GEARS) evaluations of training outcome assessments for reliability, sensitivity, and specificity. Background:. GEARS–Likert-scale skills assessment are a widely accepted tool for robotic surgical training outcome evaluations. Proficiency-based progression (PBP) training is another methodology but uses binary performance metrics for evaluations. Methods:. In a prospective, randomized, and blinded study, we compared conventional with PBP training for a robotic suturing, knot-tying anastomosis task. Thirty-six surgical residents from 16 Belgium residency programs were randomized. In the skills laboratory, the PBP group trained until they demonstrated a quantitatively defined proficiency benchmark. The conventional group were yoked to the same training time but without the proficiency requirement. The final trial was video recorded and assessed with binary metrics and GEARS by robotic surgeons blinded to individual, group, and residency program. Sensitivity and specificity of the two assessment methods were evaluated with area under the curve (AUC) and receiver operating characteristics (ROC) curves. Results:. The PBP group made 42% fewer objectively assessed performance errors than the conventional group (P < 0.001) and scored 15% better on the GEARS assessment (P = 0.033). The mean interrater reliability for binary metrics and GEARS was 0.87 and 0.38, respectively. Binary total error metrics AUC was 97% and for GEARS 85%. With a sensitivity threshold of 0.8, false positives rates were 3% and 25% for, respectively, the binary and GEARS assessments. Conclusions:. Binary metrics for scoring a robotic VUA task demonstrated better psychometric properties than the GEARS assessment.http://journals.lww.com/10.1097/AS9.0000000000000307 |
| spellingShingle | Ruben De Groote, MD Stefano Puliatti, MD Marco Amato, MD Elio Mazzone, MD Alessandro Larcher, MD Rui Farinha, MD Artur Paludo, MD Liesbeth Desender, MD, PhD Nicolas Hubert, MD Ben Van Cleynenbreugel, MD, PhD Brendan P. Bunting, PhD Alexandre Mottrie, MD, PhD Anthony G. Gallagher, PhD, DSc, MAE On behalf of the Junior ERUS/ YAU working group on robot-assisted surgery of the European Association of Urology and the ERUS Education Working Group. Collaborators: Giuseppe Rosiello, MD Pieter Uvin, MD, PhD Jasper Decoene, MD Tom Tuyten, MD Mathieu D’Hondt, MD Charles Chatzopoulos, MD Bart De Troyer, MD Filippo Turri, MD Paolo Dell’Oglio, MD Nikolaos Liakos, MD Carlo Andrea Bravi, MD Edward Lambert, MD Iulia Andras, MD Fabrizio Di Maida, MD Wouter Everaerts, MD, PhD Discrimination, Reliability, Sensitivity, and Specificity of Robotic Surgical Proficiency Assessment With Global Evaluative Assessment of Robotic Skills and Binary Scoring Metrics: Results From a Randomized Controlled Trial Annals of Surgery Open |
| title | Discrimination, Reliability, Sensitivity, and Specificity of Robotic Surgical Proficiency Assessment With Global Evaluative Assessment of Robotic Skills and Binary Scoring Metrics: Results From a Randomized Controlled Trial |
| title_full | Discrimination, Reliability, Sensitivity, and Specificity of Robotic Surgical Proficiency Assessment With Global Evaluative Assessment of Robotic Skills and Binary Scoring Metrics: Results From a Randomized Controlled Trial |
| title_fullStr | Discrimination, Reliability, Sensitivity, and Specificity of Robotic Surgical Proficiency Assessment With Global Evaluative Assessment of Robotic Skills and Binary Scoring Metrics: Results From a Randomized Controlled Trial |
| title_full_unstemmed | Discrimination, Reliability, Sensitivity, and Specificity of Robotic Surgical Proficiency Assessment With Global Evaluative Assessment of Robotic Skills and Binary Scoring Metrics: Results From a Randomized Controlled Trial |
| title_short | Discrimination, Reliability, Sensitivity, and Specificity of Robotic Surgical Proficiency Assessment With Global Evaluative Assessment of Robotic Skills and Binary Scoring Metrics: Results From a Randomized Controlled Trial |
| title_sort | discrimination reliability sensitivity and specificity of robotic surgical proficiency assessment with global evaluative assessment of robotic skills and binary scoring metrics results from a randomized controlled trial |
| url | http://journals.lww.com/10.1097/AS9.0000000000000307 |
| work_keys_str_mv | AT rubendegrootemd discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT stefanopuliattimd discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT marcoamatomd discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT eliomazzonemd discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT alessandrolarchermd discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT ruifarinhamd discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT arturpaludomd discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT liesbethdesendermdphd discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT nicolashubertmd discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT benvancleynenbreugelmdphd discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT brendanpbuntingphd discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT alexandremottriemdphd discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT anthonyggallagherphddscmae discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT onbehalfofthejuniorerusyauworkinggrouponrobotassistedsurgeryoftheeuropeanassociationofurologyandtheeruseducationworkinggroupcollaborators discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT giusepperosiellomd discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT pieteruvinmdphd discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT jasperdecoenemd discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT tomtuytenmd discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT mathieudhondtmd discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT charleschatzopoulosmd discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT bartdetroyermd discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT filippoturrimd discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT paolodellogliomd discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT nikolaosliakosmd discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT carloandreabravimd discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT edwardlambertmd discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT iuliaandrasmd discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT fabriziodimaidamd discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial AT woutereveraertsmdphd discriminationreliabilitysensitivityandspecificityofroboticsurgicalproficiencyassessmentwithglobalevaluativeassessmentofroboticskillsandbinaryscoringmetricsresultsfromarandomizedcontrolledtrial |