TEsoNet: knowledge transfer in surgical phase recognition from laparoscopic sleeve gastrectomy to the laparoscopic part of Ivor–Lewis esophagectomy
Abstract Background Surgical phase recognition using computer vision presents an essential requirement for artificial intelligence-assisted analysis of surgical workflow. Its performance is heavily dependent on large amounts of annotated video data, which...
| Main Authors: | , , , , , , , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Springer US
2023
|
| Online Access: | https://hdl.handle.net/1721.1/148620 |
| _version_ | 1826205398820978688 |
|---|---|
| author | Eckhoff, J. A. Ban, Y. Rosman, G. Müller, D. T. Hashimoto, D. A. Witkowski, E. Babic, B. Rus, D. Bruns, C. Fuchs, H. F. Meireles, O. |
| author2 | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory |
| author_facet | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Eckhoff, J. A. Ban, Y. Rosman, G. Müller, D. T. Hashimoto, D. A. Witkowski, E. Babic, B. Rus, D. Bruns, C. Fuchs, H. F. Meireles, O. |
| author_sort | Eckhoff, J. A. |
| collection | MIT |
| description | Abstract
Background
Surgical phase recognition using computer vision presents an essential requirement for artificial intelligence-assisted analysis of surgical workflow. Its performance is heavily dependent on large amounts of annotated video data, which remain a limited resource, especially concerning highly specialized procedures. Knowledge transfer from common to more complex procedures can promote data efficiency. Phase recognition models trained on large, readily available datasets may be extrapolated and transferred to smaller datasets of different procedures to improve generalizability. The conditions under which transfer learning is appropriate and feasible remain to be established.
Methods
We defined ten operative phases for the laparoscopic part of Ivor-Lewis Esophagectomy through expert consensus. A dataset of 40 videos was annotated accordingly. The knowledge transfer capability of an established model architecture for phase recognition (CNN + LSTM) was adapted to generate a “Transferal Esophagectomy Network” (TEsoNet) for co-training and transfer learning from laparoscopic Sleeve Gastrectomy to the laparoscopic part of Ivor-Lewis Esophagectomy, exploring different training set compositions and training weights.
Results
The explored model architecture is capable of accurate phase detection in complex procedures, such as Esophagectomy, even with low quantities of training data. Knowledge transfer between two upper gastrointestinal procedures is feasible and achieves reasonable accuracy with respect to operative phases with high procedural overlap.
Conclusion
Robust phase recognition models can achieve reasonable yet phase-specific accuracy through transfer learning and co-training between two related procedures, even when exposed to small amounts of training data of the target procedure. Further exploration is required to determine appropriate data amounts, key characteristics of the training procedure and temporal annotation methods required for successful transferal phase recognition. Transfer learning across different procedures addressing small datasets may increase data efficiency. Finally, to enable the surgical application of AI for intraoperative risk mitigation, coverage of rare, specialized procedures needs to be explored.
Graphical abstract |
| first_indexed | 2024-09-23T13:12:04Z |
| format | Article |
| id | mit-1721.1/148620 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T13:12:04Z |
| publishDate | 2023 |
| publisher | Springer US |
| record_format | dspace |
| spelling | mit-1721.1/1486202024-01-12T21:01:03Z TEsoNet: knowledge transfer in surgical phase recognition from laparoscopic sleeve gastrectomy to the laparoscopic part of Ivor–Lewis esophagectomy Eckhoff, J. A. Ban, Y. Rosman, G. Müller, D. T. Hashimoto, D. A. Witkowski, E. Babic, B. Rus, D. Bruns, C. Fuchs, H. F. Meireles, O. Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Abstract Background Surgical phase recognition using computer vision presents an essential requirement for artificial intelligence-assisted analysis of surgical workflow. Its performance is heavily dependent on large amounts of annotated video data, which remain a limited resource, especially concerning highly specialized procedures. Knowledge transfer from common to more complex procedures can promote data efficiency. Phase recognition models trained on large, readily available datasets may be extrapolated and transferred to smaller datasets of different procedures to improve generalizability. The conditions under which transfer learning is appropriate and feasible remain to be established. Methods We defined ten operative phases for the laparoscopic part of Ivor-Lewis Esophagectomy through expert consensus. A dataset of 40 videos was annotated accordingly. The knowledge transfer capability of an established model architecture for phase recognition (CNN + LSTM) was adapted to generate a “Transferal Esophagectomy Network” (TEsoNet) for co-training and transfer learning from laparoscopic Sleeve Gastrectomy to the laparoscopic part of Ivor-Lewis Esophagectomy, exploring different training set compositions and training weights. Results The explored model architecture is capable of accurate phase detection in complex procedures, such as Esophagectomy, even with low quantities of training data. Knowledge transfer between two upper gastrointestinal procedures is feasible and achieves reasonable accuracy with respect to operative phases with high procedural overlap. Conclusion Robust phase recognition models can achieve reasonable yet phase-specific accuracy through transfer learning and co-training between two related procedures, even when exposed to small amounts of training data of the target procedure. Further exploration is required to determine appropriate data amounts, key characteristics of the training procedure and temporal annotation methods required for successful transferal phase recognition. Transfer learning across different procedures addressing small datasets may increase data efficiency. Finally, to enable the surgical application of AI for intraoperative risk mitigation, coverage of rare, specialized procedures needs to be explored. Graphical abstract 2023-03-20T14:36:00Z 2023-03-20T14:36:00Z 2023-03-17 2023-03-19T04:18:41Z Article http://purl.org/eprint/type/JournalArticle https://hdl.handle.net/1721.1/148620 Eckhoff, J. A., Ban, Y., Rosman, G., Müller, D. T., Hashimoto, D. A. et al. 2023. "TEsoNet: knowledge transfer in surgical phase recognition from laparoscopic sleeve gastrectomy to the laparoscopic part of Ivor–Lewis esophagectomy." PUBLISHER_CC en https://doi.org/10.1007/s00464-023-09971-2 Creative Commons Attribution https://creativecommons.org/licenses/by/4.0/ The Author(s) application/pdf Springer US Springer US |
| spellingShingle | Eckhoff, J. A. Ban, Y. Rosman, G. Müller, D. T. Hashimoto, D. A. Witkowski, E. Babic, B. Rus, D. Bruns, C. Fuchs, H. F. Meireles, O. TEsoNet: knowledge transfer in surgical phase recognition from laparoscopic sleeve gastrectomy to the laparoscopic part of Ivor–Lewis esophagectomy |
| title | TEsoNet: knowledge transfer in surgical phase recognition from laparoscopic sleeve gastrectomy to the laparoscopic part of Ivor–Lewis esophagectomy |
| title_full | TEsoNet: knowledge transfer in surgical phase recognition from laparoscopic sleeve gastrectomy to the laparoscopic part of Ivor–Lewis esophagectomy |
| title_fullStr | TEsoNet: knowledge transfer in surgical phase recognition from laparoscopic sleeve gastrectomy to the laparoscopic part of Ivor–Lewis esophagectomy |
| title_full_unstemmed | TEsoNet: knowledge transfer in surgical phase recognition from laparoscopic sleeve gastrectomy to the laparoscopic part of Ivor–Lewis esophagectomy |
| title_short | TEsoNet: knowledge transfer in surgical phase recognition from laparoscopic sleeve gastrectomy to the laparoscopic part of Ivor–Lewis esophagectomy |
| title_sort | tesonet knowledge transfer in surgical phase recognition from laparoscopic sleeve gastrectomy to the laparoscopic part of ivor lewis esophagectomy |
| url | https://hdl.handle.net/1721.1/148620 |
| work_keys_str_mv | AT eckhoffja tesonetknowledgetransferinsurgicalphaserecognitionfromlaparoscopicsleevegastrectomytothelaparoscopicpartofivorlewisesophagectomy AT bany tesonetknowledgetransferinsurgicalphaserecognitionfromlaparoscopicsleevegastrectomytothelaparoscopicpartofivorlewisesophagectomy AT rosmang tesonetknowledgetransferinsurgicalphaserecognitionfromlaparoscopicsleevegastrectomytothelaparoscopicpartofivorlewisesophagectomy AT mullerdt tesonetknowledgetransferinsurgicalphaserecognitionfromlaparoscopicsleevegastrectomytothelaparoscopicpartofivorlewisesophagectomy AT hashimotoda tesonetknowledgetransferinsurgicalphaserecognitionfromlaparoscopicsleevegastrectomytothelaparoscopicpartofivorlewisesophagectomy AT witkowskie tesonetknowledgetransferinsurgicalphaserecognitionfromlaparoscopicsleevegastrectomytothelaparoscopicpartofivorlewisesophagectomy AT babicb tesonetknowledgetransferinsurgicalphaserecognitionfromlaparoscopicsleevegastrectomytothelaparoscopicpartofivorlewisesophagectomy AT rusd tesonetknowledgetransferinsurgicalphaserecognitionfromlaparoscopicsleevegastrectomytothelaparoscopicpartofivorlewisesophagectomy AT brunsc tesonetknowledgetransferinsurgicalphaserecognitionfromlaparoscopicsleevegastrectomytothelaparoscopicpartofivorlewisesophagectomy AT fuchshf tesonetknowledgetransferinsurgicalphaserecognitionfromlaparoscopicsleevegastrectomytothelaparoscopicpartofivorlewisesophagectomy AT meireleso tesonetknowledgetransferinsurgicalphaserecognitionfromlaparoscopicsleevegastrectomytothelaparoscopicpartofivorlewisesophagectomy |