Endoscope Automation Framework with Hierarchical Control and Interactive Perception for Multi-Tool Tracking in Minimally Invasive Surgery
In the context of Minimally Invasive Surgery, surgeons mainly rely on visual feedback during medical operations. In common procedures such as tissue resection, the automation of endoscopic control is crucial yet challenging, particularly due to the interactive dynamics of multi-agent operations and...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-12-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/23/24/9865 |
| _version_ | 1797379347225509888 |
|---|---|
| author | Khusniddin Fozilov Jacinto Colan Ana Davila Kazunari Misawa Jie Qiu Yuichiro Hayashi Kensaku Mori Yasuhisa Hasegawa |
| author_facet | Khusniddin Fozilov Jacinto Colan Ana Davila Kazunari Misawa Jie Qiu Yuichiro Hayashi Kensaku Mori Yasuhisa Hasegawa |
| author_sort | Khusniddin Fozilov |
| collection | DOAJ |
| description | In the context of Minimally Invasive Surgery, surgeons mainly rely on visual feedback during medical operations. In common procedures such as tissue resection, the automation of endoscopic control is crucial yet challenging, particularly due to the interactive dynamics of multi-agent operations and the necessity for real-time adaptation. This paper introduces a novel framework that unites a Hierarchical Quadratic Programming controller with an advanced interactive perception module. This integration addresses the need for adaptive visual field control and robust tool tracking in the operating scene, ensuring that surgeons and assistants have optimal viewpoint throughout the surgical task. The proposed framework handles multiple objectives within predefined thresholds, ensuring efficient tracking even amidst changes in operating backgrounds, varying lighting conditions, and partial occlusions. Empirical validations in scenarios involving single, double, and quadruple tool tracking during tissue resection tasks have underscored the system’s robustness and adaptability. The positive feedback from user studies, coupled with the low cognitive and physical strain reported by surgeons and assistants, highlight the system’s potential for real-world application. |
| first_indexed | 2024-03-08T20:21:51Z |
| format | Article |
| id | doaj.art-3721741d99ad42a2aebf973b65d4de61 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-08T20:21:51Z |
| publishDate | 2023-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-3721741d99ad42a2aebf973b65d4de612023-12-22T14:41:16ZengMDPI AGSensors1424-82202023-12-012324986510.3390/s23249865Endoscope Automation Framework with Hierarchical Control and Interactive Perception for Multi-Tool Tracking in Minimally Invasive SurgeryKhusniddin Fozilov0Jacinto Colan1Ana Davila2Kazunari Misawa3Jie Qiu4Yuichiro Hayashi5Kensaku Mori6Yasuhisa Hasegawa7Department of Micro-Nano Mechanical Science and Engineering, Nagoya University, Furo-cho, Chikusa-ku, Nagoya 464-8603, Aichi, JapanDepartment of Micro-Nano Mechanical Science and Engineering, Nagoya University, Furo-cho, Chikusa-ku, Nagoya 464-8603, Aichi, JapanInstitutes of Innovation for Future Society, Nagoya University, Furo-cho, Chikusa-ku, Nagoya 464-8603, Aichi, JapanAichi Cancer Center Hospital, Chikusa Ward, Nagoya 464-8681, Aichi, JapanGraduate School of Informatics, Nagoya University, Chikusa Ward, Nagoya 464-8601, Aichi, JapanGraduate School of Informatics, Nagoya University, Chikusa Ward, Nagoya 464-8601, Aichi, JapanGraduate School of Informatics, Nagoya University, Chikusa Ward, Nagoya 464-8601, Aichi, JapanInstitutes of Innovation for Future Society, Nagoya University, Furo-cho, Chikusa-ku, Nagoya 464-8603, Aichi, JapanIn the context of Minimally Invasive Surgery, surgeons mainly rely on visual feedback during medical operations. In common procedures such as tissue resection, the automation of endoscopic control is crucial yet challenging, particularly due to the interactive dynamics of multi-agent operations and the necessity for real-time adaptation. This paper introduces a novel framework that unites a Hierarchical Quadratic Programming controller with an advanced interactive perception module. This integration addresses the need for adaptive visual field control and robust tool tracking in the operating scene, ensuring that surgeons and assistants have optimal viewpoint throughout the surgical task. The proposed framework handles multiple objectives within predefined thresholds, ensuring efficient tracking even amidst changes in operating backgrounds, varying lighting conditions, and partial occlusions. Empirical validations in scenarios involving single, double, and quadruple tool tracking during tissue resection tasks have underscored the system’s robustness and adaptability. The positive feedback from user studies, coupled with the low cognitive and physical strain reported by surgeons and assistants, highlight the system’s potential for real-world application.https://www.mdpi.com/1424-8220/23/24/9865autonomous systemsHierarchical Quadratic Programmingrobotic surgerysegmentation and trackingvisual servoing |
| spellingShingle | Khusniddin Fozilov Jacinto Colan Ana Davila Kazunari Misawa Jie Qiu Yuichiro Hayashi Kensaku Mori Yasuhisa Hasegawa Endoscope Automation Framework with Hierarchical Control and Interactive Perception for Multi-Tool Tracking in Minimally Invasive Surgery Sensors autonomous systems Hierarchical Quadratic Programming robotic surgery segmentation and tracking visual servoing |
| title | Endoscope Automation Framework with Hierarchical Control and Interactive Perception for Multi-Tool Tracking in Minimally Invasive Surgery |
| title_full | Endoscope Automation Framework with Hierarchical Control and Interactive Perception for Multi-Tool Tracking in Minimally Invasive Surgery |
| title_fullStr | Endoscope Automation Framework with Hierarchical Control and Interactive Perception for Multi-Tool Tracking in Minimally Invasive Surgery |
| title_full_unstemmed | Endoscope Automation Framework with Hierarchical Control and Interactive Perception for Multi-Tool Tracking in Minimally Invasive Surgery |
| title_short | Endoscope Automation Framework with Hierarchical Control and Interactive Perception for Multi-Tool Tracking in Minimally Invasive Surgery |
| title_sort | endoscope automation framework with hierarchical control and interactive perception for multi tool tracking in minimally invasive surgery |
| topic | autonomous systems Hierarchical Quadratic Programming robotic surgery segmentation and tracking visual servoing |
| url | https://www.mdpi.com/1424-8220/23/24/9865 |
| work_keys_str_mv | AT khusniddinfozilov endoscopeautomationframeworkwithhierarchicalcontrolandinteractiveperceptionformultitooltrackinginminimallyinvasivesurgery AT jacintocolan endoscopeautomationframeworkwithhierarchicalcontrolandinteractiveperceptionformultitooltrackinginminimallyinvasivesurgery AT anadavila endoscopeautomationframeworkwithhierarchicalcontrolandinteractiveperceptionformultitooltrackinginminimallyinvasivesurgery AT kazunarimisawa endoscopeautomationframeworkwithhierarchicalcontrolandinteractiveperceptionformultitooltrackinginminimallyinvasivesurgery AT jieqiu endoscopeautomationframeworkwithhierarchicalcontrolandinteractiveperceptionformultitooltrackinginminimallyinvasivesurgery AT yuichirohayashi endoscopeautomationframeworkwithhierarchicalcontrolandinteractiveperceptionformultitooltrackinginminimallyinvasivesurgery AT kensakumori endoscopeautomationframeworkwithhierarchicalcontrolandinteractiveperceptionformultitooltrackinginminimallyinvasivesurgery AT yasuhisahasegawa endoscopeautomationframeworkwithhierarchicalcontrolandinteractiveperceptionformultitooltrackinginminimallyinvasivesurgery |