Real-time 3D scans of cardiac surgery using a single optical-see-through head-mounted display in a mobile setup
Microsoft HoloLens 2 (HL2) is often found in research and products as a cutting-edge device in Mixed Reality medical applications. One application is surgical telementoring, that allows a remote expert to support surgeries in real-time from afar. However, in this field of research two problems are e...
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2022-09-01
|
| Series: | Frontiers in Virtual Reality |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/frvir.2022.949360/full |
| _version_ | 1828228060511469568 |
|---|---|
| author | Bastian Dewitz Bastian Dewitz Roman Bibo Sobhan Moazemi Sebastian Kalkhoff Stephan Recker Artur Liebrecht Artur Lichtenberg Christian Geiger Frank Steinicke Hug Aubin Falko Schmid |
| author_facet | Bastian Dewitz Bastian Dewitz Roman Bibo Sobhan Moazemi Sebastian Kalkhoff Stephan Recker Artur Liebrecht Artur Lichtenberg Christian Geiger Frank Steinicke Hug Aubin Falko Schmid |
| author_sort | Bastian Dewitz |
| collection | DOAJ |
| description | Microsoft HoloLens 2 (HL2) is often found in research and products as a cutting-edge device in Mixed Reality medical applications. One application is surgical telementoring, that allows a remote expert to support surgeries in real-time from afar. However, in this field of research two problems are encountered: First, many systems rely on additional sensors to record the surgery in 3D which makes the deployment cumbersome. Second, clinical testing under real-world surgery conditions is only performed in a small number of research works. In this article, we present a standalone system that allows the capturing of 3D recordings of open cardiac surgeries under clinical conditions using only the existing sensors of HL2. We show difficulties that arose during development, especially related to the optical system of the HL2, and present how they can be solved. The system has successfully been used to record surgeries from the surgeons point of view and the obtained material can be used to reconstruct a 3D view for evaluation by an expert. In a preliminary study, we present a recording of a captured surgery under real-world clinical conditions to expert surgeons which estimate the quality of the recordings and their overall applicability for diagnosis and support. The study shows benefits from a 3D reconstruction compared to video-only transmission regarding perceived quality and feeling of immersion. |
| first_indexed | 2024-04-12T18:15:57Z |
| format | Article |
| id | doaj.art-672be618455d45eca4344270968330f1 |
| institution | Directory Open Access Journal |
| issn | 2673-4192 |
| language | English |
| last_indexed | 2024-04-12T18:15:57Z |
| publishDate | 2022-09-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Virtual Reality |
| spelling | doaj.art-672be618455d45eca4344270968330f12022-12-22T03:21:37ZengFrontiers Media S.A.Frontiers in Virtual Reality2673-41922022-09-01310.3389/frvir.2022.949360949360Real-time 3D scans of cardiac surgery using a single optical-see-through head-mounted display in a mobile setupBastian Dewitz0Bastian Dewitz1Roman Bibo2Sobhan Moazemi3Sebastian Kalkhoff4Stephan Recker5Artur Liebrecht6Artur Lichtenberg7Christian Geiger8Frank Steinicke9Hug Aubin10Falko Schmid11Digital Health Lab Düsseldorf, Department of Cardiac Surgery, Medical Faculty and University Hospital Düsseldorf, Heinrich-Heine Universität Düsseldorf, Düsseldorf, GermanyInstitute for Human-Computer Interaction, Department of Informatics, Universität Hamburg, Hamburg, GermanyDigital Health Lab Düsseldorf, Department of Cardiac Surgery, Medical Faculty and University Hospital Düsseldorf, Heinrich-Heine Universität Düsseldorf, Düsseldorf, GermanyDigital Health Lab Düsseldorf, Department of Cardiac Surgery, Medical Faculty and University Hospital Düsseldorf, Heinrich-Heine Universität Düsseldorf, Düsseldorf, GermanyDigital Health Lab Düsseldorf, Department of Cardiac Surgery, Medical Faculty and University Hospital Düsseldorf, Heinrich-Heine Universität Düsseldorf, Düsseldorf, GermanyComputer Science Faculty, Dortmund University of Applied Science and Arts, Dortmund, GermanyDigital Health Lab Düsseldorf, Department of Cardiac Surgery, Medical Faculty and University Hospital Düsseldorf, Heinrich-Heine Universität Düsseldorf, Düsseldorf, GermanyDigital Health Lab Düsseldorf, Department of Cardiac Surgery, Medical Faculty and University Hospital Düsseldorf, Heinrich-Heine Universität Düsseldorf, Düsseldorf, GermanyFaculty of Media, University of Applied Sciences Düsseldorf, Düsseldorf, GermanyInstitute for Human-Computer Interaction, Department of Informatics, Universität Hamburg, Hamburg, GermanyDigital Health Lab Düsseldorf, Department of Cardiac Surgery, Medical Faculty and University Hospital Düsseldorf, Heinrich-Heine Universität Düsseldorf, Düsseldorf, GermanyDigital Health Lab Düsseldorf, Department of Cardiac Surgery, Medical Faculty and University Hospital Düsseldorf, Heinrich-Heine Universität Düsseldorf, Düsseldorf, GermanyMicrosoft HoloLens 2 (HL2) is often found in research and products as a cutting-edge device in Mixed Reality medical applications. One application is surgical telementoring, that allows a remote expert to support surgeries in real-time from afar. However, in this field of research two problems are encountered: First, many systems rely on additional sensors to record the surgery in 3D which makes the deployment cumbersome. Second, clinical testing under real-world surgery conditions is only performed in a small number of research works. In this article, we present a standalone system that allows the capturing of 3D recordings of open cardiac surgeries under clinical conditions using only the existing sensors of HL2. We show difficulties that arose during development, especially related to the optical system of the HL2, and present how they can be solved. The system has successfully been used to record surgeries from the surgeons point of view and the obtained material can be used to reconstruct a 3D view for evaluation by an expert. In a preliminary study, we present a recording of a captured surgery under real-world clinical conditions to expert surgeons which estimate the quality of the recordings and their overall applicability for diagnosis and support. The study shows benefits from a 3D reconstruction compared to video-only transmission regarding perceived quality and feeling of immersion.https://www.frontiersin.org/articles/10.3389/frvir.2022.949360/fullHoloLens 2telemedicinetelementoringcardiac surgeryaugmented realityvirtual reality |
| spellingShingle | Bastian Dewitz Bastian Dewitz Roman Bibo Sobhan Moazemi Sebastian Kalkhoff Stephan Recker Artur Liebrecht Artur Lichtenberg Christian Geiger Frank Steinicke Hug Aubin Falko Schmid Real-time 3D scans of cardiac surgery using a single optical-see-through head-mounted display in a mobile setup Frontiers in Virtual Reality HoloLens 2 telemedicine telementoring cardiac surgery augmented reality virtual reality |
| title | Real-time 3D scans of cardiac surgery using a single optical-see-through head-mounted display in a mobile setup |
| title_full | Real-time 3D scans of cardiac surgery using a single optical-see-through head-mounted display in a mobile setup |
| title_fullStr | Real-time 3D scans of cardiac surgery using a single optical-see-through head-mounted display in a mobile setup |
| title_full_unstemmed | Real-time 3D scans of cardiac surgery using a single optical-see-through head-mounted display in a mobile setup |
| title_short | Real-time 3D scans of cardiac surgery using a single optical-see-through head-mounted display in a mobile setup |
| title_sort | real time 3d scans of cardiac surgery using a single optical see through head mounted display in a mobile setup |
| topic | HoloLens 2 telemedicine telementoring cardiac surgery augmented reality virtual reality |
| url | https://www.frontiersin.org/articles/10.3389/frvir.2022.949360/full |
| work_keys_str_mv | AT bastiandewitz realtime3dscansofcardiacsurgeryusingasingleopticalseethroughheadmounteddisplayinamobilesetup AT bastiandewitz realtime3dscansofcardiacsurgeryusingasingleopticalseethroughheadmounteddisplayinamobilesetup AT romanbibo realtime3dscansofcardiacsurgeryusingasingleopticalseethroughheadmounteddisplayinamobilesetup AT sobhanmoazemi realtime3dscansofcardiacsurgeryusingasingleopticalseethroughheadmounteddisplayinamobilesetup AT sebastiankalkhoff realtime3dscansofcardiacsurgeryusingasingleopticalseethroughheadmounteddisplayinamobilesetup AT stephanrecker realtime3dscansofcardiacsurgeryusingasingleopticalseethroughheadmounteddisplayinamobilesetup AT arturliebrecht realtime3dscansofcardiacsurgeryusingasingleopticalseethroughheadmounteddisplayinamobilesetup AT arturlichtenberg realtime3dscansofcardiacsurgeryusingasingleopticalseethroughheadmounteddisplayinamobilesetup AT christiangeiger realtime3dscansofcardiacsurgeryusingasingleopticalseethroughheadmounteddisplayinamobilesetup AT franksteinicke realtime3dscansofcardiacsurgeryusingasingleopticalseethroughheadmounteddisplayinamobilesetup AT hugaubin realtime3dscansofcardiacsurgeryusingasingleopticalseethroughheadmounteddisplayinamobilesetup AT falkoschmid realtime3dscansofcardiacsurgeryusingasingleopticalseethroughheadmounteddisplayinamobilesetup |