Extended Reality Telemedicine Collaboration System Using Patient Avatar Based on 3D Body Pose Estimation
Recent advances in extended reality (XR) technology have opened the possibility of significantly improving telemedicine systems. This is primarily achieved by transferring 3D information about patient state, which is utilized to create more immersive experiences on VR/AR headsets. In this paper, we...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-12-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/24/1/27 |
| _version_ | 1797358220432375808 |
|---|---|
| author | Matko Šarić Mladen Russo Luka Kraljević Davor Meter |
| author_facet | Matko Šarić Mladen Russo Luka Kraljević Davor Meter |
| author_sort | Matko Šarić |
| collection | DOAJ |
| description | Recent advances in extended reality (XR) technology have opened the possibility of significantly improving telemedicine systems. This is primarily achieved by transferring 3D information about patient state, which is utilized to create more immersive experiences on VR/AR headsets. In this paper, we propose an XR-based telemedicine collaboration system in which the patient is represented as a 3D avatar in an XR space shared by local and remote clinicians. The proposed system consists of an AR client application running on Microsoft HoloLens 2 used by a local clinician, a VR client application running on the HTC vive Pro used by a remote clinician, and a backend part running on the server. The patient is captured by a camera on the AR side, and the 3D body pose estimation is performed on frames from this camera stream to form a 3D patient avatar. Additionally, the AR and VR sides can interact with the patient avatar via virtual hands, and annotations can be performed on a 3D model. The main contribution of our work is the use of 3D body pose estimation for the creation of a 3D patient avatar. In this way, 3D body reconstruction using depth cameras is avoided, which reduces system complexity and hardware and network resources. Another contribution is the novel architecture of the proposed system, where audio and video streaming are realized using WebRTC protocol. The performance evaluation showed that the proposed system ensures high frame rates for both AR and VR client applications, while the processing latency remains at an acceptable level. |
| first_indexed | 2024-03-08T14:58:43Z |
| format | Article |
| id | doaj.art-c8487c4b5eb2485b876325653aee0d66 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-08T14:58:43Z |
| publishDate | 2023-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-c8487c4b5eb2485b876325653aee0d662024-01-10T15:08:13ZengMDPI AGSensors1424-82202023-12-012412710.3390/s24010027Extended Reality Telemedicine Collaboration System Using Patient Avatar Based on 3D Body Pose EstimationMatko Šarić0Mladen Russo1Luka Kraljević2Davor Meter3Faculty of Electrical Engineering, Mechanical Engineering and Naval Architecture, University of Split, Ruđera Boškovića 32, 21000 Split, CroatiaFaculty of Electrical Engineering, Mechanical Engineering and Naval Architecture, University of Split, Ruđera Boškovića 32, 21000 Split, CroatiaFaculty of Electrical Engineering, Mechanical Engineering and Naval Architecture, University of Split, Ruđera Boškovića 32, 21000 Split, CroatiaFaculty of Electrical Engineering, Mechanical Engineering and Naval Architecture, University of Split, Ruđera Boškovića 32, 21000 Split, CroatiaRecent advances in extended reality (XR) technology have opened the possibility of significantly improving telemedicine systems. This is primarily achieved by transferring 3D information about patient state, which is utilized to create more immersive experiences on VR/AR headsets. In this paper, we propose an XR-based telemedicine collaboration system in which the patient is represented as a 3D avatar in an XR space shared by local and remote clinicians. The proposed system consists of an AR client application running on Microsoft HoloLens 2 used by a local clinician, a VR client application running on the HTC vive Pro used by a remote clinician, and a backend part running on the server. The patient is captured by a camera on the AR side, and the 3D body pose estimation is performed on frames from this camera stream to form a 3D patient avatar. Additionally, the AR and VR sides can interact with the patient avatar via virtual hands, and annotations can be performed on a 3D model. The main contribution of our work is the use of 3D body pose estimation for the creation of a 3D patient avatar. In this way, 3D body reconstruction using depth cameras is avoided, which reduces system complexity and hardware and network resources. Another contribution is the novel architecture of the proposed system, where audio and video streaming are realized using WebRTC protocol. The performance evaluation showed that the proposed system ensures high frame rates for both AR and VR client applications, while the processing latency remains at an acceptable level.https://www.mdpi.com/1424-8220/24/1/27extended realitytelemedicine systemavatar3D body pose estimation |
| spellingShingle | Matko Šarić Mladen Russo Luka Kraljević Davor Meter Extended Reality Telemedicine Collaboration System Using Patient Avatar Based on 3D Body Pose Estimation Sensors extended reality telemedicine system avatar 3D body pose estimation |
| title | Extended Reality Telemedicine Collaboration System Using Patient Avatar Based on 3D Body Pose Estimation |
| title_full | Extended Reality Telemedicine Collaboration System Using Patient Avatar Based on 3D Body Pose Estimation |
| title_fullStr | Extended Reality Telemedicine Collaboration System Using Patient Avatar Based on 3D Body Pose Estimation |
| title_full_unstemmed | Extended Reality Telemedicine Collaboration System Using Patient Avatar Based on 3D Body Pose Estimation |
| title_short | Extended Reality Telemedicine Collaboration System Using Patient Avatar Based on 3D Body Pose Estimation |
| title_sort | extended reality telemedicine collaboration system using patient avatar based on 3d body pose estimation |
| topic | extended reality telemedicine system avatar 3D body pose estimation |
| url | https://www.mdpi.com/1424-8220/24/1/27 |
| work_keys_str_mv | AT matkosaric extendedrealitytelemedicinecollaborationsystemusingpatientavatarbasedon3dbodyposeestimation AT mladenrusso extendedrealitytelemedicinecollaborationsystemusingpatientavatarbasedon3dbodyposeestimation AT lukakraljevic extendedrealitytelemedicinecollaborationsystemusingpatientavatarbasedon3dbodyposeestimation AT davormeter extendedrealitytelemedicinecollaborationsystemusingpatientavatarbasedon3dbodyposeestimation |