HoloKinect: Holographic 3D Video Conferencing
Recent world events have caused a dramatic rise in the use of video conferencing solutions such as Zoom and FaceTime. Although 3D capture and display technologies are becoming common in consumer products (e.g., Apple iPhone TrueDepth sensors, Microsoft Kinect devices, and Meta Quest VR headsets), 3D...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-10-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/22/21/8118 |
| _version_ | 1797466526667767808 |
|---|---|
| author | Stephen Siemonsma Tyler Bell |
| author_facet | Stephen Siemonsma Tyler Bell |
| author_sort | Stephen Siemonsma |
| collection | DOAJ |
| description | Recent world events have caused a dramatic rise in the use of video conferencing solutions such as Zoom and FaceTime. Although 3D capture and display technologies are becoming common in consumer products (e.g., Apple iPhone TrueDepth sensors, Microsoft Kinect devices, and Meta Quest VR headsets), 3D telecommunication has not yet seen any appreciable adoption. Researchers have made great progress in developing advanced 3D telepresence systems, but often with burdensome hardware and network requirements. In this work, we present HoloKinect, an open-source, user-friendly, and GPU-accelerated platform for enabling live, two-way 3D video conferencing on commodity hardware and a standard broadband internet connection. A Microsoft Azure Kinect serves as the capture device and a Looking Glass Portrait multiscopically displays the final reconstructed 3D mesh for a hologram-like effect. HoloKinect packs color and depth information into a single video stream, leveraging multiwavelength depth (MWD) encoding to store depth maps in standard RGB video frames. The video stream is compressed with highly optimized and hardware-accelerated video codecs such as H.264. A search of the depth and video encoding parameter space was performed to analyze the quantitative and qualitative losses resulting from HoloKinect’s lossy compression scheme. Visual results were acceptable at all tested bitrates (3–30 Mbps), while the best results were achieved with higher video bitrates and full 4:4:4 chroma sampling. RMSE values of the recovered depth measurements were low across all settings permutations. |
| first_indexed | 2024-03-09T18:41:02Z |
| format | Article |
| id | doaj.art-5ccbd841ba70435199a1e356a6267319 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-09T18:41:02Z |
| publishDate | 2022-10-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-5ccbd841ba70435199a1e356a62673192023-11-24T06:43:00ZengMDPI AGSensors1424-82202022-10-012221811810.3390/s22218118HoloKinect: Holographic 3D Video ConferencingStephen Siemonsma0Tyler Bell1Department of Electrical and Computer Engineering, University of Iowa, Iowa City, IA 52242, USADepartment of Electrical and Computer Engineering, University of Iowa, Iowa City, IA 52242, USARecent world events have caused a dramatic rise in the use of video conferencing solutions such as Zoom and FaceTime. Although 3D capture and display technologies are becoming common in consumer products (e.g., Apple iPhone TrueDepth sensors, Microsoft Kinect devices, and Meta Quest VR headsets), 3D telecommunication has not yet seen any appreciable adoption. Researchers have made great progress in developing advanced 3D telepresence systems, but often with burdensome hardware and network requirements. In this work, we present HoloKinect, an open-source, user-friendly, and GPU-accelerated platform for enabling live, two-way 3D video conferencing on commodity hardware and a standard broadband internet connection. A Microsoft Azure Kinect serves as the capture device and a Looking Glass Portrait multiscopically displays the final reconstructed 3D mesh for a hologram-like effect. HoloKinect packs color and depth information into a single video stream, leveraging multiwavelength depth (MWD) encoding to store depth maps in standard RGB video frames. The video stream is compressed with highly optimized and hardware-accelerated video codecs such as H.264. A search of the depth and video encoding parameter space was performed to analyze the quantitative and qualitative losses resulting from HoloKinect’s lossy compression scheme. Visual results were acceptable at all tested bitrates (3–30 Mbps), while the best results were achieved with higher video bitrates and full 4:4:4 chroma sampling. RMSE values of the recovered depth measurements were low across all settings permutations.https://www.mdpi.com/1424-8220/22/21/81183D video conferencing3D video streamingdepth encodingtelepresenceMicrosoft Kinectmultiscopic displays |
| spellingShingle | Stephen Siemonsma Tyler Bell HoloKinect: Holographic 3D Video Conferencing Sensors 3D video conferencing 3D video streaming depth encoding telepresence Microsoft Kinect multiscopic displays |
| title | HoloKinect: Holographic 3D Video Conferencing |
| title_full | HoloKinect: Holographic 3D Video Conferencing |
| title_fullStr | HoloKinect: Holographic 3D Video Conferencing |
| title_full_unstemmed | HoloKinect: Holographic 3D Video Conferencing |
| title_short | HoloKinect: Holographic 3D Video Conferencing |
| title_sort | holokinect holographic 3d video conferencing |
| topic | 3D video conferencing 3D video streaming depth encoding telepresence Microsoft Kinect multiscopic displays |
| url | https://www.mdpi.com/1424-8220/22/21/8118 |
| work_keys_str_mv | AT stephensiemonsma holokinectholographic3dvideoconferencing AT tylerbell holokinectholographic3dvideoconferencing |