Upper Body Pose Estimation Using Deep Learning for a Virtual Reality Avatar
With the popularity of virtual reality (VR) games and devices, demand is increasing for estimating and displaying user motion in VR applications. Most pose estimation methods for VR avatars exploit inverse kinematics (IK) and online motion capture methods. In contrast to existing approaches, we aim...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-02-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/13/4/2460 |
| _version_ | 1797622556538175488 |
|---|---|
| author | Taravat Anvari Kyoungju Park Ganghyun Kim |
| author_facet | Taravat Anvari Kyoungju Park Ganghyun Kim |
| author_sort | Taravat Anvari |
| collection | DOAJ |
| description | With the popularity of virtual reality (VR) games and devices, demand is increasing for estimating and displaying user motion in VR applications. Most pose estimation methods for VR avatars exploit inverse kinematics (IK) and online motion capture methods. In contrast to existing approaches, we aim for a stable process with less computation, usable in a small space. Therefore, our strategy has minimum latency for VR device users, from high-performance to low-performance, in multi-user applications over the network. In this study, we estimate the upper body pose of a VR user in real time using a deep learning method. We propose a novel method inspired by a classical regression model and trained with 3D motion capture data. Thus, our design uses a convolutional neural network (CNN)-based architecture from the joint information of motion capture data and modifies the network input and output to obtain input from a head and both hands. After feeding the model with properly normalized inputs, a head-mounted display (HMD), and two controllers, we render the user’s corresponding avatar in VR applications. We used our proposed pose estimation method to build single-user and multi-user applications, measure their performance, conduct a user study, and compare the results with previous methods for VR avatars. |
| first_indexed | 2024-03-11T09:11:59Z |
| format | Article |
| id | doaj.art-619011913b9d4de697d3c4db56a68fcc |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-11T09:11:59Z |
| publishDate | 2023-02-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-619011913b9d4de697d3c4db56a68fcc2023-11-16T18:56:01ZengMDPI AGApplied Sciences2076-34172023-02-01134246010.3390/app13042460Upper Body Pose Estimation Using Deep Learning for a Virtual Reality AvatarTaravat Anvari0Kyoungju Park1Ganghyun Kim2Department of Computer Science and Engineering, Chung-Ang University, Seoul 06974, Republic of KoreaDepartment of Computer Science and Engineering, Chung-Ang University, Seoul 06974, Republic of KoreaDepartment of Computer Science and Engineering, Chung-Ang University, Seoul 06974, Republic of KoreaWith the popularity of virtual reality (VR) games and devices, demand is increasing for estimating and displaying user motion in VR applications. Most pose estimation methods for VR avatars exploit inverse kinematics (IK) and online motion capture methods. In contrast to existing approaches, we aim for a stable process with less computation, usable in a small space. Therefore, our strategy has minimum latency for VR device users, from high-performance to low-performance, in multi-user applications over the network. In this study, we estimate the upper body pose of a VR user in real time using a deep learning method. We propose a novel method inspired by a classical regression model and trained with 3D motion capture data. Thus, our design uses a convolutional neural network (CNN)-based architecture from the joint information of motion capture data and modifies the network input and output to obtain input from a head and both hands. After feeding the model with properly normalized inputs, a head-mounted display (HMD), and two controllers, we render the user’s corresponding avatar in VR applications. We used our proposed pose estimation method to build single-user and multi-user applications, measure their performance, conduct a user study, and compare the results with previous methods for VR avatars.https://www.mdpi.com/2076-3417/13/4/2460avatarimmersionpose estimationvirtual reality |
| spellingShingle | Taravat Anvari Kyoungju Park Ganghyun Kim Upper Body Pose Estimation Using Deep Learning for a Virtual Reality Avatar Applied Sciences avatar immersion pose estimation virtual reality |
| title | Upper Body Pose Estimation Using Deep Learning for a Virtual Reality Avatar |
| title_full | Upper Body Pose Estimation Using Deep Learning for a Virtual Reality Avatar |
| title_fullStr | Upper Body Pose Estimation Using Deep Learning for a Virtual Reality Avatar |
| title_full_unstemmed | Upper Body Pose Estimation Using Deep Learning for a Virtual Reality Avatar |
| title_short | Upper Body Pose Estimation Using Deep Learning for a Virtual Reality Avatar |
| title_sort | upper body pose estimation using deep learning for a virtual reality avatar |
| topic | avatar immersion pose estimation virtual reality |
| url | https://www.mdpi.com/2076-3417/13/4/2460 |
| work_keys_str_mv | AT taravatanvari upperbodyposeestimationusingdeeplearningforavirtualrealityavatar AT kyoungjupark upperbodyposeestimationusingdeeplearningforavirtualrealityavatar AT ganghyunkim upperbodyposeestimationusingdeeplearningforavirtualrealityavatar |