Construction of human digital twin model based on multimodal data and its application in locomotion mode identification
With the increasing attention to the state and role of people in intelligent manufacturing, there is a strong demand for human-cyber-physical systems (HCPS) that focus on human-robot interaction. The existing intelligent manufacturing system cannot satisfy efficient human-robot collaborative work. H...
| Main Authors: | , , , , , , |
|---|---|
| Other Authors: | |
| Format: | Journal Article |
| Language: | English |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/173796 |
| _version_ | 1824453965581910016 |
|---|---|
| author | Zhong, Ruirui Hu, Bingtao Feng, Yixiong Zheng, Hao Hong, Zhaoxi Lou, Shanhe Tan, Jianrong |
| author2 | School of Mechanical and Aerospace Engineering |
| author_facet | School of Mechanical and Aerospace Engineering Zhong, Ruirui Hu, Bingtao Feng, Yixiong Zheng, Hao Hong, Zhaoxi Lou, Shanhe Tan, Jianrong |
| author_sort | Zhong, Ruirui |
| collection | NTU |
| description | With the increasing attention to the state and role of people in intelligent manufacturing, there is a strong demand for human-cyber-physical systems (HCPS) that focus on human-robot interaction. The existing intelligent manufacturing system cannot satisfy efficient human-robot collaborative work. However, unlike machines equipped with sensors, human characteristic information is difficult to be perceived and digitized instantly. In view of the high complexity and uncertainty of the human body, this paper proposes a framework for building a human digital twin (HDT) model based on multimodal data and expounds on the key technologies. Data acquisition system is built to dynamically acquire and update the body state data and physiological data of the human body and realize the digital expression of multi-source heterogeneous human body information. A bidirectional long short-term memory and convolutional neural network (BiLSTM-CNN) based network is devised to fuse multimodal human data and extract the spatiotemporal features, and the human locomotion mode identification is taken as an application case. A series of optimization experiments are carried out to improve the performance of the proposed BiLSTM-CNN-based network model. The proposed model is compared with traditional locomotion mode identification models. The experimental results proved the superiority of the HDT framework for human locomotion mode identification. |
| first_indexed | 2025-02-19T03:14:48Z |
| format | Journal Article |
| id | ntu-10356/173796 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2025-02-19T03:14:48Z |
| publishDate | 2024 |
| record_format | dspace |
| spelling | ntu-10356/1737962024-03-02T16:48:17Z Construction of human digital twin model based on multimodal data and its application in locomotion mode identification Zhong, Ruirui Hu, Bingtao Feng, Yixiong Zheng, Hao Hong, Zhaoxi Lou, Shanhe Tan, Jianrong School of Mechanical and Aerospace Engineering Engineering Human digital twin Human-cyber-physical system With the increasing attention to the state and role of people in intelligent manufacturing, there is a strong demand for human-cyber-physical systems (HCPS) that focus on human-robot interaction. The existing intelligent manufacturing system cannot satisfy efficient human-robot collaborative work. However, unlike machines equipped with sensors, human characteristic information is difficult to be perceived and digitized instantly. In view of the high complexity and uncertainty of the human body, this paper proposes a framework for building a human digital twin (HDT) model based on multimodal data and expounds on the key technologies. Data acquisition system is built to dynamically acquire and update the body state data and physiological data of the human body and realize the digital expression of multi-source heterogeneous human body information. A bidirectional long short-term memory and convolutional neural network (BiLSTM-CNN) based network is devised to fuse multimodal human data and extract the spatiotemporal features, and the human locomotion mode identification is taken as an application case. A series of optimization experiments are carried out to improve the performance of the proposed BiLSTM-CNN-based network model. The proposed model is compared with traditional locomotion mode identification models. The experimental results proved the superiority of the HDT framework for human locomotion mode identification. Published version Supported by National Natural Science Foundation of China (Grant Nos. 52205288, 52130501, 52075479) and Zhejiang Provincial Key Research & Development Program (Grant No. 2021C01110). 2024-02-27T08:00:18Z 2024-02-27T08:00:18Z 2023 Journal Article Zhong, R., Hu, B., Feng, Y., Zheng, H., Hong, Z., Lou, S. & Tan, J. (2023). Construction of human digital twin model based on multimodal data and its application in locomotion mode identification. Chinese Journal of Mechanical Engineering, 36(1). https://dx.doi.org/10.1186/s10033-023-00951-0 1000-9345 https://hdl.handle.net/10356/173796 10.1186/s10033-023-00951-0 2-s2.0-85174495655 1 36 en Chinese Journal of Mechanical Engineering © The Author(s) 2023. Open Access. This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. application/pdf |
| spellingShingle | Engineering Human digital twin Human-cyber-physical system Zhong, Ruirui Hu, Bingtao Feng, Yixiong Zheng, Hao Hong, Zhaoxi Lou, Shanhe Tan, Jianrong Construction of human digital twin model based on multimodal data and its application in locomotion mode identification |
| title | Construction of human digital twin model based on multimodal data and its application in locomotion mode identification |
| title_full | Construction of human digital twin model based on multimodal data and its application in locomotion mode identification |
| title_fullStr | Construction of human digital twin model based on multimodal data and its application in locomotion mode identification |
| title_full_unstemmed | Construction of human digital twin model based on multimodal data and its application in locomotion mode identification |
| title_short | Construction of human digital twin model based on multimodal data and its application in locomotion mode identification |
| title_sort | construction of human digital twin model based on multimodal data and its application in locomotion mode identification |
| topic | Engineering Human digital twin Human-cyber-physical system |
| url | https://hdl.handle.net/10356/173796 |
| work_keys_str_mv | AT zhongruirui constructionofhumandigitaltwinmodelbasedonmultimodaldataanditsapplicationinlocomotionmodeidentification AT hubingtao constructionofhumandigitaltwinmodelbasedonmultimodaldataanditsapplicationinlocomotionmodeidentification AT fengyixiong constructionofhumandigitaltwinmodelbasedonmultimodaldataanditsapplicationinlocomotionmodeidentification AT zhenghao constructionofhumandigitaltwinmodelbasedonmultimodaldataanditsapplicationinlocomotionmodeidentification AT hongzhaoxi constructionofhumandigitaltwinmodelbasedonmultimodaldataanditsapplicationinlocomotionmodeidentification AT loushanhe constructionofhumandigitaltwinmodelbasedonmultimodaldataanditsapplicationinlocomotionmodeidentification AT tanjianrong constructionofhumandigitaltwinmodelbasedonmultimodaldataanditsapplicationinlocomotionmodeidentification |