Smart Task Assistance in Mixed Reality for Astronauts
Mixed reality (MR) registers virtual information and real objects and is an effective way to supplement astronaut training. Spatial anchors are generally used to perform virtual–real fusion in static scenes but cannot handle movable objects. To address this issue, we propose a smart task assistance...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-04-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/23/9/4344 |
| _version_ | 1797601661435248640 |
|---|---|
| author | Qingwei Sun Wei Chen Jiangang Chao Wanhong Lin Zhenying Xu Ruizhi Cao |
| author_facet | Qingwei Sun Wei Chen Jiangang Chao Wanhong Lin Zhenying Xu Ruizhi Cao |
| author_sort | Qingwei Sun |
| collection | DOAJ |
| description | Mixed reality (MR) registers virtual information and real objects and is an effective way to supplement astronaut training. Spatial anchors are generally used to perform virtual–real fusion in static scenes but cannot handle movable objects. To address this issue, we propose a smart task assistance method based on object detection and point cloud alignment. Specifically, both fixed and movable objects are detected automatically. In parallel, poses are estimated with no dependence on preset spatial position information. Firstly, YOLOv5s is used to detect the object and segment the point cloud of the corresponding structure, called the partial point cloud. Then, an iterative closest point (ICP) algorithm between the partial point cloud and the template point cloud is used to calculate the object’s pose and execute the virtual–real fusion. The results demonstrate that the proposed method achieves automatic pose estimation for both fixed and movable objects without background information and preset spatial anchors. Most volunteers reported that our approach was practical, and it thus expands the application of astronaut training. |
| first_indexed | 2024-03-11T04:06:47Z |
| format | Article |
| id | doaj.art-551b9f241cb8426aa2555104bdb54918 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-11T04:06:47Z |
| publishDate | 2023-04-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-551b9f241cb8426aa2555104bdb549182023-11-17T23:43:05ZengMDPI AGSensors1424-82202023-04-01239434410.3390/s23094344Smart Task Assistance in Mixed Reality for AstronautsQingwei Sun0Wei Chen1Jiangang Chao2Wanhong Lin3Zhenying Xu4Ruizhi Cao5Department of Aerospace Science and Technology, Space Engineering University, Beijing 101416, ChinaChina Astronaut Research and Training Center, Beijing 100094, ChinaChina Astronaut Research and Training Center, Beijing 100094, ChinaChina Astronaut Research and Training Center, Beijing 100094, ChinaChina Astronaut Research and Training Center, Beijing 100094, ChinaSchool of Computer Science and Engineering, Beihang University, Beijing 100083, ChinaMixed reality (MR) registers virtual information and real objects and is an effective way to supplement astronaut training. Spatial anchors are generally used to perform virtual–real fusion in static scenes but cannot handle movable objects. To address this issue, we propose a smart task assistance method based on object detection and point cloud alignment. Specifically, both fixed and movable objects are detected automatically. In parallel, poses are estimated with no dependence on preset spatial position information. Firstly, YOLOv5s is used to detect the object and segment the point cloud of the corresponding structure, called the partial point cloud. Then, an iterative closest point (ICP) algorithm between the partial point cloud and the template point cloud is used to calculate the object’s pose and execute the virtual–real fusion. The results demonstrate that the proposed method achieves automatic pose estimation for both fixed and movable objects without background information and preset spatial anchors. Most volunteers reported that our approach was practical, and it thus expands the application of astronaut training.https://www.mdpi.com/1424-8220/23/9/4344mixed realityastronaut trainingobject detectionpose estimationpoint cloud alignment |
| spellingShingle | Qingwei Sun Wei Chen Jiangang Chao Wanhong Lin Zhenying Xu Ruizhi Cao Smart Task Assistance in Mixed Reality for Astronauts Sensors mixed reality astronaut training object detection pose estimation point cloud alignment |
| title | Smart Task Assistance in Mixed Reality for Astronauts |
| title_full | Smart Task Assistance in Mixed Reality for Astronauts |
| title_fullStr | Smart Task Assistance in Mixed Reality for Astronauts |
| title_full_unstemmed | Smart Task Assistance in Mixed Reality for Astronauts |
| title_short | Smart Task Assistance in Mixed Reality for Astronauts |
| title_sort | smart task assistance in mixed reality for astronauts |
| topic | mixed reality astronaut training object detection pose estimation point cloud alignment |
| url | https://www.mdpi.com/1424-8220/23/9/4344 |
| work_keys_str_mv | AT qingweisun smarttaskassistanceinmixedrealityforastronauts AT weichen smarttaskassistanceinmixedrealityforastronauts AT jiangangchao smarttaskassistanceinmixedrealityforastronauts AT wanhonglin smarttaskassistanceinmixedrealityforastronauts AT zhenyingxu smarttaskassistanceinmixedrealityforastronauts AT ruizhicao smarttaskassistanceinmixedrealityforastronauts |