Motion Estimation and Hand Gesture Recognition-Based Human–UAV Interaction Approach in Real Time
As an alternative to traditional remote controller, research on vision-based hand gesture recognition is being actively conducted in the field of interaction between human and unmanned aerial vehicle (UAV). However, vision-based gesture system has a challenging problem in recognizing the motion of d...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-03-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/22/7/2513 |
| _version_ | 1827622530386493440 |
|---|---|
| author | Minjeong Yoo Yuseung Na Hamin Song Gamin Kim Junseong Yun Sangho Kim Changjoo Moon Kichun Jo |
| author_facet | Minjeong Yoo Yuseung Na Hamin Song Gamin Kim Junseong Yun Sangho Kim Changjoo Moon Kichun Jo |
| author_sort | Minjeong Yoo |
| collection | DOAJ |
| description | As an alternative to traditional remote controller, research on vision-based hand gesture recognition is being actively conducted in the field of interaction between human and unmanned aerial vehicle (UAV). However, vision-based gesture system has a challenging problem in recognizing the motion of dynamic gesture because it is difficult to estimate the pose of multi-dimensional hand gestures in 2D images. This leads to complex algorithms, including tracking in addition to detection, to recognize dynamic gestures, but they are not suitable for human–UAV interaction (HUI) systems that require safe design with high real-time performance. Therefore, in this paper, we propose a hybrid hand gesture system that combines an inertial measurement unit (IMU)-based motion capture system and a vision-based gesture system to increase real-time performance. First, IMU-based commands and vision-based commands are divided according to whether drone operation commands are continuously input. Second, IMU-based control commands are intuitively mapped to allow the UAV to move in the same direction by utilizing estimated orientation sensed by a thumb-mounted micro-IMU, and vision-based control commands are mapped with hand’s appearance through real-time object detection. The proposed system is verified in a simulation environment through efficiency evaluation with dynamic gestures of the existing vision-based system in addition to usability comparison with traditional joystick controller conducted for applicants with no experience in manipulation. As a result, it proves that it is a safer and more intuitive HUI design with a 0.089 ms processing speed and average lap time that takes about 19 s less than the joystick controller. In other words, it shows that it is viable as an alternative to existing HUI. |
| first_indexed | 2024-03-09T11:27:02Z |
| format | Article |
| id | doaj.art-e33e03e363e54cfd8dde3c92e76fc224 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-09T11:27:02Z |
| publishDate | 2022-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-e33e03e363e54cfd8dde3c92e76fc2242023-11-30T23:59:56ZengMDPI AGSensors1424-82202022-03-01227251310.3390/s22072513Motion Estimation and Hand Gesture Recognition-Based Human–UAV Interaction Approach in Real TimeMinjeong Yoo0Yuseung Na1Hamin Song2Gamin Kim3Junseong Yun4Sangho Kim5Changjoo Moon6Kichun Jo7Department of Smart Vehicle Engineering, Konkuk University, 120, Neungdong-ro, Gwangjin-gu, Seoul 05029, KoreaDepartment of Smart Vehicle Engineering, Konkuk University, 120, Neungdong-ro, Gwangjin-gu, Seoul 05029, KoreaDepartment of Smart Vehicle Engineering, Konkuk University, 120, Neungdong-ro, Gwangjin-gu, Seoul 05029, KoreaDepartment of Smart Vehicle Engineering, Konkuk University, 120, Neungdong-ro, Gwangjin-gu, Seoul 05029, KoreaDepartment of Smart Vehicle Engineering, Konkuk University, 120, Neungdong-ro, Gwangjin-gu, Seoul 05029, KoreaDepartment of Smart Vehicle Engineering, Konkuk University, 120, Neungdong-ro, Gwangjin-gu, Seoul 05029, KoreaDepartment of Smart Vehicle Engineering, Konkuk University, 120, Neungdong-ro, Gwangjin-gu, Seoul 05029, KoreaDepartment of Smart Vehicle Engineering, Konkuk University, 120, Neungdong-ro, Gwangjin-gu, Seoul 05029, KoreaAs an alternative to traditional remote controller, research on vision-based hand gesture recognition is being actively conducted in the field of interaction between human and unmanned aerial vehicle (UAV). However, vision-based gesture system has a challenging problem in recognizing the motion of dynamic gesture because it is difficult to estimate the pose of multi-dimensional hand gestures in 2D images. This leads to complex algorithms, including tracking in addition to detection, to recognize dynamic gestures, but they are not suitable for human–UAV interaction (HUI) systems that require safe design with high real-time performance. Therefore, in this paper, we propose a hybrid hand gesture system that combines an inertial measurement unit (IMU)-based motion capture system and a vision-based gesture system to increase real-time performance. First, IMU-based commands and vision-based commands are divided according to whether drone operation commands are continuously input. Second, IMU-based control commands are intuitively mapped to allow the UAV to move in the same direction by utilizing estimated orientation sensed by a thumb-mounted micro-IMU, and vision-based control commands are mapped with hand’s appearance through real-time object detection. The proposed system is verified in a simulation environment through efficiency evaluation with dynamic gestures of the existing vision-based system in addition to usability comparison with traditional joystick controller conducted for applicants with no experience in manipulation. As a result, it proves that it is a safer and more intuitive HUI design with a 0.089 ms processing speed and average lap time that takes about 19 s less than the joystick controller. In other words, it shows that it is viable as an alternative to existing HUI.https://www.mdpi.com/1424-8220/22/7/2513human–UAV interactionhybrid-based hand gesture recognitionhand-gesture-based recognitionIMU-based motion capture systemdeep learning |
| spellingShingle | Minjeong Yoo Yuseung Na Hamin Song Gamin Kim Junseong Yun Sangho Kim Changjoo Moon Kichun Jo Motion Estimation and Hand Gesture Recognition-Based Human–UAV Interaction Approach in Real Time Sensors human–UAV interaction hybrid-based hand gesture recognition hand-gesture-based recognition IMU-based motion capture system deep learning |
| title | Motion Estimation and Hand Gesture Recognition-Based Human–UAV Interaction Approach in Real Time |
| title_full | Motion Estimation and Hand Gesture Recognition-Based Human–UAV Interaction Approach in Real Time |
| title_fullStr | Motion Estimation and Hand Gesture Recognition-Based Human–UAV Interaction Approach in Real Time |
| title_full_unstemmed | Motion Estimation and Hand Gesture Recognition-Based Human–UAV Interaction Approach in Real Time |
| title_short | Motion Estimation and Hand Gesture Recognition-Based Human–UAV Interaction Approach in Real Time |
| title_sort | motion estimation and hand gesture recognition based human uav interaction approach in real time |
| topic | human–UAV interaction hybrid-based hand gesture recognition hand-gesture-based recognition IMU-based motion capture system deep learning |
| url | https://www.mdpi.com/1424-8220/22/7/2513 |
| work_keys_str_mv | AT minjeongyoo motionestimationandhandgesturerecognitionbasedhumanuavinteractionapproachinrealtime AT yuseungna motionestimationandhandgesturerecognitionbasedhumanuavinteractionapproachinrealtime AT haminsong motionestimationandhandgesturerecognitionbasedhumanuavinteractionapproachinrealtime AT gaminkim motionestimationandhandgesturerecognitionbasedhumanuavinteractionapproachinrealtime AT junseongyun motionestimationandhandgesturerecognitionbasedhumanuavinteractionapproachinrealtime AT sanghokim motionestimationandhandgesturerecognitionbasedhumanuavinteractionapproachinrealtime AT changjoomoon motionestimationandhandgesturerecognitionbasedhumanuavinteractionapproachinrealtime AT kichunjo motionestimationandhandgesturerecognitionbasedhumanuavinteractionapproachinrealtime |