Hand Gesture Interface for Robot Path Definition in Collaborative Applications: Implementation and Comparative Study
The article explores the possibilities of using hand gestures as a control interface for robotic systems in a collaborative workspace. The development of hand gesture control interfaces has become increasingly important in everyday life as well as professional contexts such as manufacturing processe...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-04-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/23/9/4219 |
| _version_ | 1797601711298183168 |
|---|---|
| author | Aleš Vysocký Tomáš Poštulka Jakub Chlebek Tomáš Kot Jan Maslowski Stefan Grushko |
| author_facet | Aleš Vysocký Tomáš Poštulka Jakub Chlebek Tomáš Kot Jan Maslowski Stefan Grushko |
| author_sort | Aleš Vysocký |
| collection | DOAJ |
| description | The article explores the possibilities of using hand gestures as a control interface for robotic systems in a collaborative workspace. The development of hand gesture control interfaces has become increasingly important in everyday life as well as professional contexts such as manufacturing processes. We present a system designed to facilitate collaboration between humans and robots in manufacturing processes that require frequent revisions of the robot path and that allows direct definition of the waypoints, which differentiates our system from the existing ones. We introduce a novel and intuitive approach to human–robot cooperation through the use of simple gestures. As part of a robotic workspace, a proposed interface was developed and implemented utilising three RGB-D sensors for monitoring the operator’s hand movements within the workspace. The system employs distributed data processing through multiple Jetson Nano units, with each unit processing data from a single camera. MediaPipe solution is utilised to localise the hand landmarks in the RGB image, enabling gesture recognition. We compare the conventional methods of defining robot trajectories with their developed gesture-based system through an experiment with 20 volunteers. The experiment involved verification of the system under realistic conditions in a real workspace closely resembling the intended industrial application. Data collected during the experiment included both objective and subjective parameters. The results indicate that the gesture-based interface enables users to define a given path objectively faster than conventional methods. We critically analyse the features and limitations of the developed system and suggest directions for future research. Overall, the experimental results indicate the usefulness of the developed system as it can speed up the definition of the robot’s path. |
| first_indexed | 2024-03-11T04:07:33Z |
| format | Article |
| id | doaj.art-061ba8bc2cce486dbc33edf7965996d8 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-11T04:07:33Z |
| publishDate | 2023-04-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-061ba8bc2cce486dbc33edf7965996d82023-11-17T23:41:19ZengMDPI AGSensors1424-82202023-04-01239421910.3390/s23094219Hand Gesture Interface for Robot Path Definition in Collaborative Applications: Implementation and Comparative StudyAleš Vysocký0Tomáš Poštulka1Jakub Chlebek2Tomáš Kot3Jan Maslowski4Stefan Grushko5Department of Robotics, Faculty of Mechanical Engineering, VSB—Technical University of Ostrava, 17. Listopadu 2172/15, 708 00 Ostrava, Czech RepublicDepartment of Robotics, Faculty of Mechanical Engineering, VSB—Technical University of Ostrava, 17. Listopadu 2172/15, 708 00 Ostrava, Czech RepublicDepartment of Robotics, Faculty of Mechanical Engineering, VSB—Technical University of Ostrava, 17. Listopadu 2172/15, 708 00 Ostrava, Czech RepublicDepartment of Robotics, Faculty of Mechanical Engineering, VSB—Technical University of Ostrava, 17. Listopadu 2172/15, 708 00 Ostrava, Czech RepublicDepartment of Robotics, Faculty of Mechanical Engineering, VSB—Technical University of Ostrava, 17. Listopadu 2172/15, 708 00 Ostrava, Czech RepublicDepartment of Robotics, Faculty of Mechanical Engineering, VSB—Technical University of Ostrava, 17. Listopadu 2172/15, 708 00 Ostrava, Czech RepublicThe article explores the possibilities of using hand gestures as a control interface for robotic systems in a collaborative workspace. The development of hand gesture control interfaces has become increasingly important in everyday life as well as professional contexts such as manufacturing processes. We present a system designed to facilitate collaboration between humans and robots in manufacturing processes that require frequent revisions of the robot path and that allows direct definition of the waypoints, which differentiates our system from the existing ones. We introduce a novel and intuitive approach to human–robot cooperation through the use of simple gestures. As part of a robotic workspace, a proposed interface was developed and implemented utilising three RGB-D sensors for monitoring the operator’s hand movements within the workspace. The system employs distributed data processing through multiple Jetson Nano units, with each unit processing data from a single camera. MediaPipe solution is utilised to localise the hand landmarks in the RGB image, enabling gesture recognition. We compare the conventional methods of defining robot trajectories with their developed gesture-based system through an experiment with 20 volunteers. The experiment involved verification of the system under realistic conditions in a real workspace closely resembling the intended industrial application. Data collected during the experiment included both objective and subjective parameters. The results indicate that the gesture-based interface enables users to define a given path objectively faster than conventional methods. We critically analyse the features and limitations of the developed system and suggest directions for future research. Overall, the experimental results indicate the usefulness of the developed system as it can speed up the definition of the robot’s path.https://www.mdpi.com/1424-8220/23/9/4219human-robot collaborationhuman-robot interactionhand trackinghand recognitiongesture |
| spellingShingle | Aleš Vysocký Tomáš Poštulka Jakub Chlebek Tomáš Kot Jan Maslowski Stefan Grushko Hand Gesture Interface for Robot Path Definition in Collaborative Applications: Implementation and Comparative Study Sensors human-robot collaboration human-robot interaction hand tracking hand recognition gesture |
| title | Hand Gesture Interface for Robot Path Definition in Collaborative Applications: Implementation and Comparative Study |
| title_full | Hand Gesture Interface for Robot Path Definition in Collaborative Applications: Implementation and Comparative Study |
| title_fullStr | Hand Gesture Interface for Robot Path Definition in Collaborative Applications: Implementation and Comparative Study |
| title_full_unstemmed | Hand Gesture Interface for Robot Path Definition in Collaborative Applications: Implementation and Comparative Study |
| title_short | Hand Gesture Interface for Robot Path Definition in Collaborative Applications: Implementation and Comparative Study |
| title_sort | hand gesture interface for robot path definition in collaborative applications implementation and comparative study |
| topic | human-robot collaboration human-robot interaction hand tracking hand recognition gesture |
| url | https://www.mdpi.com/1424-8220/23/9/4219 |
| work_keys_str_mv | AT alesvysocky handgestureinterfaceforrobotpathdefinitionincollaborativeapplicationsimplementationandcomparativestudy AT tomaspostulka handgestureinterfaceforrobotpathdefinitionincollaborativeapplicationsimplementationandcomparativestudy AT jakubchlebek handgestureinterfaceforrobotpathdefinitionincollaborativeapplicationsimplementationandcomparativestudy AT tomaskot handgestureinterfaceforrobotpathdefinitionincollaborativeapplicationsimplementationandcomparativestudy AT janmaslowski handgestureinterfaceforrobotpathdefinitionincollaborativeapplicationsimplementationandcomparativestudy AT stefangrushko handgestureinterfaceforrobotpathdefinitionincollaborativeapplicationsimplementationandcomparativestudy |