Man-machine cooperative method based on deep learning in flexible manufacturing system
Human-Robot Collaboration (HRC) is essential for enhancing productivity and flexibility in smart manufacturing, which poses requirements on accurately predicting the future movements of human operators, especially the trajectories of their upper limbs. However, existing model-based studies on human...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis-Master by Coursework |
| Language: | English |
| Published: |
Nanyang Technological University
2024
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/181138 |
| _version_ | 1826116421883527168 |
|---|---|
| author | He, Chongshan |
| author2 | Ling Keck Voon |
| author_facet | Ling Keck Voon He, Chongshan |
| author_sort | He, Chongshan |
| collection | NTU |
| description | Human-Robot Collaboration (HRC) is essential for enhancing productivity and flexibility in smart manufacturing, which poses requirements on accurately predicting the future movements of human operators, especially the trajectories of their upper limbs. However, existing model-based studies on human manipulation prediction lacks consideration of stochasticity and variability while the emerging deep learning-based methods are demanding on data size, which yet makes real-time deployment challenging. To explore worker arm motion prediction in human-robot collaboration scenarios, a new Human Arm Motion under Manufacturing environment (HUAMM) dataset was designed. This dataset is based on real static working scenes in an automated factory and includes video sequences of various assembly line operations with and without occlusions, totaling over 10 hours. The dataset contains both color and depth information and, after preprocessing, combining the advantages of both model-based and deep learning-based methods, a method for predicting human arm motion, specifically, the position of a worker's wrist in less than 0.5 second, is proposed by hybridizing a Long Short-Term Memory (LSTM) network with an Inverse Kinematics (IK) model. Using historical coordinate sequences of the wrist joint in three-dimensional space in the past multiple frames as input, a neural network is trained to output the predicted coordinates of the wrist joint for the next frame. Then IK (Inverse Kinematics) is used to calculate the arm's motion trajectory based on the predicted wrist coordinates. As the predicted wrist coordinates are sequentially used as the input for the next prediction cycle, the prediction is realized over a sliding time window. Evaluation was conducted using both proprietary and open datasets, results demonstrated that our LSTM-IK method achieved high prediction accuracy, with an average distance error of approximately 5 cm, and can adapt to various task scenarios and individual differences. Additionally, comparison with ground truth illustrated the model's ability to handle complex motion patterns, even with partial occlusions or rapid movements. |
| first_indexed | 2025-03-09T11:38:50Z |
| format | Thesis-Master by Coursework |
| id | ntu-10356/181138 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2025-03-09T11:38:50Z |
| publishDate | 2024 |
| publisher | Nanyang Technological University |
| record_format | dspace |
| spelling | ntu-10356/1811382024-11-15T15:46:58Z Man-machine cooperative method based on deep learning in flexible manufacturing system He, Chongshan Ling Keck Voon School of Electrical and Electronic Engineering EKVLING@ntu.edu.sg Engineering Arm motion prediction Human-robot collaboration Smart manufacturing Long-short term memory Inverse kinematics Human-Robot Collaboration (HRC) is essential for enhancing productivity and flexibility in smart manufacturing, which poses requirements on accurately predicting the future movements of human operators, especially the trajectories of their upper limbs. However, existing model-based studies on human manipulation prediction lacks consideration of stochasticity and variability while the emerging deep learning-based methods are demanding on data size, which yet makes real-time deployment challenging. To explore worker arm motion prediction in human-robot collaboration scenarios, a new Human Arm Motion under Manufacturing environment (HUAMM) dataset was designed. This dataset is based on real static working scenes in an automated factory and includes video sequences of various assembly line operations with and without occlusions, totaling over 10 hours. The dataset contains both color and depth information and, after preprocessing, combining the advantages of both model-based and deep learning-based methods, a method for predicting human arm motion, specifically, the position of a worker's wrist in less than 0.5 second, is proposed by hybridizing a Long Short-Term Memory (LSTM) network with an Inverse Kinematics (IK) model. Using historical coordinate sequences of the wrist joint in three-dimensional space in the past multiple frames as input, a neural network is trained to output the predicted coordinates of the wrist joint for the next frame. Then IK (Inverse Kinematics) is used to calculate the arm's motion trajectory based on the predicted wrist coordinates. As the predicted wrist coordinates are sequentially used as the input for the next prediction cycle, the prediction is realized over a sliding time window. Evaluation was conducted using both proprietary and open datasets, results demonstrated that our LSTM-IK method achieved high prediction accuracy, with an average distance error of approximately 5 cm, and can adapt to various task scenarios and individual differences. Additionally, comparison with ground truth illustrated the model's ability to handle complex motion patterns, even with partial occlusions or rapid movements. Master's degree 2024-11-15T12:55:31Z 2024-11-15T12:55:31Z 2024 Thesis-Master by Coursework He, C. (2024). Man-machine cooperative method based on deep learning in flexible manufacturing system. Master's thesis, Nanyang Technological University, Singapore. https://hdl.handle.net/10356/181138 https://hdl.handle.net/10356/181138 en application/pdf Nanyang Technological University |
| spellingShingle | Engineering Arm motion prediction Human-robot collaboration Smart manufacturing Long-short term memory Inverse kinematics He, Chongshan Man-machine cooperative method based on deep learning in flexible manufacturing system |
| title | Man-machine cooperative method based on deep learning in flexible manufacturing system |
| title_full | Man-machine cooperative method based on deep learning in flexible manufacturing system |
| title_fullStr | Man-machine cooperative method based on deep learning in flexible manufacturing system |
| title_full_unstemmed | Man-machine cooperative method based on deep learning in flexible manufacturing system |
| title_short | Man-machine cooperative method based on deep learning in flexible manufacturing system |
| title_sort | man machine cooperative method based on deep learning in flexible manufacturing system |
| topic | Engineering Arm motion prediction Human-robot collaboration Smart manufacturing Long-short term memory Inverse kinematics |
| url | https://hdl.handle.net/10356/181138 |
| work_keys_str_mv | AT hechongshan manmachinecooperativemethodbasedondeeplearninginflexiblemanufacturingsystem |