Reinforcement Learning for Collaborative Robots Pick-and-Place Applications: A Case Study
The number of applications in which industrial robots share their working environment with people is increasing. Robots appropriate for such applications are equipped with safety systems according to ISO/TS 15066:2016 and are often referred to as collaborative robots (cobots). Due to the nature of h...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-03-01
|
| Series: | Automation |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2673-4052/3/1/11 |
| _version_ | 1797472863734726656 |
|---|---|
| author | Natanael Magno Gomes Felipe Nascimento Martins José Lima Heinrich Wörtche |
| author_facet | Natanael Magno Gomes Felipe Nascimento Martins José Lima Heinrich Wörtche |
| author_sort | Natanael Magno Gomes |
| collection | DOAJ |
| description | The number of applications in which industrial robots share their working environment with people is increasing. Robots appropriate for such applications are equipped with safety systems according to ISO/TS 15066:2016 and are often referred to as collaborative robots (cobots). Due to the nature of human-robot collaboration, the working environment of cobots is subjected to unforeseeable modifications caused by people. Vision systems are often used to increase the adaptability of cobots, but they usually require knowledge of the objects to be manipulated. The application of machine learning techniques can increase the flexibility by enabling the control system of a cobot to continuously learn and adapt to unexpected changes in the working environment. In this paper we address this issue by investigating the use of Reinforcement Learning (RL) to control a cobot to perform pick-and-place tasks. We present the implementation of a control system that can adapt to changes in position and enables a cobot to grasp objects which were not part of the training. Our proposed system uses deep Q-learning to process color and depth images and generates an <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mi>ϵ</mi></semantics></math></inline-formula>-greedy policy to define robot actions. The Q-values are estimated using Convolution Neural Networks (CNNs) based on pre-trained models for feature extraction. To reduce training time, we implement a simulation environment to first train the RL agent, then we apply the resulting system on a real cobot. System performance is compared when using the pre-trained CNN models ResNext, DenseNet, MobileNet, and MNASNet. Simulation and experimental results validate the proposed approach and show that our system reaches a grasping success rate of 89.9% when manipulating a never-seen object operating with the pre-trained CNN model MobileNet. |
| first_indexed | 2024-03-09T20:07:10Z |
| format | Article |
| id | doaj.art-d78ce41f5fcc42229e9c082682dd8270 |
| institution | Directory Open Access Journal |
| issn | 2673-4052 |
| language | English |
| last_indexed | 2024-03-09T20:07:10Z |
| publishDate | 2022-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Automation |
| spelling | doaj.art-d78ce41f5fcc42229e9c082682dd82702023-11-24T00:28:33ZengMDPI AGAutomation2673-40522022-03-013122324110.3390/automation3010011Reinforcement Learning for Collaborative Robots Pick-and-Place Applications: A Case StudyNatanael Magno Gomes0Felipe Nascimento Martins1José Lima2Heinrich Wörtche3Sensors and Smart Systems Group, Institute of Engineering, Hanze University of Applied Sciences, 9747 AS Groningen, The NetherlandsSensors and Smart Systems Group, Institute of Engineering, Hanze University of Applied Sciences, 9747 AS Groningen, The NetherlandsThe Research Centre in Digitalization and Intelligent Robotics (CeDRI), Instituto Politécnico de Bragança, 5300-252 Bragança, PortugalSensors and Smart Systems Group, Institute of Engineering, Hanze University of Applied Sciences, 9747 AS Groningen, The NetherlandsThe number of applications in which industrial robots share their working environment with people is increasing. Robots appropriate for such applications are equipped with safety systems according to ISO/TS 15066:2016 and are often referred to as collaborative robots (cobots). Due to the nature of human-robot collaboration, the working environment of cobots is subjected to unforeseeable modifications caused by people. Vision systems are often used to increase the adaptability of cobots, but they usually require knowledge of the objects to be manipulated. The application of machine learning techniques can increase the flexibility by enabling the control system of a cobot to continuously learn and adapt to unexpected changes in the working environment. In this paper we address this issue by investigating the use of Reinforcement Learning (RL) to control a cobot to perform pick-and-place tasks. We present the implementation of a control system that can adapt to changes in position and enables a cobot to grasp objects which were not part of the training. Our proposed system uses deep Q-learning to process color and depth images and generates an <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mi>ϵ</mi></semantics></math></inline-formula>-greedy policy to define robot actions. The Q-values are estimated using Convolution Neural Networks (CNNs) based on pre-trained models for feature extraction. To reduce training time, we implement a simulation environment to first train the RL agent, then we apply the resulting system on a real cobot. System performance is compared when using the pre-trained CNN models ResNext, DenseNet, MobileNet, and MNASNet. Simulation and experimental results validate the proposed approach and show that our system reaches a grasping success rate of 89.9% when manipulating a never-seen object operating with the pre-trained CNN model MobileNet.https://www.mdpi.com/2673-4052/3/1/11Reinforcement LearningDeep Neural Networkscomputer visionindustrial robotscollaborative robotspick-and-place |
| spellingShingle | Natanael Magno Gomes Felipe Nascimento Martins José Lima Heinrich Wörtche Reinforcement Learning for Collaborative Robots Pick-and-Place Applications: A Case Study Automation Reinforcement Learning Deep Neural Networks computer vision industrial robots collaborative robots pick-and-place |
| title | Reinforcement Learning for Collaborative Robots Pick-and-Place Applications: A Case Study |
| title_full | Reinforcement Learning for Collaborative Robots Pick-and-Place Applications: A Case Study |
| title_fullStr | Reinforcement Learning for Collaborative Robots Pick-and-Place Applications: A Case Study |
| title_full_unstemmed | Reinforcement Learning for Collaborative Robots Pick-and-Place Applications: A Case Study |
| title_short | Reinforcement Learning for Collaborative Robots Pick-and-Place Applications: A Case Study |
| title_sort | reinforcement learning for collaborative robots pick and place applications a case study |
| topic | Reinforcement Learning Deep Neural Networks computer vision industrial robots collaborative robots pick-and-place |
| url | https://www.mdpi.com/2673-4052/3/1/11 |
| work_keys_str_mv | AT natanaelmagnogomes reinforcementlearningforcollaborativerobotspickandplaceapplicationsacasestudy AT felipenascimentomartins reinforcementlearningforcollaborativerobotspickandplaceapplicationsacasestudy AT joselima reinforcementlearningforcollaborativerobotspickandplaceapplicationsacasestudy AT heinrichwortche reinforcementlearningforcollaborativerobotspickandplaceapplicationsacasestudy |