Deep Instance Segmentation and Visual Servoing to Play Jenga with a Cost-Effective Robotic System
The game of Jenga is a benchmark used for developing innovative manipulation solutions for complex tasks. Indeed, it encourages the study of novel robotics methods to successfully extract blocks from a tower. A Jenga game involves many traits of complex industrial and surgical manipulation tasks, re...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-01-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/23/2/752 |
| _version_ | 1797437281231962112 |
|---|---|
| author | Luca Marchionna Giulio Pugliese Mauro Martini Simone Angarano Francesco Salvetti Marcello Chiaberge |
| author_facet | Luca Marchionna Giulio Pugliese Mauro Martini Simone Angarano Francesco Salvetti Marcello Chiaberge |
| author_sort | Luca Marchionna |
| collection | DOAJ |
| description | The game of Jenga is a benchmark used for developing innovative manipulation solutions for complex tasks. Indeed, it encourages the study of novel robotics methods to successfully extract blocks from a tower. A Jenga game involves many traits of complex industrial and surgical manipulation tasks, requiring a multi-step strategy, the combination of visual and tactile data, and the highly precise motion of a robotic arm to perform a single block extraction. In this work, we propose a novel, cost-effective architecture for playing Jenga with e.Do, a 6DOF anthropomorphic manipulator manufactured by Comau, a standard depth camera, and an inexpensive monodirectional force sensor. Our solution focuses on a visual-based control strategy to accurately align the end-effector with the desired block, enabling block extraction by pushing. To this aim, we trained an instance segmentation deep learning model on a synthetic custom dataset to segment each piece of the Jenga tower, allowing for visual tracking of the desired block’s pose during the motion of the manipulator. We integrated the visual-based strategy with a 1D force sensor to detect whether the block could be safely removed by identifying a force threshold value. Our experimentation shows that our low-cost solution allows e.DO to precisely reach removable blocks and perform up to 14 consecutive extractions in a row. |
| first_indexed | 2024-03-09T11:17:54Z |
| format | Article |
| id | doaj.art-17df5406a05b494591fcb7363c35cb76 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-09T11:17:54Z |
| publishDate | 2023-01-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-17df5406a05b494591fcb7363c35cb762023-12-01T00:26:57ZengMDPI AGSensors1424-82202023-01-0123275210.3390/s23020752Deep Instance Segmentation and Visual Servoing to Play Jenga with a Cost-Effective Robotic SystemLuca Marchionna0Giulio Pugliese1Mauro Martini2Simone Angarano3Francesco Salvetti4Marcello Chiaberge5Department of Electronics and Telecommunications (DET), Politecnico di Torino, 10129 Torino, ItalyDepartment of Electronics and Telecommunications (DET), Politecnico di Torino, 10129 Torino, ItalyDepartment of Electronics and Telecommunications (DET), Politecnico di Torino, 10129 Torino, ItalyDepartment of Electronics and Telecommunications (DET), Politecnico di Torino, 10129 Torino, ItalyDepartment of Electronics and Telecommunications (DET), Politecnico di Torino, 10129 Torino, ItalyDepartment of Electronics and Telecommunications (DET), Politecnico di Torino, 10129 Torino, ItalyThe game of Jenga is a benchmark used for developing innovative manipulation solutions for complex tasks. Indeed, it encourages the study of novel robotics methods to successfully extract blocks from a tower. A Jenga game involves many traits of complex industrial and surgical manipulation tasks, requiring a multi-step strategy, the combination of visual and tactile data, and the highly precise motion of a robotic arm to perform a single block extraction. In this work, we propose a novel, cost-effective architecture for playing Jenga with e.Do, a 6DOF anthropomorphic manipulator manufactured by Comau, a standard depth camera, and an inexpensive monodirectional force sensor. Our solution focuses on a visual-based control strategy to accurately align the end-effector with the desired block, enabling block extraction by pushing. To this aim, we trained an instance segmentation deep learning model on a synthetic custom dataset to segment each piece of the Jenga tower, allowing for visual tracking of the desired block’s pose during the motion of the manipulator. We integrated the visual-based strategy with a 1D force sensor to detect whether the block could be safely removed by identifying a force threshold value. Our experimentation shows that our low-cost solution allows e.DO to precisely reach removable blocks and perform up to 14 consecutive extractions in a row.https://www.mdpi.com/1424-8220/23/2/752Jengarobotic armdeep instance segmentationvisual servoingsensor fusion |
| spellingShingle | Luca Marchionna Giulio Pugliese Mauro Martini Simone Angarano Francesco Salvetti Marcello Chiaberge Deep Instance Segmentation and Visual Servoing to Play Jenga with a Cost-Effective Robotic System Sensors Jenga robotic arm deep instance segmentation visual servoing sensor fusion |
| title | Deep Instance Segmentation and Visual Servoing to Play Jenga with a Cost-Effective Robotic System |
| title_full | Deep Instance Segmentation and Visual Servoing to Play Jenga with a Cost-Effective Robotic System |
| title_fullStr | Deep Instance Segmentation and Visual Servoing to Play Jenga with a Cost-Effective Robotic System |
| title_full_unstemmed | Deep Instance Segmentation and Visual Servoing to Play Jenga with a Cost-Effective Robotic System |
| title_short | Deep Instance Segmentation and Visual Servoing to Play Jenga with a Cost-Effective Robotic System |
| title_sort | deep instance segmentation and visual servoing to play jenga with a cost effective robotic system |
| topic | Jenga robotic arm deep instance segmentation visual servoing sensor fusion |
| url | https://www.mdpi.com/1424-8220/23/2/752 |
| work_keys_str_mv | AT lucamarchionna deepinstancesegmentationandvisualservoingtoplayjengawithacosteffectiveroboticsystem AT giuliopugliese deepinstancesegmentationandvisualservoingtoplayjengawithacosteffectiveroboticsystem AT mauromartini deepinstancesegmentationandvisualservoingtoplayjengawithacosteffectiveroboticsystem AT simoneangarano deepinstancesegmentationandvisualservoingtoplayjengawithacosteffectiveroboticsystem AT francescosalvetti deepinstancesegmentationandvisualservoingtoplayjengawithacosteffectiveroboticsystem AT marcellochiaberge deepinstancesegmentationandvisualservoingtoplayjengawithacosteffectiveroboticsystem |