Determining Exception Context in Assembly Operations from Multimodal Data
Robot assembly tasks can fail due to unpredictable errors and can only continue with the manual intervention of a human operator. Recently, we proposed an exception strategy learning framework based on statistical learning and context determination, which can successfully resolve such situations. Th...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-10-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/22/20/7962 |
| _version_ | 1797469890515304448 |
|---|---|
| author | Mihael Simonič Matevž Majcen Hrovat Sašo Džeroski Aleš Ude Bojan Nemec |
| author_facet | Mihael Simonič Matevž Majcen Hrovat Sašo Džeroski Aleš Ude Bojan Nemec |
| author_sort | Mihael Simonič |
| collection | DOAJ |
| description | Robot assembly tasks can fail due to unpredictable errors and can only continue with the manual intervention of a human operator. Recently, we proposed an exception strategy learning framework based on statistical learning and context determination, which can successfully resolve such situations. This paper deals with context determination from multimodal data, which is the key component of our framework. We propose a novel approach to generate unified low-dimensional context descriptions based on image and force-torque data. For this purpose, we combine a state-of-the-art neural network model for image segmentation and contact point estimation using force-torque measurements. An ensemble of decision trees is used to combine features from the two modalities. To validate the proposed approach, we have collected datasets of deliberately induced insertion failures both for the classic peg-in-hole insertion task and for an industrially relevant task of car starter assembly. We demonstrate that the proposed approach generates reliable low-dimensional descriptors, suitable as queries necessary in statistical learning. |
| first_indexed | 2024-03-09T19:30:25Z |
| format | Article |
| id | doaj.art-17338b8af136482ca04637765dd09290 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-09T19:30:25Z |
| publishDate | 2022-10-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-17338b8af136482ca04637765dd092902023-11-24T02:29:24ZengMDPI AGSensors1424-82202022-10-012220796210.3390/s22207962Determining Exception Context in Assembly Operations from Multimodal DataMihael Simonič0Matevž Majcen Hrovat1Sašo Džeroski2Aleš Ude3Bojan Nemec4Department of Automatics, Biocybernetics and Robotics, Jožef Stefan Institute, Jamova Cesta 39, 1000 Ljubljana, SloveniaDepartment of Automatics, Biocybernetics and Robotics, Jožef Stefan Institute, Jamova Cesta 39, 1000 Ljubljana, SloveniaDepartment of Knowledge Technologies, Jožef Stefan Institute, Jamova 39, 1000 Ljubljana, SloveniaDepartment of Automatics, Biocybernetics and Robotics, Jožef Stefan Institute, Jamova Cesta 39, 1000 Ljubljana, SloveniaDepartment of Automatics, Biocybernetics and Robotics, Jožef Stefan Institute, Jamova Cesta 39, 1000 Ljubljana, SloveniaRobot assembly tasks can fail due to unpredictable errors and can only continue with the manual intervention of a human operator. Recently, we proposed an exception strategy learning framework based on statistical learning and context determination, which can successfully resolve such situations. This paper deals with context determination from multimodal data, which is the key component of our framework. We propose a novel approach to generate unified low-dimensional context descriptions based on image and force-torque data. For this purpose, we combine a state-of-the-art neural network model for image segmentation and contact point estimation using force-torque measurements. An ensemble of decision trees is used to combine features from the two modalities. To validate the proposed approach, we have collected datasets of deliberately induced insertion failures both for the classic peg-in-hole insertion task and for an industrially relevant task of car starter assembly. We demonstrate that the proposed approach generates reliable low-dimensional descriptors, suitable as queries necessary in statistical learning.https://www.mdpi.com/1424-8220/22/20/7962sensor fusionpredictive clustering treesautonomous exception handlingautonomous assemblypeg-in-hole |
| spellingShingle | Mihael Simonič Matevž Majcen Hrovat Sašo Džeroski Aleš Ude Bojan Nemec Determining Exception Context in Assembly Operations from Multimodal Data Sensors sensor fusion predictive clustering trees autonomous exception handling autonomous assembly peg-in-hole |
| title | Determining Exception Context in Assembly Operations from Multimodal Data |
| title_full | Determining Exception Context in Assembly Operations from Multimodal Data |
| title_fullStr | Determining Exception Context in Assembly Operations from Multimodal Data |
| title_full_unstemmed | Determining Exception Context in Assembly Operations from Multimodal Data |
| title_short | Determining Exception Context in Assembly Operations from Multimodal Data |
| title_sort | determining exception context in assembly operations from multimodal data |
| topic | sensor fusion predictive clustering trees autonomous exception handling autonomous assembly peg-in-hole |
| url | https://www.mdpi.com/1424-8220/22/20/7962 |
| work_keys_str_mv | AT mihaelsimonic determiningexceptioncontextinassemblyoperationsfrommultimodaldata AT matevzmajcenhrovat determiningexceptioncontextinassemblyoperationsfrommultimodaldata AT sasodzeroski determiningexceptioncontextinassemblyoperationsfrommultimodaldata AT alesude determiningexceptioncontextinassemblyoperationsfrommultimodaldata AT bojannemec determiningexceptioncontextinassemblyoperationsfrommultimodaldata |