Self-reconfigurable robot vision pipeline for safer adaptation to varying pavements width and surface conditions
Abstract This work presents the vision pipeline for our in-house developed autonomous reconfigurable pavement sweeping robot named Panthera. As the goal of Panthera is to be an autonomous self-reconfigurable robot, it has to understand the type of pavement it is moving in so that it can adapt smooth...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2022-08-01
|
| Series: | Scientific Reports |
| Online Access: | https://doi.org/10.1038/s41598-022-17858-w |
| _version_ | 1811285904589848576 |
|---|---|
| author | Lim Yi Braulio Félix Gómez Balakrishnan Ramalingam Madan Mohan Rayguru Mohan Rajesh Elara Abdullah Aamir Hayat |
| author_facet | Lim Yi Braulio Félix Gómez Balakrishnan Ramalingam Madan Mohan Rayguru Mohan Rajesh Elara Abdullah Aamir Hayat |
| author_sort | Lim Yi |
| collection | DOAJ |
| description | Abstract This work presents the vision pipeline for our in-house developed autonomous reconfigurable pavement sweeping robot named Panthera. As the goal of Panthera is to be an autonomous self-reconfigurable robot, it has to understand the type of pavement it is moving in so that it can adapt smoothly to changing pavement width and perform cleaning operations more efficiently and safely. deep learning (DL) based vision pipeline is proposed for the Panthera robot to recognize pavement features, including pavement type identification, pavement surface condition prediction, and pavement width estimation. The DeepLabv3+ semantic segmentation algorithm was customized to identify the pavement type classification, an eight-layer CNN was proposed for pavement surface condition prediction. Furthermore, pavement width estimation was computed by fusing the segmented pavement region on the depth map. In the end, the fuzzy inference system was implemented by taking input as the pavement width and its conditions detected and output as the safe operational speed. The vision pipeline was trained using the DL provided with the custom pavement images dataset. The performance was evaluated using offline test and real-time field trial images captured through the reconfigurable robot Panthera stereo vision sensor. In the experimental analysis, the DL-based vision pipeline components scored 88.02% and 93.22% accuracy for pavement segmentation and pavement surface condition assessment, respectively, and took approximately 10 ms computation time to process the single image frame from the vision sensor using the onboard computer. |
| first_indexed | 2024-04-13T02:51:47Z |
| format | Article |
| id | doaj.art-e8bac49fa67647379107d3bd5a0ee27e |
| institution | Directory Open Access Journal |
| issn | 2045-2322 |
| language | English |
| last_indexed | 2024-04-13T02:51:47Z |
| publishDate | 2022-08-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj.art-e8bac49fa67647379107d3bd5a0ee27e2022-12-22T03:05:50ZengNature PortfolioScientific Reports2045-23222022-08-0112111710.1038/s41598-022-17858-wSelf-reconfigurable robot vision pipeline for safer adaptation to varying pavements width and surface conditionsLim Yi0Braulio Félix Gómez1Balakrishnan Ramalingam2Madan Mohan Rayguru3Mohan Rajesh Elara4Abdullah Aamir Hayat5Engineering Product Development Pillar, Singapore University of Technology and Design (SUTD)Engineering Product Development Pillar, Singapore University of Technology and Design (SUTD)Engineering Product Development Pillar, Singapore University of Technology and Design (SUTD)Electrical Engineering, Delhi Technological UniversityEngineering Product Development Pillar, Singapore University of Technology and Design (SUTD)Engineering Product Development Pillar, Singapore University of Technology and Design (SUTD)Abstract This work presents the vision pipeline for our in-house developed autonomous reconfigurable pavement sweeping robot named Panthera. As the goal of Panthera is to be an autonomous self-reconfigurable robot, it has to understand the type of pavement it is moving in so that it can adapt smoothly to changing pavement width and perform cleaning operations more efficiently and safely. deep learning (DL) based vision pipeline is proposed for the Panthera robot to recognize pavement features, including pavement type identification, pavement surface condition prediction, and pavement width estimation. The DeepLabv3+ semantic segmentation algorithm was customized to identify the pavement type classification, an eight-layer CNN was proposed for pavement surface condition prediction. Furthermore, pavement width estimation was computed by fusing the segmented pavement region on the depth map. In the end, the fuzzy inference system was implemented by taking input as the pavement width and its conditions detected and output as the safe operational speed. The vision pipeline was trained using the DL provided with the custom pavement images dataset. The performance was evaluated using offline test and real-time field trial images captured through the reconfigurable robot Panthera stereo vision sensor. In the experimental analysis, the DL-based vision pipeline components scored 88.02% and 93.22% accuracy for pavement segmentation and pavement surface condition assessment, respectively, and took approximately 10 ms computation time to process the single image frame from the vision sensor using the onboard computer.https://doi.org/10.1038/s41598-022-17858-w |
| spellingShingle | Lim Yi Braulio Félix Gómez Balakrishnan Ramalingam Madan Mohan Rayguru Mohan Rajesh Elara Abdullah Aamir Hayat Self-reconfigurable robot vision pipeline for safer adaptation to varying pavements width and surface conditions Scientific Reports |
| title | Self-reconfigurable robot vision pipeline for safer adaptation to varying pavements width and surface conditions |
| title_full | Self-reconfigurable robot vision pipeline for safer adaptation to varying pavements width and surface conditions |
| title_fullStr | Self-reconfigurable robot vision pipeline for safer adaptation to varying pavements width and surface conditions |
| title_full_unstemmed | Self-reconfigurable robot vision pipeline for safer adaptation to varying pavements width and surface conditions |
| title_short | Self-reconfigurable robot vision pipeline for safer adaptation to varying pavements width and surface conditions |
| title_sort | self reconfigurable robot vision pipeline for safer adaptation to varying pavements width and surface conditions |
| url | https://doi.org/10.1038/s41598-022-17858-w |
| work_keys_str_mv | AT limyi selfreconfigurablerobotvisionpipelineforsaferadaptationtovaryingpavementswidthandsurfaceconditions AT brauliofelixgomez selfreconfigurablerobotvisionpipelineforsaferadaptationtovaryingpavementswidthandsurfaceconditions AT balakrishnanramalingam selfreconfigurablerobotvisionpipelineforsaferadaptationtovaryingpavementswidthandsurfaceconditions AT madanmohanrayguru selfreconfigurablerobotvisionpipelineforsaferadaptationtovaryingpavementswidthandsurfaceconditions AT mohanrajeshelara selfreconfigurablerobotvisionpipelineforsaferadaptationtovaryingpavementswidthandsurfaceconditions AT abdullahaamirhayat selfreconfigurablerobotvisionpipelineforsaferadaptationtovaryingpavementswidthandsurfaceconditions |