Fruit Detection and Pose Estimation for Grape Cluster–Harvesting Robot Using Binocular Imagery Based on Deep Neural Networks
Reliable and robust fruit-detection algorithms in nonstructural environments are essential for the efficient use of harvesting robots. The pose of fruits is crucial to guide robots to approach target fruits for collision-free picking. To achieve accurate picking, this study investigates an approach...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2021-06-01
|
| Series: | Frontiers in Robotics and AI |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/frobt.2021.626989/full |
| _version_ | 1818657448722956288 |
|---|---|
| author | Wei Yin Hanjin Wen Zhengtong Ning Jian Ye Zhiqiang Dong Lufeng Luo |
| author_facet | Wei Yin Hanjin Wen Zhengtong Ning Jian Ye Zhiqiang Dong Lufeng Luo |
| author_sort | Wei Yin |
| collection | DOAJ |
| description | Reliable and robust fruit-detection algorithms in nonstructural environments are essential for the efficient use of harvesting robots. The pose of fruits is crucial to guide robots to approach target fruits for collision-free picking. To achieve accurate picking, this study investigates an approach to detect fruit and estimate its pose. First, the state-of-the-art mask region convolutional neural network (Mask R-CNN) is deployed to segment binocular images to output the mask image of the target fruit. Next, a grape point cloud extracted from the images was filtered and denoised to obtain an accurate grape point cloud. Finally, the accurate grape point cloud was used with the RANSAC algorithm for grape cylinder model fitting, and the axis of the cylinder model was used to estimate the pose of the grape. A dataset was acquired in a vineyard to evaluate the performance of the proposed approach in a nonstructural environment. The fruit detection results of 210 test images show that the average precision, recall, and intersection over union (IOU) are 89.53, 95.33, and 82.00%, respectively. The detection and point cloud segmentation for each grape took approximately 1.7 s. The demonstrated performance of the developed method indicates that it can be applied to grape-harvesting robots. |
| first_indexed | 2024-12-17T03:41:39Z |
| format | Article |
| id | doaj.art-e91af12129594498b401e709d15dc2d7 |
| institution | Directory Open Access Journal |
| issn | 2296-9144 |
| language | English |
| last_indexed | 2024-12-17T03:41:39Z |
| publishDate | 2021-06-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Robotics and AI |
| spelling | doaj.art-e91af12129594498b401e709d15dc2d72022-12-21T22:05:00ZengFrontiers Media S.A.Frontiers in Robotics and AI2296-91442021-06-01810.3389/frobt.2021.626989626989Fruit Detection and Pose Estimation for Grape Cluster–Harvesting Robot Using Binocular Imagery Based on Deep Neural NetworksWei YinHanjin WenZhengtong NingJian YeZhiqiang DongLufeng LuoReliable and robust fruit-detection algorithms in nonstructural environments are essential for the efficient use of harvesting robots. The pose of fruits is crucial to guide robots to approach target fruits for collision-free picking. To achieve accurate picking, this study investigates an approach to detect fruit and estimate its pose. First, the state-of-the-art mask region convolutional neural network (Mask R-CNN) is deployed to segment binocular images to output the mask image of the target fruit. Next, a grape point cloud extracted from the images was filtered and denoised to obtain an accurate grape point cloud. Finally, the accurate grape point cloud was used with the RANSAC algorithm for grape cylinder model fitting, and the axis of the cylinder model was used to estimate the pose of the grape. A dataset was acquired in a vineyard to evaluate the performance of the proposed approach in a nonstructural environment. The fruit detection results of 210 test images show that the average precision, recall, and intersection over union (IOU) are 89.53, 95.33, and 82.00%, respectively. The detection and point cloud segmentation for each grape took approximately 1.7 s. The demonstrated performance of the developed method indicates that it can be applied to grape-harvesting robots.https://www.frontiersin.org/articles/10.3389/frobt.2021.626989/fullgrape clusterregion convolutional networkbinocular stereo cameragrape model reconstructionpose estimation |
| spellingShingle | Wei Yin Hanjin Wen Zhengtong Ning Jian Ye Zhiqiang Dong Lufeng Luo Fruit Detection and Pose Estimation for Grape Cluster–Harvesting Robot Using Binocular Imagery Based on Deep Neural Networks Frontiers in Robotics and AI grape cluster region convolutional network binocular stereo camera grape model reconstruction pose estimation |
| title | Fruit Detection and Pose Estimation for Grape Cluster–Harvesting Robot Using Binocular Imagery Based on Deep Neural Networks |
| title_full | Fruit Detection and Pose Estimation for Grape Cluster–Harvesting Robot Using Binocular Imagery Based on Deep Neural Networks |
| title_fullStr | Fruit Detection and Pose Estimation for Grape Cluster–Harvesting Robot Using Binocular Imagery Based on Deep Neural Networks |
| title_full_unstemmed | Fruit Detection and Pose Estimation for Grape Cluster–Harvesting Robot Using Binocular Imagery Based on Deep Neural Networks |
| title_short | Fruit Detection and Pose Estimation for Grape Cluster–Harvesting Robot Using Binocular Imagery Based on Deep Neural Networks |
| title_sort | fruit detection and pose estimation for grape cluster harvesting robot using binocular imagery based on deep neural networks |
| topic | grape cluster region convolutional network binocular stereo camera grape model reconstruction pose estimation |
| url | https://www.frontiersin.org/articles/10.3389/frobt.2021.626989/full |
| work_keys_str_mv | AT weiyin fruitdetectionandposeestimationforgrapeclusterharvestingrobotusingbinocularimagerybasedondeepneuralnetworks AT hanjinwen fruitdetectionandposeestimationforgrapeclusterharvestingrobotusingbinocularimagerybasedondeepneuralnetworks AT zhengtongning fruitdetectionandposeestimationforgrapeclusterharvestingrobotusingbinocularimagerybasedondeepneuralnetworks AT jianye fruitdetectionandposeestimationforgrapeclusterharvestingrobotusingbinocularimagerybasedondeepneuralnetworks AT zhiqiangdong fruitdetectionandposeestimationforgrapeclusterharvestingrobotusingbinocularimagerybasedondeepneuralnetworks AT lufengluo fruitdetectionandposeestimationforgrapeclusterharvestingrobotusingbinocularimagerybasedondeepneuralnetworks |