Realtime Picking Point Decision Algorithm of Trellis Grape for High-Speed Robotic Cut-and-Catch Harvesting
For high-speed robotic cut-and-catch harvesting, efficient trellis grape recognition and picking point positioning are crucial factors. In this study, a new method for the rapid positioning of picking points based on synchronous inference for multi-grapes was proposed. Firstly, a three-dimensional r...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-06-01
|
| Series: | Agronomy |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2073-4395/13/6/1618 |
| _version_ | 1797596589070483456 |
|---|---|
| author | Zhujie Xu Jizhan Liu Jie Wang Lianjiang Cai Yucheng Jin Shengyi Zhao Binbin Xie |
| author_facet | Zhujie Xu Jizhan Liu Jie Wang Lianjiang Cai Yucheng Jin Shengyi Zhao Binbin Xie |
| author_sort | Zhujie Xu |
| collection | DOAJ |
| description | For high-speed robotic cut-and-catch harvesting, efficient trellis grape recognition and picking point positioning are crucial factors. In this study, a new method for the rapid positioning of picking points based on synchronous inference for multi-grapes was proposed. Firstly, a three-dimensional region of interest for a finite number of grapes was constructed according to the “eye to hand” configuration. Then, a feature-enhanced recognition deep learning model called YOLO v4-SE combined with multi-channel inputs of RGB and depth images was put forward to identify occluded or overlapping grapes and synchronously infer picking points upwards of the prediction boxes of the multi-grapes imaged completely in the three-dimensional region of interest (ROI). Finally, the accuracy of each dimension of the picking points was corrected, and the global continuous picking sequence was planned in the three-dimensional ROI. The recognition experiment in the field showed that YOLO v4-SE has good detection performance in various samples with different interference. The positioning experiment, using a different number of grape bunches from the field, demonstrated that the average recognition success rate is 97% and the average positioning success rate is 93.5%; the average recognition time is 0.0864 s; and the average positioning time is 0.0842 s. The average positioning errors of the <i>x, y,</i> and <i>z</i> directions are 2.598, 2.012, and 1.378 mm, respectively. The average positioning error of the Euclidean distance between the true picking point and the predicted picking point is 7.69 mm. In field synchronous harvesting experiments with different fruiting densities, the average recognition success rate is 97%; the average positioning success rate is 93.606%; and the average picking success rate is 92.78%. The average picking speed is 6.18 <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi mathvariant="normal">s</mi><mo>×</mo><msup><mrow><mi>bunch</mi></mrow><mrow><mo>−</mo><mn>1</mn></mrow></msup></mrow></semantics></math></inline-formula>, which meets the harvesting requirements for high-speed cut-and-catch harvesting robots. This method is promising for overcoming time-consuming harvesting caused by the problematic positioning of the grape stem. |
| first_indexed | 2024-03-11T02:52:18Z |
| format | Article |
| id | doaj.art-f4ecbc81524b4013b92b622b660b2e3e |
| institution | Directory Open Access Journal |
| issn | 2073-4395 |
| language | English |
| last_indexed | 2024-03-11T02:52:18Z |
| publishDate | 2023-06-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Agronomy |
| spelling | doaj.art-f4ecbc81524b4013b92b622b660b2e3e2023-11-18T08:55:43ZengMDPI AGAgronomy2073-43952023-06-01136161810.3390/agronomy13061618Realtime Picking Point Decision Algorithm of Trellis Grape for High-Speed Robotic Cut-and-Catch HarvestingZhujie Xu0Jizhan Liu1Jie Wang2Lianjiang Cai3Yucheng Jin4Shengyi Zhao5Binbin Xie6Key Laboratory of Modern Agricultural Equipment and Technology, Ministry of Education, Jiangsu University, Zhenjiang 212013, ChinaKey Laboratory of Modern Agricultural Equipment and Technology, Ministry of Education, Jiangsu University, Zhenjiang 212013, ChinaKey Laboratory of Modern Agricultural Equipment and Technology, Ministry of Education, Jiangsu University, Zhenjiang 212013, ChinaKey Laboratory of Modern Agricultural Equipment and Technology, Ministry of Education, Jiangsu University, Zhenjiang 212013, ChinaKey Laboratory of Modern Agricultural Equipment and Technology, Ministry of Education, Jiangsu University, Zhenjiang 212013, ChinaKey Laboratory of Modern Agricultural Equipment and Technology, Ministry of Education, Jiangsu University, Zhenjiang 212013, ChinaKey Laboratory of Modern Agricultural Equipment and Technology, Ministry of Education, Jiangsu University, Zhenjiang 212013, ChinaFor high-speed robotic cut-and-catch harvesting, efficient trellis grape recognition and picking point positioning are crucial factors. In this study, a new method for the rapid positioning of picking points based on synchronous inference for multi-grapes was proposed. Firstly, a three-dimensional region of interest for a finite number of grapes was constructed according to the “eye to hand” configuration. Then, a feature-enhanced recognition deep learning model called YOLO v4-SE combined with multi-channel inputs of RGB and depth images was put forward to identify occluded or overlapping grapes and synchronously infer picking points upwards of the prediction boxes of the multi-grapes imaged completely in the three-dimensional region of interest (ROI). Finally, the accuracy of each dimension of the picking points was corrected, and the global continuous picking sequence was planned in the three-dimensional ROI. The recognition experiment in the field showed that YOLO v4-SE has good detection performance in various samples with different interference. The positioning experiment, using a different number of grape bunches from the field, demonstrated that the average recognition success rate is 97% and the average positioning success rate is 93.5%; the average recognition time is 0.0864 s; and the average positioning time is 0.0842 s. The average positioning errors of the <i>x, y,</i> and <i>z</i> directions are 2.598, 2.012, and 1.378 mm, respectively. The average positioning error of the Euclidean distance between the true picking point and the predicted picking point is 7.69 mm. In field synchronous harvesting experiments with different fruiting densities, the average recognition success rate is 97%; the average positioning success rate is 93.606%; and the average picking success rate is 92.78%. The average picking speed is 6.18 <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi mathvariant="normal">s</mi><mo>×</mo><msup><mrow><mi>bunch</mi></mrow><mrow><mo>−</mo><mn>1</mn></mrow></msup></mrow></semantics></math></inline-formula>, which meets the harvesting requirements for high-speed cut-and-catch harvesting robots. This method is promising for overcoming time-consuming harvesting caused by the problematic positioning of the grape stem.https://www.mdpi.com/2073-4395/13/6/1618trellis grapecut-and-catchYOLO v4picking pointpositioning |
| spellingShingle | Zhujie Xu Jizhan Liu Jie Wang Lianjiang Cai Yucheng Jin Shengyi Zhao Binbin Xie Realtime Picking Point Decision Algorithm of Trellis Grape for High-Speed Robotic Cut-and-Catch Harvesting Agronomy trellis grape cut-and-catch YOLO v4 picking point positioning |
| title | Realtime Picking Point Decision Algorithm of Trellis Grape for High-Speed Robotic Cut-and-Catch Harvesting |
| title_full | Realtime Picking Point Decision Algorithm of Trellis Grape for High-Speed Robotic Cut-and-Catch Harvesting |
| title_fullStr | Realtime Picking Point Decision Algorithm of Trellis Grape for High-Speed Robotic Cut-and-Catch Harvesting |
| title_full_unstemmed | Realtime Picking Point Decision Algorithm of Trellis Grape for High-Speed Robotic Cut-and-Catch Harvesting |
| title_short | Realtime Picking Point Decision Algorithm of Trellis Grape for High-Speed Robotic Cut-and-Catch Harvesting |
| title_sort | realtime picking point decision algorithm of trellis grape for high speed robotic cut and catch harvesting |
| topic | trellis grape cut-and-catch YOLO v4 picking point positioning |
| url | https://www.mdpi.com/2073-4395/13/6/1618 |
| work_keys_str_mv | AT zhujiexu realtimepickingpointdecisionalgorithmoftrellisgrapeforhighspeedroboticcutandcatchharvesting AT jizhanliu realtimepickingpointdecisionalgorithmoftrellisgrapeforhighspeedroboticcutandcatchharvesting AT jiewang realtimepickingpointdecisionalgorithmoftrellisgrapeforhighspeedroboticcutandcatchharvesting AT lianjiangcai realtimepickingpointdecisionalgorithmoftrellisgrapeforhighspeedroboticcutandcatchharvesting AT yuchengjin realtimepickingpointdecisionalgorithmoftrellisgrapeforhighspeedroboticcutandcatchharvesting AT shengyizhao realtimepickingpointdecisionalgorithmoftrellisgrapeforhighspeedroboticcutandcatchharvesting AT binbinxie realtimepickingpointdecisionalgorithmoftrellisgrapeforhighspeedroboticcutandcatchharvesting |