Design of a Virtual Multi-Interaction Operation System for Hand–Eye Coordination of Grape Harvesting Robots
In harvesting operations, simulation verification of hand–eye coordination in a virtual canopy is critical for harvesting robot research. More realistic scenarios, vision-based driving motion, and cross-platform interaction information are needed to achieve such simulations, which are very challengi...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-03-01
|
| Series: | Agronomy |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2073-4395/13/3/829 |
| _version_ | 1797613985405599744 |
|---|---|
| author | Jizhan Liu Jin Liang Shengyi Zhao Yingxing Jiang Jie Wang Yucheng Jin |
| author_facet | Jizhan Liu Jin Liang Shengyi Zhao Yingxing Jiang Jie Wang Yucheng Jin |
| author_sort | Jizhan Liu |
| collection | DOAJ |
| description | In harvesting operations, simulation verification of hand–eye coordination in a virtual canopy is critical for harvesting robot research. More realistic scenarios, vision-based driving motion, and cross-platform interaction information are needed to achieve such simulations, which are very challenging. Current simulations are more focused on path planning operations for consistency scenarios, which are far from satisfying the requirements. To this end, a new approach of visual servo multi-interaction simulation in real scenarios is proposed. In this study, a dual-arm grape harvesting robot in the laboratory is used as an example. To overcome these challenges, a multi-software federation is first proposed to establish their communication and cross-software sending of image information, coordinate information, and control commands. Then, the fruit recognition and positioning algorithm, forward and inverse kinematic model and simulation model are embedded in OpenCV and MATLAB, respectively, to drive the simulation run of the robot in V-REP, thus realizing the multi-interaction simulation of hand–eye coordination in virtual trellis vineyard. Finally, the simulation is verified, and the results show that the average running time of a string-picking simulation system is 6.5 s, and the success rate of accurate picking point grasping reached 83.3%. A complex closed loop of “scene-image recognition-grasping” is formed by data processing and transmission of various information. It can effectively realize the continuous hand–eye coordination multi-interaction simulation of the harvesting robot under the virtual environment. |
| first_indexed | 2024-03-11T07:02:17Z |
| format | Article |
| id | doaj.art-2a708292407d4b81a6f377bb0a41b006 |
| institution | Directory Open Access Journal |
| issn | 2073-4395 |
| language | English |
| last_indexed | 2024-03-11T07:02:17Z |
| publishDate | 2023-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Agronomy |
| spelling | doaj.art-2a708292407d4b81a6f377bb0a41b0062023-11-17T09:06:49ZengMDPI AGAgronomy2073-43952023-03-0113382910.3390/agronomy13030829Design of a Virtual Multi-Interaction Operation System for Hand–Eye Coordination of Grape Harvesting RobotsJizhan Liu0Jin Liang1Shengyi Zhao2Yingxing Jiang3Jie Wang4Yucheng Jin5Key Laboratory of Modern Agricultural Equipment and Technology, Ministry of Education, Jiangsu University, Zhenjiang 212013, ChinaKey Laboratory of Modern Agricultural Equipment and Technology, Ministry of Education, Jiangsu University, Zhenjiang 212013, ChinaKey Laboratory of Modern Agricultural Equipment and Technology, Ministry of Education, Jiangsu University, Zhenjiang 212013, ChinaKey Laboratory of Modern Agricultural Equipment and Technology, Ministry of Education, Jiangsu University, Zhenjiang 212013, ChinaKey Laboratory of Modern Agricultural Equipment and Technology, Ministry of Education, Jiangsu University, Zhenjiang 212013, ChinaKey Laboratory of Modern Agricultural Equipment and Technology, Ministry of Education, Jiangsu University, Zhenjiang 212013, ChinaIn harvesting operations, simulation verification of hand–eye coordination in a virtual canopy is critical for harvesting robot research. More realistic scenarios, vision-based driving motion, and cross-platform interaction information are needed to achieve such simulations, which are very challenging. Current simulations are more focused on path planning operations for consistency scenarios, which are far from satisfying the requirements. To this end, a new approach of visual servo multi-interaction simulation in real scenarios is proposed. In this study, a dual-arm grape harvesting robot in the laboratory is used as an example. To overcome these challenges, a multi-software federation is first proposed to establish their communication and cross-software sending of image information, coordinate information, and control commands. Then, the fruit recognition and positioning algorithm, forward and inverse kinematic model and simulation model are embedded in OpenCV and MATLAB, respectively, to drive the simulation run of the robot in V-REP, thus realizing the multi-interaction simulation of hand–eye coordination in virtual trellis vineyard. Finally, the simulation is verified, and the results show that the average running time of a string-picking simulation system is 6.5 s, and the success rate of accurate picking point grasping reached 83.3%. A complex closed loop of “scene-image recognition-grasping” is formed by data processing and transmission of various information. It can effectively realize the continuous hand–eye coordination multi-interaction simulation of the harvesting robot under the virtual environment.https://www.mdpi.com/2073-4395/13/3/829harvesting robotshand–eye coordinationsimulationmulti-interactiongrape |
| spellingShingle | Jizhan Liu Jin Liang Shengyi Zhao Yingxing Jiang Jie Wang Yucheng Jin Design of a Virtual Multi-Interaction Operation System for Hand–Eye Coordination of Grape Harvesting Robots Agronomy harvesting robots hand–eye coordination simulation multi-interaction grape |
| title | Design of a Virtual Multi-Interaction Operation System for Hand–Eye Coordination of Grape Harvesting Robots |
| title_full | Design of a Virtual Multi-Interaction Operation System for Hand–Eye Coordination of Grape Harvesting Robots |
| title_fullStr | Design of a Virtual Multi-Interaction Operation System for Hand–Eye Coordination of Grape Harvesting Robots |
| title_full_unstemmed | Design of a Virtual Multi-Interaction Operation System for Hand–Eye Coordination of Grape Harvesting Robots |
| title_short | Design of a Virtual Multi-Interaction Operation System for Hand–Eye Coordination of Grape Harvesting Robots |
| title_sort | design of a virtual multi interaction operation system for hand eye coordination of grape harvesting robots |
| topic | harvesting robots hand–eye coordination simulation multi-interaction grape |
| url | https://www.mdpi.com/2073-4395/13/3/829 |
| work_keys_str_mv | AT jizhanliu designofavirtualmultiinteractionoperationsystemforhandeyecoordinationofgrapeharvestingrobots AT jinliang designofavirtualmultiinteractionoperationsystemforhandeyecoordinationofgrapeharvestingrobots AT shengyizhao designofavirtualmultiinteractionoperationsystemforhandeyecoordinationofgrapeharvestingrobots AT yingxingjiang designofavirtualmultiinteractionoperationsystemforhandeyecoordinationofgrapeharvestingrobots AT jiewang designofavirtualmultiinteractionoperationsystemforhandeyecoordinationofgrapeharvestingrobots AT yuchengjin designofavirtualmultiinteractionoperationsystemforhandeyecoordinationofgrapeharvestingrobots |