Field-scale yield prediction of winter wheat under different irrigation regimes based on dynamic fusion of multimodal UAV imagery
Field-scale crop yield prediction is critical to site-specific field management, which has been facilitated by recent studies fusing unmanned aerial vehicles (UAVs) based multimodal data. However, these studies equivalently stacked multimodal data and underused canopy spatial information. In this st...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2023-04-01
|
| Series: | International Journal of Applied Earth Observations and Geoinformation |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S1569843223001140 |
| _version_ | 1797842758678872064 |
|---|---|
| author | Juncheng Ma Binhui Liu Lin Ji Zhicheng Zhu Yongfeng Wu Weihua Jiao |
| author_facet | Juncheng Ma Binhui Liu Lin Ji Zhicheng Zhu Yongfeng Wu Weihua Jiao |
| author_sort | Juncheng Ma |
| collection | DOAJ |
| description | Field-scale crop yield prediction is critical to site-specific field management, which has been facilitated by recent studies fusing unmanned aerial vehicles (UAVs) based multimodal data. However, these studies equivalently stacked multimodal data and underused canopy spatial information. In this study, multimodal imagery fusion (MIF) attention was proposed to dynamically fuse UAV-based RGB, hyperspectral near-infrared (HNIR), and thermal imagery. Based on the MIF attention, a novel model termed MultimodalNet was proposed for field-scale yield prediction of winter wheat. To compare multimodal imagery-based and multimodal features-based methods, a stacking-based ensemble learning model was built using UAV-based canopy spectral, thermal, and texture features. The results showed that the MultimodalNet achieved accurate results at the reproductive stage and performed better than any single modality in the fusion. The MultimodalNet performed best at the flowering stage, with a coefficient of determination of 0.7411 and a mean absolute percentage error of 6.05%. The HNIR and thermal imagery were essential in yield prediction of winter wheat at the reproductive stage. Compared to equivalent stacking fusion, dynamic fusion through adaptively adjusting modality attention improved the model accuracy and adaptability across winter wheat cultivars and water treatments. Equivalently stacking more modalities did not necessarily yield improved performance than dynamically fusing fewer modalities. Methods using multimodal UAV imagery with rich spatial information were more applicable than methods using multimodal features to field-scale yield prediction. This study indicates that the MultimodalNet makes a powerful tool for field-scale yield prediction of winter wheat. |
| first_indexed | 2024-04-09T16:54:32Z |
| format | Article |
| id | doaj.art-2a1879943a854fae9980315a0b87c49a |
| institution | Directory Open Access Journal |
| issn | 1569-8432 |
| language | English |
| last_indexed | 2024-04-09T16:54:32Z |
| publishDate | 2023-04-01 |
| publisher | Elsevier |
| record_format | Article |
| series | International Journal of Applied Earth Observations and Geoinformation |
| spelling | doaj.art-2a1879943a854fae9980315a0b87c49a2023-04-21T06:43:15ZengElsevierInternational Journal of Applied Earth Observations and Geoinformation1569-84322023-04-01118103292Field-scale yield prediction of winter wheat under different irrigation regimes based on dynamic fusion of multimodal UAV imageryJuncheng Ma0Binhui Liu1Lin Ji2Zhicheng Zhu3Yongfeng Wu4Weihua Jiao5Institute of Environment and Sustainable Development in Agriculture, Chinese Academy of Agricultural Sciences, Beijing 100081, ChinaDryland Farming Institute, Hebei Academy of Agriculture and Forestry Sciences, Hengshui 053000, ChinaInstitute of Environment and Sustainable Development in Agriculture, Chinese Academy of Agricultural Sciences, Beijing 100081, ChinaInstitute of Environment and Sustainable Development in Agriculture, Chinese Academy of Agricultural Sciences, Beijing 100081, China; School of Surveying and Land Information Engineering, Henan Polytechnic University, Jiaozuo 454000, ChinaInstitute of Environment and Sustainable Development in Agriculture, Chinese Academy of Agricultural Sciences, Beijing 100081, China; Corresponding author at: Institute of Environment and Sustainable Development in Agriculture, Chinese Academy of Agricultural Sciences, 12, Zhongguancun South Street, Haidian District, Beijing 100081, China.Center for Agricultural and Rural Economic Research, Shandong University of Finance and Economics, Jinan 250014, ChinaField-scale crop yield prediction is critical to site-specific field management, which has been facilitated by recent studies fusing unmanned aerial vehicles (UAVs) based multimodal data. However, these studies equivalently stacked multimodal data and underused canopy spatial information. In this study, multimodal imagery fusion (MIF) attention was proposed to dynamically fuse UAV-based RGB, hyperspectral near-infrared (HNIR), and thermal imagery. Based on the MIF attention, a novel model termed MultimodalNet was proposed for field-scale yield prediction of winter wheat. To compare multimodal imagery-based and multimodal features-based methods, a stacking-based ensemble learning model was built using UAV-based canopy spectral, thermal, and texture features. The results showed that the MultimodalNet achieved accurate results at the reproductive stage and performed better than any single modality in the fusion. The MultimodalNet performed best at the flowering stage, with a coefficient of determination of 0.7411 and a mean absolute percentage error of 6.05%. The HNIR and thermal imagery were essential in yield prediction of winter wheat at the reproductive stage. Compared to equivalent stacking fusion, dynamic fusion through adaptively adjusting modality attention improved the model accuracy and adaptability across winter wheat cultivars and water treatments. Equivalently stacking more modalities did not necessarily yield improved performance than dynamically fusing fewer modalities. Methods using multimodal UAV imagery with rich spatial information were more applicable than methods using multimodal features to field-scale yield prediction. This study indicates that the MultimodalNet makes a powerful tool for field-scale yield prediction of winter wheat.http://www.sciencedirect.com/science/article/pii/S1569843223001140Yield predictionWinter wheatMultimodal UAV imageryDynamic fusionDeep learning |
| spellingShingle | Juncheng Ma Binhui Liu Lin Ji Zhicheng Zhu Yongfeng Wu Weihua Jiao Field-scale yield prediction of winter wheat under different irrigation regimes based on dynamic fusion of multimodal UAV imagery International Journal of Applied Earth Observations and Geoinformation Yield prediction Winter wheat Multimodal UAV imagery Dynamic fusion Deep learning |
| title | Field-scale yield prediction of winter wheat under different irrigation regimes based on dynamic fusion of multimodal UAV imagery |
| title_full | Field-scale yield prediction of winter wheat under different irrigation regimes based on dynamic fusion of multimodal UAV imagery |
| title_fullStr | Field-scale yield prediction of winter wheat under different irrigation regimes based on dynamic fusion of multimodal UAV imagery |
| title_full_unstemmed | Field-scale yield prediction of winter wheat under different irrigation regimes based on dynamic fusion of multimodal UAV imagery |
| title_short | Field-scale yield prediction of winter wheat under different irrigation regimes based on dynamic fusion of multimodal UAV imagery |
| title_sort | field scale yield prediction of winter wheat under different irrigation regimes based on dynamic fusion of multimodal uav imagery |
| topic | Yield prediction Winter wheat Multimodal UAV imagery Dynamic fusion Deep learning |
| url | http://www.sciencedirect.com/science/article/pii/S1569843223001140 |
| work_keys_str_mv | AT junchengma fieldscaleyieldpredictionofwinterwheatunderdifferentirrigationregimesbasedondynamicfusionofmultimodaluavimagery AT binhuiliu fieldscaleyieldpredictionofwinterwheatunderdifferentirrigationregimesbasedondynamicfusionofmultimodaluavimagery AT linji fieldscaleyieldpredictionofwinterwheatunderdifferentirrigationregimesbasedondynamicfusionofmultimodaluavimagery AT zhichengzhu fieldscaleyieldpredictionofwinterwheatunderdifferentirrigationregimesbasedondynamicfusionofmultimodaluavimagery AT yongfengwu fieldscaleyieldpredictionofwinterwheatunderdifferentirrigationregimesbasedondynamicfusionofmultimodaluavimagery AT weihuajiao fieldscaleyieldpredictionofwinterwheatunderdifferentirrigationregimesbasedondynamicfusionofmultimodaluavimagery |