Multimodal Deep Learning and Visible-Light and Hyperspectral Imaging for Fruit Maturity Estimation
Fruit maturity is a critical factor in the supply chain, consumer preference, and agriculture industry. Most classification methods on fruit maturity identify only two classes: ripe and unripe, but this paper estimates six maturity stages of papaya fruit. Deep learning architectures have gained resp...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-02-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/21/4/1288 |
| _version_ | 1797411204984995840 |
|---|---|
| author | Cinmayii A. Garillos-Manliguez John Y. Chiang |
| author_facet | Cinmayii A. Garillos-Manliguez John Y. Chiang |
| author_sort | Cinmayii A. Garillos-Manliguez |
| collection | DOAJ |
| description | Fruit maturity is a critical factor in the supply chain, consumer preference, and agriculture industry. Most classification methods on fruit maturity identify only two classes: ripe and unripe, but this paper estimates six maturity stages of papaya fruit. Deep learning architectures have gained respect and brought breakthroughs in unimodal processing. This paper suggests a novel non-destructive and multimodal classification using deep convolutional neural networks that estimate fruit maturity by feature concatenation of data acquired from two imaging modes: visible-light and hyperspectral imaging systems. Morphological changes in the sample fruits can be easily measured with RGB images, while spectral signatures that provide high sensitivity and high correlation with the internal properties of fruits can be extracted from hyperspectral images with wavelength range in between 400 nm and 900 nm—factors that must be considered when building a model. This study further modified the architectures: AlexNet, VGG16, VGG19, ResNet50, ResNeXt50, MobileNet, and MobileNetV2 to utilize multimodal data cubes composed of RGB and hyperspectral data for sensitivity analyses. These multimodal variants can achieve up to 0.90 F1 scores and 1.45% top-2 error rate for the classification of six stages. Overall, taking advantage of multimodal input coupled with powerful deep convolutional neural network models can classify fruit maturity even at refined levels of six stages. This indicates that multimodal deep learning architectures and multimodal imaging have great potential for real-time in-field fruit maturity estimation that can help estimate optimal harvest time and other in-field industrial applications. |
| first_indexed | 2024-03-09T04:41:38Z |
| format | Article |
| id | doaj.art-17c5ca9f43a343a8a08683bf23014251 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-09T04:41:38Z |
| publishDate | 2021-02-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-17c5ca9f43a343a8a08683bf230142512023-12-03T13:19:23ZengMDPI AGSensors1424-82202021-02-01214128810.3390/s21041288Multimodal Deep Learning and Visible-Light and Hyperspectral Imaging for Fruit Maturity EstimationCinmayii A. Garillos-Manliguez0John Y. Chiang1Department of Computer Science and Engineering, National Sun Yat-sen University, Kaohsiung 804, TaiwanDepartment of Computer Science and Engineering, National Sun Yat-sen University, Kaohsiung 804, TaiwanFruit maturity is a critical factor in the supply chain, consumer preference, and agriculture industry. Most classification methods on fruit maturity identify only two classes: ripe and unripe, but this paper estimates six maturity stages of papaya fruit. Deep learning architectures have gained respect and brought breakthroughs in unimodal processing. This paper suggests a novel non-destructive and multimodal classification using deep convolutional neural networks that estimate fruit maturity by feature concatenation of data acquired from two imaging modes: visible-light and hyperspectral imaging systems. Morphological changes in the sample fruits can be easily measured with RGB images, while spectral signatures that provide high sensitivity and high correlation with the internal properties of fruits can be extracted from hyperspectral images with wavelength range in between 400 nm and 900 nm—factors that must be considered when building a model. This study further modified the architectures: AlexNet, VGG16, VGG19, ResNet50, ResNeXt50, MobileNet, and MobileNetV2 to utilize multimodal data cubes composed of RGB and hyperspectral data for sensitivity analyses. These multimodal variants can achieve up to 0.90 F1 scores and 1.45% top-2 error rate for the classification of six stages. Overall, taking advantage of multimodal input coupled with powerful deep convolutional neural network models can classify fruit maturity even at refined levels of six stages. This indicates that multimodal deep learning architectures and multimodal imaging have great potential for real-time in-field fruit maturity estimation that can help estimate optimal harvest time and other in-field industrial applications.https://www.mdpi.com/1424-8220/21/4/1288multimodalitydeep learninghyperspectral imagingfruit maturityclassification |
| spellingShingle | Cinmayii A. Garillos-Manliguez John Y. Chiang Multimodal Deep Learning and Visible-Light and Hyperspectral Imaging for Fruit Maturity Estimation Sensors multimodality deep learning hyperspectral imaging fruit maturity classification |
| title | Multimodal Deep Learning and Visible-Light and Hyperspectral Imaging for Fruit Maturity Estimation |
| title_full | Multimodal Deep Learning and Visible-Light and Hyperspectral Imaging for Fruit Maturity Estimation |
| title_fullStr | Multimodal Deep Learning and Visible-Light and Hyperspectral Imaging for Fruit Maturity Estimation |
| title_full_unstemmed | Multimodal Deep Learning and Visible-Light and Hyperspectral Imaging for Fruit Maturity Estimation |
| title_short | Multimodal Deep Learning and Visible-Light and Hyperspectral Imaging for Fruit Maturity Estimation |
| title_sort | multimodal deep learning and visible light and hyperspectral imaging for fruit maturity estimation |
| topic | multimodality deep learning hyperspectral imaging fruit maturity classification |
| url | https://www.mdpi.com/1424-8220/21/4/1288 |
| work_keys_str_mv | AT cinmayiiagarillosmanliguez multimodaldeeplearningandvisiblelightandhyperspectralimagingforfruitmaturityestimation AT johnychiang multimodaldeeplearningandvisiblelightandhyperspectralimagingforfruitmaturityestimation |