Multimodal Ground-Based Cloud Classification Using Joint Fusion Convolutional Neural Network
The accurate ground-based cloud classification is a challenging task and still under development. The most current methods are limited to only taking the cloud visual features into consideration, which is not robust to the environmental factors. In this paper, we present the novel joint fusion convo...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2018-05-01
|
| Series: | Remote Sensing |
| Subjects: | |
| Online Access: | http://www.mdpi.com/2072-4292/10/6/822 |
| _version_ | 1831712643542417408 |
|---|---|
| author | Shuang Liu Mei Li Zhong Zhang Baihua Xiao Xiaozhong Cao |
| author_facet | Shuang Liu Mei Li Zhong Zhang Baihua Xiao Xiaozhong Cao |
| author_sort | Shuang Liu |
| collection | DOAJ |
| description | The accurate ground-based cloud classification is a challenging task and still under development. The most current methods are limited to only taking the cloud visual features into consideration, which is not robust to the environmental factors. In this paper, we present the novel joint fusion convolutional neural network (JFCNN) to integrate the multimodal information for ground-based cloud classification. To learn the heterogeneous features (visual features and multimodal features) from the ground-based cloud data, we designed the proposed JFCNN as a two-stream structure which contains the vision subnetwork and multimodal subnetwork. We also proposed a novel layer named joint fusion layer to jointly learn two kinds of cloud features under one framework. After training the proposed JFCNN, we extracted the visual and multimodal features from the two subnetworks and integrated them using a weighted strategy. The proposed JFCNN was validated on the multimodal ground-based cloud (MGC) dataset and achieved remarkable performance, demonstrating its effectiveness for ground-based cloud classification task. |
| first_indexed | 2024-12-20T23:25:47Z |
| format | Article |
| id | doaj.art-45849b7450014593a7054272e03dd9ea |
| institution | Directory Open Access Journal |
| issn | 2072-4292 |
| language | English |
| last_indexed | 2024-12-20T23:25:47Z |
| publishDate | 2018-05-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Remote Sensing |
| spelling | doaj.art-45849b7450014593a7054272e03dd9ea2022-12-21T19:23:25ZengMDPI AGRemote Sensing2072-42922018-05-0110682210.3390/rs10060822rs10060822Multimodal Ground-Based Cloud Classification Using Joint Fusion Convolutional Neural NetworkShuang Liu0Mei Li1Zhong Zhang2Baihua Xiao3Xiaozhong Cao4Tianjin Key Laboratory of Wireless Mobile Communications and Power Transmission, Tianjin Normal University, Tianjin 300387, ChinaTianjin Key Laboratory of Wireless Mobile Communications and Power Transmission, Tianjin Normal University, Tianjin 300387, ChinaTianjin Key Laboratory of Wireless Mobile Communications and Power Transmission, Tianjin Normal University, Tianjin 300387, ChinaThe State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, ChinaMeteorological Observation Centre, China Meteorological Administration, Beijing 100081, ChinaThe accurate ground-based cloud classification is a challenging task and still under development. The most current methods are limited to only taking the cloud visual features into consideration, which is not robust to the environmental factors. In this paper, we present the novel joint fusion convolutional neural network (JFCNN) to integrate the multimodal information for ground-based cloud classification. To learn the heterogeneous features (visual features and multimodal features) from the ground-based cloud data, we designed the proposed JFCNN as a two-stream structure which contains the vision subnetwork and multimodal subnetwork. We also proposed a novel layer named joint fusion layer to jointly learn two kinds of cloud features under one framework. After training the proposed JFCNN, we extracted the visual and multimodal features from the two subnetworks and integrated them using a weighted strategy. The proposed JFCNN was validated on the multimodal ground-based cloud (MGC) dataset and achieved remarkable performance, demonstrating its effectiveness for ground-based cloud classification task.http://www.mdpi.com/2072-4292/10/6/822ground-based cloud classificationjoint fusion convolutional neural networkmultimodal informationfeature fusion |
| spellingShingle | Shuang Liu Mei Li Zhong Zhang Baihua Xiao Xiaozhong Cao Multimodal Ground-Based Cloud Classification Using Joint Fusion Convolutional Neural Network Remote Sensing ground-based cloud classification joint fusion convolutional neural network multimodal information feature fusion |
| title | Multimodal Ground-Based Cloud Classification Using Joint Fusion Convolutional Neural Network |
| title_full | Multimodal Ground-Based Cloud Classification Using Joint Fusion Convolutional Neural Network |
| title_fullStr | Multimodal Ground-Based Cloud Classification Using Joint Fusion Convolutional Neural Network |
| title_full_unstemmed | Multimodal Ground-Based Cloud Classification Using Joint Fusion Convolutional Neural Network |
| title_short | Multimodal Ground-Based Cloud Classification Using Joint Fusion Convolutional Neural Network |
| title_sort | multimodal ground based cloud classification using joint fusion convolutional neural network |
| topic | ground-based cloud classification joint fusion convolutional neural network multimodal information feature fusion |
| url | http://www.mdpi.com/2072-4292/10/6/822 |
| work_keys_str_mv | AT shuangliu multimodalgroundbasedcloudclassificationusingjointfusionconvolutionalneuralnetwork AT meili multimodalgroundbasedcloudclassificationusingjointfusionconvolutionalneuralnetwork AT zhongzhang multimodalgroundbasedcloudclassificationusingjointfusionconvolutionalneuralnetwork AT baihuaxiao multimodalgroundbasedcloudclassificationusingjointfusionconvolutionalneuralnetwork AT xiaozhongcao multimodalgroundbasedcloudclassificationusingjointfusionconvolutionalneuralnetwork |