RBUE: a ReLU-based uncertainty estimation method for convolutional neural networks
Abstract Convolutional neural networks (CNNs) have successfully demonstrated their powerful predictive performance in a variety of tasks. However, it remains a challenge to estimate the uncertainty of these predictions simply and accurately. Deep Ensemble is widely considered the state-of-the-art me...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Springer
2023-02-01
|
| Series: | Complex & Intelligent Systems |
| Subjects: | |
| Online Access: | https://doi.org/10.1007/s40747-023-00973-0 |
| _version_ | 1797675044615225344 |
|---|---|
| author | Yufeng Xia Jun Zhang Zhiqiang Gong Tingsong Jiang Wen Yao |
| author_facet | Yufeng Xia Jun Zhang Zhiqiang Gong Tingsong Jiang Wen Yao |
| author_sort | Yufeng Xia |
| collection | DOAJ |
| description | Abstract Convolutional neural networks (CNNs) have successfully demonstrated their powerful predictive performance in a variety of tasks. However, it remains a challenge to estimate the uncertainty of these predictions simply and accurately. Deep Ensemble is widely considered the state-of-the-art method which can estimate the uncertainty accurately, but it is expensive to train and test. MC-Dropout is another popular method that is less costly but lacks the diversity of predictions resulting in less accurate uncertainty estimates. To combine the benefits of both, we introduce a ReLU-Based Uncertainty Estimation (RBUE) method. Instead of using the randomness of the Dropout module during the test phase (MC-Dropout) or using the randomness of the initial weights of CNNs (Deep Ensemble), RBUE uses the randomness of activation function to obtain diverse outputs in the testing phase to estimate uncertainty. Under the method, we propose strategy MC-DropReLU and develop strategy MC-RReLU. The uniform distribution of the activation function’s position in CNNs allows the randomness to be well transferred to the output results and gives a more diverse output, thus improving the accuracy of the uncertainty estimation. Moreover, our method is simple to implement and does not need to modify the existing model. We experimentally validate the RBUE on three widely used datasets, CIFAR10, CIFAR100, and TinyImageNet. The experiments demonstrate that our method has competitive performance but is more favorable in training time. |
| first_indexed | 2024-03-11T22:08:00Z |
| format | Article |
| id | doaj.art-7a5893a8fd26424eba9f7dfebec1de41 |
| institution | Directory Open Access Journal |
| issn | 2199-4536 2198-6053 |
| language | English |
| last_indexed | 2024-03-11T22:08:00Z |
| publishDate | 2023-02-01 |
| publisher | Springer |
| record_format | Article |
| series | Complex & Intelligent Systems |
| spelling | doaj.art-7a5893a8fd26424eba9f7dfebec1de412023-09-24T11:34:55ZengSpringerComplex & Intelligent Systems2199-45362198-60532023-02-01954735474910.1007/s40747-023-00973-0RBUE: a ReLU-based uncertainty estimation method for convolutional neural networksYufeng Xia0Jun Zhang1Zhiqiang Gong2Tingsong Jiang3Wen Yao4College of Aerospace Science and Engineering, National University of Defense TechnologyDefense Innovation Institute, Chinese Academy of Military ScienceDefense Innovation Institute, Chinese Academy of Military ScienceDefense Innovation Institute, Chinese Academy of Military ScienceDefense Innovation Institute, Chinese Academy of Military ScienceAbstract Convolutional neural networks (CNNs) have successfully demonstrated their powerful predictive performance in a variety of tasks. However, it remains a challenge to estimate the uncertainty of these predictions simply and accurately. Deep Ensemble is widely considered the state-of-the-art method which can estimate the uncertainty accurately, but it is expensive to train and test. MC-Dropout is another popular method that is less costly but lacks the diversity of predictions resulting in less accurate uncertainty estimates. To combine the benefits of both, we introduce a ReLU-Based Uncertainty Estimation (RBUE) method. Instead of using the randomness of the Dropout module during the test phase (MC-Dropout) or using the randomness of the initial weights of CNNs (Deep Ensemble), RBUE uses the randomness of activation function to obtain diverse outputs in the testing phase to estimate uncertainty. Under the method, we propose strategy MC-DropReLU and develop strategy MC-RReLU. The uniform distribution of the activation function’s position in CNNs allows the randomness to be well transferred to the output results and gives a more diverse output, thus improving the accuracy of the uncertainty estimation. Moreover, our method is simple to implement and does not need to modify the existing model. We experimentally validate the RBUE on three widely used datasets, CIFAR10, CIFAR100, and TinyImageNet. The experiments demonstrate that our method has competitive performance but is more favorable in training time.https://doi.org/10.1007/s40747-023-00973-0ReLU basedUncertainty estimationDiverse predictionsConvolutional neural networks |
| spellingShingle | Yufeng Xia Jun Zhang Zhiqiang Gong Tingsong Jiang Wen Yao RBUE: a ReLU-based uncertainty estimation method for convolutional neural networks Complex & Intelligent Systems ReLU based Uncertainty estimation Diverse predictions Convolutional neural networks |
| title | RBUE: a ReLU-based uncertainty estimation method for convolutional neural networks |
| title_full | RBUE: a ReLU-based uncertainty estimation method for convolutional neural networks |
| title_fullStr | RBUE: a ReLU-based uncertainty estimation method for convolutional neural networks |
| title_full_unstemmed | RBUE: a ReLU-based uncertainty estimation method for convolutional neural networks |
| title_short | RBUE: a ReLU-based uncertainty estimation method for convolutional neural networks |
| title_sort | rbue a relu based uncertainty estimation method for convolutional neural networks |
| topic | ReLU based Uncertainty estimation Diverse predictions Convolutional neural networks |
| url | https://doi.org/10.1007/s40747-023-00973-0 |
| work_keys_str_mv | AT yufengxia rbuearelubaseduncertaintyestimationmethodforconvolutionalneuralnetworks AT junzhang rbuearelubaseduncertaintyestimationmethodforconvolutionalneuralnetworks AT zhiqianggong rbuearelubaseduncertaintyestimationmethodforconvolutionalneuralnetworks AT tingsongjiang rbuearelubaseduncertaintyestimationmethodforconvolutionalneuralnetworks AT wenyao rbuearelubaseduncertaintyestimationmethodforconvolutionalneuralnetworks |