Sparse Bayesian Neural Networks: Bridging Model and Parameter Uncertainty through Scalable Variational Inference
Bayesian neural networks (BNNs) have recently regained a significant amount of attention in the deep learning community due to the development of scalable approximate Bayesian inference techniques. There are several advantages of using a Bayesian approach: parameter and prediction uncertainties beco...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-03-01
|
| Series: | Mathematics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2227-7390/12/6/788 |
| _version_ | 1797240113541939200 |
|---|---|
| author | Aliaksandr Hubin Geir Storvik |
| author_facet | Aliaksandr Hubin Geir Storvik |
| author_sort | Aliaksandr Hubin |
| collection | DOAJ |
| description | Bayesian neural networks (BNNs) have recently regained a significant amount of attention in the deep learning community due to the development of scalable approximate Bayesian inference techniques. There are several advantages of using a Bayesian approach: parameter and prediction uncertainties become easily available, facilitating more rigorous statistical analysis. Furthermore, prior knowledge can be incorporated. However, the construction of scalable techniques that combine both structural and parameter uncertainty remains a challenge. In this paper, we apply the concept of model uncertainty as a framework for structural learning in BNNs and, hence, make inferences in the joint space of structures/models and parameters. Moreover, we suggest an adaptation of a scalable variational inference approach with reparametrization of marginal inclusion probabilities to incorporate the model space constraints. Experimental results on a range of benchmark datasets show that we obtain comparable accuracy results with the competing models, but based on methods that are much more sparse than ordinary BNNs. |
| first_indexed | 2024-04-24T18:02:16Z |
| format | Article |
| id | doaj.art-20b23baa56d8408ca1892bed2a24418a |
| institution | Directory Open Access Journal |
| issn | 2227-7390 |
| language | English |
| last_indexed | 2024-04-24T18:02:16Z |
| publishDate | 2024-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Mathematics |
| spelling | doaj.art-20b23baa56d8408ca1892bed2a24418a2024-03-27T13:52:54ZengMDPI AGMathematics2227-73902024-03-0112678810.3390/math12060788Sparse Bayesian Neural Networks: Bridging Model and Parameter Uncertainty through Scalable Variational InferenceAliaksandr Hubin0Geir Storvik1Bioinformatics and Applied Statistics, Norwegian University of Life Sciences, 1433 Ås, NorwayDepartment of Mathematics, University of Oslo, 0316 Oslo, NorwayBayesian neural networks (BNNs) have recently regained a significant amount of attention in the deep learning community due to the development of scalable approximate Bayesian inference techniques. There are several advantages of using a Bayesian approach: parameter and prediction uncertainties become easily available, facilitating more rigorous statistical analysis. Furthermore, prior knowledge can be incorporated. However, the construction of scalable techniques that combine both structural and parameter uncertainty remains a challenge. In this paper, we apply the concept of model uncertainty as a framework for structural learning in BNNs and, hence, make inferences in the joint space of structures/models and parameters. Moreover, we suggest an adaptation of a scalable variational inference approach with reparametrization of marginal inclusion probabilities to incorporate the model space constraints. Experimental results on a range of benchmark datasets show that we obtain comparable accuracy results with the competing models, but based on methods that are much more sparse than ordinary BNNs.https://www.mdpi.com/2227-7390/12/6/788Bayesian neural networksstructural learningmodel selectionmodel averagingapproximate Bayesian inferencepredictive uncertainty |
| spellingShingle | Aliaksandr Hubin Geir Storvik Sparse Bayesian Neural Networks: Bridging Model and Parameter Uncertainty through Scalable Variational Inference Mathematics Bayesian neural networks structural learning model selection model averaging approximate Bayesian inference predictive uncertainty |
| title | Sparse Bayesian Neural Networks: Bridging Model and Parameter Uncertainty through Scalable Variational Inference |
| title_full | Sparse Bayesian Neural Networks: Bridging Model and Parameter Uncertainty through Scalable Variational Inference |
| title_fullStr | Sparse Bayesian Neural Networks: Bridging Model and Parameter Uncertainty through Scalable Variational Inference |
| title_full_unstemmed | Sparse Bayesian Neural Networks: Bridging Model and Parameter Uncertainty through Scalable Variational Inference |
| title_short | Sparse Bayesian Neural Networks: Bridging Model and Parameter Uncertainty through Scalable Variational Inference |
| title_sort | sparse bayesian neural networks bridging model and parameter uncertainty through scalable variational inference |
| topic | Bayesian neural networks structural learning model selection model averaging approximate Bayesian inference predictive uncertainty |
| url | https://www.mdpi.com/2227-7390/12/6/788 |
| work_keys_str_mv | AT aliaksandrhubin sparsebayesianneuralnetworksbridgingmodelandparameteruncertaintythroughscalablevariationalinference AT geirstorvik sparsebayesianneuralnetworksbridgingmodelandparameteruncertaintythroughscalablevariationalinference |