Incremental Kernel Principal Components Subspace Inference With Nyström Approximation for Bayesian Deep Learning
As the state-of-the-art technology of Bayesian inference, based on low-dimensional principal components analysis (PCA) subspace inference methods can provide approximately accurate predictive distribution and well calibrated uncertainty. However, the main problem of PCA method is that it is a linear...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2021-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9364973/ |
| _version_ | 1819176048813670400 |
|---|---|
| author | Yongguang Wang Shuzhen Yao Tian Xu |
| author_facet | Yongguang Wang Shuzhen Yao Tian Xu |
| author_sort | Yongguang Wang |
| collection | DOAJ |
| description | As the state-of-the-art technology of Bayesian inference, based on low-dimensional principal components analysis (PCA) subspace inference methods can provide approximately accurate predictive distribution and well calibrated uncertainty. However, the main problem of PCA method is that it is a linear subspace feature extractor, and it cannot effectively represent the nonlinearly high-dimensional parameter space of deep neural networks (DNNs). Firstly, in this paper, in order to solve the main problem of the linear characteristics of PCA in high-dimensional space, we apply kernel PCA to extract higher-order statistical information in parameter space of DNNs. Secondly, to improve the efficiency of subsequent computation, we propose a strictly ordered incremental kernel PCA (InKPCA) subspace of parameter space within stochastic gradient descent (SGD) trajectories. In the proposed InKPCA subspace, we employ two approximation inference methods: elliptical slice sampling (ESS) and variational inference (VI). Finally, to further improve the memory efficiency of computing the kernel matrix, we apply Nyström approximation to determine the suitable size of subsets in the original datasets. The novelty of this paper is that it is the first time to apply the proposed InKPCA subspace with Nyström approximation for Bayesian inference in DNNs, and the results show that it can produce more accurate predictions and well-calibrated predictive uncertainty in regression and classification tasks of deep learning. |
| first_indexed | 2024-12-22T21:04:34Z |
| format | Article |
| id | doaj.art-dc22e6654cd74d58b41c2fc6743097d0 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-22T21:04:34Z |
| publishDate | 2021-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-dc22e6654cd74d58b41c2fc6743097d02022-12-21T18:12:43ZengIEEEIEEE Access2169-35362021-01-019362413625110.1109/ACCESS.2021.30627479364973Incremental Kernel Principal Components Subspace Inference With Nyström Approximation for Bayesian Deep LearningYongguang Wang0https://orcid.org/0000-0002-0470-9178Shuzhen Yao1https://orcid.org/0000-0001-7785-9698Tian Xu2https://orcid.org/0000-0001-6108-7494School of Computer Science and Engineering, Beihang University, Beijing, ChinaSchool of Computer Science and Engineering, Beihang University, Beijing, ChinaSchool of Computer Science and Engineering, Beihang University, Beijing, ChinaAs the state-of-the-art technology of Bayesian inference, based on low-dimensional principal components analysis (PCA) subspace inference methods can provide approximately accurate predictive distribution and well calibrated uncertainty. However, the main problem of PCA method is that it is a linear subspace feature extractor, and it cannot effectively represent the nonlinearly high-dimensional parameter space of deep neural networks (DNNs). Firstly, in this paper, in order to solve the main problem of the linear characteristics of PCA in high-dimensional space, we apply kernel PCA to extract higher-order statistical information in parameter space of DNNs. Secondly, to improve the efficiency of subsequent computation, we propose a strictly ordered incremental kernel PCA (InKPCA) subspace of parameter space within stochastic gradient descent (SGD) trajectories. In the proposed InKPCA subspace, we employ two approximation inference methods: elliptical slice sampling (ESS) and variational inference (VI). Finally, to further improve the memory efficiency of computing the kernel matrix, we apply Nyström approximation to determine the suitable size of subsets in the original datasets. The novelty of this paper is that it is the first time to apply the proposed InKPCA subspace with Nyström approximation for Bayesian inference in DNNs, and the results show that it can produce more accurate predictions and well-calibrated predictive uncertainty in regression and classification tasks of deep learning.https://ieeexplore.ieee.org/document/9364973/Bayesian deep learningincremental kernel PCAelliptical slice samplingvariational inferenceNyström approximation |
| spellingShingle | Yongguang Wang Shuzhen Yao Tian Xu Incremental Kernel Principal Components Subspace Inference With Nyström Approximation for Bayesian Deep Learning IEEE Access Bayesian deep learning incremental kernel PCA elliptical slice sampling variational inference Nyström approximation |
| title | Incremental Kernel Principal Components Subspace Inference With Nyström Approximation for Bayesian Deep Learning |
| title_full | Incremental Kernel Principal Components Subspace Inference With Nyström Approximation for Bayesian Deep Learning |
| title_fullStr | Incremental Kernel Principal Components Subspace Inference With Nyström Approximation for Bayesian Deep Learning |
| title_full_unstemmed | Incremental Kernel Principal Components Subspace Inference With Nyström Approximation for Bayesian Deep Learning |
| title_short | Incremental Kernel Principal Components Subspace Inference With Nyström Approximation for Bayesian Deep Learning |
| title_sort | incremental kernel principal components subspace inference with nystr x00f6 m approximation for bayesian deep learning |
| topic | Bayesian deep learning incremental kernel PCA elliptical slice sampling variational inference Nyström approximation |
| url | https://ieeexplore.ieee.org/document/9364973/ |
| work_keys_str_mv | AT yongguangwang incrementalkernelprincipalcomponentssubspaceinferencewithnystrx00f6mapproximationforbayesiandeeplearning AT shuzhenyao incrementalkernelprincipalcomponentssubspaceinferencewithnystrx00f6mapproximationforbayesiandeeplearning AT tianxu incrementalkernelprincipalcomponentssubspaceinferencewithnystrx00f6mapproximationforbayesiandeeplearning |