<inline-formula> <tex-math notation="LaTeX">$\beta$ </tex-math></inline-formula>-Dropout: A Unified Dropout
Dropout is an effective regularization method for deep learning tasks. Several variants of dropout based on sampling with different distributions have been proposed individually and have shown good generalization performance on various learning tasks. Among these variants, the canonical Bernoulli dr...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2019-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/8666975/ |
| _version_ | 1831805960551661568 |
|---|---|
| author | Lei Liu Yuhao Luo Xu Shen Mingzhai Sun Bin Li |
| author_facet | Lei Liu Yuhao Luo Xu Shen Mingzhai Sun Bin Li |
| author_sort | Lei Liu |
| collection | DOAJ |
| description | Dropout is an effective regularization method for deep learning tasks. Several variants of dropout based on sampling with different distributions have been proposed individually and have shown good generalization performance on various learning tasks. Among these variants, the canonical Bernoulli dropout is a discrete method, while the uniform dropout and the Gaussian dropout are continuous dropout methods. When facing a new learning task, one must make a decision on which method is more suitable, which is somehow unnatural and inconvenient. In this paper, we attempt to change the selection problem to a parameter tuning problem by proposing a general form of dropout, β-dropout, to unify the discrete dropout with continuous dropout. We show that by adjusting the shape parameter β, the β-dropout can yield the Bernoulli dropout, uniform dropout, and approximate Gaussian dropout. Furthermore, it can obtain continuous regularization strength, which paves the way for self-adaptive dropout regularization. As a first attempt, we propose a self-adaptive β-dropout, in which the parameter β is tuned automatically following a pre-designed strategy. The β-dropout is tested extensively on the MNIST, CIFAR-10, SVHN, NORB, and ILSVRC-12 datasets to investigate its superior performance. The results show that the β-dropout can conduct finer control of its regularization strength, therefore obtaining better performance. |
| first_indexed | 2024-12-22T19:32:38Z |
| format | Article |
| id | doaj.art-d999cb8aaefe40ac8709562b7391ccd6 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-22T19:32:38Z |
| publishDate | 2019-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-d999cb8aaefe40ac8709562b7391ccd62022-12-21T18:15:03ZengIEEEIEEE Access2169-35362019-01-017361403615310.1109/ACCESS.2019.29048818666975<inline-formula> <tex-math notation="LaTeX">$\beta$ </tex-math></inline-formula>-Dropout: A Unified DropoutLei Liu0Yuhao Luo1Xu Shen2Mingzhai Sun3https://orcid.org/0000-0002-0522-5511Bin Li4https://orcid.org/0000-0002-2332-3959CAS Key Laboratory of Technology in Geo-spatial Information Processing and Application System, University of Science and Technology of China, Hefei, ChinaDepartment of Precision Machinery and Precision Instrumentation, University of Science and Technology of China, Hefei, ChinaCAS Key Laboratory of Technology in Geo-spatial Information Processing and Application System, University of Science and Technology of China, Hefei, ChinaDepartment of Precision Machinery and Precision Instrumentation, University of Science and Technology of China, Hefei, ChinaCAS Key Laboratory of Technology in Geo-spatial Information Processing and Application System, University of Science and Technology of China, Hefei, ChinaDropout is an effective regularization method for deep learning tasks. Several variants of dropout based on sampling with different distributions have been proposed individually and have shown good generalization performance on various learning tasks. Among these variants, the canonical Bernoulli dropout is a discrete method, while the uniform dropout and the Gaussian dropout are continuous dropout methods. When facing a new learning task, one must make a decision on which method is more suitable, which is somehow unnatural and inconvenient. In this paper, we attempt to change the selection problem to a parameter tuning problem by proposing a general form of dropout, β-dropout, to unify the discrete dropout with continuous dropout. We show that by adjusting the shape parameter β, the β-dropout can yield the Bernoulli dropout, uniform dropout, and approximate Gaussian dropout. Furthermore, it can obtain continuous regularization strength, which paves the way for self-adaptive dropout regularization. As a first attempt, we propose a self-adaptive β-dropout, in which the parameter β is tuned automatically following a pre-designed strategy. The β-dropout is tested extensively on the MNIST, CIFAR-10, SVHN, NORB, and ILSVRC-12 datasets to investigate its superior performance. The results show that the β-dropout can conduct finer control of its regularization strength, therefore obtaining better performance.https://ieeexplore.ieee.org/document/8666975/Regularizationdropoutdeep learningGaussian dropoutBernoulli dropout |
| spellingShingle | Lei Liu Yuhao Luo Xu Shen Mingzhai Sun Bin Li <inline-formula> <tex-math notation="LaTeX">$\beta$ </tex-math></inline-formula>-Dropout: A Unified Dropout IEEE Access Regularization dropout deep learning Gaussian dropout Bernoulli dropout |
| title | <inline-formula> <tex-math notation="LaTeX">$\beta$ </tex-math></inline-formula>-Dropout: A Unified Dropout |
| title_full | <inline-formula> <tex-math notation="LaTeX">$\beta$ </tex-math></inline-formula>-Dropout: A Unified Dropout |
| title_fullStr | <inline-formula> <tex-math notation="LaTeX">$\beta$ </tex-math></inline-formula>-Dropout: A Unified Dropout |
| title_full_unstemmed | <inline-formula> <tex-math notation="LaTeX">$\beta$ </tex-math></inline-formula>-Dropout: A Unified Dropout |
| title_short | <inline-formula> <tex-math notation="LaTeX">$\beta$ </tex-math></inline-formula>-Dropout: A Unified Dropout |
| title_sort | inline formula tex math notation latex beta tex math inline formula dropout a unified dropout |
| topic | Regularization dropout deep learning Gaussian dropout Bernoulli dropout |
| url | https://ieeexplore.ieee.org/document/8666975/ |
| work_keys_str_mv | AT leiliu inlineformulatexmathnotationlatexbetatexmathinlineformuladropoutaunifieddropout AT yuhaoluo inlineformulatexmathnotationlatexbetatexmathinlineformuladropoutaunifieddropout AT xushen inlineformulatexmathnotationlatexbetatexmathinlineformuladropoutaunifieddropout AT mingzhaisun inlineformulatexmathnotationlatexbetatexmathinlineformuladropoutaunifieddropout AT binli inlineformulatexmathnotationlatexbetatexmathinlineformuladropoutaunifieddropout |