A Comparison of Variational Bounds for the Information Bottleneck Functional
In this short note, we relate the variational bounds proposed in Alemi et al. (2017) and Fischer (2020) for the information bottleneck (IB) and the conditional entropy bottleneck (CEB) functional, respectively. Although the two functionals were shown to be equivalent, it was empirically observed tha...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2020-10-01
|
| Series: | Entropy |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1099-4300/22/11/1229 |
| _version_ | 1797549552455122944 |
|---|---|
| author | Bernhard C. Geiger Ian S. Fischer |
| author_facet | Bernhard C. Geiger Ian S. Fischer |
| author_sort | Bernhard C. Geiger |
| collection | DOAJ |
| description | In this short note, we relate the variational bounds proposed in Alemi et al. (2017) and Fischer (2020) for the information bottleneck (IB) and the conditional entropy bottleneck (CEB) functional, respectively. Although the two functionals were shown to be equivalent, it was empirically observed that optimizing bounds on the CEB functional achieves better generalization performance and adversarial robustness than optimizing those on the IB functional. This work tries to shed light on this issue by showing that, in the most general setting, no ordering can be established between these variational bounds, while such an ordering can be enforced by restricting the feasible sets over which the optimizations take place. The absence of such an ordering in the general setup suggests that the variational bound on the CEB functional is either more amenable to optimization or a relevant cost function for optimization in its own regard, i.e., without justification from the IB or CEB functionals. |
| first_indexed | 2024-03-10T15:16:14Z |
| format | Article |
| id | doaj.art-d380c722a5ec43aeb062966a846a443f |
| institution | Directory Open Access Journal |
| issn | 1099-4300 |
| language | English |
| last_indexed | 2024-03-10T15:16:14Z |
| publishDate | 2020-10-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Entropy |
| spelling | doaj.art-d380c722a5ec43aeb062966a846a443f2023-11-20T18:56:54ZengMDPI AGEntropy1099-43002020-10-012211122910.3390/e22111229A Comparison of Variational Bounds for the Information Bottleneck FunctionalBernhard C. Geiger0Ian S. Fischer1Know-Center GmbH, Inffeldgasse 13/6, 8010 Graz, AustriaGoogle Research, Mountain View, CA 94043, USAIn this short note, we relate the variational bounds proposed in Alemi et al. (2017) and Fischer (2020) for the information bottleneck (IB) and the conditional entropy bottleneck (CEB) functional, respectively. Although the two functionals were shown to be equivalent, it was empirically observed that optimizing bounds on the CEB functional achieves better generalization performance and adversarial robustness than optimizing those on the IB functional. This work tries to shed light on this issue by showing that, in the most general setting, no ordering can be established between these variational bounds, while such an ordering can be enforced by restricting the feasible sets over which the optimizations take place. The absence of such an ordering in the general setup suggests that the variational bound on the CEB functional is either more amenable to optimization or a relevant cost function for optimization in its own regard, i.e., without justification from the IB or CEB functionals.https://www.mdpi.com/1099-4300/22/11/1229information bottleneckdeep learningneural networks |
| spellingShingle | Bernhard C. Geiger Ian S. Fischer A Comparison of Variational Bounds for the Information Bottleneck Functional Entropy information bottleneck deep learning neural networks |
| title | A Comparison of Variational Bounds for the Information Bottleneck Functional |
| title_full | A Comparison of Variational Bounds for the Information Bottleneck Functional |
| title_fullStr | A Comparison of Variational Bounds for the Information Bottleneck Functional |
| title_full_unstemmed | A Comparison of Variational Bounds for the Information Bottleneck Functional |
| title_short | A Comparison of Variational Bounds for the Information Bottleneck Functional |
| title_sort | comparison of variational bounds for the information bottleneck functional |
| topic | information bottleneck deep learning neural networks |
| url | https://www.mdpi.com/1099-4300/22/11/1229 |
| work_keys_str_mv | AT bernhardcgeiger acomparisonofvariationalboundsfortheinformationbottleneckfunctional AT iansfischer acomparisonofvariationalboundsfortheinformationbottleneckfunctional AT bernhardcgeiger comparisonofvariationalboundsfortheinformationbottleneckfunctional AT iansfischer comparisonofvariationalboundsfortheinformationbottleneckfunctional |