The Mathematical Structure of Information Bottleneck Methods
Information Bottleneck-based methods use mutual information as a distortion function in order to extract relevant details about the structure of a complex system by compression. One of the approaches used to generate optimal compressed representations is by annealing a parameter. In this manuscript...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2012-03-01
|
| Series: | Entropy |
| Subjects: | |
| Online Access: | http://www.mdpi.com/1099-4300/14/3/456/ |
| _version_ | 1811183792526721024 |
|---|---|
| author | Albert E. Parker Tomáš Gedeon Alexander G. Dimitrov |
| author_facet | Albert E. Parker Tomáš Gedeon Alexander G. Dimitrov |
| author_sort | Albert E. Parker |
| collection | DOAJ |
| description | Information Bottleneck-based methods use mutual information as a distortion function in order to extract relevant details about the structure of a complex system by compression. One of the approaches used to generate optimal compressed representations is by annealing a parameter. In this manuscript we present a common framework for the study of annealing in information distortion problems. We identify features that should be common to any annealing optimization problem. The main mathematical tools that we use come from the analysis of dynamical systems in the presence of symmetry (equivariant bifurcation theory). Through the compression problem, we make connections to the world of combinatorial optimization and pattern recognition. The two approaches use very different vocabularies and consider different problems to be “interesting”. We provide an initial link, through the Normalized Cut Problem, where the two disciplines can exchange tools and ideas. |
| first_indexed | 2024-04-11T11:52:13Z |
| format | Article |
| id | doaj.art-9a37623f21794362ab5ea0ac45fa74c6 |
| institution | Directory Open Access Journal |
| issn | 1099-4300 |
| language | English |
| last_indexed | 2024-04-11T11:52:13Z |
| publishDate | 2012-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Entropy |
| spelling | doaj.art-9a37623f21794362ab5ea0ac45fa74c62022-12-22T04:25:16ZengMDPI AGEntropy1099-43002012-03-0114345647910.3390/e14030456The Mathematical Structure of Information Bottleneck MethodsAlbert E. ParkerTomáš GedeonAlexander G. DimitrovInformation Bottleneck-based methods use mutual information as a distortion function in order to extract relevant details about the structure of a complex system by compression. One of the approaches used to generate optimal compressed representations is by annealing a parameter. In this manuscript we present a common framework for the study of annealing in information distortion problems. We identify features that should be common to any annealing optimization problem. The main mathematical tools that we use come from the analysis of dynamical systems in the presence of symmetry (equivariant bifurcation theory). Through the compression problem, we make connections to the world of combinatorial optimization and pattern recognition. The two approaches use very different vocabularies and consider different problems to be “interesting”. We provide an initial link, through the Normalized Cut Problem, where the two disciplines can exchange tools and ideas.http://www.mdpi.com/1099-4300/14/3/456/information distortionspontaneous symmetry breakingbifurcationsphase transition |
| spellingShingle | Albert E. Parker Tomáš Gedeon Alexander G. Dimitrov The Mathematical Structure of Information Bottleneck Methods Entropy information distortion spontaneous symmetry breaking bifurcations phase transition |
| title | The Mathematical Structure of Information Bottleneck Methods |
| title_full | The Mathematical Structure of Information Bottleneck Methods |
| title_fullStr | The Mathematical Structure of Information Bottleneck Methods |
| title_full_unstemmed | The Mathematical Structure of Information Bottleneck Methods |
| title_short | The Mathematical Structure of Information Bottleneck Methods |
| title_sort | mathematical structure of information bottleneck methods |
| topic | information distortion spontaneous symmetry breaking bifurcations phase transition |
| url | http://www.mdpi.com/1099-4300/14/3/456/ |
| work_keys_str_mv | AT alberteparker themathematicalstructureofinformationbottleneckmethods AT tomasgedeon themathematicalstructureofinformationbottleneckmethods AT alexandergdimitrov themathematicalstructureofinformationbottleneckmethods AT alberteparker mathematicalstructureofinformationbottleneckmethods AT tomasgedeon mathematicalstructureofinformationbottleneckmethods AT alexandergdimitrov mathematicalstructureofinformationbottleneckmethods |