Probabilistic symmetries and invariant neural networks
Treating neural network inputs and outputs as random variables, we characterize the structure of neural networks that can be used to model data that are invariant or equivariant under the action of a compact group. Much recent research has been devoted to encoding invariance under symmetry transform...
| Main Authors: | , |
|---|---|
| Format: | Journal article |
| Jezik: | English |
| Izdano: |
Journal of Machine Learning Research
2020
|
| _version_ | 1826264895470960640 |
|---|---|
| author | Bloem-Reddy, B Teh, YW |
| author_facet | Bloem-Reddy, B Teh, YW |
| author_sort | Bloem-Reddy, B |
| collection | OXFORD |
| description | Treating neural network inputs and outputs as random variables, we characterize the structure of neural networks that can be used to model data that are invariant or equivariant under the action of a compact group. Much recent research has been devoted to encoding invariance under symmetry transformations into neural network architectures, in an effort to improve the performance of deep neural networks in data-scarce, non-i.i.d., or unsupervised settings. By considering group invariance from the perspective of probabilistic symmetry, we establish a link between functional and probabilistic symmetry, and obtain generative functional representations of probability distributions that are invariant or equivariant under the action of a compact group. Our representations completely characterize the structure of neural networks that can be used to model such distributions and yield a general program for constructing invariant stochastic or deterministic neural networks. We demonstrate that examples from the recent literature are special cases, and develop the details of the general program for exchangeable sequences and arrays.
|
| first_indexed | 2024-03-06T20:15:12Z |
| format | Journal article |
| id | oxford-uuid:2be9f9ae-8566-48d3-aa17-108a44176a34 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-03-06T20:15:12Z |
| publishDate | 2020 |
| publisher | Journal of Machine Learning Research |
| record_format | dspace |
| spelling | oxford-uuid:2be9f9ae-8566-48d3-aa17-108a44176a342022-03-26T12:33:53ZProbabilistic symmetries and invariant neural networksJournal articlehttp://purl.org/coar/resource_type/c_dcae04bcuuid:2be9f9ae-8566-48d3-aa17-108a44176a34EnglishSymplectic Elements Journal of Machine Learning Research2020Bloem-Reddy, BTeh, YWTreating neural network inputs and outputs as random variables, we characterize the structure of neural networks that can be used to model data that are invariant or equivariant under the action of a compact group. Much recent research has been devoted to encoding invariance under symmetry transformations into neural network architectures, in an effort to improve the performance of deep neural networks in data-scarce, non-i.i.d., or unsupervised settings. By considering group invariance from the perspective of probabilistic symmetry, we establish a link between functional and probabilistic symmetry, and obtain generative functional representations of probability distributions that are invariant or equivariant under the action of a compact group. Our representations completely characterize the structure of neural networks that can be used to model such distributions and yield a general program for constructing invariant stochastic or deterministic neural networks. We demonstrate that examples from the recent literature are special cases, and develop the details of the general program for exchangeable sequences and arrays. |
| spellingShingle | Bloem-Reddy, B Teh, YW Probabilistic symmetries and invariant neural networks |
| title | Probabilistic symmetries and invariant neural networks |
| title_full | Probabilistic symmetries and invariant neural networks |
| title_fullStr | Probabilistic symmetries and invariant neural networks |
| title_full_unstemmed | Probabilistic symmetries and invariant neural networks |
| title_short | Probabilistic symmetries and invariant neural networks |
| title_sort | probabilistic symmetries and invariant neural networks |
| work_keys_str_mv | AT bloemreddyb probabilisticsymmetriesandinvariantneuralnetworks AT tehyw probabilisticsymmetriesandinvariantneuralnetworks |