Regularity Normalization: Neuroscience-Inspired Unsupervised Attention across Neural Network Layers
Inspired by the adaptation phenomenon of neuronal firing, we propose the regularity normalization (RN) as an unsupervised attention mechanism (UAM) which computes the statistical regularity in the implicit space of neural networks under the Minimum Description Length (MDL) principle. Treating the ne...
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-12-01
|
| Series: | Entropy |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1099-4300/24/1/59 |
| _version_ | 1797494266650427392 |
|---|---|
| author | Baihan Lin |
| author_facet | Baihan Lin |
| author_sort | Baihan Lin |
| collection | DOAJ |
| description | Inspired by the adaptation phenomenon of neuronal firing, we propose the regularity normalization (RN) as an unsupervised attention mechanism (UAM) which computes the statistical regularity in the implicit space of neural networks under the Minimum Description Length (MDL) principle. Treating the neural network optimization process as a partially observable model selection problem, the regularity normalization constrains the implicit space by a normalization factor, the universal code length. We compute this universal code incrementally across neural network layers and demonstrate the flexibility to include data priors such as top-down attention and other oracle information. Empirically, our approach outperforms existing normalization methods in tackling limited, imbalanced and non-stationary input distribution in image classification, classic control, procedurally-generated reinforcement learning, generative modeling, handwriting generation and question answering tasks with various neural network architectures. Lastly, the unsupervised attention mechanisms is a useful probing tool for neural networks by tracking the dependency and critical learning stages across layers and recurrent time steps of deep networks. |
| first_indexed | 2024-03-10T01:31:54Z |
| format | Article |
| id | doaj.art-b0f5dfc2d44d4b49abd7ac581a589a05 |
| institution | Directory Open Access Journal |
| issn | 1099-4300 |
| language | English |
| last_indexed | 2024-03-10T01:31:54Z |
| publishDate | 2021-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Entropy |
| spelling | doaj.art-b0f5dfc2d44d4b49abd7ac581a589a052023-11-23T13:41:14ZengMDPI AGEntropy1099-43002021-12-012415910.3390/e24010059Regularity Normalization: Neuroscience-Inspired Unsupervised Attention across Neural Network LayersBaihan Lin0Department of Neuroscience, Columbia University Irving Medical Center, New York, NY 10032, USAInspired by the adaptation phenomenon of neuronal firing, we propose the regularity normalization (RN) as an unsupervised attention mechanism (UAM) which computes the statistical regularity in the implicit space of neural networks under the Minimum Description Length (MDL) principle. Treating the neural network optimization process as a partially observable model selection problem, the regularity normalization constrains the implicit space by a normalization factor, the universal code length. We compute this universal code incrementally across neural network layers and demonstrate the flexibility to include data priors such as top-down attention and other oracle information. Empirically, our approach outperforms existing normalization methods in tackling limited, imbalanced and non-stationary input distribution in image classification, classic control, procedurally-generated reinforcement learning, generative modeling, handwriting generation and question answering tasks with various neural network architectures. Lastly, the unsupervised attention mechanisms is a useful probing tool for neural networks by tracking the dependency and critical learning stages across layers and recurrent time steps of deep networks.https://www.mdpi.com/1099-4300/24/1/59neuronal codingbiologically plausible modelsminimum description lengthdeep neural networksnormalization methods |
| spellingShingle | Baihan Lin Regularity Normalization: Neuroscience-Inspired Unsupervised Attention across Neural Network Layers Entropy neuronal coding biologically plausible models minimum description length deep neural networks normalization methods |
| title | Regularity Normalization: Neuroscience-Inspired Unsupervised Attention across Neural Network Layers |
| title_full | Regularity Normalization: Neuroscience-Inspired Unsupervised Attention across Neural Network Layers |
| title_fullStr | Regularity Normalization: Neuroscience-Inspired Unsupervised Attention across Neural Network Layers |
| title_full_unstemmed | Regularity Normalization: Neuroscience-Inspired Unsupervised Attention across Neural Network Layers |
| title_short | Regularity Normalization: Neuroscience-Inspired Unsupervised Attention across Neural Network Layers |
| title_sort | regularity normalization neuroscience inspired unsupervised attention across neural network layers |
| topic | neuronal coding biologically plausible models minimum description length deep neural networks normalization methods |
| url | https://www.mdpi.com/1099-4300/24/1/59 |
| work_keys_str_mv | AT baihanlin regularitynormalizationneuroscienceinspiredunsupervisedattentionacrossneuralnetworklayers |