Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning
We systematically explored a spectrum of normalization algorithms related to Batch Normalization (BN) and propose a generalized formulation that simultaneously solves two major limitations of BN: (1) online learning and (2) recurrent learning. Our proposal is simpler and more biologically-plausible....
| Main Authors: | , , |
|---|---|
| Format: | Technical Report |
| Language: | en_US |
| Published: |
Center for Brains, Minds and Machines (CBMM), arXiv
2016
|
| Subjects: | |
| Online Access: | http://hdl.handle.net/1721.1/104906 |
| _version_ | 1826205171199246336 |
|---|---|
| author | Liao, Qianli Kawaguchi, Kenji Poggio, Tomaso |
| author_facet | Liao, Qianli Kawaguchi, Kenji Poggio, Tomaso |
| author_sort | Liao, Qianli |
| collection | MIT |
| description | We systematically explored a spectrum of normalization algorithms related to Batch Normalization (BN) and propose a generalized formulation that simultaneously solves two major limitations of BN: (1) online learning and (2) recurrent learning. Our proposal is simpler and more biologically-plausible. Unlike previous approaches, our technique can be applied out of the box to all learning scenarios (e.g., online learning, batch learning, fully-connected, convolutional, feedforward, recurrent and mixed — recurrent and convolutional) and compare favorably with existing approaches. We also propose Lp Normalization for normalizing by different orders of statistical moments. In particular, L1 normalization is well-performing, simple to implement, fast to compute, more biologically-plausible and thus ideal for GPU or hardware implementations. |
| first_indexed | 2024-09-23T13:08:51Z |
| format | Technical Report |
| id | mit-1721.1/104906 |
| institution | Massachusetts Institute of Technology |
| language | en_US |
| last_indexed | 2024-09-23T13:08:51Z |
| publishDate | 2016 |
| publisher | Center for Brains, Minds and Machines (CBMM), arXiv |
| record_format | dspace |
| spelling | mit-1721.1/1049062019-04-10T15:19:16Z Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning Liao, Qianli Kawaguchi, Kenji Poggio, Tomaso Batch Normalization (BN) recurrent learning Lp Normalization We systematically explored a spectrum of normalization algorithms related to Batch Normalization (BN) and propose a generalized formulation that simultaneously solves two major limitations of BN: (1) online learning and (2) recurrent learning. Our proposal is simpler and more biologically-plausible. Unlike previous approaches, our technique can be applied out of the box to all learning scenarios (e.g., online learning, batch learning, fully-connected, convolutional, feedforward, recurrent and mixed — recurrent and convolutional) and compare favorably with existing approaches. We also propose Lp Normalization for normalizing by different orders of statistical moments. In particular, L1 normalization is well-performing, simple to implement, fast to compute, more biologically-plausible and thus ideal for GPU or hardware implementations. This work was supported by the Center for Brains, Minds and Machines (CBMM), funded by NSF STC award CCF - 1231216. 2016-10-21T15:42:40Z 2016-10-21T15:42:40Z 2016-10-19 Technical Report Working Paper Other http://hdl.handle.net/1721.1/104906 arXiv:1610.06160v1 en_US CBMM Memo Series;057 Attribution-NonCommercial-ShareAlike 3.0 United States http://creativecommons.org/licenses/by-nc-sa/3.0/us/ application/pdf Center for Brains, Minds and Machines (CBMM), arXiv |
| spellingShingle | Batch Normalization (BN) recurrent learning Lp Normalization Liao, Qianli Kawaguchi, Kenji Poggio, Tomaso Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning |
| title | Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning |
| title_full | Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning |
| title_fullStr | Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning |
| title_full_unstemmed | Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning |
| title_short | Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning |
| title_sort | streaming normalization towards simpler and more biologically plausible normalizations for online and recurrent learning |
| topic | Batch Normalization (BN) recurrent learning Lp Normalization |
| url | http://hdl.handle.net/1721.1/104906 |
| work_keys_str_mv | AT liaoqianli streamingnormalizationtowardssimplerandmorebiologicallyplausiblenormalizationsforonlineandrecurrentlearning AT kawaguchikenji streamingnormalizationtowardssimplerandmorebiologicallyplausiblenormalizationsforonlineandrecurrentlearning AT poggiotomaso streamingnormalizationtowardssimplerandmorebiologicallyplausiblenormalizationsforonlineandrecurrentlearning |