Fisher-rao metric, geometry, and complexity of neural networks
© 2019 by the author(s). We study the relationship between geometry and capacity measures for deep neural networks from an invariance viewpoint. We introduce a new notion of capacity - the Fisher-Rao norm - that possesses desirable invariance properties and is motivated by Information Geometry. We d...
| Main Authors: | , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
2021
|
| Online Access: | https://hdl.handle.net/1721.1/138296 |
| _version_ | 1811086059538219008 |
|---|---|
| author | Liang, T Poggio, T Rakhlin, A Stokes, J |
| author2 | Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences |
| author_facet | Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences Liang, T Poggio, T Rakhlin, A Stokes, J |
| author_sort | Liang, T |
| collection | MIT |
| description | © 2019 by the author(s). We study the relationship between geometry and capacity measures for deep neural networks from an invariance viewpoint. We introduce a new notion of capacity - the Fisher-Rao norm - that possesses desirable invariance properties and is motivated by Information Geometry. We discover an analytical characterization of the new capacity measure, through which we establish norm-comparison inequalities and further show that the new measure serves as an umbrella for several existing norm-based complexity measures. We discuss upper bounds on the generalization error induced by the proposed measure. Extensive numerical experiments on CIFAR-10 support our theoretical findings. Our theoretical analysis rests on a key structural lemma about partial derivatives of multi-layer rectifier networks. |
| first_indexed | 2024-09-23T13:20:15Z |
| format | Article |
| id | mit-1721.1/138296 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T13:20:15Z |
| publishDate | 2021 |
| record_format | dspace |
| spelling | mit-1721.1/1382962023-02-03T21:13:49Z Fisher-rao metric, geometry, and complexity of neural networks Liang, T Poggio, T Rakhlin, A Stokes, J Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences McGovern Institute for Brain Research at MIT Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Statistics and Data Science Center (Massachusetts Institute of Technology) Massachusetts Institute of Technology. Institute for Data, Systems, and Society Massachusetts Institute of Technology. Laboratory for Information and Decision Systems © 2019 by the author(s). We study the relationship between geometry and capacity measures for deep neural networks from an invariance viewpoint. We introduce a new notion of capacity - the Fisher-Rao norm - that possesses desirable invariance properties and is motivated by Information Geometry. We discover an analytical characterization of the new capacity measure, through which we establish norm-comparison inequalities and further show that the new measure serves as an umbrella for several existing norm-based complexity measures. We discuss upper bounds on the generalization error induced by the proposed measure. Extensive numerical experiments on CIFAR-10 support our theoretical findings. Our theoretical analysis rests on a key structural lemma about partial derivatives of multi-layer rectifier networks. 2021-12-02T20:14:53Z 2021-12-02T20:14:53Z 2020-01-01 2021-12-02T20:10:01Z Article http://purl.org/eprint/type/ConferencePaper https://hdl.handle.net/1721.1/138296 Liang, T, Poggio, T, Rakhlin, A and Stokes, J. 2020. "Fisher-rao metric, geometry, and complexity of neural networks." AISTATS 2019 - 22nd International Conference on Artificial Intelligence and Statistics, 89. en http://proceedings.mlr.press/v89/liang19a/liang19a.pdf AISTATS 2019 - 22nd International Conference on Artificial Intelligence and Statistics Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. application/pdf Proceedings of Machine Learning Research |
| spellingShingle | Liang, T Poggio, T Rakhlin, A Stokes, J Fisher-rao metric, geometry, and complexity of neural networks |
| title | Fisher-rao metric, geometry, and complexity of neural networks |
| title_full | Fisher-rao metric, geometry, and complexity of neural networks |
| title_fullStr | Fisher-rao metric, geometry, and complexity of neural networks |
| title_full_unstemmed | Fisher-rao metric, geometry, and complexity of neural networks |
| title_short | Fisher-rao metric, geometry, and complexity of neural networks |
| title_sort | fisher rao metric geometry and complexity of neural networks |
| url | https://hdl.handle.net/1721.1/138296 |
| work_keys_str_mv | AT liangt fisherraometricgeometryandcomplexityofneuralnetworks AT poggiot fisherraometricgeometryandcomplexityofneuralnetworks AT rakhlina fisherraometricgeometryandcomplexityofneuralnetworks AT stokesj fisherraometricgeometryandcomplexityofneuralnetworks |