Overparameterized neural networks implement associative memory
© 2020 National Academy of Sciences. All rights reserved. Identifying computational mechanisms for memorization and retrieval of data is a long-standing problem at the intersection of machine learning and neuroscience. Our main finding is that standard overparameterized deep neural networks trained...
| Main Authors: | , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Proceedings of the National Academy of Sciences
2021
|
| Online Access: | https://hdl.handle.net/1721.1/134539 |
| _version_ | 1826215581740695552 |
|---|---|
| author | Radhakrishnan, Adityanarayanan Belkin, Mikhail Uhler, Caroline |
| author2 | Massachusetts Institute of Technology. Laboratory for Information and Decision Systems |
| author_facet | Massachusetts Institute of Technology. Laboratory for Information and Decision Systems Radhakrishnan, Adityanarayanan Belkin, Mikhail Uhler, Caroline |
| author_sort | Radhakrishnan, Adityanarayanan |
| collection | MIT |
| description | © 2020 National Academy of Sciences. All rights reserved. Identifying computational mechanisms for memorization and retrieval of data is a long-standing problem at the intersection of machine learning and neuroscience. Our main finding is that standard overparameterized deep neural networks trained using standard optimization methods implement such a mechanism for real-valued data. We provide empirical evidence that 1) overparameterized autoencoders store training samples as attractors and thus iterating the learned map leads to sample recovery, and that 2) the same mechanism allows for encoding sequences of examples and serves as an even more efficient mechanism for memory than autoencoding. Theoretically, we prove that when trained on a single example, autoencoders store the example as an attractor. Lastly, by treating a sequence encoder as a composition of maps, we prove that sequence encoding provides a more efficient mechanism for memory than autoencoding. |
| first_indexed | 2024-09-23T16:36:00Z |
| format | Article |
| id | mit-1721.1/134539 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T16:36:00Z |
| publishDate | 2021 |
| publisher | Proceedings of the National Academy of Sciences |
| record_format | dspace |
| spelling | mit-1721.1/1345392023-01-10T19:37:18Z Overparameterized neural networks implement associative memory Radhakrishnan, Adityanarayanan Belkin, Mikhail Uhler, Caroline Massachusetts Institute of Technology. Laboratory for Information and Decision Systems Massachusetts Institute of Technology. Institute for Data, Systems, and Society © 2020 National Academy of Sciences. All rights reserved. Identifying computational mechanisms for memorization and retrieval of data is a long-standing problem at the intersection of machine learning and neuroscience. Our main finding is that standard overparameterized deep neural networks trained using standard optimization methods implement such a mechanism for real-valued data. We provide empirical evidence that 1) overparameterized autoencoders store training samples as attractors and thus iterating the learned map leads to sample recovery, and that 2) the same mechanism allows for encoding sequences of examples and serves as an even more efficient mechanism for memory than autoencoding. Theoretically, we prove that when trained on a single example, autoencoders store the example as an attractor. Lastly, by treating a sequence encoder as a composition of maps, we prove that sequence encoding provides a more efficient mechanism for memory than autoencoding. 2021-10-27T20:05:28Z 2021-10-27T20:05:28Z 2020 2021-03-19T15:25:38Z Article http://purl.org/eprint/type/JournalArticle https://hdl.handle.net/1721.1/134539 en 10.1073/PNAS.2005013117 Proceedings of the National Academy of Sciences of the United States of America Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. application/pdf Proceedings of the National Academy of Sciences PNAS |
| spellingShingle | Radhakrishnan, Adityanarayanan Belkin, Mikhail Uhler, Caroline Overparameterized neural networks implement associative memory |
| title | Overparameterized neural networks implement associative memory |
| title_full | Overparameterized neural networks implement associative memory |
| title_fullStr | Overparameterized neural networks implement associative memory |
| title_full_unstemmed | Overparameterized neural networks implement associative memory |
| title_short | Overparameterized neural networks implement associative memory |
| title_sort | overparameterized neural networks implement associative memory |
| url | https://hdl.handle.net/1721.1/134539 |
| work_keys_str_mv | AT radhakrishnanadityanarayanan overparameterizedneuralnetworksimplementassociativememory AT belkinmikhail overparameterizedneuralnetworksimplementassociativememory AT uhlercaroline overparameterizedneuralnetworksimplementassociativememory |