Difference between memory and prediction in linear recurrent networks
Recurrent networks are trained to memorize their input better, often in the hopes that such training will increase the ability of the network to predict. We show that networks designed to memorize input can be arbitrarily bad at prediction. We also find, for several types of inputs, that one-node ne...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
American Physical Society
2018
|
| Online Access: | http://hdl.handle.net/1721.1/114553 |
| _version_ | 1826202327014440960 |
|---|---|
| author | Marzen, Sarah E. |
| author2 | Massachusetts Institute of Technology. Department of Physics |
| author_facet | Massachusetts Institute of Technology. Department of Physics Marzen, Sarah E. |
| author_sort | Marzen, Sarah E. |
| collection | MIT |
| description | Recurrent networks are trained to memorize their input better, often in the hopes that such training will increase the ability of the network to predict. We show that networks designed to memorize input can be arbitrarily bad at prediction. We also find, for several types of inputs, that one-node networks optimized for prediction are nearly at upper bounds on predictive capacity given by Wiener filters and are roughly equivalent in performance to randomly generated five-node networks. Our results suggest that maximizing memory capacity leads to very different networks than maximizing predictive capacity and that optimizing recurrent weights can decrease reservoir size by half an order of magnitude. |
| first_indexed | 2024-09-23T12:05:45Z |
| format | Article |
| id | mit-1721.1/114553 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T12:05:45Z |
| publishDate | 2018 |
| publisher | American Physical Society |
| record_format | dspace |
| spelling | mit-1721.1/1145532022-10-01T08:07:27Z Difference between memory and prediction in linear recurrent networks Marzen, Sarah E. Massachusetts Institute of Technology. Department of Physics Marzen, Sarah E. Recurrent networks are trained to memorize their input better, often in the hopes that such training will increase the ability of the network to predict. We show that networks designed to memorize input can be arbitrarily bad at prediction. We also find, for several types of inputs, that one-node networks optimized for prediction are nearly at upper bounds on predictive capacity given by Wiener filters and are roughly equivalent in performance to randomly generated five-node networks. Our results suggest that maximizing memory capacity leads to very different networks than maximizing predictive capacity and that optimizing recurrent weights can decrease reservoir size by half an order of magnitude. 2018-04-04T20:54:46Z 2018-04-04T20:54:46Z 2017-09 2017-08 2017-11-14T22:46:09Z Article http://purl.org/eprint/type/JournalArticle 2470-0045 2470-0053 http://hdl.handle.net/1721.1/114553 Marzen, Sarah et al. "Difference between memory and prediction in linear recurrent networks." Physical Review E 96, 3 (September 2017): 032308 © 2017 American Physical Society en http://dx.doi.org/10.1103/PhysRevE.96.032308 Physical Review E Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. American Physical Society application/pdf American Physical Society American Physical Society |
| spellingShingle | Marzen, Sarah E. Difference between memory and prediction in linear recurrent networks |
| title | Difference between memory and prediction in linear recurrent networks |
| title_full | Difference between memory and prediction in linear recurrent networks |
| title_fullStr | Difference between memory and prediction in linear recurrent networks |
| title_full_unstemmed | Difference between memory and prediction in linear recurrent networks |
| title_short | Difference between memory and prediction in linear recurrent networks |
| title_sort | difference between memory and prediction in linear recurrent networks |
| url | http://hdl.handle.net/1721.1/114553 |
| work_keys_str_mv | AT marzensarahe differencebetweenmemoryandpredictioninlinearrecurrentnetworks |