Rotational Unit of Memory: A Novel Representation Unit for RNNs with Scalable Applications
<jats:p> Stacking long short-term memory (LSTM) cells or gated recurrent units (GRUs) as part of a recurrent neural network (RNN) has become a standard approach to solving a number of tasks ranging from language modeling to text summarization. Although LSTMs and GRUs were designed to model lon...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MIT Press - Journals
2021
|
| Online Access: | https://hdl.handle.net/1721.1/132374 |
| _version_ | 1826207866173784064 |
|---|---|
| author | Dangovski, Rumen Jing, Li Nakov, Preslav Tatalović, Mićo Soljačić, Marin |
| author_facet | Dangovski, Rumen Jing, Li Nakov, Preslav Tatalović, Mićo Soljačić, Marin |
| author_sort | Dangovski, Rumen |
| collection | MIT |
| description | <jats:p> Stacking long short-term memory (LSTM) cells or gated recurrent units (GRUs) as part of a recurrent neural network (RNN) has become a standard approach to solving a number of tasks ranging from language modeling to text summarization. Although LSTMs and GRUs were designed to model long-range dependencies more accurately than conventional RNNs, they nevertheless have problems copying or recalling information from the long distant past. Here, we derive a phase-coded representation of the memory state, Rotational Unit of Memory (RUM), that unifies the concepts of unitary learning and associative memory. We show experimentally that RNNs based on RUMs can solve basic sequential tasks such as memory copying and memory recall much better than LSTMs/GRUs. We further demonstrate that by replacing LSTM/GRU with RUM units we can apply neural networks to real-world problems such as language modeling and text summarization, yielding results comparable to the state of the art. </jats:p> |
| first_indexed | 2024-09-23T13:56:11Z |
| format | Article |
| id | mit-1721.1/132374 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T13:56:11Z |
| publishDate | 2021 |
| publisher | MIT Press - Journals |
| record_format | dspace |
| spelling | mit-1721.1/1323742021-09-21T03:26:02Z Rotational Unit of Memory: A Novel Representation Unit for RNNs with Scalable Applications Dangovski, Rumen Jing, Li Nakov, Preslav Tatalović, Mićo Soljačić, Marin <jats:p> Stacking long short-term memory (LSTM) cells or gated recurrent units (GRUs) as part of a recurrent neural network (RNN) has become a standard approach to solving a number of tasks ranging from language modeling to text summarization. Although LSTMs and GRUs were designed to model long-range dependencies more accurately than conventional RNNs, they nevertheless have problems copying or recalling information from the long distant past. Here, we derive a phase-coded representation of the memory state, Rotational Unit of Memory (RUM), that unifies the concepts of unitary learning and associative memory. We show experimentally that RNNs based on RUMs can solve basic sequential tasks such as memory copying and memory recall much better than LSTMs/GRUs. We further demonstrate that by replacing LSTM/GRU with RUM units we can apply neural networks to real-world problems such as language modeling and text summarization, yielding results comparable to the state of the art. </jats:p> 2021-09-20T18:22:06Z 2021-09-20T18:22:06Z 2020-11-09T17:27:54Z Article http://purl.org/eprint/type/JournalArticle https://hdl.handle.net/1721.1/132374 en 10.1162/TACL_A_00258 Transactions of the Association for Computational Linguistics Creative Commons Attribution 4.0 International license https://creativecommons.org/licenses/by/4.0/ application/pdf MIT Press - Journals MIT Press |
| spellingShingle | Dangovski, Rumen Jing, Li Nakov, Preslav Tatalović, Mićo Soljačić, Marin Rotational Unit of Memory: A Novel Representation Unit for RNNs with Scalable Applications |
| title | Rotational Unit of Memory: A Novel Representation Unit for RNNs with Scalable Applications |
| title_full | Rotational Unit of Memory: A Novel Representation Unit for RNNs with Scalable Applications |
| title_fullStr | Rotational Unit of Memory: A Novel Representation Unit for RNNs with Scalable Applications |
| title_full_unstemmed | Rotational Unit of Memory: A Novel Representation Unit for RNNs with Scalable Applications |
| title_short | Rotational Unit of Memory: A Novel Representation Unit for RNNs with Scalable Applications |
| title_sort | rotational unit of memory a novel representation unit for rnns with scalable applications |
| url | https://hdl.handle.net/1721.1/132374 |
| work_keys_str_mv | AT dangovskirumen rotationalunitofmemoryanovelrepresentationunitforrnnswithscalableapplications AT jingli rotationalunitofmemoryanovelrepresentationunitforrnnswithscalableapplications AT nakovpreslav rotationalunitofmemoryanovelrepresentationunitforrnnswithscalableapplications AT tatalovicmico rotationalunitofmemoryanovelrepresentationunitforrnnswithscalableapplications AT soljacicmarin rotationalunitofmemoryanovelrepresentationunitforrnnswithscalableapplications |