Backpropagation algorithms and Reservoir Computing in Recurrent Neural Networks for the forecasting of complex spatiotemporal dynamics
We examine the efficiency of Recurrent Neural Networks in forecasting the spatiotemporal dynamics of high dimensional and reduced order complex systems using Reservoir Computing (RC) and Backpropagation through time (BPTT) for gated network architectures. We highlight advantages and limitations of e...
| Main Authors: | , , , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Elsevier BV
2020
|
| Online Access: | https://hdl.handle.net/1721.1/126694 |
| _version_ | 1826207885609140224 |
|---|---|
| author | Vlachas, P.R. Pathak, J. Hunt, B.R. Sapsis, Themistoklis Panagiotis Girvan, M. Ott, E. Koumoutsakos, P. |
| author2 | Massachusetts Institute of Technology. Department of Mechanical Engineering |
| author_facet | Massachusetts Institute of Technology. Department of Mechanical Engineering Vlachas, P.R. Pathak, J. Hunt, B.R. Sapsis, Themistoklis Panagiotis Girvan, M. Ott, E. Koumoutsakos, P. |
| author_sort | Vlachas, P.R. |
| collection | MIT |
| description | We examine the efficiency of Recurrent Neural Networks in forecasting the spatiotemporal dynamics of high dimensional and reduced order complex systems using Reservoir Computing (RC) and Backpropagation through time (BPTT) for gated network architectures. We highlight advantages and limitations of each method and discuss their implementation for parallel computing architectures. We quantify the relative prediction accuracy of these algorithms for the long-term forecasting of chaotic systems using as benchmarks the Lorenz-96 and the Kuramoto–Sivashinsky (KS) equations. We find that, when the full state dynamics are available for training, RC outperforms BPTT approaches in terms of predictive performance and in capturing of the long-term statistics, while at the same time requiring much less training time. However, in the case of reduced order data, large scale RC models can be unstable and more likely than the BPTT algorithms to diverge. In contrast, RNNs trained via BPTT show superior forecasting abilities and capture well the dynamics of reduced order systems. Furthermore, the present study quantifies for the first time the Lyapunov Spectrum of the KS equation with BPTT, achieving similar accuracy as RC. This study establishes that RNNs are a potent computational framework for the learning and forecasting of complex spatiotemporal systems. |
| first_indexed | 2024-09-23T13:56:30Z |
| format | Article |
| id | mit-1721.1/126694 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T13:56:30Z |
| publishDate | 2020 |
| publisher | Elsevier BV |
| record_format | dspace |
| spelling | mit-1721.1/1266942022-10-01T18:08:20Z Backpropagation algorithms and Reservoir Computing in Recurrent Neural Networks for the forecasting of complex spatiotemporal dynamics Vlachas, P.R. Pathak, J. Hunt, B.R. Sapsis, Themistoklis Panagiotis Girvan, M. Ott, E. Koumoutsakos, P. Massachusetts Institute of Technology. Department of Mechanical Engineering We examine the efficiency of Recurrent Neural Networks in forecasting the spatiotemporal dynamics of high dimensional and reduced order complex systems using Reservoir Computing (RC) and Backpropagation through time (BPTT) for gated network architectures. We highlight advantages and limitations of each method and discuss their implementation for parallel computing architectures. We quantify the relative prediction accuracy of these algorithms for the long-term forecasting of chaotic systems using as benchmarks the Lorenz-96 and the Kuramoto–Sivashinsky (KS) equations. We find that, when the full state dynamics are available for training, RC outperforms BPTT approaches in terms of predictive performance and in capturing of the long-term statistics, while at the same time requiring much less training time. However, in the case of reduced order data, large scale RC models can be unstable and more likely than the BPTT algorithms to diverge. In contrast, RNNs trained via BPTT show superior forecasting abilities and capture well the dynamics of reduced order systems. Furthermore, the present study quantifies for the first time the Lyapunov Spectrum of the KS equation with BPTT, achieving similar accuracy as RC. This study establishes that RNNs are a potent computational framework for the learning and forecasting of complex spatiotemporal systems. Army Research Office (Grant W911NF-17-1-0306) 2020-08-20T01:25:43Z 2020-08-20T01:25:43Z 2020-06 2020-02 2020-08-04T17:30:16Z Article http://purl.org/eprint/type/JournalArticle 0893-6080 https://hdl.handle.net/1721.1/126694 Vlachas, P. R. et al. "Backpropagation algorithms and Reservoir Computing in Recurrent Neural Networks for the forecasting of complex spatiotemporal dynamics." Neural Networks 126 (June 2020): 191-217 © 2020 Elsevier Ltd en http://dx.doi.org/10.1016/j.neunet.2020.02.016 Neural Networks Creative Commons Attribution-NonCommercial-NoDerivs License http://creativecommons.org/licenses/by-nc-nd/4.0/ application/pdf Elsevier BV arXiv |
| spellingShingle | Vlachas, P.R. Pathak, J. Hunt, B.R. Sapsis, Themistoklis Panagiotis Girvan, M. Ott, E. Koumoutsakos, P. Backpropagation algorithms and Reservoir Computing in Recurrent Neural Networks for the forecasting of complex spatiotemporal dynamics |
| title | Backpropagation algorithms and Reservoir Computing in Recurrent Neural Networks for the forecasting of complex spatiotemporal dynamics |
| title_full | Backpropagation algorithms and Reservoir Computing in Recurrent Neural Networks for the forecasting of complex spatiotemporal dynamics |
| title_fullStr | Backpropagation algorithms and Reservoir Computing in Recurrent Neural Networks for the forecasting of complex spatiotemporal dynamics |
| title_full_unstemmed | Backpropagation algorithms and Reservoir Computing in Recurrent Neural Networks for the forecasting of complex spatiotemporal dynamics |
| title_short | Backpropagation algorithms and Reservoir Computing in Recurrent Neural Networks for the forecasting of complex spatiotemporal dynamics |
| title_sort | backpropagation algorithms and reservoir computing in recurrent neural networks for the forecasting of complex spatiotemporal dynamics |
| url | https://hdl.handle.net/1721.1/126694 |
| work_keys_str_mv | AT vlachaspr backpropagationalgorithmsandreservoircomputinginrecurrentneuralnetworksfortheforecastingofcomplexspatiotemporaldynamics AT pathakj backpropagationalgorithmsandreservoircomputinginrecurrentneuralnetworksfortheforecastingofcomplexspatiotemporaldynamics AT huntbr backpropagationalgorithmsandreservoircomputinginrecurrentneuralnetworksfortheforecastingofcomplexspatiotemporaldynamics AT sapsisthemistoklispanagiotis backpropagationalgorithmsandreservoircomputinginrecurrentneuralnetworksfortheforecastingofcomplexspatiotemporaldynamics AT girvanm backpropagationalgorithmsandreservoircomputinginrecurrentneuralnetworksfortheforecastingofcomplexspatiotemporaldynamics AT otte backpropagationalgorithmsandreservoircomputinginrecurrentneuralnetworksfortheforecastingofcomplexspatiotemporaldynamics AT koumoutsakosp backpropagationalgorithmsandreservoircomputinginrecurrentneuralnetworksfortheforecastingofcomplexspatiotemporaldynamics |