The impact of sparsity in low-rank recurrent neural networks.
Neural population dynamics are often highly coordinated, allowing task-related computations to be understood as neural trajectories through low-dimensional subspaces. How the network connectivity and input structure give rise to such activity can be investigated with the aid of low-rank recurrent ne...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science (PLoS)
2022-08-01
|
| Series: | PLoS Computational Biology |
| Online Access: | https://doi.org/10.1371/journal.pcbi.1010426 |
| _version_ | 1811282985864921088 |
|---|---|
| author | Elizabeth Herbert Srdjan Ostojic |
| author_facet | Elizabeth Herbert Srdjan Ostojic |
| author_sort | Elizabeth Herbert |
| collection | DOAJ |
| description | Neural population dynamics are often highly coordinated, allowing task-related computations to be understood as neural trajectories through low-dimensional subspaces. How the network connectivity and input structure give rise to such activity can be investigated with the aid of low-rank recurrent neural networks, a recently-developed class of computational models which offer a rich theoretical framework linking the underlying connectivity structure to emergent low-dimensional dynamics. This framework has so far relied on the assumption of all-to-all connectivity, yet cortical networks are known to be highly sparse. Here we investigate the dynamics of low-rank recurrent networks in which the connections are randomly sparsified, which makes the network connectivity formally full-rank. We first analyse the impact of sparsity on the eigenvalue spectrum of low-rank connectivity matrices, and use this to examine the implications for the dynamics. We find that in the presence of sparsity, the eigenspectra in the complex plane consist of a continuous bulk and isolated outliers, a form analogous to the eigenspectra of connectivity matrices composed of a low-rank and a full-rank random component. This analogy allows us to characterise distinct dynamical regimes of the sparsified low-rank network as a function of key network parameters. Altogether, we find that the low-dimensional dynamics induced by low-rank connectivity structure are preserved even at high levels of sparsity, and can therefore support rich and robust computations even in networks sparsified to a biologically-realistic extent. |
| first_indexed | 2024-04-13T02:02:21Z |
| format | Article |
| id | doaj.art-164ccb67d3194cf7a8c188e78ef6ff75 |
| institution | Directory Open Access Journal |
| issn | 1553-734X 1553-7358 |
| language | English |
| last_indexed | 2024-04-13T02:02:21Z |
| publishDate | 2022-08-01 |
| publisher | Public Library of Science (PLoS) |
| record_format | Article |
| series | PLoS Computational Biology |
| spelling | doaj.art-164ccb67d3194cf7a8c188e78ef6ff752022-12-22T03:07:34ZengPublic Library of Science (PLoS)PLoS Computational Biology1553-734X1553-73582022-08-01188e101042610.1371/journal.pcbi.1010426The impact of sparsity in low-rank recurrent neural networks.Elizabeth HerbertSrdjan OstojicNeural population dynamics are often highly coordinated, allowing task-related computations to be understood as neural trajectories through low-dimensional subspaces. How the network connectivity and input structure give rise to such activity can be investigated with the aid of low-rank recurrent neural networks, a recently-developed class of computational models which offer a rich theoretical framework linking the underlying connectivity structure to emergent low-dimensional dynamics. This framework has so far relied on the assumption of all-to-all connectivity, yet cortical networks are known to be highly sparse. Here we investigate the dynamics of low-rank recurrent networks in which the connections are randomly sparsified, which makes the network connectivity formally full-rank. We first analyse the impact of sparsity on the eigenvalue spectrum of low-rank connectivity matrices, and use this to examine the implications for the dynamics. We find that in the presence of sparsity, the eigenspectra in the complex plane consist of a continuous bulk and isolated outliers, a form analogous to the eigenspectra of connectivity matrices composed of a low-rank and a full-rank random component. This analogy allows us to characterise distinct dynamical regimes of the sparsified low-rank network as a function of key network parameters. Altogether, we find that the low-dimensional dynamics induced by low-rank connectivity structure are preserved even at high levels of sparsity, and can therefore support rich and robust computations even in networks sparsified to a biologically-realistic extent.https://doi.org/10.1371/journal.pcbi.1010426 |
| spellingShingle | Elizabeth Herbert Srdjan Ostojic The impact of sparsity in low-rank recurrent neural networks. PLoS Computational Biology |
| title | The impact of sparsity in low-rank recurrent neural networks. |
| title_full | The impact of sparsity in low-rank recurrent neural networks. |
| title_fullStr | The impact of sparsity in low-rank recurrent neural networks. |
| title_full_unstemmed | The impact of sparsity in low-rank recurrent neural networks. |
| title_short | The impact of sparsity in low-rank recurrent neural networks. |
| title_sort | impact of sparsity in low rank recurrent neural networks |
| url | https://doi.org/10.1371/journal.pcbi.1010426 |
| work_keys_str_mv | AT elizabethherbert theimpactofsparsityinlowrankrecurrentneuralnetworks AT srdjanostojic theimpactofsparsityinlowrankrecurrentneuralnetworks AT elizabethherbert impactofsparsityinlowrankrecurrentneuralnetworks AT srdjanostojic impactofsparsityinlowrankrecurrentneuralnetworks |