Non-equilibrium physics: from spin glasses to machine and neural learning
Disordered many-body systems exhibit a wide range of emergent phenomena across different scales. These complex behaviors can be utilized for various information processing tasks such as error correction, learning, and optimization. Despite the empirical success of utilizing these systems for intelli...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Published: |
Massachusetts Institute of Technology
2023
|
| Online Access: | https://hdl.handle.net/1721.1/152568 |
| _version_ | 1826210980750688256 |
|---|---|
| author | Zhong, Weishun |
| author2 | Sompolinsky, Haim |
| author_facet | Sompolinsky, Haim Zhong, Weishun |
| author_sort | Zhong, Weishun |
| collection | MIT |
| description | Disordered many-body systems exhibit a wide range of emergent phenomena across different scales. These complex behaviors can be utilized for various information processing tasks such as error correction, learning, and optimization. Despite the empirical success of utilizing these systems for intelligent tasks, the underlying principles that govern their emergent intelligent behaviors remain largely unknown. In this thesis, we aim to characterize such emergent intelligence in disordered systems through statistical physics. We chart a roadmap for our efforts in this thesis based on two axes: learning mechanisms (long-term memory vs. working memory) and learning dynamics (artificial vs. natural). We begin our exploration from the long-term memory and artificial dynamics continent of this atlas, where we examine the structure-function relationships in feedforward neural networks, the prototypical example of neural learning. Using replica theory, information theory, and optimal transport, we study the computational consequences of imposing connectivity constraints on the network, such as distribution constraints, sign constraints, and disentangling constraints. We evaluate the performances based on metrics such as capacity, generalization, and generative ability. Next, we explore the working memory and artificial dynamics corner of the atlas and investigate the non-equilibrium driven dynamics of recurrent neural networks under external inputs. Then, we move to the working memory and natural dynamics island and study the ability of driven spin-glasses to perform discriminative tasks such as novelty detection and classification. Finally, we conclude our exploration at the long-term memory and natural dynamics kingdom and investigate the generative modeling ability in many-body localized systems. Throughout our journey, we uncover relationships between learning mechanisms and physical dynamics that could serve as guiding principles for designing intelligent systems. We hope that our investigation into the emergent intelligence of seemingly disparate learning systems can expand our current understanding of intelligence beyond neural systems and uncover a wider range of computational substrates suitable for AI applications. |
| first_indexed | 2024-09-23T14:58:30Z |
| format | Thesis |
| id | mit-1721.1/152568 |
| institution | Massachusetts Institute of Technology |
| last_indexed | 2024-09-23T14:58:30Z |
| publishDate | 2023 |
| publisher | Massachusetts Institute of Technology |
| record_format | dspace |
| spelling | mit-1721.1/1525682023-11-01T03:17:09Z Non-equilibrium physics: from spin glasses to machine and neural learning Zhong, Weishun Sompolinsky, Haim Kardar, Mehran Massachusetts Institute of Technology. Department of Physics Disordered many-body systems exhibit a wide range of emergent phenomena across different scales. These complex behaviors can be utilized for various information processing tasks such as error correction, learning, and optimization. Despite the empirical success of utilizing these systems for intelligent tasks, the underlying principles that govern their emergent intelligent behaviors remain largely unknown. In this thesis, we aim to characterize such emergent intelligence in disordered systems through statistical physics. We chart a roadmap for our efforts in this thesis based on two axes: learning mechanisms (long-term memory vs. working memory) and learning dynamics (artificial vs. natural). We begin our exploration from the long-term memory and artificial dynamics continent of this atlas, where we examine the structure-function relationships in feedforward neural networks, the prototypical example of neural learning. Using replica theory, information theory, and optimal transport, we study the computational consequences of imposing connectivity constraints on the network, such as distribution constraints, sign constraints, and disentangling constraints. We evaluate the performances based on metrics such as capacity, generalization, and generative ability. Next, we explore the working memory and artificial dynamics corner of the atlas and investigate the non-equilibrium driven dynamics of recurrent neural networks under external inputs. Then, we move to the working memory and natural dynamics island and study the ability of driven spin-glasses to perform discriminative tasks such as novelty detection and classification. Finally, we conclude our exploration at the long-term memory and natural dynamics kingdom and investigate the generative modeling ability in many-body localized systems. Throughout our journey, we uncover relationships between learning mechanisms and physical dynamics that could serve as guiding principles for designing intelligent systems. We hope that our investigation into the emergent intelligence of seemingly disparate learning systems can expand our current understanding of intelligence beyond neural systems and uncover a wider range of computational substrates suitable for AI applications. Ph.D. 2023-10-30T20:03:33Z 2023-10-30T20:03:33Z 2023-06 2023-10-25T18:00:43.872Z Thesis https://hdl.handle.net/1721.1/152568 In Copyright - Educational Use Permitted Copyright retained by author(s) https://rightsstatements.org/page/InC-EDU/1.0/ application/pdf Massachusetts Institute of Technology |
| spellingShingle | Zhong, Weishun Non-equilibrium physics: from spin glasses to machine and neural learning |
| title | Non-equilibrium physics: from spin glasses to machine and neural learning |
| title_full | Non-equilibrium physics: from spin glasses to machine and neural learning |
| title_fullStr | Non-equilibrium physics: from spin glasses to machine and neural learning |
| title_full_unstemmed | Non-equilibrium physics: from spin glasses to machine and neural learning |
| title_short | Non-equilibrium physics: from spin glasses to machine and neural learning |
| title_sort | non equilibrium physics from spin glasses to machine and neural learning |
| url | https://hdl.handle.net/1721.1/152568 |
| work_keys_str_mv | AT zhongweishun nonequilibriumphysicsfromspinglassestomachineandneurallearning |