A mathematical theory of semantic development in deep neural networks
An extensive body of empirical research has revealed remarkable regularities in the acquisition, organization, deployment, and neural representation of human semantic knowledge, thereby raising a fundamental conceptual question: What are the theoretical principles governing the ability of neural net...
| Հիմնական հեղինակներ: | , , |
|---|---|
| Ձևաչափ: | Journal article |
| Լեզու: | English |
| Հրապարակվել է: |
National Academy of Sciences
2019
|
| _version_ | 1826276619279400960 |
|---|---|
| author | Saxe, A McClelland, J Ganguli, S |
| author_facet | Saxe, A McClelland, J Ganguli, S |
| author_sort | Saxe, A |
| collection | OXFORD |
| description | An extensive body of empirical research has revealed remarkable regularities in the acquisition, organization, deployment, and neural representation of human semantic knowledge, thereby raising a fundamental conceptual question: What are the theoretical principles governing the ability of neural networks to acquire, organize, and deploy abstract knowledge by integrating across many individual experiences? We address this question by mathematically analyzing the nonlinear dynamics of learning in deep linear networks. We find exact solutions to this learning dynamics that yield a conceptual explanation for the prevalence of many disparate phenomena in semantic cognition, including the hierarchical differentiation of concepts through rapid developmental transitions, the ubiquity of semantic illusions between such transitions, the emergence of item typicality and category coherence as factors controlling the speed of semantic processing, changing patterns of inductive projection over development, and the conservation of semantic similarity in neural representations across species. Thus, surprisingly, our simple neural model qualitatively recapitulates many diverse regularities underlying semantic development, while providing analytic insight into how the statistical structure of an environment can interact with nonlinear deep-learning dynamics to give rise to these regularities. |
| first_indexed | 2024-03-06T23:16:40Z |
| format | Journal article |
| id | oxford-uuid:675edddf-32b3-4830-b781-6b47e02a3a8c |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-03-06T23:16:40Z |
| publishDate | 2019 |
| publisher | National Academy of Sciences |
| record_format | dspace |
| spelling | oxford-uuid:675edddf-32b3-4830-b781-6b47e02a3a8c2022-03-26T18:37:49ZA mathematical theory of semantic development in deep neural networksJournal articlehttp://purl.org/coar/resource_type/c_dcae04bcuuid:675edddf-32b3-4830-b781-6b47e02a3a8cEnglishSymplectic Elements at OxfordNational Academy of Sciences2019Saxe, AMcClelland, JGanguli, SAn extensive body of empirical research has revealed remarkable regularities in the acquisition, organization, deployment, and neural representation of human semantic knowledge, thereby raising a fundamental conceptual question: What are the theoretical principles governing the ability of neural networks to acquire, organize, and deploy abstract knowledge by integrating across many individual experiences? We address this question by mathematically analyzing the nonlinear dynamics of learning in deep linear networks. We find exact solutions to this learning dynamics that yield a conceptual explanation for the prevalence of many disparate phenomena in semantic cognition, including the hierarchical differentiation of concepts through rapid developmental transitions, the ubiquity of semantic illusions between such transitions, the emergence of item typicality and category coherence as factors controlling the speed of semantic processing, changing patterns of inductive projection over development, and the conservation of semantic similarity in neural representations across species. Thus, surprisingly, our simple neural model qualitatively recapitulates many diverse regularities underlying semantic development, while providing analytic insight into how the statistical structure of an environment can interact with nonlinear deep-learning dynamics to give rise to these regularities. |
| spellingShingle | Saxe, A McClelland, J Ganguli, S A mathematical theory of semantic development in deep neural networks |
| title | A mathematical theory of semantic development in deep neural networks |
| title_full | A mathematical theory of semantic development in deep neural networks |
| title_fullStr | A mathematical theory of semantic development in deep neural networks |
| title_full_unstemmed | A mathematical theory of semantic development in deep neural networks |
| title_short | A mathematical theory of semantic development in deep neural networks |
| title_sort | mathematical theory of semantic development in deep neural networks |
| work_keys_str_mv | AT saxea amathematicaltheoryofsemanticdevelopmentindeepneuralnetworks AT mcclellandj amathematicaltheoryofsemanticdevelopmentindeepneuralnetworks AT gangulis amathematicaltheoryofsemanticdevelopmentindeepneuralnetworks AT saxea mathematicaltheoryofsemanticdevelopmentindeepneuralnetworks AT mcclellandj mathematicaltheoryofsemanticdevelopmentindeepneuralnetworks AT gangulis mathematicaltheoryofsemanticdevelopmentindeepneuralnetworks |