Learning hierarchically-structured concepts
We use a recently developed synchronous Spiking Neural Network (SNN) model to study the problem of learning hierarchically-structured concepts. We introduce an abstract data model that describes simple hierarchical concepts. We define a feed-forward layered SNN model, with learning modeled using Oja...
| Main Authors: | , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Elsevier BV
2022
|
| Online Access: | https://hdl.handle.net/1721.1/144297 |
| _version_ | 1826217486136115200 |
|---|---|
| author | Lynch, Nancy Mallmann-Trenn, Frederik |
| author2 | Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science |
| author_facet | Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Lynch, Nancy Mallmann-Trenn, Frederik |
| author_sort | Lynch, Nancy |
| collection | MIT |
| description | We use a recently developed synchronous Spiking Neural Network (SNN) model to study the problem of learning hierarchically-structured concepts. We introduce an abstract data model that describes simple hierarchical concepts. We define a feed-forward layered SNN model, with learning modeled using Oja's local learning rule, a well known biologically-plausible rule for adjusting synapse weights. We define what it means for such a network to recognize hierarchical concepts; our notion of recognition is robust, in that it tolerates a bounded amount of noise. Then, we present a learning algorithm by which a layered network may learn to recognize hierarchical concepts according to our robust definition. We analyze correctness and performance rigorously; the amount of time required to learn each concept, after learning all of the sub-concepts, is approximately O1ηkℓmaxlog(k)+1ɛ+blog(k), where k is the number of sub-concepts per concept, ℓmax is the maximum hierarchical depth, η is the learning rate, ɛ describes the amount of uncertainty allowed in robust recognition, and b describes the amount of weight decrease for "irrelevant" edges. An interesting feature of this algorithm is that it allows the network to learn sub-concepts in a highly interleaved manner. This algorithm assumes that the concepts are presented in a noise-free way; we also extend these results to accommodate noise in the learning process. Finally, we give a simple lower bound saying that, in order to recognize concepts with hierarchical depth two with noise-tolerance, a neural network should have at least two layers. The results in this paper represent first steps in the theoretical study of hierarchical concepts using SNNs. The cases studied here are basic, but they suggest many directions for extensions to more elaborate and realistic cases. |
| first_indexed | 2024-09-23T17:04:25Z |
| format | Article |
| id | mit-1721.1/144297 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T17:04:25Z |
| publishDate | 2022 |
| publisher | Elsevier BV |
| record_format | dspace |
| spelling | mit-1721.1/1442972023-11-01T04:17:06Z Learning hierarchically-structured concepts Lynch, Nancy Mallmann-Trenn, Frederik Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory We use a recently developed synchronous Spiking Neural Network (SNN) model to study the problem of learning hierarchically-structured concepts. We introduce an abstract data model that describes simple hierarchical concepts. We define a feed-forward layered SNN model, with learning modeled using Oja's local learning rule, a well known biologically-plausible rule for adjusting synapse weights. We define what it means for such a network to recognize hierarchical concepts; our notion of recognition is robust, in that it tolerates a bounded amount of noise. Then, we present a learning algorithm by which a layered network may learn to recognize hierarchical concepts according to our robust definition. We analyze correctness and performance rigorously; the amount of time required to learn each concept, after learning all of the sub-concepts, is approximately O1ηkℓmaxlog(k)+1ɛ+blog(k), where k is the number of sub-concepts per concept, ℓmax is the maximum hierarchical depth, η is the learning rate, ɛ describes the amount of uncertainty allowed in robust recognition, and b describes the amount of weight decrease for "irrelevant" edges. An interesting feature of this algorithm is that it allows the network to learn sub-concepts in a highly interleaved manner. This algorithm assumes that the concepts are presented in a noise-free way; we also extend these results to accommodate noise in the learning process. Finally, we give a simple lower bound saying that, in order to recognize concepts with hierarchical depth two with noise-tolerance, a neural network should have at least two layers. The results in this paper represent first steps in the theoretical study of hierarchical concepts using SNNs. The cases studied here are basic, but they suggest many directions for extensions to more elaborate and realistic cases. 2022-08-10T15:21:09Z 2022-08-10T15:21:09Z 2021 2022-08-10T15:17:39Z Article http://purl.org/eprint/type/JournalArticle https://hdl.handle.net/1721.1/144297 Lynch, Nancy and Mallmann-Trenn, Frederik. 2021. "Learning hierarchically-structured concepts." Neural Networks, 143. en 10.1016/J.NEUNET.2021.07.033 Neural Networks Creative Commons Attribution-NonCommercial-NoDerivs License http://creativecommons.org/licenses/by-nc-nd/4.0/ application/pdf Elsevier BV Elsevier |
| spellingShingle | Lynch, Nancy Mallmann-Trenn, Frederik Learning hierarchically-structured concepts |
| title | Learning hierarchically-structured concepts |
| title_full | Learning hierarchically-structured concepts |
| title_fullStr | Learning hierarchically-structured concepts |
| title_full_unstemmed | Learning hierarchically-structured concepts |
| title_short | Learning hierarchically-structured concepts |
| title_sort | learning hierarchically structured concepts |
| url | https://hdl.handle.net/1721.1/144297 |
| work_keys_str_mv | AT lynchnancy learninghierarchicallystructuredconcepts AT mallmanntrennfrederik learninghierarchicallystructuredconcepts |