Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks
We develop biologically plausible training mechanisms for self-supervised learning (SSL) in deep networks. Specifically, by biologically plausible training we mean (i) all updates of weights are based on current activities of pre-synaptic units and current, or activity retrieved from short term memo...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2022-03-01
|
| Series: | Frontiers in Computational Neuroscience |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fncom.2022.789253/full |
| _version_ | 1818306298882555904 |
|---|---|
| author | Mufeng Tang Yibo Yang Yali Amit |
| author_facet | Mufeng Tang Yibo Yang Yali Amit |
| author_sort | Mufeng Tang |
| collection | DOAJ |
| description | We develop biologically plausible training mechanisms for self-supervised learning (SSL) in deep networks. Specifically, by biologically plausible training we mean (i) all updates of weights are based on current activities of pre-synaptic units and current, or activity retrieved from short term memory of post synaptic units, including at the top-most error computing layer, (ii) complex computations such as normalization, inner products and division are avoided, (iii) asymmetric connections between units, and (iv) most learning is carried out in an unsupervised manner. SSL with a contrastive loss satisfies the third condition as it does not require labeled data and it introduces robustness to observed perturbations of objects, which occur naturally as objects or observers move in 3D and with variable lighting over time. We propose a contrastive hinge based loss whose error involves simple local computations satisfying (ii), as opposed to the standard contrastive losses employed in the literature, which do not lend themselves easily to implementation in a network architecture due to complex computations involving ratios and inner products. Furthermore, we show that learning can be performed with one of two more plausible alternatives to backpropagation that satisfy conditions (i) and (ii). The first is difference target propagation (DTP), which trains network parameters using target-based local losses and employs a Hebbian learning rule, thus overcoming the biologically implausible symmetric weight problem in backpropagation. The second is layer-wise learning, where each layer is directly connected to a layer computing the loss error. The layers are either updated sequentially in a greedy fashion (GLL) or in random order (RLL), and each training stage involves a single hidden layer network. Backpropagation through one layer needed for each such network can either be altered with fixed random feedback weights (RF) or using updated random feedback weights (URF) as in Amity's study 2019. Both methods represent alternatives to the symmetric weight issue of backpropagation. By training convolutional neural networks (CNNs) with SSL and DTP, GLL or RLL, we find that our proposed framework achieves comparable performance to standard BP learning downstream linear classifier evaluation of the learned embeddings. |
| first_indexed | 2024-12-13T06:40:16Z |
| format | Article |
| id | doaj.art-bc7a7b5e37254c48919b25eab55c8f89 |
| institution | Directory Open Access Journal |
| issn | 1662-5188 |
| language | English |
| last_indexed | 2024-12-13T06:40:16Z |
| publishDate | 2022-03-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Computational Neuroscience |
| spelling | doaj.art-bc7a7b5e37254c48919b25eab55c8f892022-12-21T23:56:27ZengFrontiers Media S.A.Frontiers in Computational Neuroscience1662-51882022-03-011610.3389/fncom.2022.789253789253Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep NetworksMufeng TangYibo YangYali AmitWe develop biologically plausible training mechanisms for self-supervised learning (SSL) in deep networks. Specifically, by biologically plausible training we mean (i) all updates of weights are based on current activities of pre-synaptic units and current, or activity retrieved from short term memory of post synaptic units, including at the top-most error computing layer, (ii) complex computations such as normalization, inner products and division are avoided, (iii) asymmetric connections between units, and (iv) most learning is carried out in an unsupervised manner. SSL with a contrastive loss satisfies the third condition as it does not require labeled data and it introduces robustness to observed perturbations of objects, which occur naturally as objects or observers move in 3D and with variable lighting over time. We propose a contrastive hinge based loss whose error involves simple local computations satisfying (ii), as opposed to the standard contrastive losses employed in the literature, which do not lend themselves easily to implementation in a network architecture due to complex computations involving ratios and inner products. Furthermore, we show that learning can be performed with one of two more plausible alternatives to backpropagation that satisfy conditions (i) and (ii). The first is difference target propagation (DTP), which trains network parameters using target-based local losses and employs a Hebbian learning rule, thus overcoming the biologically implausible symmetric weight problem in backpropagation. The second is layer-wise learning, where each layer is directly connected to a layer computing the loss error. The layers are either updated sequentially in a greedy fashion (GLL) or in random order (RLL), and each training stage involves a single hidden layer network. Backpropagation through one layer needed for each such network can either be altered with fixed random feedback weights (RF) or using updated random feedback weights (URF) as in Amity's study 2019. Both methods represent alternatives to the symmetric weight issue of backpropagation. By training convolutional neural networks (CNNs) with SSL and DTP, GLL or RLL, we find that our proposed framework achieves comparable performance to standard BP learning downstream linear classifier evaluation of the learned embeddings.https://www.frontiersin.org/articles/10.3389/fncom.2022.789253/fulldifference target propagationlayerwise learninghinge lossback-propagation (BP)self-supervised learning |
| spellingShingle | Mufeng Tang Yibo Yang Yali Amit Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks Frontiers in Computational Neuroscience difference target propagation layerwise learning hinge loss back-propagation (BP) self-supervised learning |
| title | Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks |
| title_full | Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks |
| title_fullStr | Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks |
| title_full_unstemmed | Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks |
| title_short | Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks |
| title_sort | biologically plausible training mechanisms for self supervised learning in deep networks |
| topic | difference target propagation layerwise learning hinge loss back-propagation (BP) self-supervised learning |
| url | https://www.frontiersin.org/articles/10.3389/fncom.2022.789253/full |
| work_keys_str_mv | AT mufengtang biologicallyplausibletrainingmechanismsforselfsupervisedlearningindeepnetworks AT yiboyang biologicallyplausibletrainingmechanismsforselfsupervisedlearningindeepnetworks AT yaliamit biologicallyplausibletrainingmechanismsforselfsupervisedlearningindeepnetworks |