Learning improvement in spiking neural networks: enhancement techniques, design and performance analysis
Inspired by the architecture of biological brains, artificial neural networks (ANNs) have achieved widespread success in tasks such as pattern recognition, data analysis, and classification. Among the many neural network models developed over the years, the spiking neural network (SNN), which was in...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis-Doctor of Philosophy |
| Language: | English |
| Published: |
Nanyang Technological University
2024
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/179879 |
| _version_ | 1811677138277892096 |
|---|---|
| author | Wang, Siqi |
| author2 | Cheng Tee Hiang |
| author_facet | Cheng Tee Hiang Wang, Siqi |
| author_sort | Wang, Siqi |
| collection | NTU |
| description | Inspired by the architecture of biological brains, artificial neural networks (ANNs) have achieved widespread success in tasks such as pattern recognition, data analysis, and classification. Among the many neural network models developed over the years, the spiking neural network (SNN), which was introduced in 1996, has shown great promise in the current trend towards compact embedded AI. By incorporating both spatial and temporal information in the network construction process, many inherent shortcomings of traditional ANNs can be overcome. In SNN, the most basic signal carrier element is the spike, bringing about a revolution in neural network weights updating compared to traditional methods that are widely applied in ANNs. SNNs have shown substantial promise in processing spatio-temporal data, mimicking biological neuronal mechanisms, and reducing computational power usage. In this thesis, a comprehensive discourse of this emerging technique and several advanced performance enhancement methods are involved. Specifically, a systematic survey of SNN-related techniques, methods include time-involved batch normalization, learnable thresholding, moderate dropout, membrane potential distribution adjustment, and parametric surrogate gradient and their corresponding effects are investigated.
Firstly, the SNN training methods, correlation between SNN and ANN, SNN optimization methods, major shortcomings of current SNN models and neuromorphic hardware-related issues are comprehensively reviewed. A pre-activation normalization method, named time-involved batch normalization, is adapted from the corresponding ANN by additionally considering temporal information to redistribute presynaptic inputs, enabling the construction of deep SNNs. By examining the membrane potential distribution after time-involved batch normalization, it is demonstrated that the SNN can be effectively constructed with much deeper architecture, leading to improved model performance.
Secondly, the heterogeneity of spiking neurons across different layers is explored. This inquiry is motivated by the diverse neuronal mechanisms observed in different regions of the brain within neuroscience. To address this, a new spiking neuronal mechanism called learnable thresholding (LT) is proposed, which endows spiking neurons with the ability to update their thresholds along with synaptic connections during network convergence. This allows neurons in different layers to demonstrate diverse neuronal responses under external stimulation, and enables flexible neuronal mechanisms across layers, proper information flow within the network, and fast network convergence. In addition, since the highly deviatory sub-models generated by the dropout may inhibit the constructed SNN’s performance, moderate dropout (MD) is designed to measure and minimize the inconsistence in the output probability distributions of different network runs. With extensive experiments and evaluations, LT and MD have been proven easily migratable to any backpropagation through time (BPTT) based SNN, regardless of the network architecture used. Ablation study result shows that the SNN with LTMD implemented can achieve superior network performance with an affordable additional computational load.
Lastly, the effect of different surrogate gradients (SG) on the trained SNN’s performance is investigated, which reflects the importance of proper SG selection in SNN construction especially when BPTT training scheme is utilized. To avoid choosing SG expressions intuitively and find out the most suitable SG for the specific task, parametric surrogate gradient (PSG) is proposed to parameterize candidate SG and make the gradient parameter learnable. Therefore, SG can approach its optimal expression form by iteratively updating the gradient parameters. Due to the binary output states, SNNs always encounter quantization error, which further results in unpredictable membrane potential distributions. Such potential shift is evaluated and a method “membrane potential distribution adjustment” (PDA) is proposed to minimize the undesired pre-activations. Experimental results indicate that the proposed methods can be readily integrated with BPTT algorithm and help modulated SNNs to achieve state-of-the-art performance on both static and neuromorphic datasets with fewer timesteps. Moreover, average firing rates of spiking neurons are consistently reduced after embedding the proposed methods due to more proper neuronal update triggered by the optimal surrogate gradients. Besides, the methods exhibit high robustness and remain operable under diverse neuromorphic hardware-related noise conditions.
This doctoral thesis provides conceptual illustration and empirical analysis of all the aforementioned techniques, presenting a new pathway to build SNN with astounding high performance. The proposed methods have proven to be successful in bridging the performance gap between conventional ANNs and current SNNs, revealing the hidden potential of SNNs. The remarkable success of these methods provides impetus for further exploration of the capabilities of SNNs. |
| first_indexed | 2024-10-01T02:32:36Z |
| format | Thesis-Doctor of Philosophy |
| id | ntu-10356/179879 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T02:32:36Z |
| publishDate | 2024 |
| publisher | Nanyang Technological University |
| record_format | dspace |
| spelling | ntu-10356/1798792024-09-04T07:56:36Z Learning improvement in spiking neural networks: enhancement techniques, design and performance analysis Wang, Siqi Cheng Tee Hiang School of Electrical and Electronic Engineering ETHCHENG@ntu.edu.sg Computer and Information Science Inspired by the architecture of biological brains, artificial neural networks (ANNs) have achieved widespread success in tasks such as pattern recognition, data analysis, and classification. Among the many neural network models developed over the years, the spiking neural network (SNN), which was introduced in 1996, has shown great promise in the current trend towards compact embedded AI. By incorporating both spatial and temporal information in the network construction process, many inherent shortcomings of traditional ANNs can be overcome. In SNN, the most basic signal carrier element is the spike, bringing about a revolution in neural network weights updating compared to traditional methods that are widely applied in ANNs. SNNs have shown substantial promise in processing spatio-temporal data, mimicking biological neuronal mechanisms, and reducing computational power usage. In this thesis, a comprehensive discourse of this emerging technique and several advanced performance enhancement methods are involved. Specifically, a systematic survey of SNN-related techniques, methods include time-involved batch normalization, learnable thresholding, moderate dropout, membrane potential distribution adjustment, and parametric surrogate gradient and their corresponding effects are investigated. Firstly, the SNN training methods, correlation between SNN and ANN, SNN optimization methods, major shortcomings of current SNN models and neuromorphic hardware-related issues are comprehensively reviewed. A pre-activation normalization method, named time-involved batch normalization, is adapted from the corresponding ANN by additionally considering temporal information to redistribute presynaptic inputs, enabling the construction of deep SNNs. By examining the membrane potential distribution after time-involved batch normalization, it is demonstrated that the SNN can be effectively constructed with much deeper architecture, leading to improved model performance. Secondly, the heterogeneity of spiking neurons across different layers is explored. This inquiry is motivated by the diverse neuronal mechanisms observed in different regions of the brain within neuroscience. To address this, a new spiking neuronal mechanism called learnable thresholding (LT) is proposed, which endows spiking neurons with the ability to update their thresholds along with synaptic connections during network convergence. This allows neurons in different layers to demonstrate diverse neuronal responses under external stimulation, and enables flexible neuronal mechanisms across layers, proper information flow within the network, and fast network convergence. In addition, since the highly deviatory sub-models generated by the dropout may inhibit the constructed SNN’s performance, moderate dropout (MD) is designed to measure and minimize the inconsistence in the output probability distributions of different network runs. With extensive experiments and evaluations, LT and MD have been proven easily migratable to any backpropagation through time (BPTT) based SNN, regardless of the network architecture used. Ablation study result shows that the SNN with LTMD implemented can achieve superior network performance with an affordable additional computational load. Lastly, the effect of different surrogate gradients (SG) on the trained SNN’s performance is investigated, which reflects the importance of proper SG selection in SNN construction especially when BPTT training scheme is utilized. To avoid choosing SG expressions intuitively and find out the most suitable SG for the specific task, parametric surrogate gradient (PSG) is proposed to parameterize candidate SG and make the gradient parameter learnable. Therefore, SG can approach its optimal expression form by iteratively updating the gradient parameters. Due to the binary output states, SNNs always encounter quantization error, which further results in unpredictable membrane potential distributions. Such potential shift is evaluated and a method “membrane potential distribution adjustment” (PDA) is proposed to minimize the undesired pre-activations. Experimental results indicate that the proposed methods can be readily integrated with BPTT algorithm and help modulated SNNs to achieve state-of-the-art performance on both static and neuromorphic datasets with fewer timesteps. Moreover, average firing rates of spiking neurons are consistently reduced after embedding the proposed methods due to more proper neuronal update triggered by the optimal surrogate gradients. Besides, the methods exhibit high robustness and remain operable under diverse neuromorphic hardware-related noise conditions. This doctoral thesis provides conceptual illustration and empirical analysis of all the aforementioned techniques, presenting a new pathway to build SNN with astounding high performance. The proposed methods have proven to be successful in bridging the performance gap between conventional ANNs and current SNNs, revealing the hidden potential of SNNs. The remarkable success of these methods provides impetus for further exploration of the capabilities of SNNs. Doctor of Philosophy 2024-08-29T01:52:27Z 2024-08-29T01:52:27Z 2024 Thesis-Doctor of Philosophy Wang, S. (2024). Learning improvement in spiking neural networks: enhancement techniques, design and performance analysis. Doctoral thesis, Nanyang Technological University, Singapore. https://hdl.handle.net/10356/179879 https://hdl.handle.net/10356/179879 10.32657/10356/179879 en This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0). application/pdf Nanyang Technological University |
| spellingShingle | Computer and Information Science Wang, Siqi Learning improvement in spiking neural networks: enhancement techniques, design and performance analysis |
| title | Learning improvement in spiking neural networks: enhancement techniques, design and performance analysis |
| title_full | Learning improvement in spiking neural networks: enhancement techniques, design and performance analysis |
| title_fullStr | Learning improvement in spiking neural networks: enhancement techniques, design and performance analysis |
| title_full_unstemmed | Learning improvement in spiking neural networks: enhancement techniques, design and performance analysis |
| title_short | Learning improvement in spiking neural networks: enhancement techniques, design and performance analysis |
| title_sort | learning improvement in spiking neural networks enhancement techniques design and performance analysis |
| topic | Computer and Information Science |
| url | https://hdl.handle.net/10356/179879 |
| work_keys_str_mv | AT wangsiqi learningimprovementinspikingneuralnetworksenhancementtechniquesdesignandperformanceanalysis |