Unsupervised Adaptive Weight Pruning for Energy-Efficient Neuromorphic Systems
To tackle real-world challenges, deep and complex neural networks are generally used with a massive number of parameters, which require large memory size, extensive computational operations, and high energy consumption in neuromorphic hardware systems. In this work, we propose an unsupervised online...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2020-11-01
|
| Series: | Frontiers in Neuroscience |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fnins.2020.598876/full |
| _version_ | 1819005899902025728 |
|---|---|
| author | Wenzhe Guo Wenzhe Guo Mohammed E. Fouda Hasan Erdem Yantir Hasan Erdem Yantir Ahmed M. Eltawil Ahmed M. Eltawil Khaled Nabil Salama |
| author_facet | Wenzhe Guo Wenzhe Guo Mohammed E. Fouda Hasan Erdem Yantir Hasan Erdem Yantir Ahmed M. Eltawil Ahmed M. Eltawil Khaled Nabil Salama |
| author_sort | Wenzhe Guo |
| collection | DOAJ |
| description | To tackle real-world challenges, deep and complex neural networks are generally used with a massive number of parameters, which require large memory size, extensive computational operations, and high energy consumption in neuromorphic hardware systems. In this work, we propose an unsupervised online adaptive weight pruning method that dynamically removes non-critical weights from a spiking neural network (SNN) to reduce network complexity and improve energy efficiency. The adaptive pruning method explores neural dynamics and firing activity of SNNs and adapts the pruning threshold over time and neurons during training. The proposed adaptation scheme allows the network to effectively identify critical weights associated with each neuron by changing the pruning threshold dynamically over time and neurons. It balances the connection strength of neurons with the previous layer with adaptive thresholds and prevents weak neurons from failure after pruning. We also evaluated improvement in the energy efficiency of SNNs with our method by computing synaptic operations (SOPs). Simulation results and detailed analyses have revealed that applying adaptation in the pruning threshold can significantly improve network performance and reduce the number of SOPs. The pruned SNN with 800 excitatory neurons can achieve a 30% reduction in SOPs during training and a 55% reduction during inference, with only 0.44% accuracy loss on MNIST dataset. Compared with a previously reported online soft pruning method, the proposed adaptive pruning method shows 3.33% higher classification accuracy and 67% more reduction in SOPs. The effectiveness of our method was confirmed on different datasets and for different network sizes. Our evaluation showed that the implementation overhead of the adaptive method regarding speed, area, and energy is negligible in the network. Therefore, this work offers a promising solution for effective network compression and building highly energy-efficient neuromorphic systems in real-time applications. |
| first_indexed | 2024-12-21T00:00:08Z |
| format | Article |
| id | doaj.art-54f183d3cdf5476d9059db964612098c |
| institution | Directory Open Access Journal |
| issn | 1662-453X |
| language | English |
| last_indexed | 2024-12-21T00:00:08Z |
| publishDate | 2020-11-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Neuroscience |
| spelling | doaj.art-54f183d3cdf5476d9059db964612098c2022-12-21T19:22:38ZengFrontiers Media S.A.Frontiers in Neuroscience1662-453X2020-11-011410.3389/fnins.2020.598876598876Unsupervised Adaptive Weight Pruning for Energy-Efficient Neuromorphic SystemsWenzhe Guo0Wenzhe Guo1Mohammed E. Fouda2Hasan Erdem Yantir3Hasan Erdem Yantir4Ahmed M. Eltawil5Ahmed M. Eltawil6Khaled Nabil Salama7Sensors Lab, Advanced Membranes & Porous Materials Center, Computer, Electrical and Mathematical Sciences and Engineering Division, King Abdullah University of Science and Technology, Thuwal, Saudi ArabiaCommunication and Computing Systems Lab, Computer, Electrical and Mathematical Sciences and Engineering Division, King Abdullah University of Science and Technology, Thuwal, Saudi ArabiaDepartment of Electrical Engineering and Computer Science, University of California, Irvine, Irvine, CA, United StatesSensors Lab, Advanced Membranes & Porous Materials Center, Computer, Electrical and Mathematical Sciences and Engineering Division, King Abdullah University of Science and Technology, Thuwal, Saudi ArabiaCommunication and Computing Systems Lab, Computer, Electrical and Mathematical Sciences and Engineering Division, King Abdullah University of Science and Technology, Thuwal, Saudi ArabiaCommunication and Computing Systems Lab, Computer, Electrical and Mathematical Sciences and Engineering Division, King Abdullah University of Science and Technology, Thuwal, Saudi ArabiaDepartment of Electrical Engineering and Computer Science, University of California, Irvine, Irvine, CA, United StatesSensors Lab, Advanced Membranes & Porous Materials Center, Computer, Electrical and Mathematical Sciences and Engineering Division, King Abdullah University of Science and Technology, Thuwal, Saudi ArabiaTo tackle real-world challenges, deep and complex neural networks are generally used with a massive number of parameters, which require large memory size, extensive computational operations, and high energy consumption in neuromorphic hardware systems. In this work, we propose an unsupervised online adaptive weight pruning method that dynamically removes non-critical weights from a spiking neural network (SNN) to reduce network complexity and improve energy efficiency. The adaptive pruning method explores neural dynamics and firing activity of SNNs and adapts the pruning threshold over time and neurons during training. The proposed adaptation scheme allows the network to effectively identify critical weights associated with each neuron by changing the pruning threshold dynamically over time and neurons. It balances the connection strength of neurons with the previous layer with adaptive thresholds and prevents weak neurons from failure after pruning. We also evaluated improvement in the energy efficiency of SNNs with our method by computing synaptic operations (SOPs). Simulation results and detailed analyses have revealed that applying adaptation in the pruning threshold can significantly improve network performance and reduce the number of SOPs. The pruned SNN with 800 excitatory neurons can achieve a 30% reduction in SOPs during training and a 55% reduction during inference, with only 0.44% accuracy loss on MNIST dataset. Compared with a previously reported online soft pruning method, the proposed adaptive pruning method shows 3.33% higher classification accuracy and 67% more reduction in SOPs. The effectiveness of our method was confirmed on different datasets and for different network sizes. Our evaluation showed that the implementation overhead of the adaptive method regarding speed, area, and energy is negligible in the network. Therefore, this work offers a promising solution for effective network compression and building highly energy-efficient neuromorphic systems in real-time applications.https://www.frontiersin.org/articles/10.3389/fnins.2020.598876/fullneuromorphic computingspiking neural networkspruningunsupervised learningSTDPpattern recognition |
| spellingShingle | Wenzhe Guo Wenzhe Guo Mohammed E. Fouda Hasan Erdem Yantir Hasan Erdem Yantir Ahmed M. Eltawil Ahmed M. Eltawil Khaled Nabil Salama Unsupervised Adaptive Weight Pruning for Energy-Efficient Neuromorphic Systems Frontiers in Neuroscience neuromorphic computing spiking neural networks pruning unsupervised learning STDP pattern recognition |
| title | Unsupervised Adaptive Weight Pruning for Energy-Efficient Neuromorphic Systems |
| title_full | Unsupervised Adaptive Weight Pruning for Energy-Efficient Neuromorphic Systems |
| title_fullStr | Unsupervised Adaptive Weight Pruning for Energy-Efficient Neuromorphic Systems |
| title_full_unstemmed | Unsupervised Adaptive Weight Pruning for Energy-Efficient Neuromorphic Systems |
| title_short | Unsupervised Adaptive Weight Pruning for Energy-Efficient Neuromorphic Systems |
| title_sort | unsupervised adaptive weight pruning for energy efficient neuromorphic systems |
| topic | neuromorphic computing spiking neural networks pruning unsupervised learning STDP pattern recognition |
| url | https://www.frontiersin.org/articles/10.3389/fnins.2020.598876/full |
| work_keys_str_mv | AT wenzheguo unsupervisedadaptiveweightpruningforenergyefficientneuromorphicsystems AT wenzheguo unsupervisedadaptiveweightpruningforenergyefficientneuromorphicsystems AT mohammedefouda unsupervisedadaptiveweightpruningforenergyefficientneuromorphicsystems AT hasanerdemyantir unsupervisedadaptiveweightpruningforenergyefficientneuromorphicsystems AT hasanerdemyantir unsupervisedadaptiveweightpruningforenergyefficientneuromorphicsystems AT ahmedmeltawil unsupervisedadaptiveweightpruningforenergyefficientneuromorphicsystems AT ahmedmeltawil unsupervisedadaptiveweightpruningforenergyefficientneuromorphicsystems AT khalednabilsalama unsupervisedadaptiveweightpruningforenergyefficientneuromorphicsystems |