Solving the spike feature information vanishing problem in spiking deep Q network with potential based normalization
Brain-inspired spiking neural networks (SNNs) are successfully applied to many pattern recognition domains. The SNNs-based deep structure has achieved considerable results in perceptual tasks, such as image classification and target detection. However, applying deep SNNs in reinforcement learning (R...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2022-08-01
|
| Series: | Frontiers in Neuroscience |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fnins.2022.953368/full |
| _version_ | 1817998245900582912 |
|---|---|
| author | Yinqian Sun Yinqian Sun Yi Zeng Yi Zeng Yi Zeng Yi Zeng Yi Zeng Yang Li Yang Li |
| author_facet | Yinqian Sun Yinqian Sun Yi Zeng Yi Zeng Yi Zeng Yi Zeng Yi Zeng Yang Li Yang Li |
| author_sort | Yinqian Sun |
| collection | DOAJ |
| description | Brain-inspired spiking neural networks (SNNs) are successfully applied to many pattern recognition domains. The SNNs-based deep structure has achieved considerable results in perceptual tasks, such as image classification and target detection. However, applying deep SNNs in reinforcement learning (RL) tasks is still a problem to be explored. Although there have been previous studies on the combination of SNNs and RL, most focus on robotic control problems with shallow networks or using the ANN-SNN conversion method to implement spiking deep Q networks (SDQN). In this study, we mathematically analyzed the problem of the disappearance of spiking signal features in SDQN and proposed a potential-based layer normalization (pbLN) method to train spiking deep Q networks directly. Experiment shows that compared with state-of-art ANN-SNN conversion method and other SDQN works, the proposed pbLN spiking deep Q networks (PL-SDQN) achieved better performance on Atari game tasks. |
| first_indexed | 2024-04-14T02:50:05Z |
| format | Article |
| id | doaj.art-6494b8ad7fed4e29ba7f3f8ead41c88d |
| institution | Directory Open Access Journal |
| issn | 1662-453X |
| language | English |
| last_indexed | 2024-04-14T02:50:05Z |
| publishDate | 2022-08-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Neuroscience |
| spelling | doaj.art-6494b8ad7fed4e29ba7f3f8ead41c88d2022-12-22T02:16:18ZengFrontiers Media S.A.Frontiers in Neuroscience1662-453X2022-08-011610.3389/fnins.2022.953368953368Solving the spike feature information vanishing problem in spiking deep Q network with potential based normalizationYinqian Sun0Yinqian Sun1Yi Zeng2Yi Zeng3Yi Zeng4Yi Zeng5Yi Zeng6Yang Li7Yang Li8Research Center for Brain-Inspired Intelligence, Institute of Automation, Chinese Academy of Sciences, Beijing, ChinaSchool of Future Technology, University of Chinese Academy of Sciences, Beijing, ChinaResearch Center for Brain-Inspired Intelligence, Institute of Automation, Chinese Academy of Sciences, Beijing, ChinaSchool of Future Technology, University of Chinese Academy of Sciences, Beijing, ChinaSchool of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, ChinaNational Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, ChinaCenter for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Shanghai, ChinaResearch Center for Brain-Inspired Intelligence, Institute of Automation, Chinese Academy of Sciences, Beijing, ChinaSchool of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, ChinaBrain-inspired spiking neural networks (SNNs) are successfully applied to many pattern recognition domains. The SNNs-based deep structure has achieved considerable results in perceptual tasks, such as image classification and target detection. However, applying deep SNNs in reinforcement learning (RL) tasks is still a problem to be explored. Although there have been previous studies on the combination of SNNs and RL, most focus on robotic control problems with shallow networks or using the ANN-SNN conversion method to implement spiking deep Q networks (SDQN). In this study, we mathematically analyzed the problem of the disappearance of spiking signal features in SDQN and proposed a potential-based layer normalization (pbLN) method to train spiking deep Q networks directly. Experiment shows that compared with state-of-art ANN-SNN conversion method and other SDQN works, the proposed pbLN spiking deep Q networks (PL-SDQN) achieved better performance on Atari game tasks.https://www.frontiersin.org/articles/10.3389/fnins.2022.953368/fullbrain-inspired decision modelSDQNreinforcement learningpotential normalizationspiking activity |
| spellingShingle | Yinqian Sun Yinqian Sun Yi Zeng Yi Zeng Yi Zeng Yi Zeng Yi Zeng Yang Li Yang Li Solving the spike feature information vanishing problem in spiking deep Q network with potential based normalization Frontiers in Neuroscience brain-inspired decision model SDQN reinforcement learning potential normalization spiking activity |
| title | Solving the spike feature information vanishing problem in spiking deep Q network with potential based normalization |
| title_full | Solving the spike feature information vanishing problem in spiking deep Q network with potential based normalization |
| title_fullStr | Solving the spike feature information vanishing problem in spiking deep Q network with potential based normalization |
| title_full_unstemmed | Solving the spike feature information vanishing problem in spiking deep Q network with potential based normalization |
| title_short | Solving the spike feature information vanishing problem in spiking deep Q network with potential based normalization |
| title_sort | solving the spike feature information vanishing problem in spiking deep q network with potential based normalization |
| topic | brain-inspired decision model SDQN reinforcement learning potential normalization spiking activity |
| url | https://www.frontiersin.org/articles/10.3389/fnins.2022.953368/full |
| work_keys_str_mv | AT yinqiansun solvingthespikefeatureinformationvanishingprobleminspikingdeepqnetworkwithpotentialbasednormalization AT yinqiansun solvingthespikefeatureinformationvanishingprobleminspikingdeepqnetworkwithpotentialbasednormalization AT yizeng solvingthespikefeatureinformationvanishingprobleminspikingdeepqnetworkwithpotentialbasednormalization AT yizeng solvingthespikefeatureinformationvanishingprobleminspikingdeepqnetworkwithpotentialbasednormalization AT yizeng solvingthespikefeatureinformationvanishingprobleminspikingdeepqnetworkwithpotentialbasednormalization AT yizeng solvingthespikefeatureinformationvanishingprobleminspikingdeepqnetworkwithpotentialbasednormalization AT yizeng solvingthespikefeatureinformationvanishingprobleminspikingdeepqnetworkwithpotentialbasednormalization AT yangli solvingthespikefeatureinformationvanishingprobleminspikingdeepqnetworkwithpotentialbasednormalization AT yangli solvingthespikefeatureinformationvanishingprobleminspikingdeepqnetworkwithpotentialbasednormalization |