A spiking neural network model of model-free reinforcement learning with high-dimensional sensory input and perceptual ambiguity.
A theoretical framework of reinforcement learning plays an important role in understanding action selection in animals. Spiking neural networks provide a theoretically grounded means to test computational hypotheses on neurally plausible algorithms of reinforcement learning through numerical simulat...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science (PLoS)
2015-01-01
|
| Series: | PLoS ONE |
| Online Access: | http://europepmc.org/articles/PMC4347982?pdf=render |
| _version_ | 1819068958875058176 |
|---|---|
| author | Takashi Nakano Makoto Otsuka Junichiro Yoshimoto Kenji Doya |
| author_facet | Takashi Nakano Makoto Otsuka Junichiro Yoshimoto Kenji Doya |
| author_sort | Takashi Nakano |
| collection | DOAJ |
| description | A theoretical framework of reinforcement learning plays an important role in understanding action selection in animals. Spiking neural networks provide a theoretically grounded means to test computational hypotheses on neurally plausible algorithms of reinforcement learning through numerical simulation. However, most of these models cannot handle observations which are noisy, or occurred in the past, even though these are inevitable and constraining features of learning in real environments. This class of problem is formally known as partially observable reinforcement learning (PORL) problems. It provides a generalization of reinforcement learning to partially observable domains. In addition, observations in the real world tend to be rich and high-dimensional. In this work, we use a spiking neural network model to approximate the free energy of a restricted Boltzmann machine and apply it to the solution of PORL problems with high-dimensional observations. Our spiking network model solves maze tasks with perceptually ambiguous high-dimensional observations without knowledge of the true environment. An extended model with working memory also solves history-dependent tasks. The way spiking neural networks handle PORL problems may provide a glimpse into the underlying laws of neural information processing which can only be discovered through such a top-down approach. |
| first_indexed | 2024-12-21T16:42:25Z |
| format | Article |
| id | doaj.art-7848836fc84643f28cc6bd323acb3877 |
| institution | Directory Open Access Journal |
| issn | 1932-6203 |
| language | English |
| last_indexed | 2024-12-21T16:42:25Z |
| publishDate | 2015-01-01 |
| publisher | Public Library of Science (PLoS) |
| record_format | Article |
| series | PLoS ONE |
| spelling | doaj.art-7848836fc84643f28cc6bd323acb38772022-12-21T18:57:05ZengPublic Library of Science (PLoS)PLoS ONE1932-62032015-01-01103e011562010.1371/journal.pone.0115620A spiking neural network model of model-free reinforcement learning with high-dimensional sensory input and perceptual ambiguity.Takashi NakanoMakoto OtsukaJunichiro YoshimotoKenji DoyaA theoretical framework of reinforcement learning plays an important role in understanding action selection in animals. Spiking neural networks provide a theoretically grounded means to test computational hypotheses on neurally plausible algorithms of reinforcement learning through numerical simulation. However, most of these models cannot handle observations which are noisy, or occurred in the past, even though these are inevitable and constraining features of learning in real environments. This class of problem is formally known as partially observable reinforcement learning (PORL) problems. It provides a generalization of reinforcement learning to partially observable domains. In addition, observations in the real world tend to be rich and high-dimensional. In this work, we use a spiking neural network model to approximate the free energy of a restricted Boltzmann machine and apply it to the solution of PORL problems with high-dimensional observations. Our spiking network model solves maze tasks with perceptually ambiguous high-dimensional observations without knowledge of the true environment. An extended model with working memory also solves history-dependent tasks. The way spiking neural networks handle PORL problems may provide a glimpse into the underlying laws of neural information processing which can only be discovered through such a top-down approach.http://europepmc.org/articles/PMC4347982?pdf=render |
| spellingShingle | Takashi Nakano Makoto Otsuka Junichiro Yoshimoto Kenji Doya A spiking neural network model of model-free reinforcement learning with high-dimensional sensory input and perceptual ambiguity. PLoS ONE |
| title | A spiking neural network model of model-free reinforcement learning with high-dimensional sensory input and perceptual ambiguity. |
| title_full | A spiking neural network model of model-free reinforcement learning with high-dimensional sensory input and perceptual ambiguity. |
| title_fullStr | A spiking neural network model of model-free reinforcement learning with high-dimensional sensory input and perceptual ambiguity. |
| title_full_unstemmed | A spiking neural network model of model-free reinforcement learning with high-dimensional sensory input and perceptual ambiguity. |
| title_short | A spiking neural network model of model-free reinforcement learning with high-dimensional sensory input and perceptual ambiguity. |
| title_sort | spiking neural network model of model free reinforcement learning with high dimensional sensory input and perceptual ambiguity |
| url | http://europepmc.org/articles/PMC4347982?pdf=render |
| work_keys_str_mv | AT takashinakano aspikingneuralnetworkmodelofmodelfreereinforcementlearningwithhighdimensionalsensoryinputandperceptualambiguity AT makotootsuka aspikingneuralnetworkmodelofmodelfreereinforcementlearningwithhighdimensionalsensoryinputandperceptualambiguity AT junichiroyoshimoto aspikingneuralnetworkmodelofmodelfreereinforcementlearningwithhighdimensionalsensoryinputandperceptualambiguity AT kenjidoya aspikingneuralnetworkmodelofmodelfreereinforcementlearningwithhighdimensionalsensoryinputandperceptualambiguity AT takashinakano spikingneuralnetworkmodelofmodelfreereinforcementlearningwithhighdimensionalsensoryinputandperceptualambiguity AT makotootsuka spikingneuralnetworkmodelofmodelfreereinforcementlearningwithhighdimensionalsensoryinputandperceptualambiguity AT junichiroyoshimoto spikingneuralnetworkmodelofmodelfreereinforcementlearningwithhighdimensionalsensoryinputandperceptualambiguity AT kenjidoya spikingneuralnetworkmodelofmodelfreereinforcementlearningwithhighdimensionalsensoryinputandperceptualambiguity |