Embodied Synaptic Plasticity With Online Reinforcement Learning
The endeavor to understand the brain involves multiple collaborating research fields. Classically, synaptic plasticity rules derived by theoretical neuroscientists are evaluated in isolation on pattern classification tasks. This contrasts with the biological brain which purpose is to control a body...
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2019-10-01
|
| Series: | Frontiers in Neurorobotics |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/article/10.3389/fnbot.2019.00081/full |
| _version_ | 1818580801097302016 |
|---|---|
| author | Jacques Kaiser Michael Hoff Michael Hoff Andreas Konle J. Camilo Vasquez Tieck David Kappel David Kappel David Kappel Daniel Reichard Anand Subramoney Robert Legenstein Arne Roennau Wolfgang Maass Rüdiger Dillmann |
| author_facet | Jacques Kaiser Michael Hoff Michael Hoff Andreas Konle J. Camilo Vasquez Tieck David Kappel David Kappel David Kappel Daniel Reichard Anand Subramoney Robert Legenstein Arne Roennau Wolfgang Maass Rüdiger Dillmann |
| author_sort | Jacques Kaiser |
| collection | DOAJ |
| description | The endeavor to understand the brain involves multiple collaborating research fields. Classically, synaptic plasticity rules derived by theoretical neuroscientists are evaluated in isolation on pattern classification tasks. This contrasts with the biological brain which purpose is to control a body in closed-loop. This paper contributes to bringing the fields of computational neuroscience and robotics closer together by integrating open-source software components from these two fields. The resulting framework allows to evaluate the validity of biologically-plausibe plasticity models in closed-loop robotics environments. We demonstrate this framework to evaluate Synaptic Plasticity with Online REinforcement learning (SPORE), a reward-learning rule based on synaptic sampling, on two visuomotor tasks: reaching and lane following. We show that SPORE is capable of learning to perform policies within the course of simulated hours for both tasks. Provisional parameter explorations indicate that the learning rate and the temperature driving the stochastic processes that govern synaptic learning dynamics need to be regulated for performance improvements to be retained. We conclude by discussing the recent deep reinforcement learning techniques which would be beneficial to increase the functionality of SPORE on visuomotor tasks. |
| first_indexed | 2024-12-16T07:23:22Z |
| format | Article |
| id | doaj.art-061bbe4b3482419db555731a5454eee4 |
| institution | Directory Open Access Journal |
| issn | 1662-5218 |
| language | English |
| last_indexed | 2024-12-16T07:23:22Z |
| publishDate | 2019-10-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Neurorobotics |
| spelling | doaj.art-061bbe4b3482419db555731a5454eee42022-12-21T22:39:34ZengFrontiers Media S.A.Frontiers in Neurorobotics1662-52182019-10-011310.3389/fnbot.2019.00081452229Embodied Synaptic Plasticity With Online Reinforcement LearningJacques Kaiser0Michael Hoff1Michael Hoff2Andreas Konle3J. Camilo Vasquez Tieck4David Kappel5David Kappel6David Kappel7Daniel Reichard8Anand Subramoney9Robert Legenstein10Arne Roennau11Wolfgang Maass12Rüdiger Dillmann13FZI Research Center for Information Technology, Karlsruhe, GermanyFZI Research Center for Information Technology, Karlsruhe, GermanyInstitute for Theoretical Computer Science, Graz University of Technology, Graz, AustriaFZI Research Center for Information Technology, Karlsruhe, GermanyFZI Research Center for Information Technology, Karlsruhe, GermanyInstitute for Theoretical Computer Science, Graz University of Technology, Graz, AustriaBernstein Center for Computational Neuroscience, III Physikalisches Institut-Biophysik, Georg-August Universität, Göttingen, GermanyTechnische Universität Dresden, Chair of Highly Parallel VLSI Systems and Neuromorphic Circuits, Dresden, GermanyFZI Research Center for Information Technology, Karlsruhe, GermanyInstitute for Theoretical Computer Science, Graz University of Technology, Graz, AustriaInstitute for Theoretical Computer Science, Graz University of Technology, Graz, AustriaFZI Research Center for Information Technology, Karlsruhe, GermanyInstitute for Theoretical Computer Science, Graz University of Technology, Graz, AustriaFZI Research Center for Information Technology, Karlsruhe, GermanyThe endeavor to understand the brain involves multiple collaborating research fields. Classically, synaptic plasticity rules derived by theoretical neuroscientists are evaluated in isolation on pattern classification tasks. This contrasts with the biological brain which purpose is to control a body in closed-loop. This paper contributes to bringing the fields of computational neuroscience and robotics closer together by integrating open-source software components from these two fields. The resulting framework allows to evaluate the validity of biologically-plausibe plasticity models in closed-loop robotics environments. We demonstrate this framework to evaluate Synaptic Plasticity with Online REinforcement learning (SPORE), a reward-learning rule based on synaptic sampling, on two visuomotor tasks: reaching and lane following. We show that SPORE is capable of learning to perform policies within the course of simulated hours for both tasks. Provisional parameter explorations indicate that the learning rate and the temperature driving the stochastic processes that govern synaptic learning dynamics need to be regulated for performance improvements to be retained. We conclude by discussing the recent deep reinforcement learning techniques which would be beneficial to increase the functionality of SPORE on visuomotor tasks.https://www.frontiersin.org/article/10.3389/fnbot.2019.00081/fullneuroroboticssynaptic plasticityspiking neural networksneuromorphic visionreinforcement learning |
| spellingShingle | Jacques Kaiser Michael Hoff Michael Hoff Andreas Konle J. Camilo Vasquez Tieck David Kappel David Kappel David Kappel Daniel Reichard Anand Subramoney Robert Legenstein Arne Roennau Wolfgang Maass Rüdiger Dillmann Embodied Synaptic Plasticity With Online Reinforcement Learning Frontiers in Neurorobotics neurorobotics synaptic plasticity spiking neural networks neuromorphic vision reinforcement learning |
| title | Embodied Synaptic Plasticity With Online Reinforcement Learning |
| title_full | Embodied Synaptic Plasticity With Online Reinforcement Learning |
| title_fullStr | Embodied Synaptic Plasticity With Online Reinforcement Learning |
| title_full_unstemmed | Embodied Synaptic Plasticity With Online Reinforcement Learning |
| title_short | Embodied Synaptic Plasticity With Online Reinforcement Learning |
| title_sort | embodied synaptic plasticity with online reinforcement learning |
| topic | neurorobotics synaptic plasticity spiking neural networks neuromorphic vision reinforcement learning |
| url | https://www.frontiersin.org/article/10.3389/fnbot.2019.00081/full |
| work_keys_str_mv | AT jacqueskaiser embodiedsynapticplasticitywithonlinereinforcementlearning AT michaelhoff embodiedsynapticplasticitywithonlinereinforcementlearning AT michaelhoff embodiedsynapticplasticitywithonlinereinforcementlearning AT andreaskonle embodiedsynapticplasticitywithonlinereinforcementlearning AT jcamilovasqueztieck embodiedsynapticplasticitywithonlinereinforcementlearning AT davidkappel embodiedsynapticplasticitywithonlinereinforcementlearning AT davidkappel embodiedsynapticplasticitywithonlinereinforcementlearning AT davidkappel embodiedsynapticplasticitywithonlinereinforcementlearning AT danielreichard embodiedsynapticplasticitywithonlinereinforcementlearning AT anandsubramoney embodiedsynapticplasticitywithonlinereinforcementlearning AT robertlegenstein embodiedsynapticplasticitywithonlinereinforcementlearning AT arneroennau embodiedsynapticplasticitywithonlinereinforcementlearning AT wolfgangmaass embodiedsynapticplasticitywithonlinereinforcementlearning AT rudigerdillmann embodiedsynapticplasticitywithonlinereinforcementlearning |