Shifting responsibly: the importance of striatal modularity to reinforcement learning in uncertain environments
We propose here that the modular organization of the striatum reflects a context-sensitive modular learning architecture in which clustered striosome–matrisome domains participate in modular reinforcement learning (RL). Based on anatomical and physiological evidence, it has been suggested that the m...
| Main Authors: | , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | en_US |
| Published: |
Frontiers Media S.A.
2011
|
| Online Access: | http://hdl.handle.net/1721.1/66562 https://orcid.org/0000-0002-4326-7720 |
| _version_ | 1826213555647545344 |
|---|---|
| author | Amemori, Ken-ichi Gibb, Leif G. Graybiel, Ann M. |
| author2 | Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences |
| author_facet | Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences Amemori, Ken-ichi Gibb, Leif G. Graybiel, Ann M. |
| author_sort | Amemori, Ken-ichi |
| collection | MIT |
| description | We propose here that the modular organization of the striatum reflects a context-sensitive modular learning architecture in which clustered striosome–matrisome domains participate in modular reinforcement learning (RL). Based on anatomical and physiological evidence, it has been suggested that the modular organization of the striatum could represent a learning architecture. There is not, however, a coherent view of how such a learning architecture could relate to the organization of striatal outputs into the direct and indirect pathways of the basal ganglia, nor a clear formulation of how such a modular architecture relates to the RL functions attributed to the striatum. Here, we hypothesize that striosome–matrisome modules not only learn to bias behavior toward specific actions, as in standard RL, but also learn to assess their own relevance to the environmental context and modulate their own learning and activity on this basis. We further hypothesize that the contextual relevance or “responsibility” of modules is determined by errors in predictions of environmental features and that such responsibility is assigned by striosomes and conveyed to matrisomes via local circuit interneurons. To examine these hypotheses and to identify the general requirements for realizing this architecture in the nervous system, we developed a simple modular RL model. We then constructed a network model of basal ganglia circuitry that includes these modules and the direct and indirect pathways. Based on simple assumptions, this model suggests that while the direct pathway may promote actions based on striatal action values, the indirect pathway may act as a gating network that facilitates or suppresses behavioral modules on the basis of striatal responsibility signals. Our modeling functionally unites the modular compartmental organization of the striatum with the direct–indirect pathway divisions of the basal ganglia, a step that we suggest will have important clinical implications. |
| first_indexed | 2024-09-23T15:51:06Z |
| format | Article |
| id | mit-1721.1/66562 |
| institution | Massachusetts Institute of Technology |
| language | en_US |
| last_indexed | 2024-09-23T15:51:06Z |
| publishDate | 2011 |
| publisher | Frontiers Media S.A. |
| record_format | dspace |
| spelling | mit-1721.1/665622022-10-02T04:34:54Z Shifting responsibly: the importance of striatal modularity to reinforcement learning in uncertain environments Amemori, Ken-ichi Gibb, Leif G. Graybiel, Ann M. Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences McGovern Institute for Brain Research at MIT Graybiel, Ann M. Graybiel, Ann M. Amemori, Ken-ichi Gibb, Leif G. We propose here that the modular organization of the striatum reflects a context-sensitive modular learning architecture in which clustered striosome–matrisome domains participate in modular reinforcement learning (RL). Based on anatomical and physiological evidence, it has been suggested that the modular organization of the striatum could represent a learning architecture. There is not, however, a coherent view of how such a learning architecture could relate to the organization of striatal outputs into the direct and indirect pathways of the basal ganglia, nor a clear formulation of how such a modular architecture relates to the RL functions attributed to the striatum. Here, we hypothesize that striosome–matrisome modules not only learn to bias behavior toward specific actions, as in standard RL, but also learn to assess their own relevance to the environmental context and modulate their own learning and activity on this basis. We further hypothesize that the contextual relevance or “responsibility” of modules is determined by errors in predictions of environmental features and that such responsibility is assigned by striosomes and conveyed to matrisomes via local circuit interneurons. To examine these hypotheses and to identify the general requirements for realizing this architecture in the nervous system, we developed a simple modular RL model. We then constructed a network model of basal ganglia circuitry that includes these modules and the direct and indirect pathways. Based on simple assumptions, this model suggests that while the direct pathway may promote actions based on striatal action values, the indirect pathway may act as a gating network that facilitates or suppresses behavioral modules on the basis of striatal responsibility signals. Our modeling functionally unites the modular compartmental organization of the striatum with the direct–indirect pathway divisions of the basal ganglia, a step that we suggest will have important clinical implications. National Institutes of Health (U.S.) (NIH R01 MH060379) National Institutes of Health (U.S.) (NIH R01 NS025529) United States. Office of Naval Research (grant ONR N00014-07-1-0903) European Union (Grant 201716) National Institutes of Health (U.S.) (NIH Grant 5 R01 MH079076) 2011-10-24T20:33:45Z 2011-10-24T20:33:45Z 2011-05 2010-12 Article http://purl.org/eprint/type/JournalArticle 1662-5161 http://hdl.handle.net/1721.1/66562 Amemori, Ken-ichi, Leif G. Gibb, and Ann M. Graybiel. “Shifting Responsibly: The Importance of Striatal Modularity to Reinforcement Learning in Uncertain Environments.” Frontiers in Human Neuroscience 5, Article 47 (2011). https://orcid.org/0000-0002-4326-7720 en_US http://dx.doi.org/10.3389/fnhum.2011.00047 Frontiers in Human Neuroscience Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. application/pdf Frontiers Media S.A. Frontiers |
| spellingShingle | Amemori, Ken-ichi Gibb, Leif G. Graybiel, Ann M. Shifting responsibly: the importance of striatal modularity to reinforcement learning in uncertain environments |
| title | Shifting responsibly: the importance of striatal modularity to reinforcement learning in uncertain environments |
| title_full | Shifting responsibly: the importance of striatal modularity to reinforcement learning in uncertain environments |
| title_fullStr | Shifting responsibly: the importance of striatal modularity to reinforcement learning in uncertain environments |
| title_full_unstemmed | Shifting responsibly: the importance of striatal modularity to reinforcement learning in uncertain environments |
| title_short | Shifting responsibly: the importance of striatal modularity to reinforcement learning in uncertain environments |
| title_sort | shifting responsibly the importance of striatal modularity to reinforcement learning in uncertain environments |
| url | http://hdl.handle.net/1721.1/66562 https://orcid.org/0000-0002-4326-7720 |
| work_keys_str_mv | AT amemorikenichi shiftingresponsiblytheimportanceofstriatalmodularitytoreinforcementlearninginuncertainenvironments AT gibbleifg shiftingresponsiblytheimportanceofstriatalmodularitytoreinforcementlearninginuncertainenvironments AT graybielannm shiftingresponsiblytheimportanceofstriatalmodularitytoreinforcementlearninginuncertainenvironments |