A cholinergic feedback circuit to regulate striatal population uncertainty and optimize reinforcement learning
Convergent evidence suggests that the basal ganglia support reinforcement learning by adjusting action values according to reward prediction errors. However, adaptive behavior in stochastic environments requires the consideration of uncertainty to dynamically adjust the learning rate. We consider ho...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
eLife Sciences Publications Ltd
2015-12-01
|
| Series: | eLife |
| Subjects: | |
| Online Access: | https://elifesciences.org/articles/12029 |
| _version_ | 1811236817856364544 |
|---|---|
| author | Nicholas T Franklin Michael J Frank |
| author_facet | Nicholas T Franklin Michael J Frank |
| author_sort | Nicholas T Franklin |
| collection | DOAJ |
| description | Convergent evidence suggests that the basal ganglia support reinforcement learning by adjusting action values according to reward prediction errors. However, adaptive behavior in stochastic environments requires the consideration of uncertainty to dynamically adjust the learning rate. We consider how cholinergic tonically active interneurons (TANs) may endow the striatum with such a mechanism in computational models spanning three Marr's levels of analysis. In the neural model, TANs modulate the excitability of spiny neurons, their population response to reinforcement, and hence the effective learning rate. Long TAN pauses facilitated robustness to spurious outcomes by increasing divergence in synaptic weights between neurons coding for alternative action values, whereas short TAN pauses facilitated stochastic behavior but increased responsiveness to change-points in outcome contingencies. A feedback control system allowed TAN pauses to be dynamically modulated by uncertainty across the spiny neuron population, allowing the system to self-tune and optimize performance across stochastic environments. |
| first_indexed | 2024-04-12T12:15:51Z |
| format | Article |
| id | doaj.art-7e47009f5c4f418790f435dba9c82188 |
| institution | Directory Open Access Journal |
| issn | 2050-084X |
| language | English |
| last_indexed | 2024-04-12T12:15:51Z |
| publishDate | 2015-12-01 |
| publisher | eLife Sciences Publications Ltd |
| record_format | Article |
| series | eLife |
| spelling | doaj.art-7e47009f5c4f418790f435dba9c821882022-12-22T03:33:26ZengeLife Sciences Publications LtdeLife2050-084X2015-12-01410.7554/eLife.12029A cholinergic feedback circuit to regulate striatal population uncertainty and optimize reinforcement learningNicholas T Franklin0Michael J Frank1https://orcid.org/0000-0001-8451-0523Department of Cognitive, Linguistic and Psychological Sciences, Brown Institute for Brain Science, Brown University, Providence, United StatesDepartment of Cognitive, Linguistic and Psychological Sciences, Brown Institute for Brain Science, Brown University, Providence, United StatesConvergent evidence suggests that the basal ganglia support reinforcement learning by adjusting action values according to reward prediction errors. However, adaptive behavior in stochastic environments requires the consideration of uncertainty to dynamically adjust the learning rate. We consider how cholinergic tonically active interneurons (TANs) may endow the striatum with such a mechanism in computational models spanning three Marr's levels of analysis. In the neural model, TANs modulate the excitability of spiny neurons, their population response to reinforcement, and hence the effective learning rate. Long TAN pauses facilitated robustness to spurious outcomes by increasing divergence in synaptic weights between neurons coding for alternative action values, whereas short TAN pauses facilitated stochastic behavior but increased responsiveness to change-points in outcome contingencies. A feedback control system allowed TAN pauses to be dynamically modulated by uncertainty across the spiny neuron population, allowing the system to self-tune and optimize performance across stochastic environments.https://elifesciences.org/articles/12029reinforcement learningbasal gangliastriatumcholinergic interneuron |
| spellingShingle | Nicholas T Franklin Michael J Frank A cholinergic feedback circuit to regulate striatal population uncertainty and optimize reinforcement learning eLife reinforcement learning basal ganglia striatum cholinergic interneuron |
| title | A cholinergic feedback circuit to regulate striatal population uncertainty and optimize reinforcement learning |
| title_full | A cholinergic feedback circuit to regulate striatal population uncertainty and optimize reinforcement learning |
| title_fullStr | A cholinergic feedback circuit to regulate striatal population uncertainty and optimize reinforcement learning |
| title_full_unstemmed | A cholinergic feedback circuit to regulate striatal population uncertainty and optimize reinforcement learning |
| title_short | A cholinergic feedback circuit to regulate striatal population uncertainty and optimize reinforcement learning |
| title_sort | cholinergic feedback circuit to regulate striatal population uncertainty and optimize reinforcement learning |
| topic | reinforcement learning basal ganglia striatum cholinergic interneuron |
| url | https://elifesciences.org/articles/12029 |
| work_keys_str_mv | AT nicholastfranklin acholinergicfeedbackcircuittoregulatestriatalpopulationuncertaintyandoptimizereinforcementlearning AT michaeljfrank acholinergicfeedbackcircuittoregulatestriatalpopulationuncertaintyandoptimizereinforcementlearning AT nicholastfranklin cholinergicfeedbackcircuittoregulatestriatalpopulationuncertaintyandoptimizereinforcementlearning AT michaeljfrank cholinergicfeedbackcircuittoregulatestriatalpopulationuncertaintyandoptimizereinforcementlearning |