Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia
We continuously face the dilemma of choosing between actions that gather new information or actions that exploit existing knowledge. This `exploration-exploitation' trade-off depends on the environment: stability favours exploiting knowledge to maximise gains; volatility favours exploring new...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2012-02-01
|
| Series: | Frontiers in Neuroscience |
| Subjects: | |
| Online Access: | http://journal.frontiersin.org/Journal/10.3389/fnins.2012.00009/full |
| _version_ | 1830285608863399936 |
|---|---|
| author | Mark D Humphries Mark D Humphries Mehdi eKhamassi Mehdi eKhamassi Kevin eGurney |
| author_facet | Mark D Humphries Mark D Humphries Mehdi eKhamassi Mehdi eKhamassi Kevin eGurney |
| author_sort | Mark D Humphries |
| collection | DOAJ |
| description | We continuously face the dilemma of choosing between actions that gather new information or actions that exploit existing knowledge. This `exploration-exploitation' trade-off depends on the environment: stability favours exploiting knowledge to maximise gains; volatility favours exploring new options and discovering new outcomes. Here we set out to reconcile recent evidence for dopamine's involvement in the exploration-exploitation trade-off with the existing evidence for basal ganglia control of action selection, by testing the hypothesis that tonic dopamine in the striatum, the basal ganglia's input nucleus, sets the current exploration-exploitation tradeoff. We first advance the idea of interpreting the basal ganglia output as a probability distribution function for action selection. Using computational models of the full basal ganglia circuit, we showed that, under this interpretation, the actions of dopamine within the striatum change the basal ganglia's output to favour the level of exploration or exploitation encoded in the probability distribution. We also found that our models predict striatal dopamine controls the exploration-exploitation trade-off if we instead read out the probability distribution from the target nuclei of the basal ganglia, where their inhibitory input shapes the cortical input to these nuclei. Finally, by integrating the basal ganglia within a reinforcement learning model, we showed how dopamine's effect on the exploration-exploitation trade-off could be measurable in a forced two-choice task. These simulations also showed how tonic dopamine can appear to affect learning while only directly altering the trade-off. Thus, our models support the hypothesis that changes in tonic dopamine within the striatum can alter the exploration-exploitation trade-off by modulating the output of the basal ganglia. |
| first_indexed | 2024-12-19T03:42:18Z |
| format | Article |
| id | doaj.art-b8f630bc51544678bafa0f93cc480c6f |
| institution | Directory Open Access Journal |
| issn | 1662-453X |
| language | English |
| last_indexed | 2024-12-19T03:42:18Z |
| publishDate | 2012-02-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Neuroscience |
| spelling | doaj.art-b8f630bc51544678bafa0f93cc480c6f2022-12-21T20:37:12ZengFrontiers Media S.A.Frontiers in Neuroscience1662-453X2012-02-01610.3389/fnins.2012.0000916922Dopaminergic control of the exploration-exploitation trade-off via the basal gangliaMark D Humphries0Mark D Humphries1Mehdi eKhamassi2Mehdi eKhamassi3Kevin eGurney4Ecole Normale SuperieureUniversity of SheffieldUniversité Pierre et Marie CurieUMR7222, Centre National de la Recherche ScientifiqueUniversity of SheffieldWe continuously face the dilemma of choosing between actions that gather new information or actions that exploit existing knowledge. This `exploration-exploitation' trade-off depends on the environment: stability favours exploiting knowledge to maximise gains; volatility favours exploring new options and discovering new outcomes. Here we set out to reconcile recent evidence for dopamine's involvement in the exploration-exploitation trade-off with the existing evidence for basal ganglia control of action selection, by testing the hypothesis that tonic dopamine in the striatum, the basal ganglia's input nucleus, sets the current exploration-exploitation tradeoff. We first advance the idea of interpreting the basal ganglia output as a probability distribution function for action selection. Using computational models of the full basal ganglia circuit, we showed that, under this interpretation, the actions of dopamine within the striatum change the basal ganglia's output to favour the level of exploration or exploitation encoded in the probability distribution. We also found that our models predict striatal dopamine controls the exploration-exploitation trade-off if we instead read out the probability distribution from the target nuclei of the basal ganglia, where their inhibitory input shapes the cortical input to these nuclei. Finally, by integrating the basal ganglia within a reinforcement learning model, we showed how dopamine's effect on the exploration-exploitation trade-off could be measurable in a forced two-choice task. These simulations also showed how tonic dopamine can appear to affect learning while only directly altering the trade-off. Thus, our models support the hypothesis that changes in tonic dopamine within the striatum can alter the exploration-exploitation trade-off by modulating the output of the basal ganglia.http://journal.frontiersin.org/Journal/10.3389/fnins.2012.00009/fullDecision Makingreinforcement learningRewarduncertaintymeta-parameters |
| spellingShingle | Mark D Humphries Mark D Humphries Mehdi eKhamassi Mehdi eKhamassi Kevin eGurney Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia Frontiers in Neuroscience Decision Making reinforcement learning Reward uncertainty meta-parameters |
| title | Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia |
| title_full | Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia |
| title_fullStr | Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia |
| title_full_unstemmed | Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia |
| title_short | Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia |
| title_sort | dopaminergic control of the exploration exploitation trade off via the basal ganglia |
| topic | Decision Making reinforcement learning Reward uncertainty meta-parameters |
| url | http://journal.frontiersin.org/Journal/10.3389/fnins.2012.00009/full |
| work_keys_str_mv | AT markdhumphries dopaminergiccontroloftheexplorationexploitationtradeoffviathebasalganglia AT markdhumphries dopaminergiccontroloftheexplorationexploitationtradeoffviathebasalganglia AT mehdiekhamassi dopaminergiccontroloftheexplorationexploitationtradeoffviathebasalganglia AT mehdiekhamassi dopaminergiccontroloftheexplorationexploitationtradeoffviathebasalganglia AT kevinegurney dopaminergiccontroloftheexplorationexploitationtradeoffviathebasalganglia |