Decision Making under Uncertainty: A Neural Model based on Partially Observable Markov Decision Processes
A fundamental problem faced by animals is learning to select actions based on noisy sensory information and incomplete knowledge of the world. It has been suggested that the brain engages in Bayesian inference during perception but how such probabilistic representations are used to select actions ha...
| Hovedforfatter: | |
|---|---|
| Format: | Article |
| Sprog: | English |
| Udgivet: |
Frontiers Media S.A.
2010-11-01
|
| Serier: | Frontiers in Computational Neuroscience |
| Fag: | |
| Online adgang: | http://journal.frontiersin.org/Journal/10.3389/fncom.2010.00146/full |
| _version_ | 1830453659587051520 |
|---|---|
| author | Rajesh P N Rao |
| author_facet | Rajesh P N Rao |
| author_sort | Rajesh P N Rao |
| collection | DOAJ |
| description | A fundamental problem faced by animals is learning to select actions based on noisy sensory information and incomplete knowledge of the world. It has been suggested that the brain engages in Bayesian inference during perception but how such probabilistic representations are used to select actions has remained unclear. Here we propose a neural model of action selection and decision making based on the theory of partially observable Markov decision processes (POMDPs). Actions are selected based not on a single optimal estimate of state but on the posterior distribution over states (the belief state). We show how such a model provides a unified framework for explaining experimental results in decision making that involve both information gathering and overt actions. The model utilizes temporal difference (TD) learning for maximizing expected reward. The resulting neural architecture posits an active role for the neocortex in belief computation while ascribing a role to the basal ganglia in belief representation, value computation, and action selection. When applied to the random dots motion discrimination task, model neurons representing belief exhibit responses similar to those of LIP neurons in primate neocortex. The appropriate threshold for switching from information gathering to overt actions emerges naturally during reward maximization. Additionally, the time course of reward prediction error in the model shares similarities with dopaminergic responses in the basal ganglia during the random dots task. For tasks with a deadline, the model learns a decision making strategy that changes with elapsed time, predicting a collapsing decision threshold consistent with some experimental studies. The model provides a new framework for understanding neural decision making and suggests an important role for interactions between the neocortex and the basal ganglia in learning the mapping between probabilistic sensory representations and actions that maximize rewards. |
| first_indexed | 2024-12-21T09:12:01Z |
| format | Article |
| id | doaj.art-f0c32676e4a94641a0c6e6fdcff8bea7 |
| institution | Directory Open Access Journal |
| issn | 1662-5188 |
| language | English |
| last_indexed | 2024-12-21T09:12:01Z |
| publishDate | 2010-11-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Computational Neuroscience |
| spelling | doaj.art-f0c32676e4a94641a0c6e6fdcff8bea72022-12-21T19:09:13ZengFrontiers Media S.A.Frontiers in Computational Neuroscience1662-51882010-11-01410.3389/fncom.2010.001461954Decision Making under Uncertainty: A Neural Model based on Partially Observable Markov Decision ProcessesRajesh P N Rao0University of WashingtonA fundamental problem faced by animals is learning to select actions based on noisy sensory information and incomplete knowledge of the world. It has been suggested that the brain engages in Bayesian inference during perception but how such probabilistic representations are used to select actions has remained unclear. Here we propose a neural model of action selection and decision making based on the theory of partially observable Markov decision processes (POMDPs). Actions are selected based not on a single optimal estimate of state but on the posterior distribution over states (the belief state). We show how such a model provides a unified framework for explaining experimental results in decision making that involve both information gathering and overt actions. The model utilizes temporal difference (TD) learning for maximizing expected reward. The resulting neural architecture posits an active role for the neocortex in belief computation while ascribing a role to the basal ganglia in belief representation, value computation, and action selection. When applied to the random dots motion discrimination task, model neurons representing belief exhibit responses similar to those of LIP neurons in primate neocortex. The appropriate threshold for switching from information gathering to overt actions emerges naturally during reward maximization. Additionally, the time course of reward prediction error in the model shares similarities with dopaminergic responses in the basal ganglia during the random dots task. For tasks with a deadline, the model learns a decision making strategy that changes with elapsed time, predicting a collapsing decision threshold consistent with some experimental studies. The model provides a new framework for understanding neural decision making and suggests an important role for interactions between the neocortex and the basal ganglia in learning the mapping between probabilistic sensory representations and actions that maximize rewards.http://journal.frontiersin.org/Journal/10.3389/fncom.2010.00146/fullBasal GangliaDopaminereinforcement learningparietal cortexBayesian inferenceDecision Theory |
| spellingShingle | Rajesh P N Rao Decision Making under Uncertainty: A Neural Model based on Partially Observable Markov Decision Processes Frontiers in Computational Neuroscience Basal Ganglia Dopamine reinforcement learning parietal cortex Bayesian inference Decision Theory |
| title | Decision Making under Uncertainty: A Neural Model based on Partially Observable Markov Decision Processes |
| title_full | Decision Making under Uncertainty: A Neural Model based on Partially Observable Markov Decision Processes |
| title_fullStr | Decision Making under Uncertainty: A Neural Model based on Partially Observable Markov Decision Processes |
| title_full_unstemmed | Decision Making under Uncertainty: A Neural Model based on Partially Observable Markov Decision Processes |
| title_short | Decision Making under Uncertainty: A Neural Model based on Partially Observable Markov Decision Processes |
| title_sort | decision making under uncertainty a neural model based on partially observable markov decision processes |
| topic | Basal Ganglia Dopamine reinforcement learning parietal cortex Bayesian inference Decision Theory |
| url | http://journal.frontiersin.org/Journal/10.3389/fncom.2010.00146/full |
| work_keys_str_mv | AT rajeshpnrao decisionmakingunderuncertaintyaneuralmodelbasedonpartiallyobservablemarkovdecisionprocesses |