A novel technique for delineating the effect of variation in the learning rate on the neural correlates of reward prediction errors in model-based fMRI
IntroductionComputational models play an increasingly important role in describing variation in neural activation in human neuroimaging experiments, including evaluating individual differences in the context of psychiatric neuroimaging. In particular, reinforcement learning (RL) techniques have been...
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2023-12-01
|
| Series: | Frontiers in Psychology |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fpsyg.2023.1211528/full |
| _version_ | 1797384200980004864 |
|---|---|
| author | Henry W. Chase |
| author_facet | Henry W. Chase |
| author_sort | Henry W. Chase |
| collection | DOAJ |
| description | IntroductionComputational models play an increasingly important role in describing variation in neural activation in human neuroimaging experiments, including evaluating individual differences in the context of psychiatric neuroimaging. In particular, reinforcement learning (RL) techniques have been widely adopted to examine neural responses to reward prediction errors and stimulus or action values, and how these might vary as a function of clinical status. However, there is a lack of consensus around the importance of the precision of free parameter estimation for these methods, particularly with regard to the learning rate. In the present study, I introduce a novel technique which may be used within a general linear model (GLM) to model the effect of mis-estimation of the learning rate on reward prediction error (RPE)-related neural responses.MethodsSimulations employed a simple RL algorithm, which was used to generate hypothetical neural activations that would be expected to be observed in functional magnetic resonance imaging (fMRI) studies of RL. Similar RL models were incorporated within a GLM-based analysis method including derivatives, with individual differences in the resulting GLM-derived beta parameters being evaluated with respect to the free parameters of the RL model or being submitted to other validation analyses.ResultsInitial simulations demonstrated that the conventional approach to fitting RL models to RPE responses is more likely to reflect individual differences in a reinforcement efficacy construct (lambda) rather than learning rate (alpha). The proposed method, adding a derivative regressor to the GLM, provides a second regressor which reflects the learning rate. Validation analyses were performed including examining another comparable method which yielded highly similar results, and a demonstration of sensitivity of the method in presence of fMRI-like noise.ConclusionOverall, the findings underscore the importance of the lambda parameter for interpreting individual differences in RPE-coupled neural activity, and validate a novel neural metric of the modulation of such activity by individual differences in the learning rate. The method is expected to find application in understanding aberrant reinforcement learning across different psychiatric patient groups including major depression and substance use disorder. |
| first_indexed | 2024-03-08T21:32:57Z |
| format | Article |
| id | doaj.art-332355a5be59459c9a71e66e8e46436c |
| institution | Directory Open Access Journal |
| issn | 1664-1078 |
| language | English |
| last_indexed | 2024-03-08T21:32:57Z |
| publishDate | 2023-12-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Psychology |
| spelling | doaj.art-332355a5be59459c9a71e66e8e46436c2023-12-21T04:48:30ZengFrontiers Media S.A.Frontiers in Psychology1664-10782023-12-011410.3389/fpsyg.2023.12115281211528A novel technique for delineating the effect of variation in the learning rate on the neural correlates of reward prediction errors in model-based fMRIHenry W. ChaseIntroductionComputational models play an increasingly important role in describing variation in neural activation in human neuroimaging experiments, including evaluating individual differences in the context of psychiatric neuroimaging. In particular, reinforcement learning (RL) techniques have been widely adopted to examine neural responses to reward prediction errors and stimulus or action values, and how these might vary as a function of clinical status. However, there is a lack of consensus around the importance of the precision of free parameter estimation for these methods, particularly with regard to the learning rate. In the present study, I introduce a novel technique which may be used within a general linear model (GLM) to model the effect of mis-estimation of the learning rate on reward prediction error (RPE)-related neural responses.MethodsSimulations employed a simple RL algorithm, which was used to generate hypothetical neural activations that would be expected to be observed in functional magnetic resonance imaging (fMRI) studies of RL. Similar RL models were incorporated within a GLM-based analysis method including derivatives, with individual differences in the resulting GLM-derived beta parameters being evaluated with respect to the free parameters of the RL model or being submitted to other validation analyses.ResultsInitial simulations demonstrated that the conventional approach to fitting RL models to RPE responses is more likely to reflect individual differences in a reinforcement efficacy construct (lambda) rather than learning rate (alpha). The proposed method, adding a derivative regressor to the GLM, provides a second regressor which reflects the learning rate. Validation analyses were performed including examining another comparable method which yielded highly similar results, and a demonstration of sensitivity of the method in presence of fMRI-like noise.ConclusionOverall, the findings underscore the importance of the lambda parameter for interpreting individual differences in RPE-coupled neural activity, and validate a novel neural metric of the modulation of such activity by individual differences in the learning rate. The method is expected to find application in understanding aberrant reinforcement learning across different psychiatric patient groups including major depression and substance use disorder.https://www.frontiersin.org/articles/10.3389/fpsyg.2023.1211528/fullreinforcement learningreinforcement sensitivityprediction errorsfMRIgeneral linear model |
| spellingShingle | Henry W. Chase A novel technique for delineating the effect of variation in the learning rate on the neural correlates of reward prediction errors in model-based fMRI Frontiers in Psychology reinforcement learning reinforcement sensitivity prediction errors fMRI general linear model |
| title | A novel technique for delineating the effect of variation in the learning rate on the neural correlates of reward prediction errors in model-based fMRI |
| title_full | A novel technique for delineating the effect of variation in the learning rate on the neural correlates of reward prediction errors in model-based fMRI |
| title_fullStr | A novel technique for delineating the effect of variation in the learning rate on the neural correlates of reward prediction errors in model-based fMRI |
| title_full_unstemmed | A novel technique for delineating the effect of variation in the learning rate on the neural correlates of reward prediction errors in model-based fMRI |
| title_short | A novel technique for delineating the effect of variation in the learning rate on the neural correlates of reward prediction errors in model-based fMRI |
| title_sort | novel technique for delineating the effect of variation in the learning rate on the neural correlates of reward prediction errors in model based fmri |
| topic | reinforcement learning reinforcement sensitivity prediction errors fMRI general linear model |
| url | https://www.frontiersin.org/articles/10.3389/fpsyg.2023.1211528/full |
| work_keys_str_mv | AT henrywchase anoveltechniquefordelineatingtheeffectofvariationinthelearningrateontheneuralcorrelatesofrewardpredictionerrorsinmodelbasedfmri AT henrywchase noveltechniquefordelineatingtheeffectofvariationinthelearningrateontheneuralcorrelatesofrewardpredictionerrorsinmodelbasedfmri |