Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making.
Classic reinforcement learning (RL) theories cannot explain human behavior in the absence of external reward or when the environment changes. Here, we employ a deep sequential decision-making paradigm with sparse reward and abrupt environmental changes. To explain the behavior of human participants...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science (PLoS)
2021-06-01
|
| Series: | PLoS Computational Biology |
| Online Access: | https://doi.org/10.1371/journal.pcbi.1009070 |
| _version_ | 1818932116646264832 |
|---|---|
| author | He A Xu Alireza Modirshanechi Marco P Lehmann Wulfram Gerstner Michael H Herzog |
| author_facet | He A Xu Alireza Modirshanechi Marco P Lehmann Wulfram Gerstner Michael H Herzog |
| author_sort | He A Xu |
| collection | DOAJ |
| description | Classic reinforcement learning (RL) theories cannot explain human behavior in the absence of external reward or when the environment changes. Here, we employ a deep sequential decision-making paradigm with sparse reward and abrupt environmental changes. To explain the behavior of human participants in these environments, we show that RL theories need to include surprise and novelty, each with a distinct role. While novelty drives exploration before the first encounter of a reward, surprise increases the rate of learning of a world-model as well as of model-free action-values. Even though the world-model is available for model-based RL, we find that human decisions are dominated by model-free action choices. The world-model is only marginally used for planning, but it is important to detect surprising events. Our theory predicts human action choices with high probability and allows us to dissociate surprise, novelty, and reward in EEG signals. |
| first_indexed | 2024-12-20T04:27:22Z |
| format | Article |
| id | doaj.art-c394c02f72114f7f930a45a422d833f0 |
| institution | Directory Open Access Journal |
| issn | 1553-734X 1553-7358 |
| language | English |
| last_indexed | 2024-12-20T04:27:22Z |
| publishDate | 2021-06-01 |
| publisher | Public Library of Science (PLoS) |
| record_format | Article |
| series | PLoS Computational Biology |
| spelling | doaj.art-c394c02f72114f7f930a45a422d833f02022-12-21T19:53:29ZengPublic Library of Science (PLoS)PLoS Computational Biology1553-734X1553-73582021-06-01176e100907010.1371/journal.pcbi.1009070Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making.He A XuAlireza ModirshanechiMarco P LehmannWulfram GerstnerMichael H HerzogClassic reinforcement learning (RL) theories cannot explain human behavior in the absence of external reward or when the environment changes. Here, we employ a deep sequential decision-making paradigm with sparse reward and abrupt environmental changes. To explain the behavior of human participants in these environments, we show that RL theories need to include surprise and novelty, each with a distinct role. While novelty drives exploration before the first encounter of a reward, surprise increases the rate of learning of a world-model as well as of model-free action-values. Even though the world-model is available for model-based RL, we find that human decisions are dominated by model-free action choices. The world-model is only marginally used for planning, but it is important to detect surprising events. Our theory predicts human action choices with high probability and allows us to dissociate surprise, novelty, and reward in EEG signals.https://doi.org/10.1371/journal.pcbi.1009070 |
| spellingShingle | He A Xu Alireza Modirshanechi Marco P Lehmann Wulfram Gerstner Michael H Herzog Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making. PLoS Computational Biology |
| title | Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making. |
| title_full | Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making. |
| title_fullStr | Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making. |
| title_full_unstemmed | Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making. |
| title_short | Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making. |
| title_sort | novelty is not surprise human exploratory and adaptive behavior in sequential decision making |
| url | https://doi.org/10.1371/journal.pcbi.1009070 |
| work_keys_str_mv | AT heaxu noveltyisnotsurprisehumanexploratoryandadaptivebehaviorinsequentialdecisionmaking AT alirezamodirshanechi noveltyisnotsurprisehumanexploratoryandadaptivebehaviorinsequentialdecisionmaking AT marcoplehmann noveltyisnotsurprisehumanexploratoryandadaptivebehaviorinsequentialdecisionmaking AT wulframgerstner noveltyisnotsurprisehumanexploratoryandadaptivebehaviorinsequentialdecisionmaking AT michaelhherzog noveltyisnotsurprisehumanexploratoryandadaptivebehaviorinsequentialdecisionmaking |