Optimal habits can develop spontaneously through sensitivity to local cost
Habits and rituals are expressed universally across animal species. These behaviors are advantageous in allowing sequential behaviors to be performed without cognitive overload, and appear to rely on neural circuits that are relatively benign but vulnerable to takeover by extreme contexts, neuropsyc...
| Main Authors: | , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | en_US |
| Published: |
Proceedings of the National Academy of Sciences (PNAS)
2012
|
| Online Access: | http://hdl.handle.net/1721.1/69896 https://orcid.org/0000-0002-7223-0556 https://orcid.org/0000-0002-4326-7720 |
| _version_ | 1811078582401761280 |
|---|---|
| author | Desrochers, Theresa M. Goodman, Noah D. Graybiel, Ann M. Jin, Dezhe Z. |
| author2 | Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences |
| author_facet | Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences Desrochers, Theresa M. Goodman, Noah D. Graybiel, Ann M. Jin, Dezhe Z. |
| author_sort | Desrochers, Theresa M. |
| collection | MIT |
| description | Habits and rituals are expressed universally across animal species. These behaviors are advantageous in allowing sequential behaviors to be performed without cognitive overload, and appear to rely on neural circuits that are relatively benign but vulnerable to takeover by extreme contexts, neuropsychiatric sequelae, and processes leading to addiction. Reinforcement learning (RL) is thought to underlie the formation of optimal habits. However, this theoretic formulation has principally been tested experimentally in simple stimulus-response tasks with relatively few available responses. We asked whether RL could also account for the emergence of habitual action sequences in realistically complex situations in which no repetitive stimulus-response links were present and in which many response options were present. We exposed naïve macaque monkeys to such experimental conditions by introducing a unique free saccade scan task. Despite the highly uncertain conditions and no instruction, the monkeys developed a succession of stereotypical, self-chosen saccade sequence patterns. Remarkably, these continued to morph for months, long after session-averaged reward and cost (eye movement distance) reached asymptote. Prima facie, these continued behavioral changes appeared to challenge RL. However, trial-by-trial analysis showed that pattern changes on adjacent trials were predicted by lowered cost, and RL simulations that reduced the cost reproduced the monkeys’ behavior. Ultimately, the patterns settled into stereotypical saccade sequences that minimized the cost of obtaining the reward on average. These findings suggest that brain mechanisms underlying the emergence of habits, and perhaps unwanted repetitive behaviors in clinical disorders, could follow RL algorithms capturing extremely local explore/exploit tradeoffs. |
| first_indexed | 2024-09-23T11:02:33Z |
| format | Article |
| id | mit-1721.1/69896 |
| institution | Massachusetts Institute of Technology |
| language | en_US |
| last_indexed | 2024-09-23T11:02:33Z |
| publishDate | 2012 |
| publisher | Proceedings of the National Academy of Sciences (PNAS) |
| record_format | dspace |
| spelling | mit-1721.1/698962022-10-01T00:42:11Z Optimal habits can develop spontaneously through sensitivity to local cost Desrochers, Theresa M. Goodman, Noah D. Graybiel, Ann M. Jin, Dezhe Z. Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences McGovern Institute for Brain Research at MIT Graybiel, Ann M. Desrochers, Theresa M. Goodman, Noah D. Graybiel, Ann M. Habits and rituals are expressed universally across animal species. These behaviors are advantageous in allowing sequential behaviors to be performed without cognitive overload, and appear to rely on neural circuits that are relatively benign but vulnerable to takeover by extreme contexts, neuropsychiatric sequelae, and processes leading to addiction. Reinforcement learning (RL) is thought to underlie the formation of optimal habits. However, this theoretic formulation has principally been tested experimentally in simple stimulus-response tasks with relatively few available responses. We asked whether RL could also account for the emergence of habitual action sequences in realistically complex situations in which no repetitive stimulus-response links were present and in which many response options were present. We exposed naïve macaque monkeys to such experimental conditions by introducing a unique free saccade scan task. Despite the highly uncertain conditions and no instruction, the monkeys developed a succession of stereotypical, self-chosen saccade sequence patterns. Remarkably, these continued to morph for months, long after session-averaged reward and cost (eye movement distance) reached asymptote. Prima facie, these continued behavioral changes appeared to challenge RL. However, trial-by-trial analysis showed that pattern changes on adjacent trials were predicted by lowered cost, and RL simulations that reduced the cost reproduced the monkeys’ behavior. Ultimately, the patterns settled into stereotypical saccade sequences that minimized the cost of obtaining the reward on average. These findings suggest that brain mechanisms underlying the emergence of habits, and perhaps unwanted repetitive behaviors in clinical disorders, could follow RL algorithms capturing extremely local explore/exploit tradeoffs. National Eye Institute (Grant EY012848) United States. Office of Naval Research (Grant N000014-07-10903) National Defense Science and Engineering Graduate Fellowship McGovern Institute for Brain Research at MIT (Friends of the McGovern Fellowship) Alfred P. Sloan Foundation (Sloan Research Fellowship) 2012-03-30T18:17:53Z 2012-03-30T18:17:53Z 2010-09 2010-08 Article http://purl.org/eprint/type/JournalArticle 0027-8424 1091-6490 http://hdl.handle.net/1721.1/69896 Desrochers, T. M. et al. “From the Cover: Optimal Habits Can Develop Spontaneously Through Sensitivity to Local Cost.” Proceedings of the National Academy of Sciences 107.47 (2010): 20512–20517. Web. 30 Mar. 2012. https://orcid.org/0000-0002-7223-0556 https://orcid.org/0000-0002-4326-7720 en_US http://dx.doi.org/10.1073/pnas.1013470107 Proceedings of the National Academy of Sciences Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. application/pdf Proceedings of the National Academy of Sciences (PNAS) PNAS |
| spellingShingle | Desrochers, Theresa M. Goodman, Noah D. Graybiel, Ann M. Jin, Dezhe Z. Optimal habits can develop spontaneously through sensitivity to local cost |
| title | Optimal habits can develop spontaneously through sensitivity to local cost |
| title_full | Optimal habits can develop spontaneously through sensitivity to local cost |
| title_fullStr | Optimal habits can develop spontaneously through sensitivity to local cost |
| title_full_unstemmed | Optimal habits can develop spontaneously through sensitivity to local cost |
| title_short | Optimal habits can develop spontaneously through sensitivity to local cost |
| title_sort | optimal habits can develop spontaneously through sensitivity to local cost |
| url | http://hdl.handle.net/1721.1/69896 https://orcid.org/0000-0002-7223-0556 https://orcid.org/0000-0002-4326-7720 |
| work_keys_str_mv | AT desrocherstheresam optimalhabitscandevelopspontaneouslythroughsensitivitytolocalcost AT goodmannoahd optimalhabitscandevelopspontaneouslythroughsensitivitytolocalcost AT graybielannm optimalhabitscandevelopspontaneouslythroughsensitivitytolocalcost AT jindezhez optimalhabitscandevelopspontaneouslythroughsensitivitytolocalcost |