A Novel Model for Arbitration Between Planning and Habitual Control Systems
It is well-established that human decision making and instrumental control uses multiple systems, some which use habitual action selection and some which require deliberate planning. Deliberate planning systems use predictions of action-outcomes using an internal model of the agent's environmen...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2019-07-01
|
| Series: | Frontiers in Neurorobotics |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/article/10.3389/fnbot.2019.00052/full |
| _version_ | 1828915615973244928 |
|---|---|
| author | Farzaneh Sheikhnezhad Fard Thomas P. Trappenberg |
| author_facet | Farzaneh Sheikhnezhad Fard Thomas P. Trappenberg |
| author_sort | Farzaneh Sheikhnezhad Fard |
| collection | DOAJ |
| description | It is well-established that human decision making and instrumental control uses multiple systems, some which use habitual action selection and some which require deliberate planning. Deliberate planning systems use predictions of action-outcomes using an internal model of the agent's environment, while habitual action selection systems learn to automate by repeating previously rewarded actions. Habitual control is computationally efficient but are not very flexible in changing environments. Conversely, deliberate planning may be computationally expensive, but flexible in dynamic environments. This paper proposes a general architecture comprising both control paradigms by introducing an arbitrator that controls which subsystem is used at any time. This system is implemented for a target-reaching task with a simulated two-joint robotic arm that comprises a supervised internal model and deep reinforcement learning. Through permutation of target-reaching conditions, we demonstrate that the proposed is capable of rapidly learning kinematics of the system without a priori knowledge, and is robust to (A) changing environmental reward and kinematics, and (B) occluded vision. The arbitrator model is compared to exclusive deliberate planning with the internal model and exclusive habitual control instances of the model. The results show how such a model can harness the benefits of both systems, using fast decisions in reliable circumstances while optimizing performance in changing environments. In addition, the proposed model learns very fast. Finally, the system which includes internal models is able to reach the target under the visual occlusion, while the pure habitual system is unable to operate sufficiently under such conditions. |
| first_indexed | 2024-12-13T20:17:24Z |
| format | Article |
| id | doaj.art-f8ac12246a324c44a6301ea928e75072 |
| institution | Directory Open Access Journal |
| issn | 1662-5218 |
| language | English |
| last_indexed | 2024-12-13T20:17:24Z |
| publishDate | 2019-07-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Neurorobotics |
| spelling | doaj.art-f8ac12246a324c44a6301ea928e750722022-12-21T23:32:47ZengFrontiers Media S.A.Frontiers in Neurorobotics1662-52182019-07-011310.3389/fnbot.2019.00052456999A Novel Model for Arbitration Between Planning and Habitual Control SystemsFarzaneh Sheikhnezhad FardThomas P. TrappenbergIt is well-established that human decision making and instrumental control uses multiple systems, some which use habitual action selection and some which require deliberate planning. Deliberate planning systems use predictions of action-outcomes using an internal model of the agent's environment, while habitual action selection systems learn to automate by repeating previously rewarded actions. Habitual control is computationally efficient but are not very flexible in changing environments. Conversely, deliberate planning may be computationally expensive, but flexible in dynamic environments. This paper proposes a general architecture comprising both control paradigms by introducing an arbitrator that controls which subsystem is used at any time. This system is implemented for a target-reaching task with a simulated two-joint robotic arm that comprises a supervised internal model and deep reinforcement learning. Through permutation of target-reaching conditions, we demonstrate that the proposed is capable of rapidly learning kinematics of the system without a priori knowledge, and is robust to (A) changing environmental reward and kinematics, and (B) occluded vision. The arbitrator model is compared to exclusive deliberate planning with the internal model and exclusive habitual control instances of the model. The results show how such a model can harness the benefits of both systems, using fast decisions in reliable circumstances while optimizing performance in changing environments. In addition, the proposed model learns very fast. Finally, the system which includes internal models is able to reach the target under the visual occlusion, while the pure habitual system is unable to operate sufficiently under such conditions.https://www.frontiersin.org/article/10.3389/fnbot.2019.00052/fullmachine learningreinforcement learningsupervised learninghabitual controllerplanninginternal models |
| spellingShingle | Farzaneh Sheikhnezhad Fard Thomas P. Trappenberg A Novel Model for Arbitration Between Planning and Habitual Control Systems Frontiers in Neurorobotics machine learning reinforcement learning supervised learning habitual controller planning internal models |
| title | A Novel Model for Arbitration Between Planning and Habitual Control Systems |
| title_full | A Novel Model for Arbitration Between Planning and Habitual Control Systems |
| title_fullStr | A Novel Model for Arbitration Between Planning and Habitual Control Systems |
| title_full_unstemmed | A Novel Model for Arbitration Between Planning and Habitual Control Systems |
| title_short | A Novel Model for Arbitration Between Planning and Habitual Control Systems |
| title_sort | novel model for arbitration between planning and habitual control systems |
| topic | machine learning reinforcement learning supervised learning habitual controller planning internal models |
| url | https://www.frontiersin.org/article/10.3389/fnbot.2019.00052/full |
| work_keys_str_mv | AT farzanehsheikhnezhadfard anovelmodelforarbitrationbetweenplanningandhabitualcontrolsystems AT thomasptrappenberg anovelmodelforarbitrationbetweenplanningandhabitualcontrolsystems AT farzanehsheikhnezhadfard novelmodelforarbitrationbetweenplanningandhabitualcontrolsystems AT thomasptrappenberg novelmodelforarbitrationbetweenplanningandhabitualcontrolsystems |