Adaptive Locomotion Learning for Quadruped Robots by Combining DRL with a Cosine Oscillator Based Rhythm Controller
Animals have evolved to adapt to complex and uncertain environments, acquiring locomotion skills for diverse surroundings. To endow a robot’s animal-like locomotion ability, in this paper, we propose a learning algorithm for quadruped robots based on deep reinforcement learning (DRL) and a rhythm co...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-10-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/13/19/11045 |
| _version_ | 1797576184815419392 |
|---|---|
| author | Xiaoping Zhang Yitong Wu Huijiang Wang Fumiya Iida Li Wang |
| author_facet | Xiaoping Zhang Yitong Wu Huijiang Wang Fumiya Iida Li Wang |
| author_sort | Xiaoping Zhang |
| collection | DOAJ |
| description | Animals have evolved to adapt to complex and uncertain environments, acquiring locomotion skills for diverse surroundings. To endow a robot’s animal-like locomotion ability, in this paper, we propose a learning algorithm for quadruped robots based on deep reinforcement learning (DRL) and a rhythm controller that is based on a cosine oscillator. For a quadruped robot, two cosine oscillators are utilized at the hip joint and the knee joint of one leg, respectively, and, finally, eight oscillators form the controller to realize the quadruped robot’s locomotion rhythm during moving. The coupling between the cosine oscillators of the rhythm controller is realized by the phase difference, which is simpler and easier to realize when dealing with the complex coupling relationship between different joints. DRL is used to help learn the controller parameters and, in the reward function design, we address the challenge of terrain adaptation without relying on the complex camera-based vision processing but based on the proprioceptive information, where a state estimator is introduced to achieve the robot’s posture and help finally utilize the food-end coordinate. Experiments are carried out in CoppeliaSim, and all of the flat, uphill and downhill conditions are considered. The results show that the robot can successfully accomplish all the above skills and, at the same time, with the reward function designed, the robot’s pitch angle, yaw angle and roll angle are very small, which means that the robot is relatively stable during walking. Then, the robot is transplanted to a new scene; the results show that although the environment is previously unencountered, the robot can still fulfill the task, which demonstrates the effectiveness and robustness of this proposed method. |
| first_indexed | 2024-03-10T21:48:40Z |
| format | Article |
| id | doaj.art-76fb166e139442b4a6516dd9b56e7575 |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-10T21:48:40Z |
| publishDate | 2023-10-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-76fb166e139442b4a6516dd9b56e75752023-11-19T14:07:51ZengMDPI AGApplied Sciences2076-34172023-10-0113191104510.3390/app131911045Adaptive Locomotion Learning for Quadruped Robots by Combining DRL with a Cosine Oscillator Based Rhythm ControllerXiaoping Zhang0Yitong Wu1Huijiang Wang2Fumiya Iida3Li Wang4School of Electrical and Control Engineering, North China University of Technology, Beijing 100144, ChinaSchool of Electrical and Control Engineering, North China University of Technology, Beijing 100144, ChinaDepartment of Engineering, University of Cambridge, Cambridge CB2 1PZ, UKDepartment of Engineering, University of Cambridge, Cambridge CB2 1PZ, UKSchool of Electrical and Control Engineering, North China University of Technology, Beijing 100144, ChinaAnimals have evolved to adapt to complex and uncertain environments, acquiring locomotion skills for diverse surroundings. To endow a robot’s animal-like locomotion ability, in this paper, we propose a learning algorithm for quadruped robots based on deep reinforcement learning (DRL) and a rhythm controller that is based on a cosine oscillator. For a quadruped robot, two cosine oscillators are utilized at the hip joint and the knee joint of one leg, respectively, and, finally, eight oscillators form the controller to realize the quadruped robot’s locomotion rhythm during moving. The coupling between the cosine oscillators of the rhythm controller is realized by the phase difference, which is simpler and easier to realize when dealing with the complex coupling relationship between different joints. DRL is used to help learn the controller parameters and, in the reward function design, we address the challenge of terrain adaptation without relying on the complex camera-based vision processing but based on the proprioceptive information, where a state estimator is introduced to achieve the robot’s posture and help finally utilize the food-end coordinate. Experiments are carried out in CoppeliaSim, and all of the flat, uphill and downhill conditions are considered. The results show that the robot can successfully accomplish all the above skills and, at the same time, with the reward function designed, the robot’s pitch angle, yaw angle and roll angle are very small, which means that the robot is relatively stable during walking. Then, the robot is transplanted to a new scene; the results show that although the environment is previously unencountered, the robot can still fulfill the task, which demonstrates the effectiveness and robustness of this proposed method.https://www.mdpi.com/2076-3417/13/19/11045quadruped robotcosine oscillatorrhythm controllerrobot locomotiondeep reinforcement learning |
| spellingShingle | Xiaoping Zhang Yitong Wu Huijiang Wang Fumiya Iida Li Wang Adaptive Locomotion Learning for Quadruped Robots by Combining DRL with a Cosine Oscillator Based Rhythm Controller Applied Sciences quadruped robot cosine oscillator rhythm controller robot locomotion deep reinforcement learning |
| title | Adaptive Locomotion Learning for Quadruped Robots by Combining DRL with a Cosine Oscillator Based Rhythm Controller |
| title_full | Adaptive Locomotion Learning for Quadruped Robots by Combining DRL with a Cosine Oscillator Based Rhythm Controller |
| title_fullStr | Adaptive Locomotion Learning for Quadruped Robots by Combining DRL with a Cosine Oscillator Based Rhythm Controller |
| title_full_unstemmed | Adaptive Locomotion Learning for Quadruped Robots by Combining DRL with a Cosine Oscillator Based Rhythm Controller |
| title_short | Adaptive Locomotion Learning for Quadruped Robots by Combining DRL with a Cosine Oscillator Based Rhythm Controller |
| title_sort | adaptive locomotion learning for quadruped robots by combining drl with a cosine oscillator based rhythm controller |
| topic | quadruped robot cosine oscillator rhythm controller robot locomotion deep reinforcement learning |
| url | https://www.mdpi.com/2076-3417/13/19/11045 |
| work_keys_str_mv | AT xiaopingzhang adaptivelocomotionlearningforquadrupedrobotsbycombiningdrlwithacosineoscillatorbasedrhythmcontroller AT yitongwu adaptivelocomotionlearningforquadrupedrobotsbycombiningdrlwithacosineoscillatorbasedrhythmcontroller AT huijiangwang adaptivelocomotionlearningforquadrupedrobotsbycombiningdrlwithacosineoscillatorbasedrhythmcontroller AT fumiyaiida adaptivelocomotionlearningforquadrupedrobotsbycombiningdrlwithacosineoscillatorbasedrhythmcontroller AT liwang adaptivelocomotionlearningforquadrupedrobotsbycombiningdrlwithacosineoscillatorbasedrhythmcontroller |