An Energy Efficient EdgeAI Autoencoder Accelerator for Reinforcement Learning
In EdgeAI embedded devices that exploit reinforcement learning (RL), it is essential to reduce the number of actions taken by the agent in the real world and minimize the compute-intensive policies learning process. Convolutional autoencoders (AEs) has demonstrated great improvement for speeding up...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2021-01-01
|
| Series: | IEEE Open Journal of Circuits and Systems |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9335309/ |
| _version_ | 1818331671710138368 |
|---|---|
| author | Nitheesh Kumar Manjunath Aidin Shiri Morteza Hosseini Bharat Prakash Nicholas R. Waytowich Tinoosh Mohsenin |
| author_facet | Nitheesh Kumar Manjunath Aidin Shiri Morteza Hosseini Bharat Prakash Nicholas R. Waytowich Tinoosh Mohsenin |
| author_sort | Nitheesh Kumar Manjunath |
| collection | DOAJ |
| description | In EdgeAI embedded devices that exploit reinforcement learning (RL), it is essential to reduce the number of actions taken by the agent in the real world and minimize the compute-intensive policies learning process. Convolutional autoencoders (AEs) has demonstrated great improvement for speeding up the policy learning time when attached to the RL agent, by compressing the high dimensional input data into a small latent representation for feeding the RL agent. Despite reducing the policy learning time, AE adds a significant computational and memory complexity to the model which contributes to the increase in the total computation and the model size. In this article, we propose a model for speeding up the policy learning process of RL agent with the use of AE neural networks, which engages binary and ternary precision to address the high complexity overhead without deteriorating the policy that an RL agent learns. Binary Neural Networks (BNNs) and Ternary Neural Networks (TNNs) compress weights into 1 and 2 bits representations, which result in significant compression of the model size and memory as well as simplifying multiply-accumulate (MAC) operations. We evaluate the performance of our model in three RL environments including DonkeyCar, Miniworld sidewalk, and Miniworld Object Pickup, which emulate various real-world applications with different levels of complexity. With proper hyperparameter optimization and architecture exploration, TNN models achieve near the same average reward, Peak Signal to Noise Ratio (PSNR) and Mean Squared Error (MSE) performance as the full-precision model while reducing the model size by 10x compared to full-precision and 3x compared to BNNs. However, in BNN models the average reward drops up to 12% - 25% compared to the full-precision even after increasing its model size by 4x. We designed and implemented a scalable hardware accelerator which is configurable in terms of the number of processing elements (PEs) and memory data width to achieve the best power, performance, and energy efficiency trade-off for EdgeAI embedded devices. The proposed hardware implemented on Artix-7 FPGA dissipates 250 μJ energy while meeting 30 frames per second (FPS) throughput requirements. The hardware is configurable to reach an efficiency of over 1 TOP/J on FPGA implementation. The proposed hardware accelerator is synthesized and placed-and-routed in 14 nm FinFET ASIC technology which brings down the power dissipation to 3.9 μJ and maximum throughput of 1,250 FPS. Compared to the state of the art TNN implementations on the same target platform, our hardware is 5x and 4.4x (2.2x if technology scaled) more energy efficient on FPGA and ASIC, respectively. |
| first_indexed | 2024-12-13T13:23:33Z |
| format | Article |
| id | doaj.art-07ff7d91b0764d1fac43a58fabeacddc |
| institution | Directory Open Access Journal |
| issn | 2644-1225 |
| language | English |
| last_indexed | 2024-12-13T13:23:33Z |
| publishDate | 2021-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Open Journal of Circuits and Systems |
| spelling | doaj.art-07ff7d91b0764d1fac43a58fabeacddc2022-12-21T23:44:20ZengIEEEIEEE Open Journal of Circuits and Systems2644-12252021-01-01218219510.1109/OJCAS.2020.30437379335309An Energy Efficient EdgeAI Autoencoder Accelerator for Reinforcement LearningNitheesh Kumar Manjunath0https://orcid.org/0000-0001-5551-2124Aidin Shiri1https://orcid.org/0000-0001-5402-0988Morteza Hosseini2Bharat Prakash3Nicholas R. Waytowich4Tinoosh Mohsenin5Department of Computer Science and Electrical Engineering, University of Maryland, Baltimore, MD, USADepartment of Computer Science and Electrical Engineering, University of Maryland, Baltimore, MD, USADepartment of Computer Science and Electrical Engineering, University of Maryland, Baltimore, MD, USADepartment of Computer Science and Electrical Engineering, University of Maryland, Baltimore, MD, USAU.S. Army Research Laboratory, Aberdeen, MD, USADepartment of Computer Science and Electrical Engineering, University of Maryland, Baltimore, MD, USAIn EdgeAI embedded devices that exploit reinforcement learning (RL), it is essential to reduce the number of actions taken by the agent in the real world and minimize the compute-intensive policies learning process. Convolutional autoencoders (AEs) has demonstrated great improvement for speeding up the policy learning time when attached to the RL agent, by compressing the high dimensional input data into a small latent representation for feeding the RL agent. Despite reducing the policy learning time, AE adds a significant computational and memory complexity to the model which contributes to the increase in the total computation and the model size. In this article, we propose a model for speeding up the policy learning process of RL agent with the use of AE neural networks, which engages binary and ternary precision to address the high complexity overhead without deteriorating the policy that an RL agent learns. Binary Neural Networks (BNNs) and Ternary Neural Networks (TNNs) compress weights into 1 and 2 bits representations, which result in significant compression of the model size and memory as well as simplifying multiply-accumulate (MAC) operations. We evaluate the performance of our model in three RL environments including DonkeyCar, Miniworld sidewalk, and Miniworld Object Pickup, which emulate various real-world applications with different levels of complexity. With proper hyperparameter optimization and architecture exploration, TNN models achieve near the same average reward, Peak Signal to Noise Ratio (PSNR) and Mean Squared Error (MSE) performance as the full-precision model while reducing the model size by 10x compared to full-precision and 3x compared to BNNs. However, in BNN models the average reward drops up to 12% - 25% compared to the full-precision even after increasing its model size by 4x. We designed and implemented a scalable hardware accelerator which is configurable in terms of the number of processing elements (PEs) and memory data width to achieve the best power, performance, and energy efficiency trade-off for EdgeAI embedded devices. The proposed hardware implemented on Artix-7 FPGA dissipates 250 μJ energy while meeting 30 frames per second (FPS) throughput requirements. The hardware is configurable to reach an efficiency of over 1 TOP/J on FPGA implementation. The proposed hardware accelerator is synthesized and placed-and-routed in 14 nm FinFET ASIC technology which brings down the power dissipation to 3.9 μJ and maximum throughput of 1,250 FPS. Compared to the state of the art TNN implementations on the same target platform, our hardware is 5x and 4.4x (2.2x if technology scaled) more energy efficient on FPGA and ASIC, respectively.https://ieeexplore.ieee.org/document/9335309/Reinforcement learningautonomous systemsautoencoderbinary neural networks (BNNs)ternary neural networks (TNNs)EdgeAI |
| spellingShingle | Nitheesh Kumar Manjunath Aidin Shiri Morteza Hosseini Bharat Prakash Nicholas R. Waytowich Tinoosh Mohsenin An Energy Efficient EdgeAI Autoencoder Accelerator for Reinforcement Learning IEEE Open Journal of Circuits and Systems Reinforcement learning autonomous systems autoencoder binary neural networks (BNNs) ternary neural networks (TNNs) EdgeAI |
| title | An Energy Efficient EdgeAI Autoencoder Accelerator for Reinforcement Learning |
| title_full | An Energy Efficient EdgeAI Autoencoder Accelerator for Reinforcement Learning |
| title_fullStr | An Energy Efficient EdgeAI Autoencoder Accelerator for Reinforcement Learning |
| title_full_unstemmed | An Energy Efficient EdgeAI Autoencoder Accelerator for Reinforcement Learning |
| title_short | An Energy Efficient EdgeAI Autoencoder Accelerator for Reinforcement Learning |
| title_sort | energy efficient edgeai autoencoder accelerator for reinforcement learning |
| topic | Reinforcement learning autonomous systems autoencoder binary neural networks (BNNs) ternary neural networks (TNNs) EdgeAI |
| url | https://ieeexplore.ieee.org/document/9335309/ |
| work_keys_str_mv | AT nitheeshkumarmanjunath anenergyefficientedgeaiautoencoderacceleratorforreinforcementlearning AT aidinshiri anenergyefficientedgeaiautoencoderacceleratorforreinforcementlearning AT mortezahosseini anenergyefficientedgeaiautoencoderacceleratorforreinforcementlearning AT bharatprakash anenergyefficientedgeaiautoencoderacceleratorforreinforcementlearning AT nicholasrwaytowich anenergyefficientedgeaiautoencoderacceleratorforreinforcementlearning AT tinooshmohsenin anenergyefficientedgeaiautoencoderacceleratorforreinforcementlearning AT nitheeshkumarmanjunath energyefficientedgeaiautoencoderacceleratorforreinforcementlearning AT aidinshiri energyefficientedgeaiautoencoderacceleratorforreinforcementlearning AT mortezahosseini energyefficientedgeaiautoencoderacceleratorforreinforcementlearning AT bharatprakash energyefficientedgeaiautoencoderacceleratorforreinforcementlearning AT nicholasrwaytowich energyefficientedgeaiautoencoderacceleratorforreinforcementlearning AT tinooshmohsenin energyefficientedgeaiautoencoderacceleratorforreinforcementlearning |