Accurate Inference With Inaccurate RRAM Devices: A Joint Algorithm-Design Solution
Resistive random access memory (RRAM) is a promising technology for energy-efficient neuromorphic accelerators. However, when a pretrained deep neural network (DNN) model is programmed to an RRAM array for inference, the model suffers from accuracy degradation due to RRAM nonidealities, such as devi...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2020-01-01
|
| Series: | IEEE Journal on Exploratory Solid-State Computational Devices and Circuits |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9069242/ |
| _version_ | 1818331663862595584 |
|---|---|
| author | Gouranga Charan Abinash Mohanty Xiaocong Du Gokul Krishnan Rajiv V. Joshi Yu Cao |
| author_facet | Gouranga Charan Abinash Mohanty Xiaocong Du Gokul Krishnan Rajiv V. Joshi Yu Cao |
| author_sort | Gouranga Charan |
| collection | DOAJ |
| description | Resistive random access memory (RRAM) is a promising technology for energy-efficient neuromorphic accelerators. However, when a pretrained deep neural network (DNN) model is programmed to an RRAM array for inference, the model suffers from accuracy degradation due to RRAM nonidealities, such as device variations, quantization error, and stuck-at-faults. Previous solutions involving multiple read-verify-write (R-V-W) to the RRAM cells require cell-by-cell compensation and, thus, an excessive amount of processing time. In this article, we propose a joint algorithm-design solution to mitigate the accuracy degradation. We first leverage knowledge distillation (KD), where the model is trained with the RRAM nonidealities to increase the robustness of the model under device variations. Furthermore, we propose random sparse adaptation (RSA), which integrates a small on-chip memory with the main RRAM array for postmapping adaptation. Only the on-chip memory is updated to recover the inference accuracy. The joint algorithm-design solution achieves the state-of-the-art accuracy of 99.41% for MNIST (LeNet-5) and 91.86% for CIFAR-10 (VGG-16) with up to 5% parameters as overhead while providing a 15-150× speedup compared with R-V-W. |
| first_indexed | 2024-12-13T13:23:26Z |
| format | Article |
| id | doaj.art-ed9f39b1d39d4305b2941cd434d8fb3d |
| institution | Directory Open Access Journal |
| issn | 2329-9231 |
| language | English |
| last_indexed | 2024-12-13T13:23:26Z |
| publishDate | 2020-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Journal on Exploratory Solid-State Computational Devices and Circuits |
| spelling | doaj.art-ed9f39b1d39d4305b2941cd434d8fb3d2022-12-21T23:44:21ZengIEEEIEEE Journal on Exploratory Solid-State Computational Devices and Circuits2329-92312020-01-0161273510.1109/JXCDC.2020.29876059069242Accurate Inference With Inaccurate RRAM Devices: A Joint Algorithm-Design SolutionGouranga Charan0https://orcid.org/0000-0002-1335-0670Abinash Mohanty1https://orcid.org/0000-0003-3035-0645Xiaocong Du2https://orcid.org/0000-0002-1079-0347Gokul Krishnan3https://orcid.org/0000-0003-1813-1140Rajiv V. Joshi4Yu Cao5https://orcid.org/0000-0001-6968-1180School of Electrical, Computer and Energy Engineering, Arizona State University, Tempe, AZ, USASchool of Electrical, Computer and Energy Engineering, Arizona State University, Tempe, AZ, USASchool of Electrical, Computer and Energy Engineering, Arizona State University, Tempe, AZ, USASchool of Electrical, Computer and Energy Engineering, Arizona State University, Tempe, AZ, USAIBM Thomas J. Watson Research Center, Yorktown Heights, NY, USASchool of Electrical, Computer and Energy Engineering, Arizona State University, Tempe, AZ, USAResistive random access memory (RRAM) is a promising technology for energy-efficient neuromorphic accelerators. However, when a pretrained deep neural network (DNN) model is programmed to an RRAM array for inference, the model suffers from accuracy degradation due to RRAM nonidealities, such as device variations, quantization error, and stuck-at-faults. Previous solutions involving multiple read-verify-write (R-V-W) to the RRAM cells require cell-by-cell compensation and, thus, an excessive amount of processing time. In this article, we propose a joint algorithm-design solution to mitigate the accuracy degradation. We first leverage knowledge distillation (KD), where the model is trained with the RRAM nonidealities to increase the robustness of the model under device variations. Furthermore, we propose random sparse adaptation (RSA), which integrates a small on-chip memory with the main RRAM array for postmapping adaptation. Only the on-chip memory is updated to recover the inference accuracy. The joint algorithm-design solution achieves the state-of-the-art accuracy of 99.41% for MNIST (LeNet-5) and 91.86% for CIFAR-10 (VGG-16) with up to 5% parameters as overhead while providing a 15-150× speedup compared with R-V-W.https://ieeexplore.ieee.org/document/9069242/Convolution neural networksdevice nonidealitiesmodel robustnessneuromorphic computingrandom sparse adaptation (RSA)resistive random access memory (RRAM) |
| spellingShingle | Gouranga Charan Abinash Mohanty Xiaocong Du Gokul Krishnan Rajiv V. Joshi Yu Cao Accurate Inference With Inaccurate RRAM Devices: A Joint Algorithm-Design Solution IEEE Journal on Exploratory Solid-State Computational Devices and Circuits Convolution neural networks device nonidealities model robustness neuromorphic computing random sparse adaptation (RSA) resistive random access memory (RRAM) |
| title | Accurate Inference With Inaccurate RRAM Devices: A Joint Algorithm-Design Solution |
| title_full | Accurate Inference With Inaccurate RRAM Devices: A Joint Algorithm-Design Solution |
| title_fullStr | Accurate Inference With Inaccurate RRAM Devices: A Joint Algorithm-Design Solution |
| title_full_unstemmed | Accurate Inference With Inaccurate RRAM Devices: A Joint Algorithm-Design Solution |
| title_short | Accurate Inference With Inaccurate RRAM Devices: A Joint Algorithm-Design Solution |
| title_sort | accurate inference with inaccurate rram devices a joint algorithm design solution |
| topic | Convolution neural networks device nonidealities model robustness neuromorphic computing random sparse adaptation (RSA) resistive random access memory (RRAM) |
| url | https://ieeexplore.ieee.org/document/9069242/ |
| work_keys_str_mv | AT gourangacharan accurateinferencewithinaccuraterramdevicesajointalgorithmdesignsolution AT abinashmohanty accurateinferencewithinaccuraterramdevicesajointalgorithmdesignsolution AT xiaocongdu accurateinferencewithinaccuraterramdevicesajointalgorithmdesignsolution AT gokulkrishnan accurateinferencewithinaccuraterramdevicesajointalgorithmdesignsolution AT rajivvjoshi accurateinferencewithinaccuraterramdevicesajointalgorithmdesignsolution AT yucao accurateinferencewithinaccuraterramdevicesajointalgorithmdesignsolution |