Direct Gradient Calculation: Simple and Variation‐Tolerant On‐Chip Training Method for Neural Networks
On‐chip training of neural networks (NNs) is regarded as a promising training method for neuromorphic systems with analog synaptic devices. Herein, a novel on‐chip training method called direct gradient calculation (DGC) is proposed to substitute conventional backpropagation (BP). In this method, th...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Wiley
2021-08-01
|
| Series: | Advanced Intelligent Systems |
| Subjects: | |
| Online Access: | https://doi.org/10.1002/aisy.202100064 |
| _version_ | 1818738042366590976 |
|---|---|
| author | Hyungyo Kim Joon Hwang Dongseok Kwon Jangsaeng Kim Min-Kyu Park Jiseong Im Byung-Gook Park Jong-Ho Lee |
| author_facet | Hyungyo Kim Joon Hwang Dongseok Kwon Jangsaeng Kim Min-Kyu Park Jiseong Im Byung-Gook Park Jong-Ho Lee |
| author_sort | Hyungyo Kim |
| collection | DOAJ |
| description | On‐chip training of neural networks (NNs) is regarded as a promising training method for neuromorphic systems with analog synaptic devices. Herein, a novel on‐chip training method called direct gradient calculation (DGC) is proposed to substitute conventional backpropagation (BP). In this method, the gradients of a cost function with respect to the weights are calculated directly by sequentially applying a small temporal change to each weight and then measuring the change in cost value. DGC achieves a similar accuracy to that of BP while performing a handwritten digit classification task, validating its training feasibility. In particular, DGC can be applied to analog hardware‐based convolutional NNs (CNNs), which is considered to be a challenging task, enabling appropriate on‐chip training. A hybrid method is also proposed that efficiently combines DGC and BP for training CNNs, and the method achieves a similar accuracy to that of BP and DGC while enhancing the training speed. Furthermore, networks utilizing DGC maintain a higher level of accuracy than those using BP in the presence of variations in hardware (such as synaptic device conductance and neuron circuit component variations) while requiring fewer circuit components. |
| first_indexed | 2024-12-18T01:02:39Z |
| format | Article |
| id | doaj.art-2ad44c2088ed41dcb455687719aea898 |
| institution | Directory Open Access Journal |
| issn | 2640-4567 |
| language | English |
| last_indexed | 2024-12-18T01:02:39Z |
| publishDate | 2021-08-01 |
| publisher | Wiley |
| record_format | Article |
| series | Advanced Intelligent Systems |
| spelling | doaj.art-2ad44c2088ed41dcb455687719aea8982022-12-21T21:26:19ZengWileyAdvanced Intelligent Systems2640-45672021-08-0138n/an/a10.1002/aisy.202100064Direct Gradient Calculation: Simple and Variation‐Tolerant On‐Chip Training Method for Neural NetworksHyungyo Kim0Joon Hwang1Dongseok Kwon2Jangsaeng Kim3Min-Kyu Park4Jiseong Im5Byung-Gook Park6Jong-Ho Lee7Department of Electrical and Computer Engineering and Inter-University Semiconductor Research Center Seoul National University 1 Gwanak-ro, Gwanak-gu Seoul 08826 South KoreaDepartment of Electrical and Computer Engineering and Inter-University Semiconductor Research Center Seoul National University 1 Gwanak-ro, Gwanak-gu Seoul 08826 South KoreaDepartment of Electrical and Computer Engineering and Inter-University Semiconductor Research Center Seoul National University 1 Gwanak-ro, Gwanak-gu Seoul 08826 South KoreaDepartment of Electrical and Computer Engineering and Inter-University Semiconductor Research Center Seoul National University 1 Gwanak-ro, Gwanak-gu Seoul 08826 South KoreaDepartment of Electrical and Computer Engineering and Inter-University Semiconductor Research Center Seoul National University 1 Gwanak-ro, Gwanak-gu Seoul 08826 South KoreaDepartment of Electrical and Computer Engineering and Inter-University Semiconductor Research Center Seoul National University 1 Gwanak-ro, Gwanak-gu Seoul 08826 South KoreaDepartment of Electrical and Computer Engineering and Inter-University Semiconductor Research Center Seoul National University 1 Gwanak-ro, Gwanak-gu Seoul 08826 South KoreaDepartment of Electrical and Computer Engineering and Inter-University Semiconductor Research Center Seoul National University 1 Gwanak-ro, Gwanak-gu Seoul 08826 South KoreaOn‐chip training of neural networks (NNs) is regarded as a promising training method for neuromorphic systems with analog synaptic devices. Herein, a novel on‐chip training method called direct gradient calculation (DGC) is proposed to substitute conventional backpropagation (BP). In this method, the gradients of a cost function with respect to the weights are calculated directly by sequentially applying a small temporal change to each weight and then measuring the change in cost value. DGC achieves a similar accuracy to that of BP while performing a handwritten digit classification task, validating its training feasibility. In particular, DGC can be applied to analog hardware‐based convolutional NNs (CNNs), which is considered to be a challenging task, enabling appropriate on‐chip training. A hybrid method is also proposed that efficiently combines DGC and BP for training CNNs, and the method achieves a similar accuracy to that of BP and DGC while enhancing the training speed. Furthermore, networks utilizing DGC maintain a higher level of accuracy than those using BP in the presence of variations in hardware (such as synaptic device conductance and neuron circuit component variations) while requiring fewer circuit components.https://doi.org/10.1002/aisy.202100064analog synaptic devicesbackpropagationhardware-based neural networkson-chip training |
| spellingShingle | Hyungyo Kim Joon Hwang Dongseok Kwon Jangsaeng Kim Min-Kyu Park Jiseong Im Byung-Gook Park Jong-Ho Lee Direct Gradient Calculation: Simple and Variation‐Tolerant On‐Chip Training Method for Neural Networks Advanced Intelligent Systems analog synaptic devices backpropagation hardware-based neural networks on-chip training |
| title | Direct Gradient Calculation: Simple and Variation‐Tolerant On‐Chip Training Method for Neural Networks |
| title_full | Direct Gradient Calculation: Simple and Variation‐Tolerant On‐Chip Training Method for Neural Networks |
| title_fullStr | Direct Gradient Calculation: Simple and Variation‐Tolerant On‐Chip Training Method for Neural Networks |
| title_full_unstemmed | Direct Gradient Calculation: Simple and Variation‐Tolerant On‐Chip Training Method for Neural Networks |
| title_short | Direct Gradient Calculation: Simple and Variation‐Tolerant On‐Chip Training Method for Neural Networks |
| title_sort | direct gradient calculation simple and variation tolerant on chip training method for neural networks |
| topic | analog synaptic devices backpropagation hardware-based neural networks on-chip training |
| url | https://doi.org/10.1002/aisy.202100064 |
| work_keys_str_mv | AT hyungyokim directgradientcalculationsimpleandvariationtolerantonchiptrainingmethodforneuralnetworks AT joonhwang directgradientcalculationsimpleandvariationtolerantonchiptrainingmethodforneuralnetworks AT dongseokkwon directgradientcalculationsimpleandvariationtolerantonchiptrainingmethodforneuralnetworks AT jangsaengkim directgradientcalculationsimpleandvariationtolerantonchiptrainingmethodforneuralnetworks AT minkyupark directgradientcalculationsimpleandvariationtolerantonchiptrainingmethodforneuralnetworks AT jiseongim directgradientcalculationsimpleandvariationtolerantonchiptrainingmethodforneuralnetworks AT byunggookpark directgradientcalculationsimpleandvariationtolerantonchiptrainingmethodforneuralnetworks AT jongholee directgradientcalculationsimpleandvariationtolerantonchiptrainingmethodforneuralnetworks |