Slimmer CNNs Through Feature Approximation and Kernel Size Reduction
Convolutional Neural Networks (CNNs) have been shown to achieve state of the art results on several image processing tasks such as classification, localization, and segmentation. Convolutional and fully connected layers form the building blocks of these networks. The convolution layers are responsib...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Open Journal of Circuits and Systems |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10173478/ |
| _version_ | 1827379775920930816 |
|---|---|
| author | Dara Nagaraju Nitin Chandrachoodan |
| author_facet | Dara Nagaraju Nitin Chandrachoodan |
| author_sort | Dara Nagaraju |
| collection | DOAJ |
| description | Convolutional Neural Networks (CNNs) have been shown to achieve state of the art results on several image processing tasks such as classification, localization, and segmentation. Convolutional and fully connected layers form the building blocks of these networks. The convolution layers are responsible for the majority of the computations even though they have fewer parameters. As inference is used much more than training (which happens only once), it is important to reduce the computations of the network for this phase. This work presents a systematic procedure to trim CNNs by identifying the least important features in the convolution layers and replacing them either with approximations or kernels of reduced size. We also propose an algorithm to integrate the lower kernel approximation technique for a given accuracy budget. We show that using the linear approximation method can achieve a 15% – 80% savings with a median of 52% reduction while the lower kernel method can achieve 33% – 95% reduction with a median of 65% in the required number of computations with only a marginal 1% loss in accuracy across several benchmark datasets. We have also demonstrated the proposed methods on VGG-16 architecture for various datasets. On VGG-16 we have achieved 4.2% - 45% savings in MAC computations (with a median of 18.5%) with only a marginal 0.5% loss in accuracy. We also show how an existing hardware accelerator for DNNs (DianNao) can be modified with low added complexity to take advantage of the kernel approximations, and estimate the speedups that can be obtained in such a way on custom embedded hardware. |
| first_indexed | 2024-03-08T13:22:59Z |
| format | Article |
| id | doaj.art-c880e10e056d4eee9407eed60ce8f51e |
| institution | Directory Open Access Journal |
| issn | 2644-1225 |
| language | English |
| last_indexed | 2024-03-08T13:22:59Z |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Open Journal of Circuits and Systems |
| spelling | doaj.art-c880e10e056d4eee9407eed60ce8f51e2024-01-18T00:01:23ZengIEEEIEEE Open Journal of Circuits and Systems2644-12252023-01-01418820210.1109/OJCAS.2023.329210910173478Slimmer CNNs Through Feature Approximation and Kernel Size ReductionDara Nagaraju0https://orcid.org/0000-0002-2221-5040Nitin Chandrachoodan1https://orcid.org/0000-0002-9258-7317Department of Electrical Engineering, Indian Institute of Technology Madras, Chennai, IndiaDepartment of Electrical Engineering, Indian Institute of Technology Madras, Chennai, IndiaConvolutional Neural Networks (CNNs) have been shown to achieve state of the art results on several image processing tasks such as classification, localization, and segmentation. Convolutional and fully connected layers form the building blocks of these networks. The convolution layers are responsible for the majority of the computations even though they have fewer parameters. As inference is used much more than training (which happens only once), it is important to reduce the computations of the network for this phase. This work presents a systematic procedure to trim CNNs by identifying the least important features in the convolution layers and replacing them either with approximations or kernels of reduced size. We also propose an algorithm to integrate the lower kernel approximation technique for a given accuracy budget. We show that using the linear approximation method can achieve a 15% – 80% savings with a median of 52% reduction while the lower kernel method can achieve 33% – 95% reduction with a median of 65% in the required number of computations with only a marginal 1% loss in accuracy across several benchmark datasets. We have also demonstrated the proposed methods on VGG-16 architecture for various datasets. On VGG-16 we have achieved 4.2% - 45% savings in MAC computations (with a median of 18.5%) with only a marginal 0.5% loss in accuracy. We also show how an existing hardware accelerator for DNNs (DianNao) can be modified with low added complexity to take advantage of the kernel approximations, and estimate the speedups that can be obtained in such a way on custom embedded hardware.https://ieeexplore.ieee.org/document/10173478/Feature approximationfeature orderingsystem level optimizationsubset selection |
| spellingShingle | Dara Nagaraju Nitin Chandrachoodan Slimmer CNNs Through Feature Approximation and Kernel Size Reduction IEEE Open Journal of Circuits and Systems Feature approximation feature ordering system level optimization subset selection |
| title | Slimmer CNNs Through Feature Approximation and Kernel Size Reduction |
| title_full | Slimmer CNNs Through Feature Approximation and Kernel Size Reduction |
| title_fullStr | Slimmer CNNs Through Feature Approximation and Kernel Size Reduction |
| title_full_unstemmed | Slimmer CNNs Through Feature Approximation and Kernel Size Reduction |
| title_short | Slimmer CNNs Through Feature Approximation and Kernel Size Reduction |
| title_sort | slimmer cnns through feature approximation and kernel size reduction |
| topic | Feature approximation feature ordering system level optimization subset selection |
| url | https://ieeexplore.ieee.org/document/10173478/ |
| work_keys_str_mv | AT daranagaraju slimmercnnsthroughfeatureapproximationandkernelsizereduction AT nitinchandrachoodan slimmercnnsthroughfeatureapproximationandkernelsizereduction |