Gradual Channel Pruning While Training Using Feature Relevance Scores for Convolutional Neural Networks
The enormous inference cost of deep neural networks can be mitigated by network compression. Pruning connections is one of the predominant approaches used for network compression. However, existing pruning techniques suffer from one or more of the following limitations: 1) They increase the time and...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2020-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9199834/ |
| _version_ | 1819132647856668672 |
|---|---|
| author | Sai Aparna Aketi Sourjya Roy Anand Raghunathan Kaushik Roy |
| author_facet | Sai Aparna Aketi Sourjya Roy Anand Raghunathan Kaushik Roy |
| author_sort | Sai Aparna Aketi |
| collection | DOAJ |
| description | The enormous inference cost of deep neural networks can be mitigated by network compression. Pruning connections is one of the predominant approaches used for network compression. However, existing pruning techniques suffer from one or more of the following limitations: 1) They increase the time and energy consumed by the compute-heavy training stage due to the addition of the pruning and fine-tuning steps, 2) They prune layer-wise based on local information about a particular layer's statistics, ignoring the effect of error propagation through the network, 3) They lack an efficient means to determine the global importance of channels, 4) Due to the use of unstructured pruning, they may not lead to any energy advantage when implemented on mainstream platforms (GPUs and TPUs), requiring specialized hardware to reap the benefits. To address the above issues, we present a simple-yet-effective methodology for gradual channel pruning while training using a data-driven metric referred to as feature relevance score. The proposed technique eliminates the need for additional retraining by pruning the least important channels in a structured manner at fixed intervals during the regular training phase. Pruning is guided by feature relevance scores, which help in efficiently evaluating the contribution of each channel towards the discriminative power of the network. We demonstrate the effectiveness of the proposed methodology on architectures such as VGG and ResNet using datasets such as CIFAR-10, CIFAR-100, and ImageNet, and successfully achieve significant model compression while trading off less than 1% accuracy. |
| first_indexed | 2024-12-22T09:34:44Z |
| format | Article |
| id | doaj.art-10258216b1fd433ab0fd3644739a047d |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-22T09:34:44Z |
| publishDate | 2020-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-10258216b1fd433ab0fd3644739a047d2022-12-21T18:30:51ZengIEEEIEEE Access2169-35362020-01-01817192417193210.1109/ACCESS.2020.30249929199834Gradual Channel Pruning While Training Using Feature Relevance Scores for Convolutional Neural NetworksSai Aparna Aketi0https://orcid.org/0000-0003-3446-0243Sourjya Roy1Anand Raghunathan2Kaushik Roy3https://orcid.org/0000-0002-0735-9695School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USASchool of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USASchool of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USASchool of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USAThe enormous inference cost of deep neural networks can be mitigated by network compression. Pruning connections is one of the predominant approaches used for network compression. However, existing pruning techniques suffer from one or more of the following limitations: 1) They increase the time and energy consumed by the compute-heavy training stage due to the addition of the pruning and fine-tuning steps, 2) They prune layer-wise based on local information about a particular layer's statistics, ignoring the effect of error propagation through the network, 3) They lack an efficient means to determine the global importance of channels, 4) Due to the use of unstructured pruning, they may not lead to any energy advantage when implemented on mainstream platforms (GPUs and TPUs), requiring specialized hardware to reap the benefits. To address the above issues, we present a simple-yet-effective methodology for gradual channel pruning while training using a data-driven metric referred to as feature relevance score. The proposed technique eliminates the need for additional retraining by pruning the least important channels in a structured manner at fixed intervals during the regular training phase. Pruning is guided by feature relevance scores, which help in efficiently evaluating the contribution of each channel towards the discriminative power of the network. We demonstrate the effectiveness of the proposed methodology on architectures such as VGG and ResNet using datasets such as CIFAR-10, CIFAR-100, and ImageNet, and successfully achieve significant model compression while trading off less than 1% accuracy.https://ieeexplore.ieee.org/document/9199834/Convolutional neural networks (CNNs)deep learningefficient deep learningneural networksmodel architecturemodel compression |
| spellingShingle | Sai Aparna Aketi Sourjya Roy Anand Raghunathan Kaushik Roy Gradual Channel Pruning While Training Using Feature Relevance Scores for Convolutional Neural Networks IEEE Access Convolutional neural networks (CNNs) deep learning efficient deep learning neural networks model architecture model compression |
| title | Gradual Channel Pruning While Training Using Feature Relevance Scores for Convolutional Neural Networks |
| title_full | Gradual Channel Pruning While Training Using Feature Relevance Scores for Convolutional Neural Networks |
| title_fullStr | Gradual Channel Pruning While Training Using Feature Relevance Scores for Convolutional Neural Networks |
| title_full_unstemmed | Gradual Channel Pruning While Training Using Feature Relevance Scores for Convolutional Neural Networks |
| title_short | Gradual Channel Pruning While Training Using Feature Relevance Scores for Convolutional Neural Networks |
| title_sort | gradual channel pruning while training using feature relevance scores for convolutional neural networks |
| topic | Convolutional neural networks (CNNs) deep learning efficient deep learning neural networks model architecture model compression |
| url | https://ieeexplore.ieee.org/document/9199834/ |
| work_keys_str_mv | AT saiaparnaaketi gradualchannelpruningwhiletrainingusingfeaturerelevancescoresforconvolutionalneuralnetworks AT sourjyaroy gradualchannelpruningwhiletrainingusingfeaturerelevancescoresforconvolutionalneuralnetworks AT anandraghunathan gradualchannelpruningwhiletrainingusingfeaturerelevancescoresforconvolutionalneuralnetworks AT kaushikroy gradualchannelpruningwhiletrainingusingfeaturerelevancescoresforconvolutionalneuralnetworks |