Smooth Group <i>L</i><sub>1/2</sub> Regularization for Pruning Convolutional Neural Networks
In this paper, a novel smooth group <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><msub><mi>L</mi><mrow><mn>1</mn><mo>/</mo><mn>2</mn></mrow></msu...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-01-01
|
| Series: | Symmetry |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2073-8994/14/1/154 |
| _version_ | 1797490065090281472 |
|---|---|
| author | Yuan Bao Zhaobin Liu Zhongxuan Luo Sibo Yang |
| author_facet | Yuan Bao Zhaobin Liu Zhongxuan Luo Sibo Yang |
| author_sort | Yuan Bao |
| collection | DOAJ |
| description | In this paper, a novel smooth group <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><msub><mi>L</mi><mrow><mn>1</mn><mo>/</mo><mn>2</mn></mrow></msub></semantics></math></inline-formula> (<inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>S</mi><mi>G</mi><msub><mi>L</mi><mrow><mn>1</mn><mo>/</mo><mn>2</mn></mrow></msub></mrow></semantics></math></inline-formula>) regularization method is proposed for pruning hidden nodes of the fully connected layer in convolution neural networks. Usually, the selection of nodes and weights is based on experience, and the convolution filter is symmetric in the convolution neural network. The main contribution of <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>S</mi><mi>G</mi><msub><mi>L</mi><mrow><mn>1</mn><mo>/</mo><mn>2</mn></mrow></msub></mrow></semantics></math></inline-formula> is to try to approximate the weights to 0 at the group level. Therefore, we will be able to prune the hidden node if the corresponding weights are all close to 0. Furthermore, the feasibility analysis of this new method is carried out under some reasonable assumptions due to the smooth function. The numerical results demonstrate the superiority of the <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>S</mi><mi>G</mi><msub><mi>L</mi><mrow><mn>1</mn><mo>/</mo><mn>2</mn></mrow></msub></mrow></semantics></math></inline-formula> method with respect to sparsity, without damaging the classification performance. |
| first_indexed | 2024-03-10T00:25:36Z |
| format | Article |
| id | doaj.art-4a9ca79519044c8eb4608f616e68a0a7 |
| institution | Directory Open Access Journal |
| issn | 2073-8994 |
| language | English |
| last_indexed | 2024-03-10T00:25:36Z |
| publishDate | 2022-01-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Symmetry |
| spelling | doaj.art-4a9ca79519044c8eb4608f616e68a0a72023-11-23T15:34:20ZengMDPI AGSymmetry2073-89942022-01-0114115410.3390/sym14010154Smooth Group <i>L</i><sub>1/2</sub> Regularization for Pruning Convolutional Neural NetworksYuan Bao0Zhaobin Liu1Zhongxuan Luo2Sibo Yang3School of Information Science and Technology, Dalian Maritime University, Dalian 116026, ChinaSchool of Information Science and Technology, Dalian Maritime University, Dalian 116026, ChinaSchool of Software, Dalian University of Technology, Dalian 116620, ChinaSchool of Science, Dalian Maritime University, Dalian 116026, ChinaIn this paper, a novel smooth group <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><msub><mi>L</mi><mrow><mn>1</mn><mo>/</mo><mn>2</mn></mrow></msub></semantics></math></inline-formula> (<inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>S</mi><mi>G</mi><msub><mi>L</mi><mrow><mn>1</mn><mo>/</mo><mn>2</mn></mrow></msub></mrow></semantics></math></inline-formula>) regularization method is proposed for pruning hidden nodes of the fully connected layer in convolution neural networks. Usually, the selection of nodes and weights is based on experience, and the convolution filter is symmetric in the convolution neural network. The main contribution of <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>S</mi><mi>G</mi><msub><mi>L</mi><mrow><mn>1</mn><mo>/</mo><mn>2</mn></mrow></msub></mrow></semantics></math></inline-formula> is to try to approximate the weights to 0 at the group level. Therefore, we will be able to prune the hidden node if the corresponding weights are all close to 0. Furthermore, the feasibility analysis of this new method is carried out under some reasonable assumptions due to the smooth function. The numerical results demonstrate the superiority of the <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi>S</mi><mi>G</mi><msub><mi>L</mi><mrow><mn>1</mn><mo>/</mo><mn>2</mn></mrow></msub></mrow></semantics></math></inline-formula> method with respect to sparsity, without damaging the classification performance.https://www.mdpi.com/2073-8994/14/1/154convolutional neural network (CNN)fully connected layersmooth group <i>L</i><sub>1/2</sub> regularizationsparsity |
| spellingShingle | Yuan Bao Zhaobin Liu Zhongxuan Luo Sibo Yang Smooth Group <i>L</i><sub>1/2</sub> Regularization for Pruning Convolutional Neural Networks Symmetry convolutional neural network (CNN) fully connected layer smooth group <i>L</i><sub>1/2</sub> regularization sparsity |
| title | Smooth Group <i>L</i><sub>1/2</sub> Regularization for Pruning Convolutional Neural Networks |
| title_full | Smooth Group <i>L</i><sub>1/2</sub> Regularization for Pruning Convolutional Neural Networks |
| title_fullStr | Smooth Group <i>L</i><sub>1/2</sub> Regularization for Pruning Convolutional Neural Networks |
| title_full_unstemmed | Smooth Group <i>L</i><sub>1/2</sub> Regularization for Pruning Convolutional Neural Networks |
| title_short | Smooth Group <i>L</i><sub>1/2</sub> Regularization for Pruning Convolutional Neural Networks |
| title_sort | smooth group i l i sub 1 2 sub regularization for pruning convolutional neural networks |
| topic | convolutional neural network (CNN) fully connected layer smooth group <i>L</i><sub>1/2</sub> regularization sparsity |
| url | https://www.mdpi.com/2073-8994/14/1/154 |
| work_keys_str_mv | AT yuanbao smoothgroupilisub12subregularizationforpruningconvolutionalneuralnetworks AT zhaobinliu smoothgroupilisub12subregularizationforpruningconvolutionalneuralnetworks AT zhongxuanluo smoothgroupilisub12subregularizationforpruningconvolutionalneuralnetworks AT siboyang smoothgroupilisub12subregularizationforpruningconvolutionalneuralnetworks |