Stable Low-Rank CP Decomposition for Compression of Convolutional Neural Networks Based on Sensitivity
Modern convolutional neural networks (CNNs) play a crucial role in computer vision applications. The intricacy of the application scenarios and the growing dataset both significantly raise the complexity of CNNs. As a result, they are often overparameterized and have significant computational costs....
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-02-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/14/4/1491 |
| _version_ | 1797299025472389120 |
|---|---|
| author | Chenbin Yang Huiyi Liu |
| author_facet | Chenbin Yang Huiyi Liu |
| author_sort | Chenbin Yang |
| collection | DOAJ |
| description | Modern convolutional neural networks (CNNs) play a crucial role in computer vision applications. The intricacy of the application scenarios and the growing dataset both significantly raise the complexity of CNNs. As a result, they are often overparameterized and have significant computational costs. One potential solution for optimizing and compressing the CNNs is to replace convolutional layers with low-rank tensor decomposition. The most suitable technique for this is Canonical Polyadic (CP) decomposition. However, there are two primary issues with CP decomposition that lead to a significant loss in accuracy. Firstly, the selection of tensor ranks for CP decomposition is an unsolved issue. Secondly, degeneracy and instability are common problems in the CP decomposition of contractional tensors, which makes fine-tuning the compressed model difficult. In this study, a novel approach was proposed for compressing CNNs by using CP decomposition. The first step involves using the sensitivity of convolutional layers to determine the tensor ranks for CP decomposition effectively. Subsequently, to address the degeneracy issue and enhance the stability of the CP decomposition, two novel techniques were incorporated: optimization with sensitivity constraints and iterative fine-tuning based on sensitivity order. Finally, the proposed method was examined on common CNN structures for image classification tasks and demonstrated that it provides stable performance and significantly fewer reductions in classification accuracy. |
| first_indexed | 2024-03-07T22:43:33Z |
| format | Article |
| id | doaj.art-720aadfc957141379e5e0228fbb59313 |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-07T22:43:33Z |
| publishDate | 2024-02-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-720aadfc957141379e5e0228fbb593132024-02-23T15:06:10ZengMDPI AGApplied Sciences2076-34172024-02-01144149110.3390/app14041491Stable Low-Rank CP Decomposition for Compression of Convolutional Neural Networks Based on SensitivityChenbin Yang0Huiyi Liu1College of Computer and Information, Hohai University, Nanjing 211100, ChinaCollege of Computer and Information, Hohai University, Nanjing 211100, ChinaModern convolutional neural networks (CNNs) play a crucial role in computer vision applications. The intricacy of the application scenarios and the growing dataset both significantly raise the complexity of CNNs. As a result, they are often overparameterized and have significant computational costs. One potential solution for optimizing and compressing the CNNs is to replace convolutional layers with low-rank tensor decomposition. The most suitable technique for this is Canonical Polyadic (CP) decomposition. However, there are two primary issues with CP decomposition that lead to a significant loss in accuracy. Firstly, the selection of tensor ranks for CP decomposition is an unsolved issue. Secondly, degeneracy and instability are common problems in the CP decomposition of contractional tensors, which makes fine-tuning the compressed model difficult. In this study, a novel approach was proposed for compressing CNNs by using CP decomposition. The first step involves using the sensitivity of convolutional layers to determine the tensor ranks for CP decomposition effectively. Subsequently, to address the degeneracy issue and enhance the stability of the CP decomposition, two novel techniques were incorporated: optimization with sensitivity constraints and iterative fine-tuning based on sensitivity order. Finally, the proposed method was examined on common CNN structures for image classification tasks and demonstrated that it provides stable performance and significantly fewer reductions in classification accuracy.https://www.mdpi.com/2076-3417/14/4/1491convolutional neural networksmodel compressionCP decompositionrank selectionsensitivity |
| spellingShingle | Chenbin Yang Huiyi Liu Stable Low-Rank CP Decomposition for Compression of Convolutional Neural Networks Based on Sensitivity Applied Sciences convolutional neural networks model compression CP decomposition rank selection sensitivity |
| title | Stable Low-Rank CP Decomposition for Compression of Convolutional Neural Networks Based on Sensitivity |
| title_full | Stable Low-Rank CP Decomposition for Compression of Convolutional Neural Networks Based on Sensitivity |
| title_fullStr | Stable Low-Rank CP Decomposition for Compression of Convolutional Neural Networks Based on Sensitivity |
| title_full_unstemmed | Stable Low-Rank CP Decomposition for Compression of Convolutional Neural Networks Based on Sensitivity |
| title_short | Stable Low-Rank CP Decomposition for Compression of Convolutional Neural Networks Based on Sensitivity |
| title_sort | stable low rank cp decomposition for compression of convolutional neural networks based on sensitivity |
| topic | convolutional neural networks model compression CP decomposition rank selection sensitivity |
| url | https://www.mdpi.com/2076-3417/14/4/1491 |
| work_keys_str_mv | AT chenbinyang stablelowrankcpdecompositionforcompressionofconvolutionalneuralnetworksbasedonsensitivity AT huiyiliu stablelowrankcpdecompositionforcompressionofconvolutionalneuralnetworksbasedonsensitivity |