Accelerating Super-Resolution Network Inference via Sensitivity-Based Weight Sparsity Allocation
Weight sparsification has been extensively studied in image classification and object detection to accelerate network inference. However, for image generation tasks, such as image super-resolution, forcing some weights to zeros is a non-trivial task that typically causes significant degradation in r...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10298064/ |
| _version_ | 1797632784080044032 |
|---|---|
| author | Tuan Nghia Nguyen Xuan Truong Nguyen Kyujoong Lee Hyuk-Jae Lee |
| author_facet | Tuan Nghia Nguyen Xuan Truong Nguyen Kyujoong Lee Hyuk-Jae Lee |
| author_sort | Tuan Nghia Nguyen |
| collection | DOAJ |
| description | Weight sparsification has been extensively studied in image classification and object detection to accelerate network inference. However, for image generation tasks, such as image super-resolution, forcing some weights to zeros is a non-trivial task that typically causes significant degradation in restoration quality, that is, peak signal-to-noise (PSNR). In this study, we first introduce a sensitivity metric to measure PSNR degradation for layer-wise sparsity changes and observe that the sensitivities vary significantly across network layers. We demonstrate that a uniform sparsity allocation method generally causes a non-negligible decrease in accuracy, that is, approximately 0.17 dB, for 65% of the zero weights. In addition, finding an optimal solution to the sparsity allocation problem is not feasible because the design space is exponential with respect to the number of weights and layers. To address this problem, we proposed a simple yet effective sparsity allocation method based on layer-wise sensitivity. The experimental results demonstrate that the proposed method achieves up to 35% computation reduction with an average accuracy drop of 0.02 dB varying between 0.01 to 0.04 dB across the well-known datasets Set5, Set14, B100, and Urban100. Moreover, when integrated with the activation sparsity SMSR, the proposed method reduced the computation by 46% on average. |
| first_indexed | 2024-03-11T11:42:40Z |
| format | Article |
| id | doaj.art-999abb0cde984734a2cb383c059f3fd5 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-03-11T11:42:40Z |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-999abb0cde984734a2cb383c059f3fd52023-11-10T00:00:52ZengIEEEIEEE Access2169-35362023-01-011112296212297310.1109/ACCESS.2023.332817310298064Accelerating Super-Resolution Network Inference via Sensitivity-Based Weight Sparsity AllocationTuan Nghia Nguyen0https://orcid.org/0000-0003-2130-4364Xuan Truong Nguyen1https://orcid.org/0000-0002-7527-6971Kyujoong Lee2https://orcid.org/0000-0002-3080-3010Hyuk-Jae Lee3https://orcid.org/0000-0001-8895-9117Inter-University Semiconductor Research Center, Seoul National University, Seoul, Republic of KoreaDepartment of Next-Generation Semiconductor Convergence and Open Sharing Systems (COSS), Seoul National University, Seoul, Republic of KoreaSchool of AI Convergence, Sungshin Women’s University, Seoul, Republic of KoreaInter-University Semiconductor Research Center, Seoul National University, Seoul, Republic of KoreaWeight sparsification has been extensively studied in image classification and object detection to accelerate network inference. However, for image generation tasks, such as image super-resolution, forcing some weights to zeros is a non-trivial task that typically causes significant degradation in restoration quality, that is, peak signal-to-noise (PSNR). In this study, we first introduce a sensitivity metric to measure PSNR degradation for layer-wise sparsity changes and observe that the sensitivities vary significantly across network layers. We demonstrate that a uniform sparsity allocation method generally causes a non-negligible decrease in accuracy, that is, approximately 0.17 dB, for 65% of the zero weights. In addition, finding an optimal solution to the sparsity allocation problem is not feasible because the design space is exponential with respect to the number of weights and layers. To address this problem, we proposed a simple yet effective sparsity allocation method based on layer-wise sensitivity. The experimental results demonstrate that the proposed method achieves up to 35% computation reduction with an average accuracy drop of 0.02 dB varying between 0.01 to 0.04 dB across the well-known datasets Set5, Set14, B100, and Urban100. Moreover, when integrated with the activation sparsity SMSR, the proposed method reduced the computation by 46% on average.https://ieeexplore.ieee.org/document/10298064/Computational reductionefficient neural networksparsity |
| spellingShingle | Tuan Nghia Nguyen Xuan Truong Nguyen Kyujoong Lee Hyuk-Jae Lee Accelerating Super-Resolution Network Inference via Sensitivity-Based Weight Sparsity Allocation IEEE Access Computational reduction efficient neural network sparsity |
| title | Accelerating Super-Resolution Network Inference via Sensitivity-Based Weight Sparsity Allocation |
| title_full | Accelerating Super-Resolution Network Inference via Sensitivity-Based Weight Sparsity Allocation |
| title_fullStr | Accelerating Super-Resolution Network Inference via Sensitivity-Based Weight Sparsity Allocation |
| title_full_unstemmed | Accelerating Super-Resolution Network Inference via Sensitivity-Based Weight Sparsity Allocation |
| title_short | Accelerating Super-Resolution Network Inference via Sensitivity-Based Weight Sparsity Allocation |
| title_sort | accelerating super resolution network inference via sensitivity based weight sparsity allocation |
| topic | Computational reduction efficient neural network sparsity |
| url | https://ieeexplore.ieee.org/document/10298064/ |
| work_keys_str_mv | AT tuannghianguyen acceleratingsuperresolutionnetworkinferenceviasensitivitybasedweightsparsityallocation AT xuantruongnguyen acceleratingsuperresolutionnetworkinferenceviasensitivitybasedweightsparsityallocation AT kyujoonglee acceleratingsuperresolutionnetworkinferenceviasensitivitybasedweightsparsityallocation AT hyukjaelee acceleratingsuperresolutionnetworkinferenceviasensitivitybasedweightsparsityallocation |