An Energy-Efficient Edge Computing Paradigm for Convolution-Based Image Upsampling
State-of-the-art deep learning solutions for image upsampling are currently trained using either resize or sub-pixel convolution to learn kernels that generate high fidelity images with minimal artifacts. However, performing inference with these learned convolution kernels requires memory-intensive...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2021-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9592768/ |
| _version_ | 1819035616968441856 |
|---|---|
| author | Ian Colbert Kenneth Kreutz-Delgado Srinjoy Das |
| author_facet | Ian Colbert Kenneth Kreutz-Delgado Srinjoy Das |
| author_sort | Ian Colbert |
| collection | DOAJ |
| description | State-of-the-art deep learning solutions for image upsampling are currently trained using either resize or sub-pixel convolution to learn kernels that generate high fidelity images with minimal artifacts. However, performing inference with these learned convolution kernels requires memory-intensive feature map transformations that dominate time and energy costs in real-time applications. To alleviate this pressure on memory bandwidth, we propose a novel energy-efficient edge computing paradigm that confines the use of resize or sub-pixel convolution to training in the cloud by transforming learned convolution kernels to deconvolution kernels before deploying them for inference as a functionally equivalent deconvolution. These kernel transformations, intended as a one-time cost when shifting from training to inference, enable a systems designer to use each algorithm in their optimal context by preserving the image fidelity learned when training in the cloud while minimizing data transfer penalties during inference at the edge. We compare the inference properties of these convolution-based image upsampling algorithms and introduce a novel deconvolution inference algorithm, which we refer to as REVD2. To demonstrate the benefits of our approach, we upsample images selected from the BSD300 dataset using a pre-trained single-image super resolution network provided by the PyTorch model zoo. Using quantitative models of incurred time and energy costs to analyze this deep neural network, we estimate that using REVD2 for inference at the edge improves system latency by <inline-formula> <tex-math notation="LaTeX">$2.1\times $ </tex-math></inline-formula> or <inline-formula> <tex-math notation="LaTeX">$2.8\times $ </tex-math></inline-formula> and energy efficiency by <inline-formula> <tex-math notation="LaTeX">$2.1\times $ </tex-math></inline-formula> or <inline-formula> <tex-math notation="LaTeX">$2.7\times $ </tex-math></inline-formula> when respectively compared to sub-pixel or resize convolution counterparts. |
| first_indexed | 2024-12-21T07:52:28Z |
| format | Article |
| id | doaj.art-6918181698ef4588b0398f020c59e7d9 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-21T07:52:28Z |
| publishDate | 2021-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-6918181698ef4588b0398f020c59e7d92022-12-21T19:11:02ZengIEEEIEEE Access2169-35362021-01-01914796714798410.1109/ACCESS.2021.31239389592768An Energy-Efficient Edge Computing Paradigm for Convolution-Based Image UpsamplingIan Colbert0https://orcid.org/0000-0002-1669-5519Kenneth Kreutz-Delgado1https://orcid.org/0000-0001-8135-6823Srinjoy Das2https://orcid.org/0000-0003-3821-8112Department of Electrical and Computer Engineering, University of California at San Diego, San Diego, CA, USADepartment of Electrical and Computer Engineering, University of California at San Diego, San Diego, CA, USASchool of Mathematical and Data Sciences, West Virginia University, Morgantown, WV, USAState-of-the-art deep learning solutions for image upsampling are currently trained using either resize or sub-pixel convolution to learn kernels that generate high fidelity images with minimal artifacts. However, performing inference with these learned convolution kernels requires memory-intensive feature map transformations that dominate time and energy costs in real-time applications. To alleviate this pressure on memory bandwidth, we propose a novel energy-efficient edge computing paradigm that confines the use of resize or sub-pixel convolution to training in the cloud by transforming learned convolution kernels to deconvolution kernels before deploying them for inference as a functionally equivalent deconvolution. These kernel transformations, intended as a one-time cost when shifting from training to inference, enable a systems designer to use each algorithm in their optimal context by preserving the image fidelity learned when training in the cloud while minimizing data transfer penalties during inference at the edge. We compare the inference properties of these convolution-based image upsampling algorithms and introduce a novel deconvolution inference algorithm, which we refer to as REVD2. To demonstrate the benefits of our approach, we upsample images selected from the BSD300 dataset using a pre-trained single-image super resolution network provided by the PyTorch model zoo. Using quantitative models of incurred time and energy costs to analyze this deep neural network, we estimate that using REVD2 for inference at the edge improves system latency by <inline-formula> <tex-math notation="LaTeX">$2.1\times $ </tex-math></inline-formula> or <inline-formula> <tex-math notation="LaTeX">$2.8\times $ </tex-math></inline-formula> and energy efficiency by <inline-formula> <tex-math notation="LaTeX">$2.1\times $ </tex-math></inline-formula> or <inline-formula> <tex-math notation="LaTeX">$2.7\times $ </tex-math></inline-formula> when respectively compared to sub-pixel or resize convolution counterparts.https://ieeexplore.ieee.org/document/9592768/Computer visiondeconvolutiondeep learningedge computingenergy efficiencyimage upsampling |
| spellingShingle | Ian Colbert Kenneth Kreutz-Delgado Srinjoy Das An Energy-Efficient Edge Computing Paradigm for Convolution-Based Image Upsampling IEEE Access Computer vision deconvolution deep learning edge computing energy efficiency image upsampling |
| title | An Energy-Efficient Edge Computing Paradigm for Convolution-Based Image Upsampling |
| title_full | An Energy-Efficient Edge Computing Paradigm for Convolution-Based Image Upsampling |
| title_fullStr | An Energy-Efficient Edge Computing Paradigm for Convolution-Based Image Upsampling |
| title_full_unstemmed | An Energy-Efficient Edge Computing Paradigm for Convolution-Based Image Upsampling |
| title_short | An Energy-Efficient Edge Computing Paradigm for Convolution-Based Image Upsampling |
| title_sort | energy efficient edge computing paradigm for convolution based image upsampling |
| topic | Computer vision deconvolution deep learning edge computing energy efficiency image upsampling |
| url | https://ieeexplore.ieee.org/document/9592768/ |
| work_keys_str_mv | AT iancolbert anenergyefficientedgecomputingparadigmforconvolutionbasedimageupsampling AT kennethkreutzdelgado anenergyefficientedgecomputingparadigmforconvolutionbasedimageupsampling AT srinjoydas anenergyefficientedgecomputingparadigmforconvolutionbasedimageupsampling AT iancolbert energyefficientedgecomputingparadigmforconvolutionbasedimageupsampling AT kennethkreutzdelgado energyefficientedgecomputingparadigmforconvolutionbasedimageupsampling AT srinjoydas energyefficientedgecomputingparadigmforconvolutionbasedimageupsampling |