Multi-Frame Depth Super-Resolution for ToF Sensor With Total Variation Regularized L1 Function

In this paper, we propose a multi-frame depth super-resolution (SR) method based on L<sub>1</sub> data fidelity with the total variation regularization (TV-L<sub>1</sub>) model. The majority of time-of-flight (ToF) sensors exhibit limited spatial resolution compared to RGB se...

Full description

Bibliographic Details
Main Authors: Jonghyun Kim, Jaeduk Han, Moon Gi Kang
Format: Article
Language:English
Published: IEEE 2020-01-01
Series:IEEE Access
Subjects:
Online Access:https://ieeexplore.ieee.org/document/9189817/
_version_ 1818616710871121920
author Jonghyun Kim
Jaeduk Han
Moon Gi Kang
author_facet Jonghyun Kim
Jaeduk Han
Moon Gi Kang
author_sort Jonghyun Kim
collection DOAJ
description In this paper, we propose a multi-frame depth super-resolution (SR) method based on L<sub>1</sub> data fidelity with the total variation regularization (TV-L<sub>1</sub>) model. The majority of time-of-flight (ToF) sensors exhibit limited spatial resolution compared to RGB sensors and the improvement of the depth image resolution is an inherently ill-posed problem. To overcome this under-determined problem, the solution space is limited by the regularization term through prior knowledge and the data fidelity term using statistical information of the noise. Firstly, the statistical characteristics of ToF depth images are analyzed to specify the appropriate observation model. Thereafter, the objective function for multi-frame depth SR based on the TV-L<sub>1</sub> model is designed by considering the prior knowledge of the depth images. This approach enables the sharpness of the edges to be preserved and the noise amplification to be suppressed simultaneously. Furthermore, an efficient solver based on half-quadratic splitting is proposed. The algorithm minimizes the objective function for the multi-frame SR problem consisting of the TV regularization term and L<sub>1</sub> data fidelity term. The proposed method is verified on a synthetic dataset and real-world data acquired from a ToF sensor. The experimental results demonstrate that the proposed method can substantially reconstruct high-resolution depth images in terms of preserving sharp depth discontinuities, without any obvious artifacts, and can increase robustness to noise.
first_indexed 2024-12-16T16:54:08Z
format Article
id doaj.art-030e8c10944448f3be696be56109c540
institution Directory Open Access Journal
issn 2169-3536
language English
last_indexed 2024-12-16T16:54:08Z
publishDate 2020-01-01
publisher IEEE
record_format Article
series IEEE Access
spelling doaj.art-030e8c10944448f3be696be56109c5402022-12-21T22:23:55ZengIEEEIEEE Access2169-35362020-01-01816581016582610.1109/ACCESS.2020.30229109189817Multi-Frame Depth Super-Resolution for ToF Sensor With Total Variation Regularized L1 FunctionJonghyun Kim0https://orcid.org/0000-0002-3788-0211Jaeduk Han1https://orcid.org/0000-0001-8798-1275Moon Gi Kang2https://orcid.org/0000-0002-5771-929XSchool of Electrical and Electronic Engineering, Yonsei University, Seoul, South KoreaSchool of Electrical and Electronic Engineering, Yonsei University, Seoul, South KoreaSchool of Electrical and Electronic Engineering, Yonsei University, Seoul, South KoreaIn this paper, we propose a multi-frame depth super-resolution (SR) method based on L<sub>1</sub> data fidelity with the total variation regularization (TV-L<sub>1</sub>) model. The majority of time-of-flight (ToF) sensors exhibit limited spatial resolution compared to RGB sensors and the improvement of the depth image resolution is an inherently ill-posed problem. To overcome this under-determined problem, the solution space is limited by the regularization term through prior knowledge and the data fidelity term using statistical information of the noise. Firstly, the statistical characteristics of ToF depth images are analyzed to specify the appropriate observation model. Thereafter, the objective function for multi-frame depth SR based on the TV-L<sub>1</sub> model is designed by considering the prior knowledge of the depth images. This approach enables the sharpness of the edges to be preserved and the noise amplification to be suppressed simultaneously. Furthermore, an efficient solver based on half-quadratic splitting is proposed. The algorithm minimizes the objective function for the multi-frame SR problem consisting of the TV regularization term and L<sub>1</sub> data fidelity term. The proposed method is verified on a synthetic dataset and real-world data acquired from a ToF sensor. The experimental results demonstrate that the proposed method can substantially reconstruct high-resolution depth images in terms of preserving sharp depth discontinuities, without any obvious artifacts, and can increase robustness to noise.https://ieeexplore.ieee.org/document/9189817/Depth super-resolutiontime-of-flight sensormulti-frame super-resolutiontotal variation regularizationL1 data fidelity
spellingShingle Jonghyun Kim
Jaeduk Han
Moon Gi Kang
Multi-Frame Depth Super-Resolution for ToF Sensor With Total Variation Regularized L1 Function
IEEE Access
Depth super-resolution
time-of-flight sensor
multi-frame super-resolution
total variation regularization
L1 data fidelity
title Multi-Frame Depth Super-Resolution for ToF Sensor With Total Variation Regularized L1 Function
title_full Multi-Frame Depth Super-Resolution for ToF Sensor With Total Variation Regularized L1 Function
title_fullStr Multi-Frame Depth Super-Resolution for ToF Sensor With Total Variation Regularized L1 Function
title_full_unstemmed Multi-Frame Depth Super-Resolution for ToF Sensor With Total Variation Regularized L1 Function
title_short Multi-Frame Depth Super-Resolution for ToF Sensor With Total Variation Regularized L1 Function
title_sort multi frame depth super resolution for tof sensor with total variation regularized l1 function
topic Depth super-resolution
time-of-flight sensor
multi-frame super-resolution
total variation regularization
L1 data fidelity
url https://ieeexplore.ieee.org/document/9189817/
work_keys_str_mv AT jonghyunkim multiframedepthsuperresolutionfortofsensorwithtotalvariationregularizedl1function
AT jaedukhan multiframedepthsuperresolutionfortofsensorwithtotalvariationregularizedl1function
AT moongikang multiframedepthsuperresolutionfortofsensorwithtotalvariationregularizedl1function