Multi-Frame Depth Super-Resolution for ToF Sensor With Total Variation Regularized L1 Function
In this paper, we propose a multi-frame depth super-resolution (SR) method based on L<sub>1</sub> data fidelity with the total variation regularization (TV-L<sub>1</sub>) model. The majority of time-of-flight (ToF) sensors exhibit limited spatial resolution compared to RGB se...
Main Authors: | , , |
---|---|
Format: | Article |
Language: | English |
Published: |
IEEE
2020-01-01
|
Series: | IEEE Access |
Subjects: | |
Online Access: | https://ieeexplore.ieee.org/document/9189817/ |
_version_ | 1818616710871121920 |
---|---|
author | Jonghyun Kim Jaeduk Han Moon Gi Kang |
author_facet | Jonghyun Kim Jaeduk Han Moon Gi Kang |
author_sort | Jonghyun Kim |
collection | DOAJ |
description | In this paper, we propose a multi-frame depth super-resolution (SR) method based on L<sub>1</sub> data fidelity with the total variation regularization (TV-L<sub>1</sub>) model. The majority of time-of-flight (ToF) sensors exhibit limited spatial resolution compared to RGB sensors and the improvement of the depth image resolution is an inherently ill-posed problem. To overcome this under-determined problem, the solution space is limited by the regularization term through prior knowledge and the data fidelity term using statistical information of the noise. Firstly, the statistical characteristics of ToF depth images are analyzed to specify the appropriate observation model. Thereafter, the objective function for multi-frame depth SR based on the TV-L<sub>1</sub> model is designed by considering the prior knowledge of the depth images. This approach enables the sharpness of the edges to be preserved and the noise amplification to be suppressed simultaneously. Furthermore, an efficient solver based on half-quadratic splitting is proposed. The algorithm minimizes the objective function for the multi-frame SR problem consisting of the TV regularization term and L<sub>1</sub> data fidelity term. The proposed method is verified on a synthetic dataset and real-world data acquired from a ToF sensor. The experimental results demonstrate that the proposed method can substantially reconstruct high-resolution depth images in terms of preserving sharp depth discontinuities, without any obvious artifacts, and can increase robustness to noise. |
first_indexed | 2024-12-16T16:54:08Z |
format | Article |
id | doaj.art-030e8c10944448f3be696be56109c540 |
institution | Directory Open Access Journal |
issn | 2169-3536 |
language | English |
last_indexed | 2024-12-16T16:54:08Z |
publishDate | 2020-01-01 |
publisher | IEEE |
record_format | Article |
series | IEEE Access |
spelling | doaj.art-030e8c10944448f3be696be56109c5402022-12-21T22:23:55ZengIEEEIEEE Access2169-35362020-01-01816581016582610.1109/ACCESS.2020.30229109189817Multi-Frame Depth Super-Resolution for ToF Sensor With Total Variation Regularized L1 FunctionJonghyun Kim0https://orcid.org/0000-0002-3788-0211Jaeduk Han1https://orcid.org/0000-0001-8798-1275Moon Gi Kang2https://orcid.org/0000-0002-5771-929XSchool of Electrical and Electronic Engineering, Yonsei University, Seoul, South KoreaSchool of Electrical and Electronic Engineering, Yonsei University, Seoul, South KoreaSchool of Electrical and Electronic Engineering, Yonsei University, Seoul, South KoreaIn this paper, we propose a multi-frame depth super-resolution (SR) method based on L<sub>1</sub> data fidelity with the total variation regularization (TV-L<sub>1</sub>) model. The majority of time-of-flight (ToF) sensors exhibit limited spatial resolution compared to RGB sensors and the improvement of the depth image resolution is an inherently ill-posed problem. To overcome this under-determined problem, the solution space is limited by the regularization term through prior knowledge and the data fidelity term using statistical information of the noise. Firstly, the statistical characteristics of ToF depth images are analyzed to specify the appropriate observation model. Thereafter, the objective function for multi-frame depth SR based on the TV-L<sub>1</sub> model is designed by considering the prior knowledge of the depth images. This approach enables the sharpness of the edges to be preserved and the noise amplification to be suppressed simultaneously. Furthermore, an efficient solver based on half-quadratic splitting is proposed. The algorithm minimizes the objective function for the multi-frame SR problem consisting of the TV regularization term and L<sub>1</sub> data fidelity term. The proposed method is verified on a synthetic dataset and real-world data acquired from a ToF sensor. The experimental results demonstrate that the proposed method can substantially reconstruct high-resolution depth images in terms of preserving sharp depth discontinuities, without any obvious artifacts, and can increase robustness to noise.https://ieeexplore.ieee.org/document/9189817/Depth super-resolutiontime-of-flight sensormulti-frame super-resolutiontotal variation regularizationL1 data fidelity |
spellingShingle | Jonghyun Kim Jaeduk Han Moon Gi Kang Multi-Frame Depth Super-Resolution for ToF Sensor With Total Variation Regularized L1 Function IEEE Access Depth super-resolution time-of-flight sensor multi-frame super-resolution total variation regularization L1 data fidelity |
title | Multi-Frame Depth Super-Resolution for ToF Sensor With Total Variation Regularized L1 Function |
title_full | Multi-Frame Depth Super-Resolution for ToF Sensor With Total Variation Regularized L1 Function |
title_fullStr | Multi-Frame Depth Super-Resolution for ToF Sensor With Total Variation Regularized L1 Function |
title_full_unstemmed | Multi-Frame Depth Super-Resolution for ToF Sensor With Total Variation Regularized L1 Function |
title_short | Multi-Frame Depth Super-Resolution for ToF Sensor With Total Variation Regularized L1 Function |
title_sort | multi frame depth super resolution for tof sensor with total variation regularized l1 function |
topic | Depth super-resolution time-of-flight sensor multi-frame super-resolution total variation regularization L1 data fidelity |
url | https://ieeexplore.ieee.org/document/9189817/ |
work_keys_str_mv | AT jonghyunkim multiframedepthsuperresolutionfortofsensorwithtotalvariationregularizedl1function AT jaedukhan multiframedepthsuperresolutionfortofsensorwithtotalvariationregularizedl1function AT moongikang multiframedepthsuperresolutionfortofsensorwithtotalvariationregularizedl1function |