Resolving Multi-Path Interference in Compressive Time-of-Flight Depth Imaging with a Multi-Tap Macro-Pixel Computational CMOS Image Sensor
Multi-path interference causes depth errors in indirect time-of-flight (ToF) cameras. In this paper, resolving multi-path interference caused by surface reflections using a multi-tap macro-pixel computational CMOS image sensor is demonstrated. The imaging area is implemented by an array of macro-pix...
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-03-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/22/7/2442 |
| _version_ | 1797437792723140608 |
|---|---|
| author | Masaya Horio Yu Feng Tomoya Kokado Taishi Takasawa Keita Yasutomi Shoji Kawahito Takashi Komuro Hajime Nagahara Keiichiro Kagawa |
| author_facet | Masaya Horio Yu Feng Tomoya Kokado Taishi Takasawa Keita Yasutomi Shoji Kawahito Takashi Komuro Hajime Nagahara Keiichiro Kagawa |
| author_sort | Masaya Horio |
| collection | DOAJ |
| description | Multi-path interference causes depth errors in indirect time-of-flight (ToF) cameras. In this paper, resolving multi-path interference caused by surface reflections using a multi-tap macro-pixel computational CMOS image sensor is demonstrated. The imaging area is implemented by an array of macro-pixels composed of four subpixels embodied by a four-tap lateral electric field charge modulator (LEFM). This sensor can simultaneously acquire 16 images for different temporal shutters. This method can reproduce more than 16 images based on compressive sensing with multi-frequency shutters and sub-clock shifting. In simulations, an object was placed 16 m away from the sensor, and the depth of an interference object was varied from 1 to 32 m in 1 m steps. The two reflections were separated in two stages: coarse estimation based on a compressive sensing solver and refinement by a nonlinear search to investigate the potential of our sensor. Relative standard deviation (precision) and relative mean error (accuracy) were evaluated under the influence of photon shot noise. The proposed method was verified using a prototype multi-tap macro-pixel computational CMOS image sensor in single-path and dual-path situations. In the experiment, an acrylic plate was placed 1 m or 2 m and a mirror 9.3 m from the sensor. |
| first_indexed | 2024-03-09T11:27:46Z |
| format | Article |
| id | doaj.art-a0c6075330114d4eae7a382053f36dc4 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-09T11:27:46Z |
| publishDate | 2022-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-a0c6075330114d4eae7a382053f36dc42023-11-30T23:58:57ZengMDPI AGSensors1424-82202022-03-01227244210.3390/s22072442Resolving Multi-Path Interference in Compressive Time-of-Flight Depth Imaging with a Multi-Tap Macro-Pixel Computational CMOS Image SensorMasaya Horio0Yu Feng1Tomoya Kokado2Taishi Takasawa3Keita Yasutomi4Shoji Kawahito5Takashi Komuro6Hajime Nagahara7Keiichiro Kagawa8Graduate School of Integrated Science and Technology, Shizuoka University, Shizuoka 422-8011, JapanGraduate School of Integrated Science and Technology, Shizuoka University, Shizuoka 422-8011, JapanGraduate School of Integrated Science and Technology, Shizuoka University, Shizuoka 422-8011, JapanResearch Institute of Electronics, Shizuoka University, Shizuoka 422-8011, JapanResearch Institute of Electronics, Shizuoka University, Shizuoka 422-8011, JapanResearch Institute of Electronics, Shizuoka University, Shizuoka 422-8011, JapanGraduate School of Science and Engineering, Saitama University, Saitama 338-8570, JapanInstitute of Datability Science, Osaka University, Osaka 565-0871, JapanResearch Institute of Electronics, Shizuoka University, Shizuoka 422-8011, JapanMulti-path interference causes depth errors in indirect time-of-flight (ToF) cameras. In this paper, resolving multi-path interference caused by surface reflections using a multi-tap macro-pixel computational CMOS image sensor is demonstrated. The imaging area is implemented by an array of macro-pixels composed of four subpixels embodied by a four-tap lateral electric field charge modulator (LEFM). This sensor can simultaneously acquire 16 images for different temporal shutters. This method can reproduce more than 16 images based on compressive sensing with multi-frequency shutters and sub-clock shifting. In simulations, an object was placed 16 m away from the sensor, and the depth of an interference object was varied from 1 to 32 m in 1 m steps. The two reflections were separated in two stages: coarse estimation based on a compressive sensing solver and refinement by a nonlinear search to investigate the potential of our sensor. Relative standard deviation (precision) and relative mean error (accuracy) were evaluated under the influence of photon shot noise. The proposed method was verified using a prototype multi-tap macro-pixel computational CMOS image sensor in single-path and dual-path situations. In the experiment, an acrylic plate was placed 1 m or 2 m and a mirror 9.3 m from the sensor.https://www.mdpi.com/1424-8220/22/7/2442time-of-flight depth imagingmulti-path interferencemulti-tap macro-pixelcharge modulatorscompressive sensingCMOS image sensor |
| spellingShingle | Masaya Horio Yu Feng Tomoya Kokado Taishi Takasawa Keita Yasutomi Shoji Kawahito Takashi Komuro Hajime Nagahara Keiichiro Kagawa Resolving Multi-Path Interference in Compressive Time-of-Flight Depth Imaging with a Multi-Tap Macro-Pixel Computational CMOS Image Sensor Sensors time-of-flight depth imaging multi-path interference multi-tap macro-pixel charge modulators compressive sensing CMOS image sensor |
| title | Resolving Multi-Path Interference in Compressive Time-of-Flight Depth Imaging with a Multi-Tap Macro-Pixel Computational CMOS Image Sensor |
| title_full | Resolving Multi-Path Interference in Compressive Time-of-Flight Depth Imaging with a Multi-Tap Macro-Pixel Computational CMOS Image Sensor |
| title_fullStr | Resolving Multi-Path Interference in Compressive Time-of-Flight Depth Imaging with a Multi-Tap Macro-Pixel Computational CMOS Image Sensor |
| title_full_unstemmed | Resolving Multi-Path Interference in Compressive Time-of-Flight Depth Imaging with a Multi-Tap Macro-Pixel Computational CMOS Image Sensor |
| title_short | Resolving Multi-Path Interference in Compressive Time-of-Flight Depth Imaging with a Multi-Tap Macro-Pixel Computational CMOS Image Sensor |
| title_sort | resolving multi path interference in compressive time of flight depth imaging with a multi tap macro pixel computational cmos image sensor |
| topic | time-of-flight depth imaging multi-path interference multi-tap macro-pixel charge modulators compressive sensing CMOS image sensor |
| url | https://www.mdpi.com/1424-8220/22/7/2442 |
| work_keys_str_mv | AT masayahorio resolvingmultipathinterferenceincompressivetimeofflightdepthimagingwithamultitapmacropixelcomputationalcmosimagesensor AT yufeng resolvingmultipathinterferenceincompressivetimeofflightdepthimagingwithamultitapmacropixelcomputationalcmosimagesensor AT tomoyakokado resolvingmultipathinterferenceincompressivetimeofflightdepthimagingwithamultitapmacropixelcomputationalcmosimagesensor AT taishitakasawa resolvingmultipathinterferenceincompressivetimeofflightdepthimagingwithamultitapmacropixelcomputationalcmosimagesensor AT keitayasutomi resolvingmultipathinterferenceincompressivetimeofflightdepthimagingwithamultitapmacropixelcomputationalcmosimagesensor AT shojikawahito resolvingmultipathinterferenceincompressivetimeofflightdepthimagingwithamultitapmacropixelcomputationalcmosimagesensor AT takashikomuro resolvingmultipathinterferenceincompressivetimeofflightdepthimagingwithamultitapmacropixelcomputationalcmosimagesensor AT hajimenagahara resolvingmultipathinterferenceincompressivetimeofflightdepthimagingwithamultitapmacropixelcomputationalcmosimagesensor AT keiichirokagawa resolvingmultipathinterferenceincompressivetimeofflightdepthimagingwithamultitapmacropixelcomputationalcmosimagesensor |