A 256 × 256 LiDAR Imaging System Based on a 200 mW SPAD-Based SoC with Microlens Array and Lightweight RGB-Guided Depth Completion Neural Network
Light detection and ranging (LiDAR) technology, a cutting-edge advancement in mobile applications, presents a myriad of compelling use cases, including enhancing low-light photography, capturing and sharing 3D images of fascinating objects, and elevating the overall augmented reality (AR) experience...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-08-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/23/15/6927 |
| _version_ | 1797585957389598720 |
|---|---|
| author | Jier Wang Jie Li Yifan Wu Hengwei Yu Lebei Cui Miao Sun Patrick Yin Chiang |
| author_facet | Jier Wang Jie Li Yifan Wu Hengwei Yu Lebei Cui Miao Sun Patrick Yin Chiang |
| author_sort | Jier Wang |
| collection | DOAJ |
| description | Light detection and ranging (LiDAR) technology, a cutting-edge advancement in mobile applications, presents a myriad of compelling use cases, including enhancing low-light photography, capturing and sharing 3D images of fascinating objects, and elevating the overall augmented reality (AR) experience. However, its widespread adoption has been hindered by the prohibitive costs and substantial power consumption associated with its implementation in mobile devices. To surmount these obstacles, this paper proposes a low-power, low-cost, single-photon avalanche detector (SPAD)-based system-on-chip (SoC) which packages the microlens arrays (MLAs) and a lightweight RGB-guided sparse depth imaging completion neural network for 3D LiDAR imaging. The proposed SoC integrates an 8 × 8 SPAD macropixel array with time-to-digital converters (TDCs) and a charge pump, fabricated using a 180 nm bipolar-CMOS-DMOS (BCD) process. Initially, the primary function of this SoC was limited to serving as a ranging sensor. A random MLA-based homogenizing diffuser efficiently transforms Gaussian beams into flat-topped beams with a 45° field of view (FOV), enabling flash projection at the transmitter. To further enhance resolution and broaden application possibilities, a lightweight neural network employing RGB-guided sparse depth complementation is proposed, enabling a substantial expansion of image resolution from 8 × 8 to quarter video graphics array level (QVGA; 256 × 256). Experimental results demonstrate the effectiveness and stability of the hardware encompassing the SoC and optical system, as well as the lightweight features and accuracy of the algorithmic neural network. The state-of-the-art SoC-neural network solution offers a promising and inspiring foundation for developing consumer-level 3D imaging applications on mobile devices. |
| first_indexed | 2024-03-11T00:16:22Z |
| format | Article |
| id | doaj.art-4ba53bf663584b04b68052881c3874b3 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-11T00:16:22Z |
| publishDate | 2023-08-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-4ba53bf663584b04b68052881c3874b32023-11-18T23:36:20ZengMDPI AGSensors1424-82202023-08-012315692710.3390/s23156927A 256 × 256 LiDAR Imaging System Based on a 200 mW SPAD-Based SoC with Microlens Array and Lightweight RGB-Guided Depth Completion Neural NetworkJier Wang0Jie Li1Yifan Wu2Hengwei Yu3Lebei Cui4Miao Sun5Patrick Yin Chiang6State Key Laboratory of ASIC and System, Fudan University, Shanghai 201203, ChinaState Key Laboratory of ASIC and System, Fudan University, Shanghai 201203, ChinaCollege of Electronics and Information Engineering, Tongji University, Shanghai 201804, ChinaState Key Laboratory of ASIC and System, Fudan University, Shanghai 201203, ChinaState Key Laboratory of ASIC and System, Fudan University, Shanghai 201203, ChinaState Key Laboratory of ASIC and System, Fudan University, Shanghai 201203, ChinaState Key Laboratory of ASIC and System, Fudan University, Shanghai 201203, ChinaLight detection and ranging (LiDAR) technology, a cutting-edge advancement in mobile applications, presents a myriad of compelling use cases, including enhancing low-light photography, capturing and sharing 3D images of fascinating objects, and elevating the overall augmented reality (AR) experience. However, its widespread adoption has been hindered by the prohibitive costs and substantial power consumption associated with its implementation in mobile devices. To surmount these obstacles, this paper proposes a low-power, low-cost, single-photon avalanche detector (SPAD)-based system-on-chip (SoC) which packages the microlens arrays (MLAs) and a lightweight RGB-guided sparse depth imaging completion neural network for 3D LiDAR imaging. The proposed SoC integrates an 8 × 8 SPAD macropixel array with time-to-digital converters (TDCs) and a charge pump, fabricated using a 180 nm bipolar-CMOS-DMOS (BCD) process. Initially, the primary function of this SoC was limited to serving as a ranging sensor. A random MLA-based homogenizing diffuser efficiently transforms Gaussian beams into flat-topped beams with a 45° field of view (FOV), enabling flash projection at the transmitter. To further enhance resolution and broaden application possibilities, a lightweight neural network employing RGB-guided sparse depth complementation is proposed, enabling a substantial expansion of image resolution from 8 × 8 to quarter video graphics array level (QVGA; 256 × 256). Experimental results demonstrate the effectiveness and stability of the hardware encompassing the SoC and optical system, as well as the lightweight features and accuracy of the algorithmic neural network. The state-of-the-art SoC-neural network solution offers a promising and inspiring foundation for developing consumer-level 3D imaging applications on mobile devices.https://www.mdpi.com/1424-8220/23/15/6927LiDAR3D imagingsystem-on-chipmicrolens arrayneural networkRGB-guided |
| spellingShingle | Jier Wang Jie Li Yifan Wu Hengwei Yu Lebei Cui Miao Sun Patrick Yin Chiang A 256 × 256 LiDAR Imaging System Based on a 200 mW SPAD-Based SoC with Microlens Array and Lightweight RGB-Guided Depth Completion Neural Network Sensors LiDAR 3D imaging system-on-chip microlens array neural network RGB-guided |
| title | A 256 × 256 LiDAR Imaging System Based on a 200 mW SPAD-Based SoC with Microlens Array and Lightweight RGB-Guided Depth Completion Neural Network |
| title_full | A 256 × 256 LiDAR Imaging System Based on a 200 mW SPAD-Based SoC with Microlens Array and Lightweight RGB-Guided Depth Completion Neural Network |
| title_fullStr | A 256 × 256 LiDAR Imaging System Based on a 200 mW SPAD-Based SoC with Microlens Array and Lightweight RGB-Guided Depth Completion Neural Network |
| title_full_unstemmed | A 256 × 256 LiDAR Imaging System Based on a 200 mW SPAD-Based SoC with Microlens Array and Lightweight RGB-Guided Depth Completion Neural Network |
| title_short | A 256 × 256 LiDAR Imaging System Based on a 200 mW SPAD-Based SoC with Microlens Array and Lightweight RGB-Guided Depth Completion Neural Network |
| title_sort | 256 256 lidar imaging system based on a 200 mw spad based soc with microlens array and lightweight rgb guided depth completion neural network |
| topic | LiDAR 3D imaging system-on-chip microlens array neural network RGB-guided |
| url | https://www.mdpi.com/1424-8220/23/15/6927 |
| work_keys_str_mv | AT jierwang a256256lidarimagingsystembasedona200mwspadbasedsocwithmicrolensarrayandlightweightrgbguideddepthcompletionneuralnetwork AT jieli a256256lidarimagingsystembasedona200mwspadbasedsocwithmicrolensarrayandlightweightrgbguideddepthcompletionneuralnetwork AT yifanwu a256256lidarimagingsystembasedona200mwspadbasedsocwithmicrolensarrayandlightweightrgbguideddepthcompletionneuralnetwork AT hengweiyu a256256lidarimagingsystembasedona200mwspadbasedsocwithmicrolensarrayandlightweightrgbguideddepthcompletionneuralnetwork AT lebeicui a256256lidarimagingsystembasedona200mwspadbasedsocwithmicrolensarrayandlightweightrgbguideddepthcompletionneuralnetwork AT miaosun a256256lidarimagingsystembasedona200mwspadbasedsocwithmicrolensarrayandlightweightrgbguideddepthcompletionneuralnetwork AT patrickyinchiang a256256lidarimagingsystembasedona200mwspadbasedsocwithmicrolensarrayandlightweightrgbguideddepthcompletionneuralnetwork AT jierwang 256256lidarimagingsystembasedona200mwspadbasedsocwithmicrolensarrayandlightweightrgbguideddepthcompletionneuralnetwork AT jieli 256256lidarimagingsystembasedona200mwspadbasedsocwithmicrolensarrayandlightweightrgbguideddepthcompletionneuralnetwork AT yifanwu 256256lidarimagingsystembasedona200mwspadbasedsocwithmicrolensarrayandlightweightrgbguideddepthcompletionneuralnetwork AT hengweiyu 256256lidarimagingsystembasedona200mwspadbasedsocwithmicrolensarrayandlightweightrgbguideddepthcompletionneuralnetwork AT lebeicui 256256lidarimagingsystembasedona200mwspadbasedsocwithmicrolensarrayandlightweightrgbguideddepthcompletionneuralnetwork AT miaosun 256256lidarimagingsystembasedona200mwspadbasedsocwithmicrolensarrayandlightweightrgbguideddepthcompletionneuralnetwork AT patrickyinchiang 256256lidarimagingsystembasedona200mwspadbasedsocwithmicrolensarrayandlightweightrgbguideddepthcompletionneuralnetwork |