FPGA-oriented lightweight multi-modal free-space detection network
For autonomous vehicles, free-space detection is an essential part of visual perception. With the development of multi-modal convolutional neural networks (CNNs) in recent years, the performance of driving scene semantic segmentation algorithms has been dramatically improved. Therefore most free-spa...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Taylor & Francis Group
2023-12-01
|
| Series: | Connection Science |
| Subjects: | |
| Online Access: | http://dx.doi.org/10.1080/09540091.2022.2159333 |
| _version_ | 1797683980044075008 |
|---|---|
| author | Feiyi Fang Junzhu Mao Wei Yu Jianfeng Lu |
| author_facet | Feiyi Fang Junzhu Mao Wei Yu Jianfeng Lu |
| author_sort | Feiyi Fang |
| collection | DOAJ |
| description | For autonomous vehicles, free-space detection is an essential part of visual perception. With the development of multi-modal convolutional neural networks (CNNs) in recent years, the performance of driving scene semantic segmentation algorithms has been dramatically improved. Therefore most free-space detection algorithms are developed based on multiple sensors. However, multi-modal CNNs have high data throughput and contain a large number of computationally intensive convolution calculations, limiting their feasibility for real-time applications. Field Programmable Gate Arrays (FPGAs) provide a unique combination of flexibility, performance, and low power for these problems to accommodate multi-modal data and the computational acceleration of different compression algorithms. Network lightweight methods offer great assurance for facilitating the deployment of CNNs on such resource-constrained devices. In this paper, we propose a network lightweight method for a multi-modal free-space detection algorithm. We first propose an FPGA-friendly multi-modal free-space detection lightweight network. It comprises operators that FPGA prefers and achieves a $ 95.54\% $ MaxF score on the test set of KITTI-Road free-space detection tasks and 81 ms runtime when running on 700 W GPU devices. Then we present a pruning approach for this network to reduce the number of parameters in case the complete model exceeds the FPGA chip memory. The pruning is in two parts. For the feature extractors, we propose a data-dependent filter pruner according to the principle that the low-rank feature map contains less information. To not compromise the integrity of the multi-modal information, the pruner is independent for each modality. For the segmentation decoder, we apply a channel pruning approach to remove redundant parameters. Finally, we implement our designs on an FPGA board using 8-bit quantisation, and the accelerator achieves outstanding performance. A real-time application of scene segmentation on KITTI-Road is used to evaluate our algorithm, and the model achieves a $ 94.39\% $ MaxF score and minimum 14 ms runtime on 20W FPGA devices. |
| first_indexed | 2024-03-12T00:23:44Z |
| format | Article |
| id | doaj.art-0823dd7eb5a44f45894615960debdd68 |
| institution | Directory Open Access Journal |
| issn | 0954-0091 1360-0494 |
| language | English |
| last_indexed | 2024-03-12T00:23:44Z |
| publishDate | 2023-12-01 |
| publisher | Taylor & Francis Group |
| record_format | Article |
| series | Connection Science |
| spelling | doaj.art-0823dd7eb5a44f45894615960debdd682023-09-15T10:48:01ZengTaylor & Francis GroupConnection Science0954-00911360-04942023-12-0135110.1080/09540091.2022.21593332159333FPGA-oriented lightweight multi-modal free-space detection networkFeiyi Fang0Junzhu Mao1Wei Yu2Jianfeng Lu3Nanjing University of Science and TechnologyNanjing University of Science and TechnologyBeijing RICH AI information Technology Co., LtdNanjing University of Science and TechnologyFor autonomous vehicles, free-space detection is an essential part of visual perception. With the development of multi-modal convolutional neural networks (CNNs) in recent years, the performance of driving scene semantic segmentation algorithms has been dramatically improved. Therefore most free-space detection algorithms are developed based on multiple sensors. However, multi-modal CNNs have high data throughput and contain a large number of computationally intensive convolution calculations, limiting their feasibility for real-time applications. Field Programmable Gate Arrays (FPGAs) provide a unique combination of flexibility, performance, and low power for these problems to accommodate multi-modal data and the computational acceleration of different compression algorithms. Network lightweight methods offer great assurance for facilitating the deployment of CNNs on such resource-constrained devices. In this paper, we propose a network lightweight method for a multi-modal free-space detection algorithm. We first propose an FPGA-friendly multi-modal free-space detection lightweight network. It comprises operators that FPGA prefers and achieves a $ 95.54\% $ MaxF score on the test set of KITTI-Road free-space detection tasks and 81 ms runtime when running on 700 W GPU devices. Then we present a pruning approach for this network to reduce the number of parameters in case the complete model exceeds the FPGA chip memory. The pruning is in two parts. For the feature extractors, we propose a data-dependent filter pruner according to the principle that the low-rank feature map contains less information. To not compromise the integrity of the multi-modal information, the pruner is independent for each modality. For the segmentation decoder, we apply a channel pruning approach to remove redundant parameters. Finally, we implement our designs on an FPGA board using 8-bit quantisation, and the accelerator achieves outstanding performance. A real-time application of scene segmentation on KITTI-Road is used to evaluate our algorithm, and the model achieves a $ 94.39\% $ MaxF score and minimum 14 ms runtime on 20W FPGA devices.http://dx.doi.org/10.1080/09540091.2022.2159333pruningfree-space detectionlightweight networkmulti-modal learningfpga |
| spellingShingle | Feiyi Fang Junzhu Mao Wei Yu Jianfeng Lu FPGA-oriented lightweight multi-modal free-space detection network Connection Science pruning free-space detection lightweight network multi-modal learning fpga |
| title | FPGA-oriented lightweight multi-modal free-space detection network |
| title_full | FPGA-oriented lightweight multi-modal free-space detection network |
| title_fullStr | FPGA-oriented lightweight multi-modal free-space detection network |
| title_full_unstemmed | FPGA-oriented lightweight multi-modal free-space detection network |
| title_short | FPGA-oriented lightweight multi-modal free-space detection network |
| title_sort | fpga oriented lightweight multi modal free space detection network |
| topic | pruning free-space detection lightweight network multi-modal learning fpga |
| url | http://dx.doi.org/10.1080/09540091.2022.2159333 |
| work_keys_str_mv | AT feiyifang fpgaorientedlightweightmultimodalfreespacedetectionnetwork AT junzhumao fpgaorientedlightweightmultimodalfreespacedetectionnetwork AT weiyu fpgaorientedlightweightmultimodalfreespacedetectionnetwork AT jianfenglu fpgaorientedlightweightmultimodalfreespacedetectionnetwork |