Real-Time Multi-Task ADAS Implementation on Reconfigurable Heterogeneous MPSoC Architecture
The rapid adoption of Advanced Driver Assistance Systems (ADAS) in modern vehicles, aiming to elevate driving safety and experience, necessitates the real-time processing of high-definition video data. This requirement brings about considerable computational complexity and memory demands, highlighti...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10198234/ |
| _version_ | 1797752064536739840 |
|---|---|
| author | Guner Tatar Salih Bayar |
| author_facet | Guner Tatar Salih Bayar |
| author_sort | Guner Tatar |
| collection | DOAJ |
| description | The rapid adoption of Advanced Driver Assistance Systems (ADAS) in modern vehicles, aiming to elevate driving safety and experience, necessitates the real-time processing of high-definition video data. This requirement brings about considerable computational complexity and memory demands, highlighting a critical research void for a design integrating high FPS throughput with optimal Mean Average Precision (mAP) and Mean Intersection over Union (mIoU). Performance improvement at lower costs, multi-tasking ability on a single hardware platform, and flawless incorporation into memory-constrained devices are also essential for boosting ADAS performance. Addressing these challenges, this study proposes an ADAS multi-task learning hardware-software co-design approach underpinned by the Kria KV260 Multi-Processor System-on-Chip Field Programmable Gate Array (MPSoC-FPGA) platform. The approach facilitates efficient real-time execution of deep learning algorithms specific to ADAS applications. Utilizing the BDD100K+Waymo, KITTI, and CityScapes datasets, our ADAS multi-task learning system endeavours to provide accurate and efficient multi-object detection, segmentation, and lane and drivable area detection in road images. The system deploys a segmentation-based object detection strategy, using a ResNet-18 backbone encoder and a Single Shot Detector architecture, coupled with quantization-aware training to augment inference performance without compromising accuracy. The ADAS multi-task learning offers customization options for various ADAS applications and can be further optimized for increased precision and reduced memory usage. Experimental results showcase the system’s capability to perform real-time multi-class object detection, segmentation, line detection, and drivable area detection on road images at approximately 25.4 FPS using a <inline-formula> <tex-math notation="LaTeX">$1920\times 1080\text{p}$ </tex-math></inline-formula> Full HD camera. Impressively, the quantized model has demonstrated a 51% mAP for object detection, 56.62% mIoU for image segmentation, 43.86% mIoU for line detection, and 81.56% IoU for drivable area identification, reinforcing its high efficacy and precision. The findings underscore that the proposed ADAS multi-task learning system is a practical, reliable, and effective solution for real-world applications. |
| first_indexed | 2024-03-12T16:58:16Z |
| format | Article |
| id | doaj.art-ef15d02541d74b9fbebc45b5068187bf |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-03-12T16:58:16Z |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-ef15d02541d74b9fbebc45b5068187bf2023-08-07T23:00:31ZengIEEEIEEE Access2169-35362023-01-0111807418076010.1109/ACCESS.2023.330037910198234Real-Time Multi-Task ADAS Implementation on Reconfigurable Heterogeneous MPSoC ArchitectureGuner Tatar0https://orcid.org/0000-0002-3664-1366Salih Bayar1https://orcid.org/0000-0002-4600-1880Department of Electrical Electronic Engineering, Fatih Sultan Mehmet Vakif University, Istanbul, TurkeyDepartment of Electrical and Electronic Engineering, Marmara University, Istanbul, TurkeyThe rapid adoption of Advanced Driver Assistance Systems (ADAS) in modern vehicles, aiming to elevate driving safety and experience, necessitates the real-time processing of high-definition video data. This requirement brings about considerable computational complexity and memory demands, highlighting a critical research void for a design integrating high FPS throughput with optimal Mean Average Precision (mAP) and Mean Intersection over Union (mIoU). Performance improvement at lower costs, multi-tasking ability on a single hardware platform, and flawless incorporation into memory-constrained devices are also essential for boosting ADAS performance. Addressing these challenges, this study proposes an ADAS multi-task learning hardware-software co-design approach underpinned by the Kria KV260 Multi-Processor System-on-Chip Field Programmable Gate Array (MPSoC-FPGA) platform. The approach facilitates efficient real-time execution of deep learning algorithms specific to ADAS applications. Utilizing the BDD100K+Waymo, KITTI, and CityScapes datasets, our ADAS multi-task learning system endeavours to provide accurate and efficient multi-object detection, segmentation, and lane and drivable area detection in road images. The system deploys a segmentation-based object detection strategy, using a ResNet-18 backbone encoder and a Single Shot Detector architecture, coupled with quantization-aware training to augment inference performance without compromising accuracy. The ADAS multi-task learning offers customization options for various ADAS applications and can be further optimized for increased precision and reduced memory usage. Experimental results showcase the system’s capability to perform real-time multi-class object detection, segmentation, line detection, and drivable area detection on road images at approximately 25.4 FPS using a <inline-formula> <tex-math notation="LaTeX">$1920\times 1080\text{p}$ </tex-math></inline-formula> Full HD camera. Impressively, the quantized model has demonstrated a 51% mAP for object detection, 56.62% mIoU for image segmentation, 43.86% mIoU for line detection, and 81.56% IoU for drivable area identification, reinforcing its high efficacy and precision. The findings underscore that the proposed ADAS multi-task learning system is a practical, reliable, and effective solution for real-world applications.https://ieeexplore.ieee.org/document/10198234/ADASdeep learningdeep processing unitmemory allocationmulti-task learningMPSoC-FPGA architecture |
| spellingShingle | Guner Tatar Salih Bayar Real-Time Multi-Task ADAS Implementation on Reconfigurable Heterogeneous MPSoC Architecture IEEE Access ADAS deep learning deep processing unit memory allocation multi-task learning MPSoC-FPGA architecture |
| title | Real-Time Multi-Task ADAS Implementation on Reconfigurable Heterogeneous MPSoC Architecture |
| title_full | Real-Time Multi-Task ADAS Implementation on Reconfigurable Heterogeneous MPSoC Architecture |
| title_fullStr | Real-Time Multi-Task ADAS Implementation on Reconfigurable Heterogeneous MPSoC Architecture |
| title_full_unstemmed | Real-Time Multi-Task ADAS Implementation on Reconfigurable Heterogeneous MPSoC Architecture |
| title_short | Real-Time Multi-Task ADAS Implementation on Reconfigurable Heterogeneous MPSoC Architecture |
| title_sort | real time multi task adas implementation on reconfigurable heterogeneous mpsoc architecture |
| topic | ADAS deep learning deep processing unit memory allocation multi-task learning MPSoC-FPGA architecture |
| url | https://ieeexplore.ieee.org/document/10198234/ |
| work_keys_str_mv | AT gunertatar realtimemultitaskadasimplementationonreconfigurableheterogeneousmpsocarchitecture AT salihbayar realtimemultitaskadasimplementationonreconfigurableheterogeneousmpsocarchitecture |