Optimal Configuration of Multi-Task Learning for Autonomous Driving
For autonomous driving, it is imperative to perform various high-computation image recognition tasks with high accuracy, utilizing diverse sensors to perceive the surrounding environment. Specifically, cameras are used to perform lane detection, object detection, and segmentation, and, in the absenc...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-12-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/23/24/9729 |
| _version_ | 1797379431592886272 |
|---|---|
| author | Woomin Jun Minjun Son Jisang Yoo Sungjin Lee |
| author_facet | Woomin Jun Minjun Son Jisang Yoo Sungjin Lee |
| author_sort | Woomin Jun |
| collection | DOAJ |
| description | For autonomous driving, it is imperative to perform various high-computation image recognition tasks with high accuracy, utilizing diverse sensors to perceive the surrounding environment. Specifically, cameras are used to perform lane detection, object detection, and segmentation, and, in the absence of lidar, tasks extend to inferring 3D information through depth estimation, 3D object detection, 3D reconstruction, and SLAM. However, accurately processing all these image recognition operations in real-time for autonomous driving under constrained hardware conditions is practically unfeasible. In this study, considering the characteristics of image recognition tasks performed by these sensors and the given hardware conditions, we investigated MTL (multi-task learning), which enables parallel execution of various image recognition tasks to maximize their processing speed, accuracy, and memory efficiency. Particularly, this study analyzes the combinations of image recognition tasks for autonomous driving and proposes the MDO (multi-task decision and optimization) algorithm, consisting of three steps, as a means for optimization. In the initial step, a MTS (multi-task set) is selected to minimize overall latency while meeting minimum accuracy requirements. Subsequently, additional training of the shared backbone and individual subnets is conducted to enhance accuracy with the predefined MTS. Finally, both the shared backbone and each subnet undergo compression while maintaining the already secured accuracy and latency performance. The experimental results indicate that integrated accuracy performance is critically important in the configuration and optimization of MTL, and this integrated accuracy is determined by the ITC (inter-task correlation). The MDO algorithm was designed to consider these characteristics and construct multi-task sets with tasks that exhibit high ITC. Furthermore, the implementation of the proposed MDO algorithm, coupled with additional SSL (semi-supervised learning) based training, resulted in a significant performance enhancement. This advancement manifested as approximately a 12% increase in object detection mAP performance, a 15% improvement in lane detection accuracy, and a 27% reduction in latency, surpassing the results of previous three-task learning techniques like YOLOP and HybridNet. |
| first_indexed | 2024-03-08T20:23:08Z |
| format | Article |
| id | doaj.art-7fb40ee0f40f460ab2133c3b73cdeb9d |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-08T20:23:08Z |
| publishDate | 2023-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-7fb40ee0f40f460ab2133c3b73cdeb9d2023-12-22T14:40:25ZengMDPI AGSensors1424-82202023-12-012324972910.3390/s23249729Optimal Configuration of Multi-Task Learning for Autonomous DrivingWoomin Jun0Minjun Son1Jisang Yoo2Sungjin Lee3Electronic Engineering, Dong Seoul University, Seongnam 13117, Republic of KoreaElectronic Engineering, Dong Seoul University, Seongnam 13117, Republic of KoreaElectronic Engineering, Dong Seoul University, Seongnam 13117, Republic of KoreaElectronic Engineering, Dong Seoul University, Seongnam 13117, Republic of KoreaFor autonomous driving, it is imperative to perform various high-computation image recognition tasks with high accuracy, utilizing diverse sensors to perceive the surrounding environment. Specifically, cameras are used to perform lane detection, object detection, and segmentation, and, in the absence of lidar, tasks extend to inferring 3D information through depth estimation, 3D object detection, 3D reconstruction, and SLAM. However, accurately processing all these image recognition operations in real-time for autonomous driving under constrained hardware conditions is practically unfeasible. In this study, considering the characteristics of image recognition tasks performed by these sensors and the given hardware conditions, we investigated MTL (multi-task learning), which enables parallel execution of various image recognition tasks to maximize their processing speed, accuracy, and memory efficiency. Particularly, this study analyzes the combinations of image recognition tasks for autonomous driving and proposes the MDO (multi-task decision and optimization) algorithm, consisting of three steps, as a means for optimization. In the initial step, a MTS (multi-task set) is selected to minimize overall latency while meeting minimum accuracy requirements. Subsequently, additional training of the shared backbone and individual subnets is conducted to enhance accuracy with the predefined MTS. Finally, both the shared backbone and each subnet undergo compression while maintaining the already secured accuracy and latency performance. The experimental results indicate that integrated accuracy performance is critically important in the configuration and optimization of MTL, and this integrated accuracy is determined by the ITC (inter-task correlation). The MDO algorithm was designed to consider these characteristics and construct multi-task sets with tasks that exhibit high ITC. Furthermore, the implementation of the proposed MDO algorithm, coupled with additional SSL (semi-supervised learning) based training, resulted in a significant performance enhancement. This advancement manifested as approximately a 12% increase in object detection mAP performance, a 15% improvement in lane detection accuracy, and a 27% reduction in latency, surpassing the results of previous three-task learning techniques like YOLOP and HybridNet.https://www.mdpi.com/1424-8220/23/24/9729autonomous drivingmulti-task learninglane detectionobject detectiondrivable area segmentationdepth estimation |
| spellingShingle | Woomin Jun Minjun Son Jisang Yoo Sungjin Lee Optimal Configuration of Multi-Task Learning for Autonomous Driving Sensors autonomous driving multi-task learning lane detection object detection drivable area segmentation depth estimation |
| title | Optimal Configuration of Multi-Task Learning for Autonomous Driving |
| title_full | Optimal Configuration of Multi-Task Learning for Autonomous Driving |
| title_fullStr | Optimal Configuration of Multi-Task Learning for Autonomous Driving |
| title_full_unstemmed | Optimal Configuration of Multi-Task Learning for Autonomous Driving |
| title_short | Optimal Configuration of Multi-Task Learning for Autonomous Driving |
| title_sort | optimal configuration of multi task learning for autonomous driving |
| topic | autonomous driving multi-task learning lane detection object detection drivable area segmentation depth estimation |
| url | https://www.mdpi.com/1424-8220/23/24/9729 |
| work_keys_str_mv | AT woominjun optimalconfigurationofmultitasklearningforautonomousdriving AT minjunson optimalconfigurationofmultitasklearningforautonomousdriving AT jisangyoo optimalconfigurationofmultitasklearningforautonomousdriving AT sungjinlee optimalconfigurationofmultitasklearningforautonomousdriving |