PIFNet: 3D Object Detection Using Joint Image and Point Cloud Features for Autonomous Driving
Owing to its wide range of applications, 3D object detection has attracted increasing attention in computer vision tasks. Most existing 3D object detection methods are based on Lidar point cloud data. However, these methods have some limitations in localization consistency and classification confide...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-04-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/12/7/3686 |
| _version_ | 1797440252354232320 |
|---|---|
| author | Wenqi Zheng Han Xie Yunfan Chen Jeongjin Roh Hyunchul Shin |
| author_facet | Wenqi Zheng Han Xie Yunfan Chen Jeongjin Roh Hyunchul Shin |
| author_sort | Wenqi Zheng |
| collection | DOAJ |
| description | Owing to its wide range of applications, 3D object detection has attracted increasing attention in computer vision tasks. Most existing 3D object detection methods are based on Lidar point cloud data. However, these methods have some limitations in localization consistency and classification confidence, due to the irregularity and sparsity of Light Detection and Ranging (LiDAR) point cloud data. Inspired by the complementary characteristics of Lidar and camera sensors, we propose a new end-to-end learnable framework named Point-Image Fusion Network (PIFNet) to integrate the LiDAR point cloud and camera images. To resolve the problem of inconsistency in the localization and classification, we designed an Encoder-Decoder Fusion (EDF) module to extract the image features effectively, while maintaining the fine-grained localization information of objects. Furthermore, a new effective fusion module is proposed to integrate the color and texture features from images and the depth information from the point cloud. This module can enhance the irregularity and sparsity problem of the point cloud features by capitalizing the fine-grained information from camera images. In PIFNet, each intermediate feature map is fed into the fusion module to be integrated with its corresponding point-wise features. Furthermore, point-wise features are used instead of voxel-wise features to reduce information loss. Extensive experiments using the KITTI dataset demonstrate the superiority of PIFNet over other state-of-the-art methods. Compared with several state-of-the-art methods, our approach outperformed by 1.97% in mean Average Precision (mAP) and by 2.86% in Average Precision (AP) for the hard cases on the KITTI 3D object detection benchmark. |
| first_indexed | 2024-03-09T12:04:30Z |
| format | Article |
| id | doaj.art-2e385a99ae5d40a284670976661ef59f |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-09T12:04:30Z |
| publishDate | 2022-04-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-2e385a99ae5d40a284670976661ef59f2023-11-30T22:59:08ZengMDPI AGApplied Sciences2076-34172022-04-01127368610.3390/app12073686PIFNet: 3D Object Detection Using Joint Image and Point Cloud Features for Autonomous DrivingWenqi Zheng0Han Xie1Yunfan Chen2Jeongjin Roh3Hyunchul Shin4Department of Electrical Engineering, Hanyang University, Ansan 15588, KoreaDepartment of Electrical Engineering, Hanyang University, Ansan 15588, KoreaSchool of Electrical and Electronic Engineering, Hubei University of Technology, Wuhan 430068, ChinaDepartment of Electrical Engineering, Hanyang University, Ansan 15588, KoreaDepartment of Electrical Engineering, Hanyang University, Ansan 15588, KoreaOwing to its wide range of applications, 3D object detection has attracted increasing attention in computer vision tasks. Most existing 3D object detection methods are based on Lidar point cloud data. However, these methods have some limitations in localization consistency and classification confidence, due to the irregularity and sparsity of Light Detection and Ranging (LiDAR) point cloud data. Inspired by the complementary characteristics of Lidar and camera sensors, we propose a new end-to-end learnable framework named Point-Image Fusion Network (PIFNet) to integrate the LiDAR point cloud and camera images. To resolve the problem of inconsistency in the localization and classification, we designed an Encoder-Decoder Fusion (EDF) module to extract the image features effectively, while maintaining the fine-grained localization information of objects. Furthermore, a new effective fusion module is proposed to integrate the color and texture features from images and the depth information from the point cloud. This module can enhance the irregularity and sparsity problem of the point cloud features by capitalizing the fine-grained information from camera images. In PIFNet, each intermediate feature map is fed into the fusion module to be integrated with its corresponding point-wise features. Furthermore, point-wise features are used instead of voxel-wise features to reduce information loss. Extensive experiments using the KITTI dataset demonstrate the superiority of PIFNet over other state-of-the-art methods. Compared with several state-of-the-art methods, our approach outperformed by 1.97% in mean Average Precision (mAP) and by 2.86% in Average Precision (AP) for the hard cases on the KITTI 3D object detection benchmark.https://www.mdpi.com/2076-3417/12/7/36863D object detectionlidar point cloudcamera imagesobject detection |
| spellingShingle | Wenqi Zheng Han Xie Yunfan Chen Jeongjin Roh Hyunchul Shin PIFNet: 3D Object Detection Using Joint Image and Point Cloud Features for Autonomous Driving Applied Sciences 3D object detection lidar point cloud camera images object detection |
| title | PIFNet: 3D Object Detection Using Joint Image and Point Cloud Features for Autonomous Driving |
| title_full | PIFNet: 3D Object Detection Using Joint Image and Point Cloud Features for Autonomous Driving |
| title_fullStr | PIFNet: 3D Object Detection Using Joint Image and Point Cloud Features for Autonomous Driving |
| title_full_unstemmed | PIFNet: 3D Object Detection Using Joint Image and Point Cloud Features for Autonomous Driving |
| title_short | PIFNet: 3D Object Detection Using Joint Image and Point Cloud Features for Autonomous Driving |
| title_sort | pifnet 3d object detection using joint image and point cloud features for autonomous driving |
| topic | 3D object detection lidar point cloud camera images object detection |
| url | https://www.mdpi.com/2076-3417/12/7/3686 |
| work_keys_str_mv | AT wenqizheng pifnet3dobjectdetectionusingjointimageandpointcloudfeaturesforautonomousdriving AT hanxie pifnet3dobjectdetectionusingjointimageandpointcloudfeaturesforautonomousdriving AT yunfanchen pifnet3dobjectdetectionusingjointimageandpointcloudfeaturesforautonomousdriving AT jeongjinroh pifnet3dobjectdetectionusingjointimageandpointcloudfeaturesforautonomousdriving AT hyunchulshin pifnet3dobjectdetectionusingjointimageandpointcloudfeaturesforautonomousdriving |