Adaptive Multi-Pedestrian Tracking by Multi-Sensor: Track-to-Track Fusion Using Monocular 3D Detection and MMW Radar
Accurate and reliable tracking of multi-pedestrian is of great importance for autonomous driving, human-robot interaction and video surveillance. Since different scenarios have different best-performing sensors, sensor fusion perception plans are believed to have complementary modalities and be capa...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-04-01
|
| Series: | Remote Sensing |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2072-4292/14/8/1837 |
| _version_ | 1797409469643096064 |
|---|---|
| author | Yipeng Zhu Tao Wang Shiqiang Zhu |
| author_facet | Yipeng Zhu Tao Wang Shiqiang Zhu |
| author_sort | Yipeng Zhu |
| collection | DOAJ |
| description | Accurate and reliable tracking of multi-pedestrian is of great importance for autonomous driving, human-robot interaction and video surveillance. Since different scenarios have different best-performing sensors, sensor fusion perception plans are believed to have complementary modalities and be capable of handling situations which are challenging for single sensor. In this paper, we propose a novel track-to-track fusion strategy for multi-pedestrian tracking by using a millimeter-wave (MMW) radar and a monocular camera. Pedestrians are firstly tracked by each sensor according to the sensor characteristic. Specifically, the 3D monocular pedestrian detections are obtained by a convolutional neural network (CNN). The trajectory is formed by the tracking-by-detection approach, combined with Bayesian estimation. The measurement noise of the 3D monocular detection is modeled by a detection uncertainty value obtained from the same CNN, as an approach to estimate the pedestrian state more accurately. The MMW radar utilizes the track-before-detection method due to the sparseness of the radar features. Afterwards, the pedestrian trajectories are obtained by the proposed track-to-track fusion strategy, which can work adaptively under challenging weather conditions, low-illumination conditions and clutter scenarios. A group of tests are carried out to validate our pedestrian tracking strategy. Tracking trajectories and optimal sub-pattern assignment (OSPA) metric demonstrate the accuracy and robustness of the proposed multi-sensor multi-pedestrian tracking system. |
| first_indexed | 2024-03-09T04:14:58Z |
| format | Article |
| id | doaj.art-4c9edd7676d946b7a3b1dbb474de0b28 |
| institution | Directory Open Access Journal |
| issn | 2072-4292 |
| language | English |
| last_indexed | 2024-03-09T04:14:58Z |
| publishDate | 2022-04-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Remote Sensing |
| spelling | doaj.art-4c9edd7676d946b7a3b1dbb474de0b282023-12-03T13:55:23ZengMDPI AGRemote Sensing2072-42922022-04-01148183710.3390/rs14081837Adaptive Multi-Pedestrian Tracking by Multi-Sensor: Track-to-Track Fusion Using Monocular 3D Detection and MMW RadarYipeng Zhu0Tao Wang1Shiqiang Zhu2Ocean College, Zhejiang University, Zhoushan 316000, ChinaOcean College, Zhejiang University, Zhoushan 316000, ChinaOcean College, Zhejiang University, Zhoushan 316000, ChinaAccurate and reliable tracking of multi-pedestrian is of great importance for autonomous driving, human-robot interaction and video surveillance. Since different scenarios have different best-performing sensors, sensor fusion perception plans are believed to have complementary modalities and be capable of handling situations which are challenging for single sensor. In this paper, we propose a novel track-to-track fusion strategy for multi-pedestrian tracking by using a millimeter-wave (MMW) radar and a monocular camera. Pedestrians are firstly tracked by each sensor according to the sensor characteristic. Specifically, the 3D monocular pedestrian detections are obtained by a convolutional neural network (CNN). The trajectory is formed by the tracking-by-detection approach, combined with Bayesian estimation. The measurement noise of the 3D monocular detection is modeled by a detection uncertainty value obtained from the same CNN, as an approach to estimate the pedestrian state more accurately. The MMW radar utilizes the track-before-detection method due to the sparseness of the radar features. Afterwards, the pedestrian trajectories are obtained by the proposed track-to-track fusion strategy, which can work adaptively under challenging weather conditions, low-illumination conditions and clutter scenarios. A group of tests are carried out to validate our pedestrian tracking strategy. Tracking trajectories and optimal sub-pattern assignment (OSPA) metric demonstrate the accuracy and robustness of the proposed multi-sensor multi-pedestrian tracking system.https://www.mdpi.com/2072-4292/14/8/1837pedestrian trackingsensor fusionmonocular 3D detectionMMW radartrack-to-track fusion |
| spellingShingle | Yipeng Zhu Tao Wang Shiqiang Zhu Adaptive Multi-Pedestrian Tracking by Multi-Sensor: Track-to-Track Fusion Using Monocular 3D Detection and MMW Radar Remote Sensing pedestrian tracking sensor fusion monocular 3D detection MMW radar track-to-track fusion |
| title | Adaptive Multi-Pedestrian Tracking by Multi-Sensor: Track-to-Track Fusion Using Monocular 3D Detection and MMW Radar |
| title_full | Adaptive Multi-Pedestrian Tracking by Multi-Sensor: Track-to-Track Fusion Using Monocular 3D Detection and MMW Radar |
| title_fullStr | Adaptive Multi-Pedestrian Tracking by Multi-Sensor: Track-to-Track Fusion Using Monocular 3D Detection and MMW Radar |
| title_full_unstemmed | Adaptive Multi-Pedestrian Tracking by Multi-Sensor: Track-to-Track Fusion Using Monocular 3D Detection and MMW Radar |
| title_short | Adaptive Multi-Pedestrian Tracking by Multi-Sensor: Track-to-Track Fusion Using Monocular 3D Detection and MMW Radar |
| title_sort | adaptive multi pedestrian tracking by multi sensor track to track fusion using monocular 3d detection and mmw radar |
| topic | pedestrian tracking sensor fusion monocular 3D detection MMW radar track-to-track fusion |
| url | https://www.mdpi.com/2072-4292/14/8/1837 |
| work_keys_str_mv | AT yipengzhu adaptivemultipedestriantrackingbymultisensortracktotrackfusionusingmonocular3ddetectionandmmwradar AT taowang adaptivemultipedestriantrackingbymultisensortracktotrackfusionusingmonocular3ddetectionandmmwradar AT shiqiangzhu adaptivemultipedestriantrackingbymultisensortracktotrackfusionusingmonocular3ddetectionandmmwradar |