Audiovisual Tracking of Multiple Speakers in Smart Spaces - Frank Sanabria-Macias; Marta Marron-Romera; Javier Macias-Guarasa (Sensors, 2023)
This paper presents GAVT, a highly accurate audiovisual 3D tracking system based on particle filters and a probabilistic framework, employing a single camera and a microphone array. Our first contribution is a complex visual appearance model that accurately locates the speaker’s mouth. It transforms a Viola & Jones face detector classifier kernel into a likelihood estimator, leveraging knowledge from multiple classifiers trained for different face poses. Additionally, we propose a mechanism to handle occlusions based on the new likelihood’s dispersion. The audio localization proposal utilizes a probabilistic steered response power, representing cross-correlation functions as Gaussian mixture models. Moreover, to prevent tracker interference, we introduce a novel mechanism for associating Gaussians with speakers. The evaluation is carried out using the AV16.3 and CAV3D databases for Single- and Multiple-Object Tracking tasks (SOT and MOT, respectively). GAVT significantly improves the localization performance over audio-only and video-only modalities, with up to <semantics>50.3%</semantics> average relative improvement in 3D when compared with the video-only modality. When compared to the state of the art, our audiovisual system achieves up to <semantics>69.7%</semantics> average relative improvement for the SOT and MOT tasks in the AV16.3 dataset (2D comparison), and up to <semantics>18.1%</semantics> average relative improvement in the MOT task for the CAV3D dataset (3D comparison).