Spatial audio and spatial audio-visual learning - He, Y (2024)
As humans, we extensively depend on multimodal signals to perceive, interact with, and analyze our surrounding 3D spatial environment, so as to accomplish various complex tasks. Amongst all our multimodal senses, sound and vision are the two most ubiquitous signals in real world scenarios. Equipping machines with such spatial audio-visual multimodal inferring and learning capability, or human-level audio-visual intelligence, is a vital yet challenging task. Although the widely accepted importance for spatial audio-visual intelligence, current research community’s focus has been heavily biased towards vision, with far less attention paid to spatial audio. Research in spatial audio-visual learning typically takes spatial audio as an auxiliary input to assist vision-centred tasks, ignoring the intricate and multifaceted interactions between audio and vision.