Separating the “chirp” from the “chat”: self-supervised visual grounding of sound and language

Separating the “chirp” from the “chat”: self-supervised visual grounding of sound and language

We present DenseAV, a novel dual encoder grounding architecture that learns high-resolution, semantically meaningful, and audio-visual aligned features solely through watching videos. We show that DenseAV can discover the “meaning” of words and the “location” of sounds without explicit localization...

Descripció completa

Dades bibliogràfiques
Autors principals:	Hamilton, M, Zisserman, A, Hershey, JR, Freeman, WT
Format:	Conference item
Idioma:	English
Publicat:	IEEE 2024

Ítems similars

Multi-task self-supervised visual learning
per: Doersch, C, et al.
Publicat: (2017)

Ambient Sound Provides Supervision for Visual Learning
per: Owens, Andrew Hale, et al.
Publicat: (2017)

Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning
per: Owens, Andrew, et al.
Publicat: (2021)

Self-Supervised Learning for Audio-Visual Relationships of Videos With Stereo Sounds
per: Tomoya Sato, et al.
Publicat: (2022-01-01)

Self-supervised learning of audio-visual objects from video
per: Afouras, T, et al.
Publicat: (2020)

Enhancement of sound by soft reflections in exponentially chirped crystals
per: A. Cebrecos, et al.
Publicat: (2014-12-01)

Music Gesture for Visual Sound Separation
per: Gan, Chuang, et al.
Publicat: (2021)

First observations of oblique ionospheric sounding chirp signal in Mexico
per: M.A. Sergeeva, et al.
Publicat: (2019-03-01)

Features of backscatter ionospheric sounding as studied with a chirp ionosonde
per: Ponomarchuk S.N., et al.
Publicat: (2017-09-01)

Audio-Visual Self-Supervised Terrain Type Recognition for Ground Mobile Platforms
per: Akiyoshi Kurobe, et al.
Publicat: (2021-01-01)

Self-supervised learning for spinal MRIs
per: Jamaludin, A, et al.
Publicat: (2017)

Self-Supervised Moving Vehicle Tracking With Stereo Sound
per: Gan, Chuang, et al.
Publicat: (2021)

Unsupervised discovery of visual object class hierarchies
per: Sivic, J, et al.
Publicat: (2008)

Localizing visual sounds the hard way
per: Vedaldi, A, et al.
Publicat: (2021)

Weakly supervised scale-invariant learning of models for visual recognition
per: Fergus, R, et al.
Publicat: (2006)

Enhancing bowel sound recognition with self-attention and self-supervised pre-training.
per: Yansuo Yu, et al.
Publicat: (2024-01-01)

Self-supervised co-training for video representation learning
per: Han, T, et al.
Publicat: (2020)

ESTIMATING ANTENNA COUPLING FACTOR FOR PROBLEM OF TOPSIDEIONOSPHERE SOUNDING FROM SPACE BY CHIRP SIGNALS
per: Podlesnyi A.V., et al.
Publicat: (2019-12-01)

Self-supervised learning of class embeddings from video
per: Wiles, O, et al.
Publicat: (2020)

Sight to Sound: An End-to-End Approach for Visual Piano Transcription
per: Koepke, S, et al.
Publicat: (2020)

ASDNet: An Efficient Self-Supervised Convolutional Network for Anomalous Sound Detection
per: Dewei Kong, et al.
Publicat: (2025-01-01)

Self-Supervised Transfer Learning from Natural Images for Sound Classification
per: Sungho Shin, et al.
Publicat: (2021-03-01)

Visually Indicated Sounds
per: Isola, Phillip, et al.
Publicat: (2017)

Direct Underwater Sound Velocity Measurement Based on the Acousto-Optic Self-Interference Effect between the Chirp Signal and the Optical Frequency Comb
per: Zihui Yang, et al.
Publicat: (2022-12-01)

Extraction of Individual EEG Gamma Frequencies from the Responses to Click-Based Chirp-Modulated Sounds
per: Aurimas Mockevičius, et al.
Publicat: (2023-03-01)

Chat2VIS: Generating Data Visualizations via Natural Language Using ChatGPT, Codex and GPT-3 Large Language Models
per: Paula Maddigan, et al.
Publicat: (2023-01-01)

Disentangled Speech Embeddings Using Cross-Modal Self-Supervision
per: Nagrani, A, et al.
Publicat: (2020)

Self-Supervised Audio-Visual Co-Segmentation
per: Rouditchenko, Andrew, et al.
Publicat: (2022)

Self-Supervised Audio-Visual Co-Segmentation
per: Rouditchenko, Andrew, et al.
Publicat: (2021)

Self-Supervised Autoencoders for Visual Anomaly Detection
per: Alexander Bauer, et al.
Publicat: (2024-12-01)

Combining Unsupervised and Supervised Learning for Sample Efficient Continuous Language Grounding
per: Oliver Roesler
Publicat: (2022-09-01)

Weakly-supervised fingerspelling recognition in British Sign Language videos
per: Prajwal, KR, et al.
Publicat: (2022)

Self-supervised learning of a facial attribute embedding from video
per: Wiles, O, et al.
Publicat: (2018)

Self-supervised video object segmentation by motion grouping
per: Yang, C, et al.
Publicat: (2021)

A Climate Hyperspectral Infrared Radiance Product (CHIRP) Combining the AIRS and CrIS Satellite Sounding Record
per: L. Larrabee Strow, et al.
Publicat: (2021-01-01)

Diagnostics of HF radio channel: based on data from backscatter ionospheric sounding by continuous chirp signal
per: Ponomarchuk S.N., et al.
Publicat: (2018-06-01)

Application of Optimized Adaptive Chirp Mode Decomposition Method in Chirp Signal
per: Junyuan Wang, et al.
Publicat: (2020-05-01)

Self-supervised multi-modal alignment for whole body medical imaging
per: Windsor, R, et al.
Publicat: (2021)

Now you're speaking my language: visual language identification
per: Afouras, T, et al.
Publicat: (2020)

Exploring the Utility of ChatGPT for Self-directed Online Language Learning
per: Zixi Li, et al.
Publicat: (2024-09-01)