Separating the “chirp” from the “chat”: self-supervised visual grounding of sound and language

Separating the “chirp” from the “chat”: self-supervised visual grounding of sound and language

We present DenseAV, a novel dual encoder grounding architecture that learns high-resolution, semantically meaningful, and audio-visual aligned features solely through watching videos. We show that DenseAV can discover the “meaning” of words and the “location” of sounds without explicit localization...

Полное описание

Библиографические подробности
Главные авторы:	Hamilton, M, Zisserman, A, Hershey, JR, Freeman, WT
Формат:	Conference item
Язык:	English
Опубликовано:	IEEE 2024

Схожие документы

Multi-task self-supervised visual learning
по: Doersch, C, и др.
Опубликовано: (2017)

Ambient Sound Provides Supervision for Visual Learning
по: Owens, Andrew Hale, и др.
Опубликовано: (2017)

Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning
по: Owens, Andrew, и др.
Опубликовано: (2021)

Self-Supervised Learning for Audio-Visual Relationships of Videos With Stereo Sounds
по: Tomoya Sato, и др.
Опубликовано: (2022-01-01)

Self-supervised learning of audio-visual objects from video
по: Afouras, T, и др.
Опубликовано: (2020)

Enhancement of sound by soft reflections in exponentially chirped crystals
по: A. Cebrecos, и др.
Опубликовано: (2014-12-01)

Music Gesture for Visual Sound Separation
по: Gan, Chuang, и др.
Опубликовано: (2021)

First observations of oblique ionospheric sounding chirp signal in Mexico
по: M.A. Sergeeva, и др.
Опубликовано: (2019-03-01)

Features of backscatter ionospheric sounding as studied with a chirp ionosonde
по: Ponomarchuk S.N., и др.
Опубликовано: (2017-09-01)

Audio-Visual Self-Supervised Terrain Type Recognition for Ground Mobile Platforms
по: Akiyoshi Kurobe, и др.
Опубликовано: (2021-01-01)

Self-supervised learning for spinal MRIs
по: Jamaludin, A, и др.
Опубликовано: (2017)

Self-Supervised Moving Vehicle Tracking With Stereo Sound
по: Gan, Chuang, и др.
Опубликовано: (2021)

Unsupervised discovery of visual object class hierarchies
по: Sivic, J, и др.
Опубликовано: (2008)

Localizing visual sounds the hard way
по: Vedaldi, A, и др.
Опубликовано: (2021)

Weakly supervised scale-invariant learning of models for visual recognition
по: Fergus, R, и др.
Опубликовано: (2006)

Enhancing bowel sound recognition with self-attention and self-supervised pre-training.
по: Yansuo Yu, и др.
Опубликовано: (2024-01-01)

Self-supervised co-training for video representation learning
по: Han, T, и др.
Опубликовано: (2020)

ESTIMATING ANTENNA COUPLING FACTOR FOR PROBLEM OF TOPSIDEIONOSPHERE SOUNDING FROM SPACE BY CHIRP SIGNALS
по: Podlesnyi A.V., и др.
Опубликовано: (2019-12-01)

Self-supervised learning of class embeddings from video
по: Wiles, O, и др.
Опубликовано: (2020)

Sight to Sound: An End-to-End Approach for Visual Piano Transcription
по: Koepke, S, и др.
Опубликовано: (2020)

ASDNet: An Efficient Self-Supervised Convolutional Network for Anomalous Sound Detection
по: Dewei Kong, и др.
Опубликовано: (2025-01-01)

Self-Supervised Transfer Learning from Natural Images for Sound Classification
по: Sungho Shin, и др.
Опубликовано: (2021-03-01)

Visually Indicated Sounds
по: Isola, Phillip, и др.
Опубликовано: (2017)

Direct Underwater Sound Velocity Measurement Based on the Acousto-Optic Self-Interference Effect between the Chirp Signal and the Optical Frequency Comb
по: Zihui Yang, и др.
Опубликовано: (2022-12-01)

Extraction of Individual EEG Gamma Frequencies from the Responses to Click-Based Chirp-Modulated Sounds
по: Aurimas Mockevičius, и др.
Опубликовано: (2023-03-01)

Chat2VIS: Generating Data Visualizations via Natural Language Using ChatGPT, Codex and GPT-3 Large Language Models
по: Paula Maddigan, и др.
Опубликовано: (2023-01-01)

Disentangled Speech Embeddings Using Cross-Modal Self-Supervision
по: Nagrani, A, и др.
Опубликовано: (2020)

Self-Supervised Audio-Visual Co-Segmentation
по: Rouditchenko, Andrew, и др.
Опубликовано: (2022)

Self-Supervised Audio-Visual Co-Segmentation
по: Rouditchenko, Andrew, и др.
Опубликовано: (2021)

Self-Supervised Autoencoders for Visual Anomaly Detection
по: Alexander Bauer, и др.
Опубликовано: (2024-12-01)

Combining Unsupervised and Supervised Learning for Sample Efficient Continuous Language Grounding
по: Oliver Roesler
Опубликовано: (2022-09-01)

Weakly-supervised fingerspelling recognition in British Sign Language videos
по: Prajwal, KR, и др.
Опубликовано: (2022)

Self-supervised learning of a facial attribute embedding from video
по: Wiles, O, и др.
Опубликовано: (2018)

Self-supervised video object segmentation by motion grouping
по: Yang, C, и др.
Опубликовано: (2021)

A Climate Hyperspectral Infrared Radiance Product (CHIRP) Combining the AIRS and CrIS Satellite Sounding Record
по: L. Larrabee Strow, и др.
Опубликовано: (2021-01-01)

Diagnostics of HF radio channel: based on data from backscatter ionospheric sounding by continuous chirp signal
по: Ponomarchuk S.N., и др.
Опубликовано: (2018-06-01)

Application of Optimized Adaptive Chirp Mode Decomposition Method in Chirp Signal
по: Junyuan Wang, и др.
Опубликовано: (2020-05-01)

Self-supervised multi-modal alignment for whole body medical imaging
по: Windsor, R, и др.
Опубликовано: (2021)

Now you're speaking my language: visual language identification
по: Afouras, T, и др.
Опубликовано: (2020)

Exploring the Utility of ChatGPT for Self-directed Online Language Learning
по: Zixi Li, и др.
Опубликовано: (2024-09-01)