Separating the “chirp” from the “chat”: self-supervised visual grounding of sound and language

Separating the “chirp” from the “chat”: self-supervised visual grounding of sound and language

We present DenseAV, a novel dual encoder grounding architecture that learns high-resolution, semantically meaningful, and audio-visual aligned features solely through watching videos. We show that DenseAV can discover the “meaning” of words and the “location” of sounds without explicit localization...

Szczegółowa specyfikacja

Opis bibliograficzny
Główni autorzy:	Hamilton, M, Zisserman, A, Hershey, JR, Freeman, WT
Format:	Conference item
Język:	English
Wydane:	IEEE 2024

Podobne zapisy

Multi-task self-supervised visual learning
od: Doersch, C, i wsp.
Wydane: (2017)

Ambient Sound Provides Supervision for Visual Learning
od: Owens, Andrew Hale, i wsp.
Wydane: (2017)

Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning
od: Owens, Andrew, i wsp.
Wydane: (2021)

Self-Supervised Learning for Audio-Visual Relationships of Videos With Stereo Sounds
od: Tomoya Sato, i wsp.
Wydane: (2022-01-01)

Self-supervised learning of audio-visual objects from video
od: Afouras, T, i wsp.
Wydane: (2020)

Enhancement of sound by soft reflections in exponentially chirped crystals
od: A. Cebrecos, i wsp.
Wydane: (2014-12-01)

Music Gesture for Visual Sound Separation
od: Gan, Chuang, i wsp.
Wydane: (2021)

First observations of oblique ionospheric sounding chirp signal in Mexico
od: M.A. Sergeeva, i wsp.
Wydane: (2019-03-01)

Features of backscatter ionospheric sounding as studied with a chirp ionosonde
od: Ponomarchuk S.N., i wsp.
Wydane: (2017-09-01)

Audio-Visual Self-Supervised Terrain Type Recognition for Ground Mobile Platforms
od: Akiyoshi Kurobe, i wsp.
Wydane: (2021-01-01)

Self-supervised learning for spinal MRIs
od: Jamaludin, A, i wsp.
Wydane: (2017)

Self-Supervised Moving Vehicle Tracking With Stereo Sound
od: Gan, Chuang, i wsp.
Wydane: (2021)

Unsupervised discovery of visual object class hierarchies
od: Sivic, J, i wsp.
Wydane: (2008)

Localizing visual sounds the hard way
od: Vedaldi, A, i wsp.
Wydane: (2021)

Weakly supervised scale-invariant learning of models for visual recognition
od: Fergus, R, i wsp.
Wydane: (2006)

Enhancing bowel sound recognition with self-attention and self-supervised pre-training.
od: Yansuo Yu, i wsp.
Wydane: (2024-01-01)

Self-supervised co-training for video representation learning
od: Han, T, i wsp.
Wydane: (2020)

ESTIMATING ANTENNA COUPLING FACTOR FOR PROBLEM OF TOPSIDEIONOSPHERE SOUNDING FROM SPACE BY CHIRP SIGNALS
od: Podlesnyi A.V., i wsp.
Wydane: (2019-12-01)

Self-supervised learning of class embeddings from video
od: Wiles, O, i wsp.
Wydane: (2020)

Sight to Sound: An End-to-End Approach for Visual Piano Transcription
od: Koepke, S, i wsp.
Wydane: (2020)

ASDNet: An Efficient Self-Supervised Convolutional Network for Anomalous Sound Detection
od: Dewei Kong, i wsp.
Wydane: (2025-01-01)

Self-Supervised Transfer Learning from Natural Images for Sound Classification
od: Sungho Shin, i wsp.
Wydane: (2021-03-01)

Visually Indicated Sounds
od: Isola, Phillip, i wsp.
Wydane: (2017)

Direct Underwater Sound Velocity Measurement Based on the Acousto-Optic Self-Interference Effect between the Chirp Signal and the Optical Frequency Comb
od: Zihui Yang, i wsp.
Wydane: (2022-12-01)

Extraction of Individual EEG Gamma Frequencies from the Responses to Click-Based Chirp-Modulated Sounds
od: Aurimas Mockevičius, i wsp.
Wydane: (2023-03-01)

Chat2VIS: Generating Data Visualizations via Natural Language Using ChatGPT, Codex and GPT-3 Large Language Models
od: Paula Maddigan, i wsp.
Wydane: (2023-01-01)

Self-Supervised Audio-Visual Co-Segmentation
od: Rouditchenko, Andrew, i wsp.
Wydane: (2022)

Self-Supervised Audio-Visual Co-Segmentation
od: Rouditchenko, Andrew, i wsp.
Wydane: (2021)

Self-Supervised Autoencoders for Visual Anomaly Detection
od: Alexander Bauer, i wsp.
Wydane: (2024-12-01)

Disentangled Speech Embeddings Using Cross-Modal Self-Supervision
od: Nagrani, A, i wsp.
Wydane: (2020)

Combining Unsupervised and Supervised Learning for Sample Efficient Continuous Language Grounding
od: Oliver Roesler
Wydane: (2022-09-01)

Weakly-supervised fingerspelling recognition in British Sign Language videos
od: Prajwal, KR, i wsp.
Wydane: (2022)

Self-supervised learning of a facial attribute embedding from video
od: Wiles, O, i wsp.
Wydane: (2018)

Self-supervised video object segmentation by motion grouping
od: Yang, C, i wsp.
Wydane: (2021)

A Climate Hyperspectral Infrared Radiance Product (CHIRP) Combining the AIRS and CrIS Satellite Sounding Record
od: L. Larrabee Strow, i wsp.
Wydane: (2021-01-01)

Diagnostics of HF radio channel: based on data from backscatter ionospheric sounding by continuous chirp signal
od: Ponomarchuk S.N., i wsp.
Wydane: (2018-06-01)

Application of Optimized Adaptive Chirp Mode Decomposition Method in Chirp Signal
od: Junyuan Wang, i wsp.
Wydane: (2020-05-01)

Self-supervised multi-modal alignment for whole body medical imaging
od: Windsor, R, i wsp.
Wydane: (2021)

Now you're speaking my language: visual language identification
od: Afouras, T, i wsp.
Wydane: (2020)

Exploring the Utility of ChatGPT for Self-directed Online Language Learning
od: Zixi Li, i wsp.
Wydane: (2024-09-01)