Separating the “chirp” from the “chat”: self-supervised visual grounding of sound and language

Separating the “chirp” from the “chat”: self-supervised visual grounding of sound and language

We present DenseAV, a novel dual encoder grounding architecture that learns high-resolution, semantically meaningful, and audio-visual aligned features solely through watching videos. We show that DenseAV can discover the “meaning” of words and the “location” of sounds without explicit localization...

Bibliografski detalji
Glavni autori:	Hamilton, M, Zisserman, A, Hershey, JR, Freeman, WT
Format:	Conference item
Jezik:	English
Izdano:	IEEE 2024

Slični predmeti

Multi-task self-supervised visual learning
od: Doersch, C, i dr.
Izdano: (2017)

Ambient Sound Provides Supervision for Visual Learning
od: Owens, Andrew Hale, i dr.
Izdano: (2017)

Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning
od: Owens, Andrew, i dr.
Izdano: (2021)

Self-Supervised Learning for Audio-Visual Relationships of Videos With Stereo Sounds
od: Tomoya Sato, i dr.
Izdano: (2022-01-01)

Self-supervised learning of audio-visual objects from video
od: Afouras, T, i dr.
Izdano: (2020)

Enhancement of sound by soft reflections in exponentially chirped crystals
od: A. Cebrecos, i dr.
Izdano: (2014-12-01)

Music Gesture for Visual Sound Separation
od: Gan, Chuang, i dr.
Izdano: (2021)

First observations of oblique ionospheric sounding chirp signal in Mexico
od: M.A. Sergeeva, i dr.
Izdano: (2019-03-01)

Features of backscatter ionospheric sounding as studied with a chirp ionosonde
od: Ponomarchuk S.N., i dr.
Izdano: (2017-09-01)

Audio-Visual Self-Supervised Terrain Type Recognition for Ground Mobile Platforms
od: Akiyoshi Kurobe, i dr.
Izdano: (2021-01-01)

Self-supervised learning for spinal MRIs
od: Jamaludin, A, i dr.
Izdano: (2017)

Self-Supervised Moving Vehicle Tracking With Stereo Sound
od: Gan, Chuang, i dr.
Izdano: (2021)

Unsupervised discovery of visual object class hierarchies
od: Sivic, J, i dr.
Izdano: (2008)

Localizing visual sounds the hard way
od: Vedaldi, A, i dr.
Izdano: (2021)

Weakly supervised scale-invariant learning of models for visual recognition
od: Fergus, R, i dr.
Izdano: (2006)

Enhancing bowel sound recognition with self-attention and self-supervised pre-training.
od: Yansuo Yu, i dr.
Izdano: (2024-01-01)

Self-supervised co-training for video representation learning
od: Han, T, i dr.
Izdano: (2020)

ESTIMATING ANTENNA COUPLING FACTOR FOR PROBLEM OF TOPSIDEIONOSPHERE SOUNDING FROM SPACE BY CHIRP SIGNALS
od: Podlesnyi A.V., i dr.
Izdano: (2019-12-01)

Self-supervised learning of class embeddings from video
od: Wiles, O, i dr.
Izdano: (2020)

Sight to Sound: An End-to-End Approach for Visual Piano Transcription
od: Koepke, S, i dr.
Izdano: (2020)

ASDNet: An Efficient Self-Supervised Convolutional Network for Anomalous Sound Detection
od: Dewei Kong, i dr.
Izdano: (2025-01-01)

Self-Supervised Transfer Learning from Natural Images for Sound Classification
od: Sungho Shin, i dr.
Izdano: (2021-03-01)

Visually Indicated Sounds
od: Isola, Phillip, i dr.
Izdano: (2017)

Direct Underwater Sound Velocity Measurement Based on the Acousto-Optic Self-Interference Effect between the Chirp Signal and the Optical Frequency Comb
od: Zihui Yang, i dr.
Izdano: (2022-12-01)

Extraction of Individual EEG Gamma Frequencies from the Responses to Click-Based Chirp-Modulated Sounds
od: Aurimas Mockevičius, i dr.
Izdano: (2023-03-01)

Chat2VIS: Generating Data Visualizations via Natural Language Using ChatGPT, Codex and GPT-3 Large Language Models
od: Paula Maddigan, i dr.
Izdano: (2023-01-01)

Disentangled Speech Embeddings Using Cross-Modal Self-Supervision
od: Nagrani, A, i dr.
Izdano: (2020)

Self-Supervised Audio-Visual Co-Segmentation
od: Rouditchenko, Andrew, i dr.
Izdano: (2022)

Self-Supervised Audio-Visual Co-Segmentation
od: Rouditchenko, Andrew, i dr.
Izdano: (2021)

Self-Supervised Autoencoders for Visual Anomaly Detection
od: Alexander Bauer, i dr.
Izdano: (2024-12-01)

Combining Unsupervised and Supervised Learning for Sample Efficient Continuous Language Grounding
od: Oliver Roesler
Izdano: (2022-09-01)

Weakly-supervised fingerspelling recognition in British Sign Language videos
od: Prajwal, KR, i dr.
Izdano: (2022)

Self-supervised learning of a facial attribute embedding from video
od: Wiles, O, i dr.
Izdano: (2018)

Self-supervised video object segmentation by motion grouping
od: Yang, C, i dr.
Izdano: (2021)

A Climate Hyperspectral Infrared Radiance Product (CHIRP) Combining the AIRS and CrIS Satellite Sounding Record
od: L. Larrabee Strow, i dr.
Izdano: (2021-01-01)

Diagnostics of HF radio channel: based on data from backscatter ionospheric sounding by continuous chirp signal
od: Ponomarchuk S.N., i dr.
Izdano: (2018-06-01)

Application of Optimized Adaptive Chirp Mode Decomposition Method in Chirp Signal
od: Junyuan Wang, i dr.
Izdano: (2020-05-01)

Self-supervised multi-modal alignment for whole body medical imaging
od: Windsor, R, i dr.
Izdano: (2021)

Now you're speaking my language: visual language identification
od: Afouras, T, i dr.
Izdano: (2020)

Exploring the Utility of ChatGPT for Self-directed Online Language Learning
od: Zixi Li, i dr.
Izdano: (2024-09-01)