Separating the “chirp” from the “chat”: self-supervised visual grounding of sound and language

Separating the “chirp” from the “chat”: self-supervised visual grounding of sound and language

We present DenseAV, a novel dual encoder grounding architecture that learns high-resolution, semantically meaningful, and audio-visual aligned features solely through watching videos. We show that DenseAV can discover the “meaning” of words and the “location” of sounds without explicit localization...

書誌詳細
主要な著者:	Hamilton, M, Zisserman, A, Hershey, JR, Freeman, WT
フォーマット:	Conference item
言語:	English
出版事項:	IEEE 2024

類似資料

Multi-task self-supervised visual learning
著者:: Doersch, C, 等
出版事項: (2017)

Ambient Sound Provides Supervision for Visual Learning
著者:: Owens, Andrew Hale, 等
出版事項: (2017)

Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning
著者:: Owens, Andrew, 等
出版事項: (2021)

Self-Supervised Learning for Audio-Visual Relationships of Videos With Stereo Sounds
著者:: Tomoya Sato, 等
出版事項: (2022-01-01)

Self-supervised learning of audio-visual objects from video
著者:: Afouras, T, 等
出版事項: (2020)

Enhancement of sound by soft reflections in exponentially chirped crystals
著者:: A. Cebrecos, 等
出版事項: (2014-12-01)

Music Gesture for Visual Sound Separation
著者:: Gan, Chuang, 等
出版事項: (2021)

First observations of oblique ionospheric sounding chirp signal in Mexico
著者:: M.A. Sergeeva, 等
出版事項: (2019-03-01)

Features of backscatter ionospheric sounding as studied with a chirp ionosonde
著者:: Ponomarchuk S.N., 等
出版事項: (2017-09-01)

Audio-Visual Self-Supervised Terrain Type Recognition for Ground Mobile Platforms
著者:: Akiyoshi Kurobe, 等
出版事項: (2021-01-01)

Self-supervised learning for spinal MRIs
著者:: Jamaludin, A, 等
出版事項: (2017)

Self-Supervised Moving Vehicle Tracking With Stereo Sound
著者:: Gan, Chuang, 等
出版事項: (2021)

Unsupervised discovery of visual object class hierarchies
著者:: Sivic, J, 等
出版事項: (2008)

Localizing visual sounds the hard way
著者:: Vedaldi, A, 等
出版事項: (2021)

Weakly supervised scale-invariant learning of models for visual recognition
著者:: Fergus, R, 等
出版事項: (2006)

Enhancing bowel sound recognition with self-attention and self-supervised pre-training.
著者:: Yansuo Yu, 等
出版事項: (2024-01-01)

Self-supervised co-training for video representation learning
著者:: Han, T, 等
出版事項: (2020)

ESTIMATING ANTENNA COUPLING FACTOR FOR PROBLEM OF TOPSIDEIONOSPHERE SOUNDING FROM SPACE BY CHIRP SIGNALS
著者:: Podlesnyi A.V., 等
出版事項: (2019-12-01)

Self-supervised learning of class embeddings from video
著者:: Wiles, O, 等
出版事項: (2020)

Sight to Sound: An End-to-End Approach for Visual Piano Transcription
著者:: Koepke, S, 等
出版事項: (2020)

ASDNet: An Efficient Self-Supervised Convolutional Network for Anomalous Sound Detection
著者:: Dewei Kong, 等
出版事項: (2025-01-01)

Self-Supervised Transfer Learning from Natural Images for Sound Classification
著者:: Sungho Shin, 等
出版事項: (2021-03-01)

Visually Indicated Sounds
著者:: Isola, Phillip, 等
出版事項: (2017)

Direct Underwater Sound Velocity Measurement Based on the Acousto-Optic Self-Interference Effect between the Chirp Signal and the Optical Frequency Comb
著者:: Zihui Yang, 等
出版事項: (2022-12-01)

Extraction of Individual EEG Gamma Frequencies from the Responses to Click-Based Chirp-Modulated Sounds
著者:: Aurimas Mockevičius, 等
出版事項: (2023-03-01)

Chat2VIS: Generating Data Visualizations via Natural Language Using ChatGPT, Codex and GPT-3 Large Language Models
著者:: Paula Maddigan, 等
出版事項: (2023-01-01)

Disentangled Speech Embeddings Using Cross-Modal Self-Supervision
著者:: Nagrani, A, 等
出版事項: (2020)

Self-Supervised Audio-Visual Co-Segmentation
著者:: Rouditchenko, Andrew, 等
出版事項: (2022)

Self-Supervised Audio-Visual Co-Segmentation
著者:: Rouditchenko, Andrew, 等
出版事項: (2021)

Self-Supervised Autoencoders for Visual Anomaly Detection
著者:: Alexander Bauer, 等
出版事項: (2024-12-01)

Combining Unsupervised and Supervised Learning for Sample Efficient Continuous Language Grounding
著者:: Oliver Roesler
出版事項: (2022-09-01)

Weakly-supervised fingerspelling recognition in British Sign Language videos
著者:: Prajwal, KR, 等
出版事項: (2022)

Self-supervised learning of a facial attribute embedding from video
著者:: Wiles, O, 等
出版事項: (2018)

Self-supervised video object segmentation by motion grouping
著者:: Yang, C, 等
出版事項: (2021)

A Climate Hyperspectral Infrared Radiance Product (CHIRP) Combining the AIRS and CrIS Satellite Sounding Record
著者:: L. Larrabee Strow, 等
出版事項: (2021-01-01)

Diagnostics of HF radio channel: based on data from backscatter ionospheric sounding by continuous chirp signal
著者:: Ponomarchuk S.N., 等
出版事項: (2018-06-01)

Application of Optimized Adaptive Chirp Mode Decomposition Method in Chirp Signal
著者:: Junyuan Wang, 等
出版事項: (2020-05-01)

Self-supervised multi-modal alignment for whole body medical imaging
著者:: Windsor, R, 等
出版事項: (2021)

Now you're speaking my language: visual language identification
著者:: Afouras, T, 等
出版事項: (2020)

Exploring the Utility of ChatGPT for Self-directed Online Language Learning
著者:: Zixi Li, 等
出版事項: (2024-09-01)