Audio-Visual Overlapped Speech Detection for Spontaneous Distant Speech
Although advances in deep learning have brought remarkable improvements to Overlapped Speech Detection (OSD), the performance in far-field environments is still limited owing to the lack of real-world overlapped speech and a low signal-to-noise ratio. In this paper, we present an end-to-end audiovis...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10064301/ |
| _version_ | 1827980952241963008 |
|---|---|
| author | Minyoung Kyoung Hyungbae Jeon Kiyoung Park |
| author_facet | Minyoung Kyoung Hyungbae Jeon Kiyoung Park |
| author_sort | Minyoung Kyoung |
| collection | DOAJ |
| description | Although advances in deep learning have brought remarkable improvements to Overlapped Speech Detection (OSD), the performance in far-field environments is still limited owing to the lack of real-world overlapped speech and a low signal-to-noise ratio. In this paper, we present an end-to-end audiovisual OSD system based on decision fusion between audio and video modalities. Firstly, we propose a simple yet powerful audio data augmentation method for spontaneous distant speech data. Secondly, to maximize the effectiveness of the video modality, we design a video OSD system based on a cross-speaker attention module that explores the visual correlation between multiple speakers. Lastly, we present cross-modality attention module to make the final decision more accurate. Our experimental results demonstrate that our approach outperforms current state-of-the-art methods on a real-world distant speech dataset. Moreover, our approach can robustly detect overlapped speech when compared with its counterpart, which uses audio modality alone. |
| first_indexed | 2024-04-09T22:01:01Z |
| format | Article |
| id | doaj.art-877df0adf2224d0ca90686a3096816ef |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-04-09T22:01:01Z |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-877df0adf2224d0ca90686a3096816ef2023-03-23T23:00:09ZengIEEEIEEE Access2169-35362023-01-0111274262743210.1109/ACCESS.2023.325452910064301Audio-Visual Overlapped Speech Detection for Spontaneous Distant SpeechMinyoung Kyoung0https://orcid.org/0009-0006-2814-9386Hyungbae Jeon1Kiyoung Park2Electronics and Telecommunications Research Institute (ETRI), Daejeon, Republic of KoreaElectronics and Telecommunications Research Institute (ETRI), Daejeon, Republic of KoreaElectronics and Telecommunications Research Institute (ETRI), Daejeon, Republic of KoreaAlthough advances in deep learning have brought remarkable improvements to Overlapped Speech Detection (OSD), the performance in far-field environments is still limited owing to the lack of real-world overlapped speech and a low signal-to-noise ratio. In this paper, we present an end-to-end audiovisual OSD system based on decision fusion between audio and video modalities. Firstly, we propose a simple yet powerful audio data augmentation method for spontaneous distant speech data. Secondly, to maximize the effectiveness of the video modality, we design a video OSD system based on a cross-speaker attention module that explores the visual correlation between multiple speakers. Lastly, we present cross-modality attention module to make the final decision more accurate. Our experimental results demonstrate that our approach outperforms current state-of-the-art methods on a real-world distant speech dataset. Moreover, our approach can robustly detect overlapped speech when compared with its counterpart, which uses audio modality alone.https://ieeexplore.ieee.org/document/10064301/Overlapped speech detectionfar-field audio data augmentationaudiovisual speech processingmultimodal deep learning |
| spellingShingle | Minyoung Kyoung Hyungbae Jeon Kiyoung Park Audio-Visual Overlapped Speech Detection for Spontaneous Distant Speech IEEE Access Overlapped speech detection far-field audio data augmentation audiovisual speech processing multimodal deep learning |
| title | Audio-Visual Overlapped Speech Detection for Spontaneous Distant Speech |
| title_full | Audio-Visual Overlapped Speech Detection for Spontaneous Distant Speech |
| title_fullStr | Audio-Visual Overlapped Speech Detection for Spontaneous Distant Speech |
| title_full_unstemmed | Audio-Visual Overlapped Speech Detection for Spontaneous Distant Speech |
| title_short | Audio-Visual Overlapped Speech Detection for Spontaneous Distant Speech |
| title_sort | audio visual overlapped speech detection for spontaneous distant speech |
| topic | Overlapped speech detection far-field audio data augmentation audiovisual speech processing multimodal deep learning |
| url | https://ieeexplore.ieee.org/document/10064301/ |
| work_keys_str_mv | AT minyoungkyoung audiovisualoverlappedspeechdetectionforspontaneousdistantspeech AT hyungbaejeon audiovisualoverlappedspeechdetectionforspontaneousdistantspeech AT kiyoungpark audiovisualoverlappedspeechdetectionforspontaneousdistantspeech |