My lips are concealed: audio-visual speech enhancement through obstructions
Our objective is an audio-visual model for separating a single speaker from a mixture of sounds such as other speakers and background noise. Moreover, we wish to hear the speaker even when the visual cues are temporarily absent due to occlusion. To this end we introduce a deep audio-visual speech e...
| Main Authors: | , , |
|---|---|
| Format: | Conference item |
| Published: |
ISCA
2019
|
| _version_ | 1826262881516126208 |
|---|---|
| author | Afouras, T Chung, JS Zisserman, A |
| author_facet | Afouras, T Chung, JS Zisserman, A |
| author_sort | Afouras, T |
| collection | OXFORD |
| description | Our objective is an audio-visual model for separating a single speaker from a mixture of sounds such as other speakers and background noise. Moreover, we wish to hear the speaker even when the visual cues are temporarily absent due to occlusion.
To this end we introduce a deep audio-visual speech enhancement network that is able to separate a speaker’s voice by conditioning on both the speaker’s lip movements and/or a representation of their voice. The voice representation can be obtained by either (i) enrollment, or (ii) by self-enrollment — learning the representation on-the-fly given sufficient unobstructed visual input. The model is trained by blending audios, and by introducing artificial occlusions around the mouth region that prevent the visual modality from dominating.
The method is speaker-independent, and we demonstrate it on real examples of speakers unheard (and unseen) during training. The method also improves over previous models in particular for cases of occlusion in the visual modality. |
| first_indexed | 2024-03-06T19:42:50Z |
| format | Conference item |
| id | oxford-uuid:213f5842-3978-4507-99ab-41caebd76757 |
| institution | University of Oxford |
| last_indexed | 2024-03-06T19:42:50Z |
| publishDate | 2019 |
| publisher | ISCA |
| record_format | dspace |
| spelling | oxford-uuid:213f5842-3978-4507-99ab-41caebd767572022-03-26T11:32:21ZMy lips are concealed: audio-visual speech enhancement through obstructionsConference itemhttp://purl.org/coar/resource_type/c_5794uuid:213f5842-3978-4507-99ab-41caebd76757Symplectic ElementsISCA2019Afouras, TChung, JSZisserman, AOur objective is an audio-visual model for separating a single speaker from a mixture of sounds such as other speakers and background noise. Moreover, we wish to hear the speaker even when the visual cues are temporarily absent due to occlusion. To this end we introduce a deep audio-visual speech enhancement network that is able to separate a speaker’s voice by conditioning on both the speaker’s lip movements and/or a representation of their voice. The voice representation can be obtained by either (i) enrollment, or (ii) by self-enrollment — learning the representation on-the-fly given sufficient unobstructed visual input. The model is trained by blending audios, and by introducing artificial occlusions around the mouth region that prevent the visual modality from dominating. The method is speaker-independent, and we demonstrate it on real examples of speakers unheard (and unseen) during training. The method also improves over previous models in particular for cases of occlusion in the visual modality. |
| spellingShingle | Afouras, T Chung, JS Zisserman, A My lips are concealed: audio-visual speech enhancement through obstructions |
| title | My lips are concealed: audio-visual speech enhancement through obstructions |
| title_full | My lips are concealed: audio-visual speech enhancement through obstructions |
| title_fullStr | My lips are concealed: audio-visual speech enhancement through obstructions |
| title_full_unstemmed | My lips are concealed: audio-visual speech enhancement through obstructions |
| title_short | My lips are concealed: audio-visual speech enhancement through obstructions |
| title_sort | my lips are concealed audio visual speech enhancement through obstructions |
| work_keys_str_mv | AT afourast mylipsareconcealedaudiovisualspeechenhancementthroughobstructions AT chungjs mylipsareconcealedaudiovisualspeechenhancementthroughobstructions AT zissermana mylipsareconcealedaudiovisualspeechenhancementthroughobstructions |