MeWEHV: Mel and Wave Embeddings for Human Voice Tasks
A recent trend in speech processing is the use of embeddings created through machine learning models trained on a specific task with large datasets. By leveraging the knowledge already acquired, these models can be reused in new tasks where the amount of available data is small. This paper proposes...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10198451/ |
| _version_ | 1797746097514348544 |
|---|---|
| author | Andres Carofilis Laura Fernandez-Robles Enrique Alegre Eduardo Fidalgo |
| author_facet | Andres Carofilis Laura Fernandez-Robles Enrique Alegre Eduardo Fidalgo |
| author_sort | Andres Carofilis |
| collection | DOAJ |
| description | A recent trend in speech processing is the use of embeddings created through machine learning models trained on a specific task with large datasets. By leveraging the knowledge already acquired, these models can be reused in new tasks where the amount of available data is small. This paper proposes a pipeline to create a new model, called Mel and Wave Embeddings for Human Voice Tasks (MeWEHV), capable of generating robust embeddings for speech processing. MeWEHV combines the embeddings generated by a pre-trained raw audio waveform encoder model, and deep features extracted from Mel Frequency Cepstral Coefficients (MFCCs) using Convolutional Neural Networks (CNNs). We evaluate the performance of MeWEHV on three tasks: speaker, language, and accent identification. For the first one, we use the VoxCeleb1, and VBHIR datasets and present YouSpeakers204, a new and publicly available dataset for English speaker identification that contains 19607 audio clips from 204 persons speaking in six different accents, allowing other researchers to work with a very balanced dataset, and to create new models that are robust to multiple accents. For evaluating the language identification task, we use the VoxForge, Common Language, and the LRE17 datasets. Finally, for accent identification, we use the Latin American Spanish Corpora (LASC), Common Voice, and the NISP datasets. Our approach allows a significant increase in the performance of state-of-the-art embedding generation models on all the tested datasets, with a low additional computational cost. |
| first_indexed | 2024-03-12T15:32:12Z |
| format | Article |
| id | doaj.art-2705711a25654b4f8ad2a90da3226f3f |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-03-12T15:32:12Z |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-2705711a25654b4f8ad2a90da3226f3f2023-08-09T23:00:19ZengIEEEIEEE Access2169-35362023-01-0111800898010410.1109/ACCESS.2023.330097310198451MeWEHV: Mel and Wave Embeddings for Human Voice TasksAndres Carofilis0https://orcid.org/0000-0001-9446-0152Laura Fernandez-Robles1https://orcid.org/0000-0001-6573-8477Enrique Alegre2https://orcid.org/0000-0003-2081-774XEduardo Fidalgo3https://orcid.org/0000-0003-1202-5232Department of Electrical, Systems, and Automation Engineering, School of Industrial, Computer and Aerospace Engineering, Universidad de León, Campus de Vegazana, León, SpainDepartment of Mechanical, Computer, and Aerospace Engineering, Universidad de León, Campus de Vegazana, León, SpainDepartment of Electrical, Systems, and Automation Engineering, School of Industrial, Computer and Aerospace Engineering, Universidad de León, Campus de Vegazana, León, SpainDepartment of Electrical, Systems, and Automation Engineering, School of Industrial, Computer and Aerospace Engineering, Universidad de León, Campus de Vegazana, León, SpainA recent trend in speech processing is the use of embeddings created through machine learning models trained on a specific task with large datasets. By leveraging the knowledge already acquired, these models can be reused in new tasks where the amount of available data is small. This paper proposes a pipeline to create a new model, called Mel and Wave Embeddings for Human Voice Tasks (MeWEHV), capable of generating robust embeddings for speech processing. MeWEHV combines the embeddings generated by a pre-trained raw audio waveform encoder model, and deep features extracted from Mel Frequency Cepstral Coefficients (MFCCs) using Convolutional Neural Networks (CNNs). We evaluate the performance of MeWEHV on three tasks: speaker, language, and accent identification. For the first one, we use the VoxCeleb1, and VBHIR datasets and present YouSpeakers204, a new and publicly available dataset for English speaker identification that contains 19607 audio clips from 204 persons speaking in six different accents, allowing other researchers to work with a very balanced dataset, and to create new models that are robust to multiple accents. For evaluating the language identification task, we use the VoxForge, Common Language, and the LRE17 datasets. Finally, for accent identification, we use the Latin American Spanish Corpora (LASC), Common Voice, and the NISP datasets. Our approach allows a significant increase in the performance of state-of-the-art embedding generation models on all the tested datasets, with a low additional computational cost.https://ieeexplore.ieee.org/document/10198451/EmbeddingsHuBERTspeech classificationWavLMXLSR-Wav2Vec2YouSpeakers204 |
| spellingShingle | Andres Carofilis Laura Fernandez-Robles Enrique Alegre Eduardo Fidalgo MeWEHV: Mel and Wave Embeddings for Human Voice Tasks IEEE Access Embeddings HuBERT speech classification WavLM XLSR-Wav2Vec2 YouSpeakers204 |
| title | MeWEHV: Mel and Wave Embeddings for Human Voice Tasks |
| title_full | MeWEHV: Mel and Wave Embeddings for Human Voice Tasks |
| title_fullStr | MeWEHV: Mel and Wave Embeddings for Human Voice Tasks |
| title_full_unstemmed | MeWEHV: Mel and Wave Embeddings for Human Voice Tasks |
| title_short | MeWEHV: Mel and Wave Embeddings for Human Voice Tasks |
| title_sort | mewehv mel and wave embeddings for human voice tasks |
| topic | Embeddings HuBERT speech classification WavLM XLSR-Wav2Vec2 YouSpeakers204 |
| url | https://ieeexplore.ieee.org/document/10198451/ |
| work_keys_str_mv | AT andrescarofilis mewehvmelandwaveembeddingsforhumanvoicetasks AT laurafernandezrobles mewehvmelandwaveembeddingsforhumanvoicetasks AT enriquealegre mewehvmelandwaveembeddingsforhumanvoicetasks AT eduardofidalgo mewehvmelandwaveembeddingsforhumanvoicetasks |