Sequence-to-Sequence Voice Reconstruction for Silent Speech in a Tonal Language
Silent speech decoding (SSD), based on articulatory neuromuscular activities, has become a prevalent task of brain–computer interfaces (BCIs) in recent years. Many works have been devoted to decoding surface electromyography (sEMG) from articulatory neuromuscular activities. However, restoring silen...
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-06-01
|
| Series: | Brain Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3425/12/7/818 |
| _version_ | 1797407242735058944 |
|---|---|
| author | Huiyan Li Haohong Lin You Wang Hengyang Wang Ming Zhang Han Gao Qing Ai Zhiyuan Luo Guang Li |
| author_facet | Huiyan Li Haohong Lin You Wang Hengyang Wang Ming Zhang Han Gao Qing Ai Zhiyuan Luo Guang Li |
| author_sort | Huiyan Li |
| collection | DOAJ |
| description | Silent speech decoding (SSD), based on articulatory neuromuscular activities, has become a prevalent task of brain–computer interfaces (BCIs) in recent years. Many works have been devoted to decoding surface electromyography (sEMG) from articulatory neuromuscular activities. However, restoring silent speech in tonal languages such as Mandarin Chinese is still difficult. This paper proposes an optimized sequence-to-sequence (Seq2Seq) approach to synthesize voice from the sEMG-based silent speech. We extract duration information to regulate the sEMG-based silent speech using the audio length. Then, we provide a deep-learning model with an encoder–decoder structure and a state-of-the-art vocoder to generate the audio waveform. Experiments based on six Mandarin Chinese speakers demonstrate that the proposed model can successfully decode silent speech in Mandarin Chinese and achieve a character error rate (CER) of 6.41% on average with human evaluation. |
| first_indexed | 2024-03-09T03:38:50Z |
| format | Article |
| id | doaj.art-d6b750ede55a46218068a0de5368a07f |
| institution | Directory Open Access Journal |
| issn | 2076-3425 |
| language | English |
| last_indexed | 2024-03-09T03:38:50Z |
| publishDate | 2022-06-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Brain Sciences |
| spelling | doaj.art-d6b750ede55a46218068a0de5368a07f2023-12-03T14:44:42ZengMDPI AGBrain Sciences2076-34252022-06-0112781810.3390/brainsci12070818Sequence-to-Sequence Voice Reconstruction for Silent Speech in a Tonal LanguageHuiyan Li0Haohong Lin1You Wang2Hengyang Wang3Ming Zhang4Han Gao5Qing Ai6Zhiyuan Luo7Guang Li8State Key Laboratory of Industrial Control Technology, Institute of Cyber Systems and Control, Zhejiang University, Hangzhou 310027, ChinaState Key Laboratory of Industrial Control Technology, Institute of Cyber Systems and Control, Zhejiang University, Hangzhou 310027, ChinaState Key Laboratory of Industrial Control Technology, Institute of Cyber Systems and Control, Zhejiang University, Hangzhou 310027, ChinaState Key Laboratory of Industrial Control Technology, Institute of Cyber Systems and Control, Zhejiang University, Hangzhou 310027, ChinaState Key Laboratory of Industrial Control Technology, Institute of Cyber Systems and Control, Zhejiang University, Hangzhou 310027, ChinaState Key Laboratory of Industrial Control Technology, Institute of Cyber Systems and Control, Zhejiang University, Hangzhou 310027, ChinaState Key Laboratory of Industrial Control Technology, Institute of Cyber Systems and Control, Zhejiang University, Hangzhou 310027, ChinaDepartment of Computer Science, Royal Holloway, University of London, Egham Hill, Egham TW20 0EX, Surrey, UKState Key Laboratory of Industrial Control Technology, Institute of Cyber Systems and Control, Zhejiang University, Hangzhou 310027, ChinaSilent speech decoding (SSD), based on articulatory neuromuscular activities, has become a prevalent task of brain–computer interfaces (BCIs) in recent years. Many works have been devoted to decoding surface electromyography (sEMG) from articulatory neuromuscular activities. However, restoring silent speech in tonal languages such as Mandarin Chinese is still difficult. This paper proposes an optimized sequence-to-sequence (Seq2Seq) approach to synthesize voice from the sEMG-based silent speech. We extract duration information to regulate the sEMG-based silent speech using the audio length. Then, we provide a deep-learning model with an encoder–decoder structure and a state-of-the-art vocoder to generate the audio waveform. Experiments based on six Mandarin Chinese speakers demonstrate that the proposed model can successfully decode silent speech in Mandarin Chinese and achieve a character error rate (CER) of 6.41% on average with human evaluation.https://www.mdpi.com/2076-3425/12/7/818silent speechelectromyography (EMG)neuromuscular signalsequence-to-sequence (Seq2Seq) |
| spellingShingle | Huiyan Li Haohong Lin You Wang Hengyang Wang Ming Zhang Han Gao Qing Ai Zhiyuan Luo Guang Li Sequence-to-Sequence Voice Reconstruction for Silent Speech in a Tonal Language Brain Sciences silent speech electromyography (EMG) neuromuscular signal sequence-to-sequence (Seq2Seq) |
| title | Sequence-to-Sequence Voice Reconstruction for Silent Speech in a Tonal Language |
| title_full | Sequence-to-Sequence Voice Reconstruction for Silent Speech in a Tonal Language |
| title_fullStr | Sequence-to-Sequence Voice Reconstruction for Silent Speech in a Tonal Language |
| title_full_unstemmed | Sequence-to-Sequence Voice Reconstruction for Silent Speech in a Tonal Language |
| title_short | Sequence-to-Sequence Voice Reconstruction for Silent Speech in a Tonal Language |
| title_sort | sequence to sequence voice reconstruction for silent speech in a tonal language |
| topic | silent speech electromyography (EMG) neuromuscular signal sequence-to-sequence (Seq2Seq) |
| url | https://www.mdpi.com/2076-3425/12/7/818 |
| work_keys_str_mv | AT huiyanli sequencetosequencevoicereconstructionforsilentspeechinatonallanguage AT haohonglin sequencetosequencevoicereconstructionforsilentspeechinatonallanguage AT youwang sequencetosequencevoicereconstructionforsilentspeechinatonallanguage AT hengyangwang sequencetosequencevoicereconstructionforsilentspeechinatonallanguage AT mingzhang sequencetosequencevoicereconstructionforsilentspeechinatonallanguage AT hangao sequencetosequencevoicereconstructionforsilentspeechinatonallanguage AT qingai sequencetosequencevoicereconstructionforsilentspeechinatonallanguage AT zhiyuanluo sequencetosequencevoicereconstructionforsilentspeechinatonallanguage AT guangli sequencetosequencevoicereconstructionforsilentspeechinatonallanguage |