Speech Processing for Language Learning: A Practical Approach to Computer-Assisted Pronunciation Teaching
This article contributes to the discourse on how contemporary computer and information technology may help in improving foreign language learning not only by supporting better and more flexible workflow and digitizing study materials but also through creating completely new use cases made possible b...
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-01-01
|
| Series: | Electronics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2079-9292/10/3/235 |
| _version_ | 1797409139491602432 |
|---|---|
| author | Natalia Bogach Elena Boitsova Sergey Chernonog Anton Lamtev Maria Lesnichaya Iurii Lezhenin Andrey Novopashenny Roman Svechnikov Daria Tsikach Konstantin Vasiliev Evgeny Pyshkin John Blake |
| author_facet | Natalia Bogach Elena Boitsova Sergey Chernonog Anton Lamtev Maria Lesnichaya Iurii Lezhenin Andrey Novopashenny Roman Svechnikov Daria Tsikach Konstantin Vasiliev Evgeny Pyshkin John Blake |
| author_sort | Natalia Bogach |
| collection | DOAJ |
| description | This article contributes to the discourse on how contemporary computer and information technology may help in improving foreign language learning not only by supporting better and more flexible workflow and digitizing study materials but also through creating completely new use cases made possible by technological improvements in signal processing algorithms. We discuss an approach and propose a holistic solution to teaching the phonological phenomena which are crucial for correct pronunciation, such as the phonemes; the energy and duration of syllables and pauses, which construct the phrasal rhythm; and the tone movement within an utterance, i.e., the phrasal intonation. The working prototype of StudyIntonation Computer-Assisted Pronunciation Training (CAPT) system is a tool for mobile devices, which offers a set of tasks based on a “listen and repeat” approach and gives the audio-visual feedback in real time. The present work summarizes the efforts taken to enrich the current version of this CAPT tool with two new functions: the phonetic transcription and rhythmic patterns of model and learner speech. Both are designed on a base of a third-party automatic speech recognition (ASR) library Kaldi, which was incorporated inside StudyIntonation signal processing software core. We also examine the scope of automatic speech recognition applicability within the CAPT system workflow and evaluate the Levenstein distance between the transcription made by human experts and that obtained automatically in our code. We developed an algorithm of rhythm reconstruction using acoustic and language ASR models. It is also shown that even having sufficiently correct production of phonemes, the learners do not produce a correct phrasal rhythm and intonation, and therefore, the joint training of sounds, rhythm and intonation within a single learning environment is beneficial. To mitigate the recording imperfections voice activity detection (VAD) is applied to all the speech records processed. The try-outs showed that StudyIntonation can create transcriptions and process rhythmic patterns, but some specific problems with connected speech transcription were detected. The learners feedback in the sense of pronunciation assessment was also updated and a conventional mechanism based on dynamic time warping (DTW) was combined with cross-recurrence quantification analysis (CRQA) approach, which resulted in a better discriminating ability. The CRQA metrics combined with those of DTW were shown to add to the accuracy of learner performance estimation. The major implications for computer-assisted English pronunciation teaching are discussed. |
| first_indexed | 2024-03-09T04:10:00Z |
| format | Article |
| id | doaj.art-c9e8d6adb7a049b99a984fe5bcc5ca99 |
| institution | Directory Open Access Journal |
| issn | 2079-9292 |
| language | English |
| last_indexed | 2024-03-09T04:10:00Z |
| publishDate | 2021-01-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Electronics |
| spelling | doaj.art-c9e8d6adb7a049b99a984fe5bcc5ca992023-12-03T14:01:59ZengMDPI AGElectronics2079-92922021-01-0110323510.3390/electronics10030235Speech Processing for Language Learning: A Practical Approach to Computer-Assisted Pronunciation TeachingNatalia Bogach0Elena Boitsova1Sergey Chernonog2Anton Lamtev3Maria Lesnichaya4Iurii Lezhenin5Andrey Novopashenny6Roman Svechnikov7Daria Tsikach8Konstantin Vasiliev9Evgeny Pyshkin10John Blake11Institute of Computer Science and Technology, Peter the Great St. Petersburg Polytechnic University, 195029 St. Petersburg, RussiaInstitute of Computer Science and Technology, Peter the Great St. Petersburg Polytechnic University, 195029 St. Petersburg, RussiaInstitute of Computer Science and Technology, Peter the Great St. Petersburg Polytechnic University, 195029 St. Petersburg, RussiaInstitute of Computer Science and Technology, Peter the Great St. Petersburg Polytechnic University, 195029 St. Petersburg, RussiaInstitute of Computer Science and Technology, Peter the Great St. Petersburg Polytechnic University, 195029 St. Petersburg, RussiaInstitute of Computer Science and Technology, Peter the Great St. Petersburg Polytechnic University, 195029 St. Petersburg, RussiaInstitute of Computer Science and Technology, Peter the Great St. Petersburg Polytechnic University, 195029 St. Petersburg, RussiaInstitute of Computer Science and Technology, Peter the Great St. Petersburg Polytechnic University, 195029 St. Petersburg, RussiaInstitute of Computer Science and Technology, Peter the Great St. Petersburg Polytechnic University, 195029 St. Petersburg, RussiaInstitute of Computer Science and Technology, Peter the Great St. Petersburg Polytechnic University, 195029 St. Petersburg, RussiaDivision of Information Systems, School of Computer Science and Engineering, University of Aizu, Aizuwakamatsu 965-8580, JapanCenter for Language Research, School of Computer Science and Engineering, University of Aizu, Aizuwakamatsu 965-8580, JapanThis article contributes to the discourse on how contemporary computer and information technology may help in improving foreign language learning not only by supporting better and more flexible workflow and digitizing study materials but also through creating completely new use cases made possible by technological improvements in signal processing algorithms. We discuss an approach and propose a holistic solution to teaching the phonological phenomena which are crucial for correct pronunciation, such as the phonemes; the energy and duration of syllables and pauses, which construct the phrasal rhythm; and the tone movement within an utterance, i.e., the phrasal intonation. The working prototype of StudyIntonation Computer-Assisted Pronunciation Training (CAPT) system is a tool for mobile devices, which offers a set of tasks based on a “listen and repeat” approach and gives the audio-visual feedback in real time. The present work summarizes the efforts taken to enrich the current version of this CAPT tool with two new functions: the phonetic transcription and rhythmic patterns of model and learner speech. Both are designed on a base of a third-party automatic speech recognition (ASR) library Kaldi, which was incorporated inside StudyIntonation signal processing software core. We also examine the scope of automatic speech recognition applicability within the CAPT system workflow and evaluate the Levenstein distance between the transcription made by human experts and that obtained automatically in our code. We developed an algorithm of rhythm reconstruction using acoustic and language ASR models. It is also shown that even having sufficiently correct production of phonemes, the learners do not produce a correct phrasal rhythm and intonation, and therefore, the joint training of sounds, rhythm and intonation within a single learning environment is beneficial. To mitigate the recording imperfections voice activity detection (VAD) is applied to all the speech records processed. The try-outs showed that StudyIntonation can create transcriptions and process rhythmic patterns, but some specific problems with connected speech transcription were detected. The learners feedback in the sense of pronunciation assessment was also updated and a conventional mechanism based on dynamic time warping (DTW) was combined with cross-recurrence quantification analysis (CRQA) approach, which resulted in a better discriminating ability. The CRQA metrics combined with those of DTW were shown to add to the accuracy of learner performance estimation. The major implications for computer-assisted English pronunciation teaching are discussed.https://www.mdpi.com/2079-9292/10/3/235speech processingcomputer-assisted pronunciation training (CAPT)voice activity detection (VAD)audio-visual feedbacktime warping (DTW)cross-recurrence quantification analysis (CRQA) |
| spellingShingle | Natalia Bogach Elena Boitsova Sergey Chernonog Anton Lamtev Maria Lesnichaya Iurii Lezhenin Andrey Novopashenny Roman Svechnikov Daria Tsikach Konstantin Vasiliev Evgeny Pyshkin John Blake Speech Processing for Language Learning: A Practical Approach to Computer-Assisted Pronunciation Teaching Electronics speech processing computer-assisted pronunciation training (CAPT) voice activity detection (VAD) audio-visual feedback time warping (DTW) cross-recurrence quantification analysis (CRQA) |
| title | Speech Processing for Language Learning: A Practical Approach to Computer-Assisted Pronunciation Teaching |
| title_full | Speech Processing for Language Learning: A Practical Approach to Computer-Assisted Pronunciation Teaching |
| title_fullStr | Speech Processing for Language Learning: A Practical Approach to Computer-Assisted Pronunciation Teaching |
| title_full_unstemmed | Speech Processing for Language Learning: A Practical Approach to Computer-Assisted Pronunciation Teaching |
| title_short | Speech Processing for Language Learning: A Practical Approach to Computer-Assisted Pronunciation Teaching |
| title_sort | speech processing for language learning a practical approach to computer assisted pronunciation teaching |
| topic | speech processing computer-assisted pronunciation training (CAPT) voice activity detection (VAD) audio-visual feedback time warping (DTW) cross-recurrence quantification analysis (CRQA) |
| url | https://www.mdpi.com/2079-9292/10/3/235 |
| work_keys_str_mv | AT nataliabogach speechprocessingforlanguagelearningapracticalapproachtocomputerassistedpronunciationteaching AT elenaboitsova speechprocessingforlanguagelearningapracticalapproachtocomputerassistedpronunciationteaching AT sergeychernonog speechprocessingforlanguagelearningapracticalapproachtocomputerassistedpronunciationteaching AT antonlamtev speechprocessingforlanguagelearningapracticalapproachtocomputerassistedpronunciationteaching AT marialesnichaya speechprocessingforlanguagelearningapracticalapproachtocomputerassistedpronunciationteaching AT iuriilezhenin speechprocessingforlanguagelearningapracticalapproachtocomputerassistedpronunciationteaching AT andreynovopashenny speechprocessingforlanguagelearningapracticalapproachtocomputerassistedpronunciationteaching AT romansvechnikov speechprocessingforlanguagelearningapracticalapproachtocomputerassistedpronunciationteaching AT dariatsikach speechprocessingforlanguagelearningapracticalapproachtocomputerassistedpronunciationteaching AT konstantinvasiliev speechprocessingforlanguagelearningapracticalapproachtocomputerassistedpronunciationteaching AT evgenypyshkin speechprocessingforlanguagelearningapracticalapproachtocomputerassistedpronunciationteaching AT johnblake speechprocessingforlanguagelearningapracticalapproachtocomputerassistedpronunciationteaching |