Code-Switching in Automatic Speech Recognition: The Issues and Future Directions
Code-switching (CS) in spoken language is where the speech has two or more languages within an utterance. It is an unsolved issue in automatic speech recognition (ASR) research as ASR needs to recognise speech in bilingual and multilingual settings, where the accuracy of ASR systems declines with CS...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-09-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/12/19/9541 |
| _version_ | 1827655331952459776 |
|---|---|
| author | Mumtaz Begum Mustafa Mansoor Ali Yusoof Hasan Kahtan Khalaf Ahmad Abdel Rahman Mahmoud Abushariah Miss Laiha Mat Kiah Hua Nong Ting Saravanan Muthaiyah |
| author_facet | Mumtaz Begum Mustafa Mansoor Ali Yusoof Hasan Kahtan Khalaf Ahmad Abdel Rahman Mahmoud Abushariah Miss Laiha Mat Kiah Hua Nong Ting Saravanan Muthaiyah |
| author_sort | Mumtaz Begum Mustafa |
| collection | DOAJ |
| description | Code-switching (CS) in spoken language is where the speech has two or more languages within an utterance. It is an unsolved issue in automatic speech recognition (ASR) research as ASR needs to recognise speech in bilingual and multilingual settings, where the accuracy of ASR systems declines with CS due to pronunciation variation. There are very few reviews carried out on CS, with none conducted on bilingual and multilingual CS ASR systems. This study investigates the importance of CS in bilingual and multilingual speech recognition systems. To meet the objective of this study, two research questions were formulated, which cover both the current issues and the direction of the research. Our review focuses on databases, acoustic and language modelling, and evaluation metrics. Using selected keywords, this research has identified 274 papers and selected 42 experimental papers for review, of which 24 (representing 57%) have discussed CS, while the rest look at multilingual ASR research. The selected papers cover many well-resourced and under-resourced languages, and novel techniques to manage CS in ASR systems, which are mapping, combining and merging the phone sets of the languages experimented with in the research. Our review also examines the performance of those methods. This review found a significant variation in the performance of CS in terms of word error rates, indicating an inconsistency in the ability of ASRs to handle CS. In the conclusion, we suggest several future directions that address the issues identified in this review. |
| first_indexed | 2024-03-09T22:05:16Z |
| format | Article |
| id | doaj.art-225a4084e70f4b2093ca4bd9244977ae |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-09T22:05:16Z |
| publishDate | 2022-09-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-225a4084e70f4b2093ca4bd9244977ae2023-11-23T19:41:22ZengMDPI AGApplied Sciences2076-34172022-09-011219954110.3390/app12199541Code-Switching in Automatic Speech Recognition: The Issues and Future DirectionsMumtaz Begum Mustafa0Mansoor Ali Yusoof1Hasan Kahtan Khalaf2Ahmad Abdel Rahman Mahmoud Abushariah3Miss Laiha Mat Kiah4Hua Nong Ting5Saravanan Muthaiyah6Faculty of Computer Science and Information Technology, Universiti Malaya, Kuala Lumpur 50603, MalaysiaFaculty of Business Finance and Information Technology, MAHSA University, Jenjarom 42610, MalaysiaCardiff School of Technologies, Cardiff Metropolitan University, Llandaff Campus, Western Avenue, Cardiff CF5 2YB, UKDepartment of Biomedical Engineering, Faculty of Engineering, Universiti Malaya, Kuala Lumpur 50603, MalaysiaFaculty of Computer Science and Information Technology, Universiti Malaya, Kuala Lumpur 50603, MalaysiaDepartment of Biomedical Engineering, Faculty of Engineering, Universiti Malaya, Kuala Lumpur 50603, MalaysiaFaculty of Management, BR1018, Multimedia University, Persiaran Multimedia, Cyberjaya 63100, MalaysiaCode-switching (CS) in spoken language is where the speech has two or more languages within an utterance. It is an unsolved issue in automatic speech recognition (ASR) research as ASR needs to recognise speech in bilingual and multilingual settings, where the accuracy of ASR systems declines with CS due to pronunciation variation. There are very few reviews carried out on CS, with none conducted on bilingual and multilingual CS ASR systems. This study investigates the importance of CS in bilingual and multilingual speech recognition systems. To meet the objective of this study, two research questions were formulated, which cover both the current issues and the direction of the research. Our review focuses on databases, acoustic and language modelling, and evaluation metrics. Using selected keywords, this research has identified 274 papers and selected 42 experimental papers for review, of which 24 (representing 57%) have discussed CS, while the rest look at multilingual ASR research. The selected papers cover many well-resourced and under-resourced languages, and novel techniques to manage CS in ASR systems, which are mapping, combining and merging the phone sets of the languages experimented with in the research. Our review also examines the performance of those methods. This review found a significant variation in the performance of CS in terms of word error rates, indicating an inconsistency in the ability of ASRs to handle CS. In the conclusion, we suggest several future directions that address the issues identified in this review.https://www.mdpi.com/2076-3417/12/19/9541code-switchingautomatic speech recognition systemmultilingual speech recognitionbilingual speech recognitionlanguage and acoustic modelsevaluation metrics |
| spellingShingle | Mumtaz Begum Mustafa Mansoor Ali Yusoof Hasan Kahtan Khalaf Ahmad Abdel Rahman Mahmoud Abushariah Miss Laiha Mat Kiah Hua Nong Ting Saravanan Muthaiyah Code-Switching in Automatic Speech Recognition: The Issues and Future Directions Applied Sciences code-switching automatic speech recognition system multilingual speech recognition bilingual speech recognition language and acoustic models evaluation metrics |
| title | Code-Switching in Automatic Speech Recognition: The Issues and Future Directions |
| title_full | Code-Switching in Automatic Speech Recognition: The Issues and Future Directions |
| title_fullStr | Code-Switching in Automatic Speech Recognition: The Issues and Future Directions |
| title_full_unstemmed | Code-Switching in Automatic Speech Recognition: The Issues and Future Directions |
| title_short | Code-Switching in Automatic Speech Recognition: The Issues and Future Directions |
| title_sort | code switching in automatic speech recognition the issues and future directions |
| topic | code-switching automatic speech recognition system multilingual speech recognition bilingual speech recognition language and acoustic models evaluation metrics |
| url | https://www.mdpi.com/2076-3417/12/19/9541 |
| work_keys_str_mv | AT mumtazbegummustafa codeswitchinginautomaticspeechrecognitiontheissuesandfuturedirections AT mansooraliyusoof codeswitchinginautomaticspeechrecognitiontheissuesandfuturedirections AT hasankahtankhalaf codeswitchinginautomaticspeechrecognitiontheissuesandfuturedirections AT ahmadabdelrahmanmahmoudabushariah codeswitchinginautomaticspeechrecognitiontheissuesandfuturedirections AT misslaihamatkiah codeswitchinginautomaticspeechrecognitiontheissuesandfuturedirections AT huanongting codeswitchinginautomaticspeechrecognitiontheissuesandfuturedirections AT saravananmuthaiyah codeswitchinginautomaticspeechrecognitiontheissuesandfuturedirections |