Lexical and Language Modeling of Diacritics and Morphemes in Arabic Automatic Speech Recognition
Thesis: S.M., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2014.
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Language: | eng |
| Published: |
Massachusetts Institute of Technology
2014
|
| Subjects: | |
| Online Access: | http://hdl.handle.net/1721.1/87941 |
| _version_ | 1826189754557792256 |
|---|---|
| author | Alhanai, Tuka (Tuka Waddah Talib Ali Al Hanai) |
| author2 | James R. Glass. |
| author_facet | James R. Glass. Alhanai, Tuka (Tuka Waddah Talib Ali Al Hanai) |
| author_sort | Alhanai, Tuka (Tuka Waddah Talib Ali Al Hanai) |
| collection | MIT |
| description | Thesis: S.M., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2014. |
| first_indexed | 2024-09-23T08:20:49Z |
| format | Thesis |
| id | mit-1721.1/87941 |
| institution | Massachusetts Institute of Technology |
| language | eng |
| last_indexed | 2024-09-23T08:20:49Z |
| publishDate | 2014 |
| publisher | Massachusetts Institute of Technology |
| record_format | dspace |
| spelling | mit-1721.1/879412020-03-31T14:42:57Z Lexical and Language Modeling of Diacritics and Morphemes in Arabic Automatic Speech Recognition Alhanai, Tuka (Tuka Waddah Talib Ali Al Hanai) James R. Glass. Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science. Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Electrical Engineering and Computer Science. Thesis: S.M., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2014. Cataloged from PDF version of thesis. Includes bibliographical references (pages 69-72). Arabic is a morphologically rich language which rarely displays diacritics. These two features of the language pose challenges when building Automatic Speech Recognition (ASR) systems. Morphological complexity leads to many possible combinations of stems and affixes to form words, and produces texts with high Out Of Vocabulary (OOV) rates. In addition, texts rarely display diacritics which informs the reader about short vowels, geminates, and nunnations (word ending /n/). A lack of diacritics means that 30% of textual information is missing, causing ambiguities in lexical and language modeling when attempting to model pronunciations, and the context of a particular pronunciation. Intuitively, from an English centric view, the phrase th'wrtr wrt n thwrt with 'morphological decomposition' is realized as, th wrtr wrt n th wrt. Including 'diacritics' produces, the writer wrote in the writ. Thus our investigations in this thesis are twofold. Firstly, we show the benefits and interactions between modeling all classes of diacritics (short vowels, geminates, nunnations) in the lexicon. On a Modern Standard Arabic (MSA) corpus of broadcast news, this provides a 1.9% absolute improvement in Word Error Rate (WER) (p < 0.001). We also extend this graphemic lexicon with pronunciation rules, yielding a significant improvement over a lexicon that does not explicitly nodel diacritics. This results in a of 2.4% absolute improvement in WER (p < 0.001). Secondly, we show the benefits of language modeling at the morphemic level with diacritics, over the commonly available, word-based, nondiacratized text. This yields an absolute WER improvement of 1.0% (p < 0.001). by Tuka Al Hanai. S.M. 2014-06-13T22:34:48Z 2014-06-13T22:34:48Z 2014 2014 Thesis http://hdl.handle.net/1721.1/87941 880382239 eng M.I.T. theses are protected by copyright. They may be viewed from this source for any purpose, but reproduction or distribution in any format is prohibited without written permission. See provided URL for inquiries about permission. http://dspace.mit.edu/handle/1721.1/7582 72 pages application/pdf Massachusetts Institute of Technology |
| spellingShingle | Electrical Engineering and Computer Science. Alhanai, Tuka (Tuka Waddah Talib Ali Al Hanai) Lexical and Language Modeling of Diacritics and Morphemes in Arabic Automatic Speech Recognition |
| title | Lexical and Language Modeling of Diacritics and Morphemes in Arabic Automatic Speech Recognition |
| title_full | Lexical and Language Modeling of Diacritics and Morphemes in Arabic Automatic Speech Recognition |
| title_fullStr | Lexical and Language Modeling of Diacritics and Morphemes in Arabic Automatic Speech Recognition |
| title_full_unstemmed | Lexical and Language Modeling of Diacritics and Morphemes in Arabic Automatic Speech Recognition |
| title_short | Lexical and Language Modeling of Diacritics and Morphemes in Arabic Automatic Speech Recognition |
| title_sort | lexical and language modeling of diacritics and morphemes in arabic automatic speech recognition |
| topic | Electrical Engineering and Computer Science. |
| url | http://hdl.handle.net/1721.1/87941 |
| work_keys_str_mv | AT alhanaitukatukawaddahtalibalialhanai lexicalandlanguagemodelingofdiacriticsandmorphemesinarabicautomaticspeechrecognition |