Exponential language modeling using morphological features and multi-task learning
For languages with fast vocabulary growth and limited resources, data sparsity leads to challenges in training a language model. One strategy for addressing this problem is to leverage morphological structure as features in the model. This paper explores different uses of unsupervised morphological...
| Glavni autori: | , , , |
|---|---|
| Format: | Journal article |
| Izdano: |
Institute of Electrical and Electronics Engineers
2015
|
| _version_ | 1826279676656484352 |
|---|---|
| author | Fang, H Ostendorf, M Baumann, P Pierrehumbert, J |
| author_facet | Fang, H Ostendorf, M Baumann, P Pierrehumbert, J |
| author_sort | Fang, H |
| collection | OXFORD |
| description | For languages with fast vocabulary growth and limited resources, data sparsity leads to challenges in training a language model. One strategy for addressing this problem is to leverage morphological structure as features in the model. This paper explores different uses of unsupervised morphological features in both the history and prediction space for three word-based exponential models (maximum entropy, logbilinear, and recurrent neural net (RNN)). Multi-task training is introduced as a regularizing mechanism to improve performance in the continuous-space approaches. The models are compared to non-parametric baselines. From using the RNN with morphological features and multi-task learning, experiments with conversational speech from four languages show we can obtain consistent gains of 7-11% in perplexity reduction in a limited-resource scenario (10 hrs speech), and 12-18% when the training size is increased ( 80 hrs ). Results are mixed for all other approaches, compared to a modified Kneser-Ney baseline, but morphology is useful in continuous-space models compared to their word-only baseline. Multi-task learning improves both continuous-space models. |
| first_indexed | 2024-03-07T00:02:20Z |
| format | Journal article |
| id | oxford-uuid:7661328c-3e1f-4a1b-a650-5f684fd8f058 |
| institution | University of Oxford |
| last_indexed | 2024-03-07T00:02:20Z |
| publishDate | 2015 |
| publisher | Institute of Electrical and Electronics Engineers |
| record_format | dspace |
| spelling | oxford-uuid:7661328c-3e1f-4a1b-a650-5f684fd8f0582022-03-26T20:15:26ZExponential language modeling using morphological features and multi-task learningJournal articlehttp://purl.org/coar/resource_type/c_dcae04bcuuid:7661328c-3e1f-4a1b-a650-5f684fd8f058Symplectic Elements at OxfordInstitute of Electrical and Electronics Engineers2015Fang, HOstendorf, MBaumann, PPierrehumbert, JFor languages with fast vocabulary growth and limited resources, data sparsity leads to challenges in training a language model. One strategy for addressing this problem is to leverage morphological structure as features in the model. This paper explores different uses of unsupervised morphological features in both the history and prediction space for three word-based exponential models (maximum entropy, logbilinear, and recurrent neural net (RNN)). Multi-task training is introduced as a regularizing mechanism to improve performance in the continuous-space approaches. The models are compared to non-parametric baselines. From using the RNN with morphological features and multi-task learning, experiments with conversational speech from four languages show we can obtain consistent gains of 7-11% in perplexity reduction in a limited-resource scenario (10 hrs speech), and 12-18% when the training size is increased ( 80 hrs ). Results are mixed for all other approaches, compared to a modified Kneser-Ney baseline, but morphology is useful in continuous-space models compared to their word-only baseline. Multi-task learning improves both continuous-space models. |
| spellingShingle | Fang, H Ostendorf, M Baumann, P Pierrehumbert, J Exponential language modeling using morphological features and multi-task learning |
| title | Exponential language modeling using morphological features and multi-task learning |
| title_full | Exponential language modeling using morphological features and multi-task learning |
| title_fullStr | Exponential language modeling using morphological features and multi-task learning |
| title_full_unstemmed | Exponential language modeling using morphological features and multi-task learning |
| title_short | Exponential language modeling using morphological features and multi-task learning |
| title_sort | exponential language modeling using morphological features and multi task learning |
| work_keys_str_mv | AT fangh exponentiallanguagemodelingusingmorphologicalfeaturesandmultitasklearning AT ostendorfm exponentiallanguagemodelingusingmorphologicalfeaturesandmultitasklearning AT baumannp exponentiallanguagemodelingusingmorphologicalfeaturesandmultitasklearning AT pierrehumbertj exponentiallanguagemodelingusingmorphologicalfeaturesandmultitasklearning |