Dataset of stopwords extracted from Uzbek texts
Filtering stop words is an important task when processing text queries to search for information in large data sets. It enables a reduction of the search space without losing the semantic meaning. The stop words, which have only grammatical roles and not contributing to information content still add...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2022-08-01
|
| Series: | Data in Brief |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2352340922005522 |
| _version_ | 1811216326155304960 |
|---|---|
| author | Khabibulla Madatov Shukurla Bekchanov Jernej Vičič |
| author_facet | Khabibulla Madatov Shukurla Bekchanov Jernej Vičič |
| author_sort | Khabibulla Madatov |
| collection | DOAJ |
| description | Filtering stop words is an important task when processing text queries to search for information in large data sets. It enables a reduction of the search space without losing the semantic meaning. The stop words, which have only grammatical roles and not contributing to information content still add up to the complexity of the query. Existing mathematical models that are used to tackle this problem are not suitable for all families of natural languages [1]. For example, they do not cover families of languages to which Uzbek can be included. In the present work, the collocation method of this problem is o ered for families of languages that include the Uzbek language as well. This method concerns the so-called agglutinative languages, in which the task of recognizing stop words is much more difficult, since the stop words are “masked” in the text. In this work the unigram, the bigram and the collocation methods are applied to the “School corpus” that corresponds to the type of languages being studied. |
| first_indexed | 2024-04-12T06:37:17Z |
| format | Article |
| id | doaj.art-cbe85d9f265549008eec41d7a197f63b |
| institution | Directory Open Access Journal |
| issn | 2352-3409 |
| language | English |
| last_indexed | 2024-04-12T06:37:17Z |
| publishDate | 2022-08-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Data in Brief |
| spelling | doaj.art-cbe85d9f265549008eec41d7a197f63b2022-12-22T03:43:51ZengElsevierData in Brief2352-34092022-08-0143108351Dataset of stopwords extracted from Uzbek textsKhabibulla Madatov0Shukurla Bekchanov1Jernej Vičič2Urgench state university, 14, Kh. Alimdjan str, Urgench city, 220100, UzbekistanUrgench state university, 14, Kh. Alimdjan str, Urgench city, 220100, UzbekistanResearch Centre of the Slovenian Academy of Sciences and Arts, The Fran Ramovš Institute, Novi trg 2, 1000 Ljubljana, Slovenija; University of Primorska, FAMNIT, Glagoljaska 8, 6000 Koper, Slovenia; Corresponding author.Filtering stop words is an important task when processing text queries to search for information in large data sets. It enables a reduction of the search space without losing the semantic meaning. The stop words, which have only grammatical roles and not contributing to information content still add up to the complexity of the query. Existing mathematical models that are used to tackle this problem are not suitable for all families of natural languages [1]. For example, they do not cover families of languages to which Uzbek can be included. In the present work, the collocation method of this problem is o ered for families of languages that include the Uzbek language as well. This method concerns the so-called agglutinative languages, in which the task of recognizing stop words is much more difficult, since the stop words are “masked” in the text. In this work the unigram, the bigram and the collocation methods are applied to the “School corpus” that corresponds to the type of languages being studied.http://www.sciencedirect.com/science/article/pii/S2352340922005522Stop wordsMachine LearningUnigramBigramCollocation |
| spellingShingle | Khabibulla Madatov Shukurla Bekchanov Jernej Vičič Dataset of stopwords extracted from Uzbek texts Data in Brief Stop words Machine Learning Unigram Bigram Collocation |
| title | Dataset of stopwords extracted from Uzbek texts |
| title_full | Dataset of stopwords extracted from Uzbek texts |
| title_fullStr | Dataset of stopwords extracted from Uzbek texts |
| title_full_unstemmed | Dataset of stopwords extracted from Uzbek texts |
| title_short | Dataset of stopwords extracted from Uzbek texts |
| title_sort | dataset of stopwords extracted from uzbek texts |
| topic | Stop words Machine Learning Unigram Bigram Collocation |
| url | http://www.sciencedirect.com/science/article/pii/S2352340922005522 |
| work_keys_str_mv | AT khabibullamadatov datasetofstopwordsextractedfromuzbektexts AT shukurlabekchanov datasetofstopwordsextractedfromuzbektexts AT jernejvicic datasetofstopwordsextractedfromuzbektexts |