Continuous Sign Language Recognition and Its Translation into Intonation-Colored Speech
This article is devoted to solving the problem of converting sign language into a consistent text with intonation markup for subsequent voice synthesis of sign phrases by speech with intonation. The paper proposes an improved method of continuous recognition of sign language, the results of which ar...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-07-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/23/14/6383 |
| _version_ | 1797587528248721408 |
|---|---|
| author | Nurzada Amangeldy Aru Ukenova Gulmira Bekmanova Bibigul Razakhova Marek Milosz Saule Kudubayeva |
| author_facet | Nurzada Amangeldy Aru Ukenova Gulmira Bekmanova Bibigul Razakhova Marek Milosz Saule Kudubayeva |
| author_sort | Nurzada Amangeldy |
| collection | DOAJ |
| description | This article is devoted to solving the problem of converting sign language into a consistent text with intonation markup for subsequent voice synthesis of sign phrases by speech with intonation. The paper proposes an improved method of continuous recognition of sign language, the results of which are transmitted to a natural language processor based on analyzers of morphology, syntax, and semantics of the Kazakh language, including morphological inflection and the construction of an intonation model of simple sentences. This approach has significant practical and social significance, as it can lead to the development of technologies that will help people with disabilities to communicate and improve their quality of life. As a result of the cross-validation of the model, we obtained an average test accuracy of 0.97 and an average val_accuracy of 0.90 for model evaluation. We also identified 20 sentence structures of the Kazakh language with their intonational model. |
| first_indexed | 2024-03-11T00:40:12Z |
| format | Article |
| id | doaj.art-8a72c46d7f57477392518a9aaf543b65 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-11T00:40:12Z |
| publishDate | 2023-07-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-8a72c46d7f57477392518a9aaf543b652023-11-18T21:17:00ZengMDPI AGSensors1424-82202023-07-012314638310.3390/s23146383Continuous Sign Language Recognition and Its Translation into Intonation-Colored SpeechNurzada Amangeldy0Aru Ukenova1Gulmira Bekmanova2Bibigul Razakhova3Marek Milosz4Saule Kudubayeva5Faculty of Information Technologies, L.N. Gumilyov Eurasian National University, Astana 010000, KazakhstanFaculty of Information Technologies, L.N. Gumilyov Eurasian National University, Astana 010000, KazakhstanFaculty of Information Technologies, L.N. Gumilyov Eurasian National University, Astana 010000, KazakhstanFaculty of Information Technologies, L.N. Gumilyov Eurasian National University, Astana 010000, KazakhstanDepartment of Computer Science, Lublin University of Technology, 36B Nadbystrzycka Str., 20-618 Lublin, PolandFaculty of Information Technologies, L.N. Gumilyov Eurasian National University, Astana 010000, KazakhstanThis article is devoted to solving the problem of converting sign language into a consistent text with intonation markup for subsequent voice synthesis of sign phrases by speech with intonation. The paper proposes an improved method of continuous recognition of sign language, the results of which are transmitted to a natural language processor based on analyzers of morphology, syntax, and semantics of the Kazakh language, including morphological inflection and the construction of an intonation model of simple sentences. This approach has significant practical and social significance, as it can lead to the development of technologies that will help people with disabilities to communicate and improve their quality of life. As a result of the cross-validation of the model, we obtained an average test accuracy of 0.97 and an average val_accuracy of 0.90 for model evaluation. We also identified 20 sentence structures of the Kazakh language with their intonational model.https://www.mdpi.com/1424-8220/23/14/6383sign language recognitionnatural language processingintonational speech synthesislong short-term memoryspatiotemporal features |
| spellingShingle | Nurzada Amangeldy Aru Ukenova Gulmira Bekmanova Bibigul Razakhova Marek Milosz Saule Kudubayeva Continuous Sign Language Recognition and Its Translation into Intonation-Colored Speech Sensors sign language recognition natural language processing intonational speech synthesis long short-term memory spatiotemporal features |
| title | Continuous Sign Language Recognition and Its Translation into Intonation-Colored Speech |
| title_full | Continuous Sign Language Recognition and Its Translation into Intonation-Colored Speech |
| title_fullStr | Continuous Sign Language Recognition and Its Translation into Intonation-Colored Speech |
| title_full_unstemmed | Continuous Sign Language Recognition and Its Translation into Intonation-Colored Speech |
| title_short | Continuous Sign Language Recognition and Its Translation into Intonation-Colored Speech |
| title_sort | continuous sign language recognition and its translation into intonation colored speech |
| topic | sign language recognition natural language processing intonational speech synthesis long short-term memory spatiotemporal features |
| url | https://www.mdpi.com/1424-8220/23/14/6383 |
| work_keys_str_mv | AT nurzadaamangeldy continuoussignlanguagerecognitionanditstranslationintointonationcoloredspeech AT aruukenova continuoussignlanguagerecognitionanditstranslationintointonationcoloredspeech AT gulmirabekmanova continuoussignlanguagerecognitionanditstranslationintointonationcoloredspeech AT bibigulrazakhova continuoussignlanguagerecognitionanditstranslationintointonationcoloredspeech AT marekmilosz continuoussignlanguagerecognitionanditstranslationintointonationcoloredspeech AT saulekudubayeva continuoussignlanguagerecognitionanditstranslationintointonationcoloredspeech |