Deep Learning in Sign Language Recognition: A Hybrid Approach for the Recognition of Static and Dynamic Signs
A speech impairment limits a person’s capacity for oral and auditory communication. A great improvement in communication between the deaf and the general public would be represented by a real-time sign language detector. This work proposes a deep learning-based algorithm that can identify words from...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-08-01
|
| Series: | Mathematics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2227-7390/11/17/3729 |
| _version_ | 1797582168711495680 |
|---|---|
| author | Ahmed Mateen Buttar Usama Ahmad Abdu H. Gumaei Adel Assiri Muhammad Azeem Akbar Bader Fahad Alkhamees |
| author_facet | Ahmed Mateen Buttar Usama Ahmad Abdu H. Gumaei Adel Assiri Muhammad Azeem Akbar Bader Fahad Alkhamees |
| author_sort | Ahmed Mateen Buttar |
| collection | DOAJ |
| description | A speech impairment limits a person’s capacity for oral and auditory communication. A great improvement in communication between the deaf and the general public would be represented by a real-time sign language detector. This work proposes a deep learning-based algorithm that can identify words from a person’s gestures and detect them. There have been many studies on this topic, but the development of static and dynamic sign language recognition models is still a challenging area of research. The difficulty is in obtaining an appropriate model that addresses the challenges of continuous signs that are independent of the signer. Different signers’ speeds, durations, and many other factors make it challenging to create a model with high accuracy and continuity. For the accurate and effective recognition of signs, this study uses two different deep learning-based approaches. We create a real-time American Sign Language detector using the skeleton model, which reliably categorizes continuous signs in sign language in most cases using a deep learning approach. In the second deep learning approach, we create a sign language detector for static signs using YOLOv6. This application is very helpful for sign language users and learners to practice sign language in real time. After training both algorithms separately for static and continuous signs, we create a single algorithm using a hybrid approach. The proposed model, consisting of LSTM with MediaPipe holistic landmarks, achieves around 92% accuracy for different continuous signs, and the YOLOv6 model achieves 96% accuracy over different static signs. Throughout this study, we determine which approach is best for sequential movement detection and for the classification of different signs according to sign language and shows remarkable accuracy in real time. |
| first_indexed | 2024-03-10T23:17:03Z |
| format | Article |
| id | doaj.art-177c087e9bb34b318ef4a7286ffd97fa |
| institution | Directory Open Access Journal |
| issn | 2227-7390 |
| language | English |
| last_indexed | 2024-03-10T23:17:03Z |
| publishDate | 2023-08-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Mathematics |
| spelling | doaj.art-177c087e9bb34b318ef4a7286ffd97fa2023-11-19T08:31:22ZengMDPI AGMathematics2227-73902023-08-011117372910.3390/math11173729Deep Learning in Sign Language Recognition: A Hybrid Approach for the Recognition of Static and Dynamic SignsAhmed Mateen Buttar0Usama Ahmad1Abdu H. Gumaei2Adel Assiri3Muhammad Azeem Akbar4Bader Fahad Alkhamees5Department of Computer Science, University of Agriculture Faisalabad, Faisalabad 38000, PakistanDepartment of Computer Science, University of Agriculture Faisalabad, Faisalabad 38000, PakistanDepartment of Computer Science, College of Computer Engineering and Sciences, Prince Sattam bin Abdulaziz University, Al-Kharj 11942, Saudi ArabiaManagement Information Systems Department, College of Business, King Khalid University, Abha 61421, Saudi ArabiaSoftware Engineering Department, LUT University, 15210 Lahti, FinlandDepartment of Information Systems, College of Computer and Information Sciences, King Saud University, Riyadh 11543, Saudi ArabiaA speech impairment limits a person’s capacity for oral and auditory communication. A great improvement in communication between the deaf and the general public would be represented by a real-time sign language detector. This work proposes a deep learning-based algorithm that can identify words from a person’s gestures and detect them. There have been many studies on this topic, but the development of static and dynamic sign language recognition models is still a challenging area of research. The difficulty is in obtaining an appropriate model that addresses the challenges of continuous signs that are independent of the signer. Different signers’ speeds, durations, and many other factors make it challenging to create a model with high accuracy and continuity. For the accurate and effective recognition of signs, this study uses two different deep learning-based approaches. We create a real-time American Sign Language detector using the skeleton model, which reliably categorizes continuous signs in sign language in most cases using a deep learning approach. In the second deep learning approach, we create a sign language detector for static signs using YOLOv6. This application is very helpful for sign language users and learners to practice sign language in real time. After training both algorithms separately for static and continuous signs, we create a single algorithm using a hybrid approach. The proposed model, consisting of LSTM with MediaPipe holistic landmarks, achieves around 92% accuracy for different continuous signs, and the YOLOv6 model achieves 96% accuracy over different static signs. Throughout this study, we determine which approach is best for sequential movement detection and for the classification of different signs according to sign language and shows remarkable accuracy in real time.https://www.mdpi.com/2227-7390/11/17/3729You Only Look Once (YOLO)Long Short-Term Memory (LSTM)deep learningconfusion matrixconvolutional neural network (CNN)MediaPipe holistic |
| spellingShingle | Ahmed Mateen Buttar Usama Ahmad Abdu H. Gumaei Adel Assiri Muhammad Azeem Akbar Bader Fahad Alkhamees Deep Learning in Sign Language Recognition: A Hybrid Approach for the Recognition of Static and Dynamic Signs Mathematics You Only Look Once (YOLO) Long Short-Term Memory (LSTM) deep learning confusion matrix convolutional neural network (CNN) MediaPipe holistic |
| title | Deep Learning in Sign Language Recognition: A Hybrid Approach for the Recognition of Static and Dynamic Signs |
| title_full | Deep Learning in Sign Language Recognition: A Hybrid Approach for the Recognition of Static and Dynamic Signs |
| title_fullStr | Deep Learning in Sign Language Recognition: A Hybrid Approach for the Recognition of Static and Dynamic Signs |
| title_full_unstemmed | Deep Learning in Sign Language Recognition: A Hybrid Approach for the Recognition of Static and Dynamic Signs |
| title_short | Deep Learning in Sign Language Recognition: A Hybrid Approach for the Recognition of Static and Dynamic Signs |
| title_sort | deep learning in sign language recognition a hybrid approach for the recognition of static and dynamic signs |
| topic | You Only Look Once (YOLO) Long Short-Term Memory (LSTM) deep learning confusion matrix convolutional neural network (CNN) MediaPipe holistic |
| url | https://www.mdpi.com/2227-7390/11/17/3729 |
| work_keys_str_mv | AT ahmedmateenbuttar deeplearninginsignlanguagerecognitionahybridapproachfortherecognitionofstaticanddynamicsigns AT usamaahmad deeplearninginsignlanguagerecognitionahybridapproachfortherecognitionofstaticanddynamicsigns AT abduhgumaei deeplearninginsignlanguagerecognitionahybridapproachfortherecognitionofstaticanddynamicsigns AT adelassiri deeplearninginsignlanguagerecognitionahybridapproachfortherecognitionofstaticanddynamicsigns AT muhammadazeemakbar deeplearninginsignlanguagerecognitionahybridapproachfortherecognitionofstaticanddynamicsigns AT baderfahadalkhamees deeplearninginsignlanguagerecognitionahybridapproachfortherecognitionofstaticanddynamicsigns |