Enhancing sign language recognition using CNN and SIFT: A case study on Pakistan sign language
Millions of people who have trouble hearing rely heavily on sign language as their primary means of communication. As a form of visual language, sign language is primarily used as features by the deaf and hard of hearing to communicate with one another. These features make it challenging for the ave...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2024-02-01
|
| Series: | Journal of King Saud University: Computer and Information Sciences |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S1319157824000235 |
| _version_ | 1827325547201429504 |
|---|---|
| author | Sadia Arooj Saud Altaf Shafiq Ahmad Haitham Mahmoud Adamali Shah Noor Mohamed |
| author_facet | Sadia Arooj Saud Altaf Shafiq Ahmad Haitham Mahmoud Adamali Shah Noor Mohamed |
| author_sort | Sadia Arooj |
| collection | DOAJ |
| description | Millions of people who have trouble hearing rely heavily on sign language as their primary means of communication. As a form of visual language, sign language is primarily used as features by the deaf and hard of hearing to communicate with one another. These features make it challenging for the average person to grasp what is being said. As a result, it becomes more difficult for deaf people to interact with the hearing community. In order to solve this communication barrier, scientists have been working on sign language recognition systems. The purpose of this research is to propose a hand gesture recognition framework for Pakistan Sign Language (PSL) by training a Convolutional Neural Network (CNN) on PSL gesture images obtained with the aid of a testbed built in the laboratory using the Kinect motion sensor for Urdu alphabets. Kinect images of hands were taken in this study under a variety of lighting conditions. A feature vector set was created from hand margin, size, shape, palm center points, and finger position using Scale Invariant Feature Transform (SIFT). Important features of PSL-based sign images were extracted using SIFT and then converted to vector points. The proposed improved CNN model was able to achieve an impressive accuracy of 98.74% on the PSL dataset, demonstrating an impressively low error rate of only 1.26%. A case study is demonstrated in the assessment of the system that has been provided. Finally, a comparison is performed between others and our work to show the efficiency of the proposed framework and legitimize the concept. |
| first_indexed | 2024-03-07T14:29:50Z |
| format | Article |
| id | doaj.art-3080a2dce71f4df4a472c4ae8dfffafa |
| institution | Directory Open Access Journal |
| issn | 1319-1578 |
| language | English |
| last_indexed | 2024-03-07T14:29:50Z |
| publishDate | 2024-02-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Journal of King Saud University: Computer and Information Sciences |
| spelling | doaj.art-3080a2dce71f4df4a472c4ae8dfffafa2024-03-06T05:25:39ZengElsevierJournal of King Saud University: Computer and Information Sciences1319-15782024-02-01362101934Enhancing sign language recognition using CNN and SIFT: A case study on Pakistan sign languageSadia Arooj0Saud Altaf1Shafiq Ahmad2Haitham Mahmoud3Adamali Shah Noor Mohamed4University Institute of Information Technology, Pir Mehr Ali Shah Arid Agriculture University, Rawalpindi 46300, PakistanDepartment of Information Engineering Technology, National Skills University, Islamabad 44000, Pakistan; Corresponding author.Industrial Engineering Department, College of Engineering, King Saud University, Riyadh 11421, Saudi ArabiaIndustrial Engineering Department, College of Engineering, King Saud University, Riyadh 11421, Saudi ArabiaElectrical Engineering Department, College of Engineering, King Saud University, P.O. Box 800, Riyadh 11421, Saudi ArabiaMillions of people who have trouble hearing rely heavily on sign language as their primary means of communication. As a form of visual language, sign language is primarily used as features by the deaf and hard of hearing to communicate with one another. These features make it challenging for the average person to grasp what is being said. As a result, it becomes more difficult for deaf people to interact with the hearing community. In order to solve this communication barrier, scientists have been working on sign language recognition systems. The purpose of this research is to propose a hand gesture recognition framework for Pakistan Sign Language (PSL) by training a Convolutional Neural Network (CNN) on PSL gesture images obtained with the aid of a testbed built in the laboratory using the Kinect motion sensor for Urdu alphabets. Kinect images of hands were taken in this study under a variety of lighting conditions. A feature vector set was created from hand margin, size, shape, palm center points, and finger position using Scale Invariant Feature Transform (SIFT). Important features of PSL-based sign images were extracted using SIFT and then converted to vector points. The proposed improved CNN model was able to achieve an impressive accuracy of 98.74% on the PSL dataset, demonstrating an impressively low error rate of only 1.26%. A case study is demonstrated in the assessment of the system that has been provided. Finally, a comparison is performed between others and our work to show the efficiency of the proposed framework and legitimize the concept.http://www.sciencedirect.com/science/article/pii/S1319157824000235Pakistan sign languageUrdu AlphabetsHand gesture recognitionCNNSIFT |
| spellingShingle | Sadia Arooj Saud Altaf Shafiq Ahmad Haitham Mahmoud Adamali Shah Noor Mohamed Enhancing sign language recognition using CNN and SIFT: A case study on Pakistan sign language Journal of King Saud University: Computer and Information Sciences Pakistan sign language Urdu Alphabets Hand gesture recognition CNN SIFT |
| title | Enhancing sign language recognition using CNN and SIFT: A case study on Pakistan sign language |
| title_full | Enhancing sign language recognition using CNN and SIFT: A case study on Pakistan sign language |
| title_fullStr | Enhancing sign language recognition using CNN and SIFT: A case study on Pakistan sign language |
| title_full_unstemmed | Enhancing sign language recognition using CNN and SIFT: A case study on Pakistan sign language |
| title_short | Enhancing sign language recognition using CNN and SIFT: A case study on Pakistan sign language |
| title_sort | enhancing sign language recognition using cnn and sift a case study on pakistan sign language |
| topic | Pakistan sign language Urdu Alphabets Hand gesture recognition CNN SIFT |
| url | http://www.sciencedirect.com/science/article/pii/S1319157824000235 |
| work_keys_str_mv | AT sadiaarooj enhancingsignlanguagerecognitionusingcnnandsiftacasestudyonpakistansignlanguage AT saudaltaf enhancingsignlanguagerecognitionusingcnnandsiftacasestudyonpakistansignlanguage AT shafiqahmad enhancingsignlanguagerecognitionusingcnnandsiftacasestudyonpakistansignlanguage AT haithammahmoud enhancingsignlanguagerecognitionusingcnnandsiftacasestudyonpakistansignlanguage AT adamalishahnoormohamed enhancingsignlanguagerecognitionusingcnnandsiftacasestudyonpakistansignlanguage |