Vision-based Pakistani sign language recognition using bag-of-words and support vector machines
Abstract In order to perform their daily activities, a person is required to communicating with others. This can be a major obstacle for the deaf population of the world, who communicate using sign languages (SL). Pakistani Sign Language (PSL) is used by more than 250,000 deaf Pakistanis. Developing...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2022-12-01
|
| Series: | Scientific Reports |
| Online Access: | https://doi.org/10.1038/s41598-022-15864-6 |
| _version_ | 1811204020177469440 |
|---|---|
| author | Muhammad Shaheer Mirza Sheikh Muhammad Munaf Fahad Azim Shahid Ali Saad Jawaid Khan |
| author_facet | Muhammad Shaheer Mirza Sheikh Muhammad Munaf Fahad Azim Shahid Ali Saad Jawaid Khan |
| author_sort | Muhammad Shaheer Mirza |
| collection | DOAJ |
| description | Abstract In order to perform their daily activities, a person is required to communicating with others. This can be a major obstacle for the deaf population of the world, who communicate using sign languages (SL). Pakistani Sign Language (PSL) is used by more than 250,000 deaf Pakistanis. Developing a SL recognition system would greatly facilitate these people. This study aimed to collect data of static and dynamic PSL alphabets and to develop a vision-based system for their recognition using Bag-of-Words (BoW) and Support Vector Machine (SVM) techniques. A total of 5120 images for 36 static PSL alphabet signs and 353 videos with 45,224 frames for 3 dynamic PSL alphabet signs were collected from 10 native signers of PSL. The developed system used the collected data as input, resized the data to various scales and converted the RGB images into grayscale. The resized grayscale images were segmented using Thresholding technique and features were extracted using Speeded Up Robust Feature (SURF). The obtained SURF descriptors were clustered using K-means clustering. A BoW was obtained by computing the Euclidean distance between the SURF descriptors and the clustered data. The codebooks were divided into training and testing using fivefold cross validation. The highest overall classification accuracy for static PSL signs was 97.80% at 750 × 750 image dimensions and 500 Bags. For dynamic PSL signs a 96.53% accuracy was obtained at 480 × 270 video resolution and 200 Bags. |
| first_indexed | 2024-04-12T03:04:41Z |
| format | Article |
| id | doaj.art-1e16fcbb690c45278da61049e079b138 |
| institution | Directory Open Access Journal |
| issn | 2045-2322 |
| language | English |
| last_indexed | 2024-04-12T03:04:41Z |
| publishDate | 2022-12-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj.art-1e16fcbb690c45278da61049e079b1382022-12-22T03:50:32ZengNature PortfolioScientific Reports2045-23222022-12-0112111310.1038/s41598-022-15864-6Vision-based Pakistani sign language recognition using bag-of-words and support vector machinesMuhammad Shaheer Mirza0Sheikh Muhammad Munaf1Fahad Azim2Shahid Ali3Saad Jawaid Khan4Department of Biomedical Engineering, Faculty of Engineering, Science, Technology and Management, Ziauddin UniversityDepartment of Software Engineering, Faculty of Engineering, Science, Technology and Management, Ziauddin UniversityDepartment of Electrical Engineering, Faculty of Engineering, Science, Technology and Management, Ziauddin UniversityDepartment of Speech Language and Hearing Sciences, Faculty of Health Sciences, Ziauddin UniversityDepartment of Biomedical Engineering, Faculty of Engineering, Science, Technology and Management, Ziauddin UniversityAbstract In order to perform their daily activities, a person is required to communicating with others. This can be a major obstacle for the deaf population of the world, who communicate using sign languages (SL). Pakistani Sign Language (PSL) is used by more than 250,000 deaf Pakistanis. Developing a SL recognition system would greatly facilitate these people. This study aimed to collect data of static and dynamic PSL alphabets and to develop a vision-based system for their recognition using Bag-of-Words (BoW) and Support Vector Machine (SVM) techniques. A total of 5120 images for 36 static PSL alphabet signs and 353 videos with 45,224 frames for 3 dynamic PSL alphabet signs were collected from 10 native signers of PSL. The developed system used the collected data as input, resized the data to various scales and converted the RGB images into grayscale. The resized grayscale images were segmented using Thresholding technique and features were extracted using Speeded Up Robust Feature (SURF). The obtained SURF descriptors were clustered using K-means clustering. A BoW was obtained by computing the Euclidean distance between the SURF descriptors and the clustered data. The codebooks were divided into training and testing using fivefold cross validation. The highest overall classification accuracy for static PSL signs was 97.80% at 750 × 750 image dimensions and 500 Bags. For dynamic PSL signs a 96.53% accuracy was obtained at 480 × 270 video resolution and 200 Bags.https://doi.org/10.1038/s41598-022-15864-6 |
| spellingShingle | Muhammad Shaheer Mirza Sheikh Muhammad Munaf Fahad Azim Shahid Ali Saad Jawaid Khan Vision-based Pakistani sign language recognition using bag-of-words and support vector machines Scientific Reports |
| title | Vision-based Pakistani sign language recognition using bag-of-words and support vector machines |
| title_full | Vision-based Pakistani sign language recognition using bag-of-words and support vector machines |
| title_fullStr | Vision-based Pakistani sign language recognition using bag-of-words and support vector machines |
| title_full_unstemmed | Vision-based Pakistani sign language recognition using bag-of-words and support vector machines |
| title_short | Vision-based Pakistani sign language recognition using bag-of-words and support vector machines |
| title_sort | vision based pakistani sign language recognition using bag of words and support vector machines |
| url | https://doi.org/10.1038/s41598-022-15864-6 |
| work_keys_str_mv | AT muhammadshaheermirza visionbasedpakistanisignlanguagerecognitionusingbagofwordsandsupportvectormachines AT sheikhmuhammadmunaf visionbasedpakistanisignlanguagerecognitionusingbagofwordsandsupportvectormachines AT fahadazim visionbasedpakistanisignlanguagerecognitionusingbagofwordsandsupportvectormachines AT shahidali visionbasedpakistanisignlanguagerecognitionusingbagofwordsandsupportvectormachines AT saadjawaidkhan visionbasedpakistanisignlanguagerecognitionusingbagofwordsandsupportvectormachines |