Sign2Pose: A Pose-Based Approach for Gloss Prediction Using a Transformer Model
Word-level sign language recognition (WSLR) is the backbone for continuous sign language recognition (CSLR) that infers glosses from sign videos. Finding the relevant gloss from the sign sequence and detecting explicit boundaries of the glosses from sign videos is a persistent challenge. In this pap...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-03-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/23/5/2853 |
| _version_ | 1827752048023568384 |
|---|---|
| author | Jennifer Eunice Andrew J Yuichi Sei D. Jude Hemanth |
| author_facet | Jennifer Eunice Andrew J Yuichi Sei D. Jude Hemanth |
| author_sort | Jennifer Eunice |
| collection | DOAJ |
| description | Word-level sign language recognition (WSLR) is the backbone for continuous sign language recognition (CSLR) that infers glosses from sign videos. Finding the relevant gloss from the sign sequence and detecting explicit boundaries of the glosses from sign videos is a persistent challenge. In this paper, we propose a systematic approach for gloss prediction in WLSR using the Sign2Pose Gloss prediction transformer model. The primary goal of this work is to enhance WLSR’s gloss prediction accuracy with reduced time and computational overhead. The proposed approach uses hand-crafted features rather than automated feature extraction, which is computationally expensive and less accurate. A modified key frame extraction technique is proposed that uses histogram difference and Euclidean distance metrics to select and drop redundant frames. To enhance the model’s generalization ability, pose vector augmentation using perspective transformation along with joint angle rotation is performed. Further, for normalization, we employed YOLOv3 (You Only Look Once) to detect the signing space and track the hand gestures of the signers in the frames. The proposed model experiments on WLASL datasets achieved the top 1% recognition accuracy of 80.9% in WLASL100 and 64.21% in WLASL300. The performance of the proposed model surpasses state-of-the-art approaches. The integration of key frame extraction, augmentation, and pose estimation improved the performance of the proposed gloss prediction model by increasing the model’s precision in locating minor variations in their body posture. We observed that introducing YOLOv3 improved gloss prediction accuracy and helped prevent model overfitting. Overall, the proposed model showed 17% improved performance in the WLASL 100 dataset. |
| first_indexed | 2024-03-11T07:09:19Z |
| format | Article |
| id | doaj.art-37b81edcb1744178a61f76e03f3e8b0b |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-11T07:09:19Z |
| publishDate | 2023-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-37b81edcb1744178a61f76e03f3e8b0b2023-11-17T08:40:38ZengMDPI AGSensors1424-82202023-03-01235285310.3390/s23052853Sign2Pose: A Pose-Based Approach for Gloss Prediction Using a Transformer ModelJennifer Eunice0Andrew J1Yuichi Sei2D. Jude Hemanth3Department of Electronics and Communication Engineering, Karunya Institute of Technology and Sciences, Coimbatore 641114, IndiaComputer Science and Engineering, Manipal Institute of Technology, Manipal Academy of Higher Education, Manipal 576104, IndiaDepartment of Informatics, The University of Electro-Communications, Tokyo 182-8585, JapanDepartment of Electronics and Communication Engineering, Karunya Institute of Technology and Sciences, Coimbatore 641114, IndiaWord-level sign language recognition (WSLR) is the backbone for continuous sign language recognition (CSLR) that infers glosses from sign videos. Finding the relevant gloss from the sign sequence and detecting explicit boundaries of the glosses from sign videos is a persistent challenge. In this paper, we propose a systematic approach for gloss prediction in WLSR using the Sign2Pose Gloss prediction transformer model. The primary goal of this work is to enhance WLSR’s gloss prediction accuracy with reduced time and computational overhead. The proposed approach uses hand-crafted features rather than automated feature extraction, which is computationally expensive and less accurate. A modified key frame extraction technique is proposed that uses histogram difference and Euclidean distance metrics to select and drop redundant frames. To enhance the model’s generalization ability, pose vector augmentation using perspective transformation along with joint angle rotation is performed. Further, for normalization, we employed YOLOv3 (You Only Look Once) to detect the signing space and track the hand gestures of the signers in the frames. The proposed model experiments on WLASL datasets achieved the top 1% recognition accuracy of 80.9% in WLASL100 and 64.21% in WLASL300. The performance of the proposed model surpasses state-of-the-art approaches. The integration of key frame extraction, augmentation, and pose estimation improved the performance of the proposed gloss prediction model by increasing the model’s precision in locating minor variations in their body posture. We observed that introducing YOLOv3 improved gloss prediction accuracy and helped prevent model overfitting. Overall, the proposed model showed 17% improved performance in the WLASL 100 dataset.https://www.mdpi.com/1424-8220/23/5/2853sign language recognitiongloss predictiontransformerpose-based approachpose estimationdeep learning |
| spellingShingle | Jennifer Eunice Andrew J Yuichi Sei D. Jude Hemanth Sign2Pose: A Pose-Based Approach for Gloss Prediction Using a Transformer Model Sensors sign language recognition gloss prediction transformer pose-based approach pose estimation deep learning |
| title | Sign2Pose: A Pose-Based Approach for Gloss Prediction Using a Transformer Model |
| title_full | Sign2Pose: A Pose-Based Approach for Gloss Prediction Using a Transformer Model |
| title_fullStr | Sign2Pose: A Pose-Based Approach for Gloss Prediction Using a Transformer Model |
| title_full_unstemmed | Sign2Pose: A Pose-Based Approach for Gloss Prediction Using a Transformer Model |
| title_short | Sign2Pose: A Pose-Based Approach for Gloss Prediction Using a Transformer Model |
| title_sort | sign2pose a pose based approach for gloss prediction using a transformer model |
| topic | sign language recognition gloss prediction transformer pose-based approach pose estimation deep learning |
| url | https://www.mdpi.com/1424-8220/23/5/2853 |
| work_keys_str_mv | AT jennifereunice sign2poseaposebasedapproachforglosspredictionusingatransformermodel AT andrewj sign2poseaposebasedapproachforglosspredictionusingatransformermodel AT yuichisei sign2poseaposebasedapproachforglosspredictionusingatransformermodel AT djudehemanth sign2poseaposebasedapproachforglosspredictionusingatransformermodel |