Borno-Net: A Real-Time Bengali Sign-Character Detection and Sentence Generation System Using Quantized Yolov4-Tiny and LSTMs
Sign language is the most commonly used form of communication for persons with disabilities who have hearing or speech difficulties. However, persons without hearing impairment cannot understand these signs in many cases. As a consequence, persons with disabilities experience difficulties while expr...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-04-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/13/9/5219 |
| _version_ | 1797603083212029952 |
|---|---|
| author | Nasima Begum Rashik Rahman Nusrat Jahan Saqib Sizan Khan Tanjina Helaly Ashraful Haque Nipa Khatun |
| author_facet | Nasima Begum Rashik Rahman Nusrat Jahan Saqib Sizan Khan Tanjina Helaly Ashraful Haque Nipa Khatun |
| author_sort | Nasima Begum |
| collection | DOAJ |
| description | Sign language is the most commonly used form of communication for persons with disabilities who have hearing or speech difficulties. However, persons without hearing impairment cannot understand these signs in many cases. As a consequence, persons with disabilities experience difficulties while expressing their emotions or needs. Thus, a sign character detection and text generation system is necessary to mitigate this issue. In this paper, we propose an end-to-end system that can detect Bengali sign characters from input images or video frames and generate meaningful sentences. The proposed system consists of two phases. In the first phase, a quantization technique for the YoloV4-Tiny detection model is proposed for detecting 49 different sign characters, including 36 Bengali alphabet characters, 10 numeric characters, and 3 special characters. Here, the detection model localizes hand signs and predicts the corresponding character. The second phase generates text from the predicted characters by a detection model. The Long Short-Term Memory (LSTM) model is utilized to generate meaningful text from the character signs detected in the previous phase. To train the proposed system, the BdSL 49 dataset is used, which has approximately 14,745 images of 49 different classes. The proposed quantized YoloV4-Tiny model achieves a mAP of 99.7%, and the proposed language model achieves an overall accuracy of 99.12%. In addition, performance analysis among YoloV4, YoloV4 Tiny, and YoloV7 models is provided in this research. |
| first_indexed | 2024-03-11T04:25:22Z |
| format | Article |
| id | doaj.art-a357951a14494a31973e3277a53f213b |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-11T04:25:22Z |
| publishDate | 2023-04-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-a357951a14494a31973e3277a53f213b2023-11-17T22:31:01ZengMDPI AGApplied Sciences2076-34172023-04-01139521910.3390/app13095219Borno-Net: A Real-Time Bengali Sign-Character Detection and Sentence Generation System Using Quantized Yolov4-Tiny and LSTMsNasima Begum0Rashik Rahman1Nusrat Jahan2Saqib Sizan Khan3Tanjina Helaly4Ashraful Haque5Nipa Khatun6Department of Computer Science and Engineering, University of Asia Pacific, Dhaka 1205, BangladeshDepartment of Computer Science and Engineering, University of Asia Pacific, Dhaka 1205, BangladeshDepartment of Computer Science and Engineering, University of Asia Pacific, Dhaka 1205, BangladeshDepartment of Computer Science and Engineering, University of Asia Pacific, Dhaka 1205, BangladeshDepartment of Computer Science and Engineering, University of Asia Pacific, Dhaka 1205, BangladeshDepartment of Computer Science and Engineering, University of Asia Pacific, Dhaka 1205, BangladeshDepartment of Computer Science and Engineering, University of Asia Pacific, Dhaka 1205, BangladeshSign language is the most commonly used form of communication for persons with disabilities who have hearing or speech difficulties. However, persons without hearing impairment cannot understand these signs in many cases. As a consequence, persons with disabilities experience difficulties while expressing their emotions or needs. Thus, a sign character detection and text generation system is necessary to mitigate this issue. In this paper, we propose an end-to-end system that can detect Bengali sign characters from input images or video frames and generate meaningful sentences. The proposed system consists of two phases. In the first phase, a quantization technique for the YoloV4-Tiny detection model is proposed for detecting 49 different sign characters, including 36 Bengali alphabet characters, 10 numeric characters, and 3 special characters. Here, the detection model localizes hand signs and predicts the corresponding character. The second phase generates text from the predicted characters by a detection model. The Long Short-Term Memory (LSTM) model is utilized to generate meaningful text from the character signs detected in the previous phase. To train the proposed system, the BdSL 49 dataset is used, which has approximately 14,745 images of 49 different classes. The proposed quantized YoloV4-Tiny model achieves a mAP of 99.7%, and the proposed language model achieves an overall accuracy of 99.12%. In addition, performance analysis among YoloV4, YoloV4 Tiny, and YoloV7 models is provided in this research.https://www.mdpi.com/2076-3417/13/9/5219deep learningnatural language processingsign character detectionsign languagecomputer visionYolov4-Tiny |
| spellingShingle | Nasima Begum Rashik Rahman Nusrat Jahan Saqib Sizan Khan Tanjina Helaly Ashraful Haque Nipa Khatun Borno-Net: A Real-Time Bengali Sign-Character Detection and Sentence Generation System Using Quantized Yolov4-Tiny and LSTMs Applied Sciences deep learning natural language processing sign character detection sign language computer vision Yolov4-Tiny |
| title | Borno-Net: A Real-Time Bengali Sign-Character Detection and Sentence Generation System Using Quantized Yolov4-Tiny and LSTMs |
| title_full | Borno-Net: A Real-Time Bengali Sign-Character Detection and Sentence Generation System Using Quantized Yolov4-Tiny and LSTMs |
| title_fullStr | Borno-Net: A Real-Time Bengali Sign-Character Detection and Sentence Generation System Using Quantized Yolov4-Tiny and LSTMs |
| title_full_unstemmed | Borno-Net: A Real-Time Bengali Sign-Character Detection and Sentence Generation System Using Quantized Yolov4-Tiny and LSTMs |
| title_short | Borno-Net: A Real-Time Bengali Sign-Character Detection and Sentence Generation System Using Quantized Yolov4-Tiny and LSTMs |
| title_sort | borno net a real time bengali sign character detection and sentence generation system using quantized yolov4 tiny and lstms |
| topic | deep learning natural language processing sign character detection sign language computer vision Yolov4-Tiny |
| url | https://www.mdpi.com/2076-3417/13/9/5219 |
| work_keys_str_mv | AT nasimabegum bornonetarealtimebengalisigncharacterdetectionandsentencegenerationsystemusingquantizedyolov4tinyandlstms AT rashikrahman bornonetarealtimebengalisigncharacterdetectionandsentencegenerationsystemusingquantizedyolov4tinyandlstms AT nusratjahan bornonetarealtimebengalisigncharacterdetectionandsentencegenerationsystemusingquantizedyolov4tinyandlstms AT saqibsizankhan bornonetarealtimebengalisigncharacterdetectionandsentencegenerationsystemusingquantizedyolov4tinyandlstms AT tanjinahelaly bornonetarealtimebengalisigncharacterdetectionandsentencegenerationsystemusingquantizedyolov4tinyandlstms AT ashrafulhaque bornonetarealtimebengalisigncharacterdetectionandsentencegenerationsystemusingquantizedyolov4tinyandlstms AT nipakhatun bornonetarealtimebengalisigncharacterdetectionandsentencegenerationsystemusingquantizedyolov4tinyandlstms |