Interpretation of Bahasa Isyarat Malaysia (BIM) Using SSD-MobileNet-V2 FPNLite and COCO mAP
This research proposes a study on two-way communication between deaf/mute and normal people using an Android application. Despite advancements in technology, there is still a lack of mobile applications that facilitate two-way communication between deaf/mute and normal people, especially by using B...
| Những tác giả chính: | , , , , , , |
|---|---|
| Định dạng: | Bài viết |
| Ngôn ngữ: | English |
| Được phát hành: |
Mdpi
2023
|
| Những chủ đề: | |
| Truy cập trực tuyến: | http://eprints.uthm.edu.my/11622/1/J16131_fcea86e7763277e20bdbfab7a92ccd21.pdf |
| _version_ | 1825638577997873152 |
|---|---|
| author | Saiful Bahri, Iffah Zulaikha Saon, Sharifah Mahamad, Abd Kadir Isa, Khalid Fadlilah, Umi Ahmadon, Mohd Anuaruddin Yamaguchi, Shingo |
| author_facet | Saiful Bahri, Iffah Zulaikha Saon, Sharifah Mahamad, Abd Kadir Isa, Khalid Fadlilah, Umi Ahmadon, Mohd Anuaruddin Yamaguchi, Shingo |
| author_sort | Saiful Bahri, Iffah Zulaikha |
| collection | UTHM |
| description | This research proposes a study on two-way communication between deaf/mute and normal people using an Android application. Despite advancements in technology, there is still a lack of mobile applications that facilitate two-way communication between deaf/mute and normal people,
especially by using Bahasa Isyarat Malaysia (BIM). This project consists of three parts: First, we use BIM letters, which enables the recognition of BIM letters and BIM combined letters to form a word. In this part, a MobileNet pre-trained model is implemented to train the model with a total
of 87,000 images for 29 classes, with a 10% test size and a 90% training size. The second part is BIM word hand gestures, which consists of five classes that are trained with the SSD-MobileNet-V2 FPNLite 320 × 320 pre-trained model with a speed of 22 s/frame rate and COCO mAP of 22.2, with a
total of 500 images for all five classes and first-time training set to 2000 steps, while the second- and third-time training are set to 2500 steps. The third part is Android application development using Android Studio, which contains the features of the BIM letters and BIM word hand gestures, with the trained models converted into TensorFlow Lite. This feature also includes the conversion of speech to text, whereby this feature allows converting speech to text through the Android application. Thus, BIM letters obtain 99.75% accuracy after training the models, while BIM word hand gestures obtain 61.60% accuracy. The suggested system is validated as a result of these simulations and tests. |
| first_indexed | 2024-12-08T07:42:24Z |
| format | Article |
| id | uthm.eprints-11622 |
| institution | Universiti Tun Hussein Onn Malaysia |
| language | English |
| last_indexed | 2024-12-08T07:42:24Z |
| publishDate | 2023 |
| publisher | Mdpi |
| record_format | dspace |
| spelling | uthm.eprints-116222024-09-25T07:21:22Z http://eprints.uthm.edu.my/11622/ Interpretation of Bahasa Isyarat Malaysia (BIM) Using SSD-MobileNet-V2 FPNLite and COCO mAP Saiful Bahri, Iffah Zulaikha Saon, Sharifah Mahamad, Abd Kadir Isa, Khalid Fadlilah, Umi Ahmadon, Mohd Anuaruddin Yamaguchi, Shingo T Technology (General) This research proposes a study on two-way communication between deaf/mute and normal people using an Android application. Despite advancements in technology, there is still a lack of mobile applications that facilitate two-way communication between deaf/mute and normal people, especially by using Bahasa Isyarat Malaysia (BIM). This project consists of three parts: First, we use BIM letters, which enables the recognition of BIM letters and BIM combined letters to form a word. In this part, a MobileNet pre-trained model is implemented to train the model with a total of 87,000 images for 29 classes, with a 10% test size and a 90% training size. The second part is BIM word hand gestures, which consists of five classes that are trained with the SSD-MobileNet-V2 FPNLite 320 × 320 pre-trained model with a speed of 22 s/frame rate and COCO mAP of 22.2, with a total of 500 images for all five classes and first-time training set to 2000 steps, while the second- and third-time training are set to 2500 steps. The third part is Android application development using Android Studio, which contains the features of the BIM letters and BIM word hand gestures, with the trained models converted into TensorFlow Lite. This feature also includes the conversion of speech to text, whereby this feature allows converting speech to text through the Android application. Thus, BIM letters obtain 99.75% accuracy after training the models, while BIM word hand gestures obtain 61.60% accuracy. The suggested system is validated as a result of these simulations and tests. Mdpi 2023 Article PeerReviewed text en http://eprints.uthm.edu.my/11622/1/J16131_fcea86e7763277e20bdbfab7a92ccd21.pdf Saiful Bahri, Iffah Zulaikha and Saon, Sharifah and Mahamad, Abd Kadir and Isa, Khalid and Fadlilah, Umi and Ahmadon, Mohd Anuaruddin and Yamaguchi, Shingo (2023) Interpretation of Bahasa Isyarat Malaysia (BIM) Using SSD-MobileNet-V2 FPNLite and COCO mAP. Information, 14 (319). pp. 1-20. https://doi.org/10.3390/info14060319 |
| spellingShingle | T Technology (General) Saiful Bahri, Iffah Zulaikha Saon, Sharifah Mahamad, Abd Kadir Isa, Khalid Fadlilah, Umi Ahmadon, Mohd Anuaruddin Yamaguchi, Shingo Interpretation of Bahasa Isyarat Malaysia (BIM) Using SSD-MobileNet-V2 FPNLite and COCO mAP |
| title | Interpretation of Bahasa Isyarat Malaysia (BIM) Using
SSD-MobileNet-V2 FPNLite and COCO mAP |
| title_full | Interpretation of Bahasa Isyarat Malaysia (BIM) Using
SSD-MobileNet-V2 FPNLite and COCO mAP |
| title_fullStr | Interpretation of Bahasa Isyarat Malaysia (BIM) Using
SSD-MobileNet-V2 FPNLite and COCO mAP |
| title_full_unstemmed | Interpretation of Bahasa Isyarat Malaysia (BIM) Using
SSD-MobileNet-V2 FPNLite and COCO mAP |
| title_short | Interpretation of Bahasa Isyarat Malaysia (BIM) Using
SSD-MobileNet-V2 FPNLite and COCO mAP |
| title_sort | interpretation of bahasa isyarat malaysia bim using ssd mobilenet v2 fpnlite and coco map |
| topic | T Technology (General) |
| url | http://eprints.uthm.edu.my/11622/1/J16131_fcea86e7763277e20bdbfab7a92ccd21.pdf |
| work_keys_str_mv | AT saifulbahriiffahzulaikha interpretationofbahasaisyaratmalaysiabimusingssdmobilenetv2fpnliteandcocomap AT saonsharifah interpretationofbahasaisyaratmalaysiabimusingssdmobilenetv2fpnliteandcocomap AT mahamadabdkadir interpretationofbahasaisyaratmalaysiabimusingssdmobilenetv2fpnliteandcocomap AT isakhalid interpretationofbahasaisyaratmalaysiabimusingssdmobilenetv2fpnliteandcocomap AT fadlilahumi interpretationofbahasaisyaratmalaysiabimusingssdmobilenetv2fpnliteandcocomap AT ahmadonmohdanuaruddin interpretationofbahasaisyaratmalaysiabimusingssdmobilenetv2fpnliteandcocomap AT yamaguchishingo interpretationofbahasaisyaratmalaysiabimusingssdmobilenetv2fpnliteandcocomap |