Interpretation of Bahasa Isyarat Malaysia (BIM) Using SSD-MobileNet-V2 FPNLite and COCO mAP
: This research proposes a study on two-way communication between deaf/mute and normal people using an Android application. Despite advancements in technology, there is still a lack of mobile applications that facilitate two-way communication between deaf/mute and normal people, especially by using...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Mdpi
2023
|
| Subjects: | |
| Online Access: | http://eprints.uthm.edu.my/10156/1/J16131_fcea86e7763277e20bdbfab7a92ccd21.pdf |
| _version_ | 1825638300933685248 |
|---|---|
| author | Saiful Bahri, Iffah Zulaikha Sharifah Saon, Sharifah Saon Mahamad, Abd Kadir Isa, Khalid Umi Fadlilah, Umi Fadlilah Ahmadon, Mohd Anuaruddin Shingo Yamaguchi, Shingo Yamaguchi |
| author_facet | Saiful Bahri, Iffah Zulaikha Sharifah Saon, Sharifah Saon Mahamad, Abd Kadir Isa, Khalid Umi Fadlilah, Umi Fadlilah Ahmadon, Mohd Anuaruddin Shingo Yamaguchi, Shingo Yamaguchi |
| author_sort | Saiful Bahri, Iffah Zulaikha |
| collection | UTHM |
| description | : This research proposes a study on two-way communication between deaf/mute and normal people using an Android application. Despite advancements in technology, there is still a lack of mobile applications that facilitate two-way communication between deaf/mute and normal people,
especially by using Bahasa Isyarat Malaysia (BIM). This project consists of three parts: First, we use BIM letters, which enables the recognition of BIM letters and BIM combined letters to form a word. In this part, a MobileNet pre-trained model is implemented to train the model with a total
of 87,000 images for 29 classes, with a 10% test size and a 90% training size. The second part is BIM word hand gestures, which consists of five classes that are trained with the SSD-MobileNet-V2 FPNLite 320 × 320 pre-trained model with a speed of 22 s/frame rate and COCO mAP of 22.2, with a
total of 500 images for all five classes and first-time training set to 2000 steps, while the second- and third-time training are set to 2500 steps. The third part is Android application development using Android Studio, which contains the features of the BIM letters and BIM word hand gestures, with the trained models converted into TensorFlow Lite. This feature also includes the conversion of speech to text, whereby this feature allows converting speech to text through the Android application. Thus, BIM letters obtain 99.75% accuracy after training the models, while BIM word hand gestures obtain 61.60% accuracy. The suggested system is validated as a result of these simulations and tests. |
| first_indexed | 2024-03-05T22:04:43Z |
| format | Article |
| id | uthm.eprints-10156 |
| institution | Universiti Tun Hussein Onn Malaysia |
| language | English |
| last_indexed | 2024-03-05T22:04:43Z |
| publishDate | 2023 |
| publisher | Mdpi |
| record_format | dspace |
| spelling | uthm.eprints-101562023-10-17T07:40:52Z http://eprints.uthm.edu.my/10156/ Interpretation of Bahasa Isyarat Malaysia (BIM) Using SSD-MobileNet-V2 FPNLite and COCO mAP Saiful Bahri, Iffah Zulaikha Sharifah Saon, Sharifah Saon Mahamad, Abd Kadir Isa, Khalid Umi Fadlilah, Umi Fadlilah Ahmadon, Mohd Anuaruddin Shingo Yamaguchi, Shingo Yamaguchi T Technology (General) : This research proposes a study on two-way communication between deaf/mute and normal people using an Android application. Despite advancements in technology, there is still a lack of mobile applications that facilitate two-way communication between deaf/mute and normal people, especially by using Bahasa Isyarat Malaysia (BIM). This project consists of three parts: First, we use BIM letters, which enables the recognition of BIM letters and BIM combined letters to form a word. In this part, a MobileNet pre-trained model is implemented to train the model with a total of 87,000 images for 29 classes, with a 10% test size and a 90% training size. The second part is BIM word hand gestures, which consists of five classes that are trained with the SSD-MobileNet-V2 FPNLite 320 × 320 pre-trained model with a speed of 22 s/frame rate and COCO mAP of 22.2, with a total of 500 images for all five classes and first-time training set to 2000 steps, while the second- and third-time training are set to 2500 steps. The third part is Android application development using Android Studio, which contains the features of the BIM letters and BIM word hand gestures, with the trained models converted into TensorFlow Lite. This feature also includes the conversion of speech to text, whereby this feature allows converting speech to text through the Android application. Thus, BIM letters obtain 99.75% accuracy after training the models, while BIM word hand gestures obtain 61.60% accuracy. The suggested system is validated as a result of these simulations and tests. Mdpi 2023 Article PeerReviewed text en http://eprints.uthm.edu.my/10156/1/J16131_fcea86e7763277e20bdbfab7a92ccd21.pdf Saiful Bahri, Iffah Zulaikha and Sharifah Saon, Sharifah Saon and Mahamad, Abd Kadir and Isa, Khalid and Umi Fadlilah, Umi Fadlilah and Ahmadon, Mohd Anuaruddin and Shingo Yamaguchi, Shingo Yamaguchi (2023) Interpretation of Bahasa Isyarat Malaysia (BIM) Using SSD-MobileNet-V2 FPNLite and COCO mAP. Information, 14 (319). pp. 1-20. https://doi.org/10.3390/info14060319 |
| spellingShingle | T Technology (General) Saiful Bahri, Iffah Zulaikha Sharifah Saon, Sharifah Saon Mahamad, Abd Kadir Isa, Khalid Umi Fadlilah, Umi Fadlilah Ahmadon, Mohd Anuaruddin Shingo Yamaguchi, Shingo Yamaguchi Interpretation of Bahasa Isyarat Malaysia (BIM) Using SSD-MobileNet-V2 FPNLite and COCO mAP |
| title | Interpretation of Bahasa Isyarat Malaysia (BIM) Using SSD-MobileNet-V2 FPNLite and COCO mAP |
| title_full | Interpretation of Bahasa Isyarat Malaysia (BIM) Using SSD-MobileNet-V2 FPNLite and COCO mAP |
| title_fullStr | Interpretation of Bahasa Isyarat Malaysia (BIM) Using SSD-MobileNet-V2 FPNLite and COCO mAP |
| title_full_unstemmed | Interpretation of Bahasa Isyarat Malaysia (BIM) Using SSD-MobileNet-V2 FPNLite and COCO mAP |
| title_short | Interpretation of Bahasa Isyarat Malaysia (BIM) Using SSD-MobileNet-V2 FPNLite and COCO mAP |
| title_sort | interpretation of bahasa isyarat malaysia bim using ssd mobilenet v2 fpnlite and coco map |
| topic | T Technology (General) |
| url | http://eprints.uthm.edu.my/10156/1/J16131_fcea86e7763277e20bdbfab7a92ccd21.pdf |
| work_keys_str_mv | AT saifulbahriiffahzulaikha interpretationofbahasaisyaratmalaysiabimusingssdmobilenetv2fpnliteandcocomap AT sharifahsaonsharifahsaon interpretationofbahasaisyaratmalaysiabimusingssdmobilenetv2fpnliteandcocomap AT mahamadabdkadir interpretationofbahasaisyaratmalaysiabimusingssdmobilenetv2fpnliteandcocomap AT isakhalid interpretationofbahasaisyaratmalaysiabimusingssdmobilenetv2fpnliteandcocomap AT umifadlilahumifadlilah interpretationofbahasaisyaratmalaysiabimusingssdmobilenetv2fpnliteandcocomap AT ahmadonmohdanuaruddin interpretationofbahasaisyaratmalaysiabimusingssdmobilenetv2fpnliteandcocomap AT shingoyamaguchishingoyamaguchi interpretationofbahasaisyaratmalaysiabimusingssdmobilenetv2fpnliteandcocomap |