Interpretation of Bahasa Isyarat Malaysia (BIM) Using SSD-MobileNet-V2 FPNLite and COCO mAP
This research proposes a study on two-way communication between deaf/mute and normal people using an Android application. Despite advancements in technology, there is still a lack of mobile applications that facilitate two-way communication between deaf/mute and normal people, especially by using Ba...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-05-01
|
| Series: | Information |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2078-2489/14/6/319 |
| _version_ | 1827737040891936768 |
|---|---|
| author | Iffah Zulaikha Saiful Bahri Sharifah Saon Abd Kadir Mahamad Khalid Isa Umi Fadlilah Mohd Anuaruddin Bin Ahmadon Shingo Yamaguchi |
| author_facet | Iffah Zulaikha Saiful Bahri Sharifah Saon Abd Kadir Mahamad Khalid Isa Umi Fadlilah Mohd Anuaruddin Bin Ahmadon Shingo Yamaguchi |
| author_sort | Iffah Zulaikha Saiful Bahri |
| collection | DOAJ |
| description | This research proposes a study on two-way communication between deaf/mute and normal people using an Android application. Despite advancements in technology, there is still a lack of mobile applications that facilitate two-way communication between deaf/mute and normal people, especially by using Bahasa Isyarat Malaysia (BIM). This project consists of three parts: First, we use BIM letters, which enables the recognition of BIM letters and BIM combined letters to form a word. In this part, a MobileNet pre-trained model is implemented to train the model with a total of 87,000 images for 29 classes, with a 10% test size and a 90% training size. The second part is BIM word hand gestures, which consists of five classes that are trained with the SSD-MobileNet-V2 FPNLite 320 × 320 pre-trained model with a speed of 22 s/frame rate and COCO mAP of 22.2, with a total of 500 images for all five classes and first-time training set to 2000 steps, while the second- and third-time training are set to 2500 steps. The third part is Android application development using Android Studio, which contains the features of the BIM letters and BIM word hand gestures, with the trained models converted into TensorFlow Lite. This feature also includes the conversion of speech to text, whereby this feature allows converting speech to text through the Android application. Thus, BIM letters obtain 99.75% accuracy after training the models, while BIM word hand gestures obtain 61.60% accuracy. The suggested system is validated as a result of these simulations and tests. |
| first_indexed | 2024-03-11T02:20:30Z |
| format | Article |
| id | doaj.art-8c0a386e11114e1394d9a590f1409c51 |
| institution | Directory Open Access Journal |
| issn | 2078-2489 |
| language | English |
| last_indexed | 2024-03-11T02:20:30Z |
| publishDate | 2023-05-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Information |
| spelling | doaj.art-8c0a386e11114e1394d9a590f1409c512023-11-18T10:54:27ZengMDPI AGInformation2078-24892023-05-0114631910.3390/info14060319Interpretation of Bahasa Isyarat Malaysia (BIM) Using SSD-MobileNet-V2 FPNLite and COCO mAPIffah Zulaikha Saiful Bahri0Sharifah Saon1Abd Kadir Mahamad2Khalid Isa3Umi Fadlilah4Mohd Anuaruddin Bin Ahmadon5Shingo Yamaguchi6Faculty of Electrical and Electronic Engineering, Universiti Tun Hussein Onn Malaysia, 86400 Parit Raja, Johor, MalaysiaFaculty of Electrical and Electronic Engineering, Universiti Tun Hussein Onn Malaysia, 86400 Parit Raja, Johor, MalaysiaFaculty of Electrical and Electronic Engineering, Universiti Tun Hussein Onn Malaysia, 86400 Parit Raja, Johor, MalaysiaFaculty of Electrical and Electronic Engineering, Universiti Tun Hussein Onn Malaysia, 86400 Parit Raja, Johor, MalaysiaTeknik Elektro, Fakultas Teknik, Kampus 2, Universitas Muhammadiyah Surakarta (UMS), Jl. Ahmad Yani, Tromol Pos 1, Surakarta 57169, Jawa Tengah, IndonesiaGraduate School of Sciences and Technology for Innovation, Yamaguchi University, Tokiwadai 2-16-1, Ube 755-8611, JapanGraduate School of Sciences and Technology for Innovation, Yamaguchi University, Tokiwadai 2-16-1, Ube 755-8611, JapanThis research proposes a study on two-way communication between deaf/mute and normal people using an Android application. Despite advancements in technology, there is still a lack of mobile applications that facilitate two-way communication between deaf/mute and normal people, especially by using Bahasa Isyarat Malaysia (BIM). This project consists of three parts: First, we use BIM letters, which enables the recognition of BIM letters and BIM combined letters to form a word. In this part, a MobileNet pre-trained model is implemented to train the model with a total of 87,000 images for 29 classes, with a 10% test size and a 90% training size. The second part is BIM word hand gestures, which consists of five classes that are trained with the SSD-MobileNet-V2 FPNLite 320 × 320 pre-trained model with a speed of 22 s/frame rate and COCO mAP of 22.2, with a total of 500 images for all five classes and first-time training set to 2000 steps, while the second- and third-time training are set to 2500 steps. The third part is Android application development using Android Studio, which contains the features of the BIM letters and BIM word hand gestures, with the trained models converted into TensorFlow Lite. This feature also includes the conversion of speech to text, whereby this feature allows converting speech to text through the Android application. Thus, BIM letters obtain 99.75% accuracy after training the models, while BIM word hand gestures obtain 61.60% accuracy. The suggested system is validated as a result of these simulations and tests.https://www.mdpi.com/2078-2489/14/6/319Bahasa Isyarat Malaysia (BIM)SSD-MobileNet-V2 FPNLiteCOCO mAPTensorFlow LiteAndroid application |
| spellingShingle | Iffah Zulaikha Saiful Bahri Sharifah Saon Abd Kadir Mahamad Khalid Isa Umi Fadlilah Mohd Anuaruddin Bin Ahmadon Shingo Yamaguchi Interpretation of Bahasa Isyarat Malaysia (BIM) Using SSD-MobileNet-V2 FPNLite and COCO mAP Information Bahasa Isyarat Malaysia (BIM) SSD-MobileNet-V2 FPNLite COCO mAP TensorFlow Lite Android application |
| title | Interpretation of Bahasa Isyarat Malaysia (BIM) Using SSD-MobileNet-V2 FPNLite and COCO mAP |
| title_full | Interpretation of Bahasa Isyarat Malaysia (BIM) Using SSD-MobileNet-V2 FPNLite and COCO mAP |
| title_fullStr | Interpretation of Bahasa Isyarat Malaysia (BIM) Using SSD-MobileNet-V2 FPNLite and COCO mAP |
| title_full_unstemmed | Interpretation of Bahasa Isyarat Malaysia (BIM) Using SSD-MobileNet-V2 FPNLite and COCO mAP |
| title_short | Interpretation of Bahasa Isyarat Malaysia (BIM) Using SSD-MobileNet-V2 FPNLite and COCO mAP |
| title_sort | interpretation of bahasa isyarat malaysia bim using ssd mobilenet v2 fpnlite and coco map |
| topic | Bahasa Isyarat Malaysia (BIM) SSD-MobileNet-V2 FPNLite COCO mAP TensorFlow Lite Android application |
| url | https://www.mdpi.com/2078-2489/14/6/319 |
| work_keys_str_mv | AT iffahzulaikhasaifulbahri interpretationofbahasaisyaratmalaysiabimusingssdmobilenetv2fpnliteandcocomap AT sharifahsaon interpretationofbahasaisyaratmalaysiabimusingssdmobilenetv2fpnliteandcocomap AT abdkadirmahamad interpretationofbahasaisyaratmalaysiabimusingssdmobilenetv2fpnliteandcocomap AT khalidisa interpretationofbahasaisyaratmalaysiabimusingssdmobilenetv2fpnliteandcocomap AT umifadlilah interpretationofbahasaisyaratmalaysiabimusingssdmobilenetv2fpnliteandcocomap AT mohdanuaruddinbinahmadon interpretationofbahasaisyaratmalaysiabimusingssdmobilenetv2fpnliteandcocomap AT shingoyamaguchi interpretationofbahasaisyaratmalaysiabimusingssdmobilenetv2fpnliteandcocomap |