MFIRA: Multimodal Fusion Intent Recognition Algorithm for AR Chemistry Experiments
The current virtual system for secondary school experiments poses several issues, such as limited methods of operation for students and an inability of the system to comprehend the users’ operational intentions, resulting in a greater operational burden for students and hindering the goal of the exp...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-07-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/13/14/8200 |
| _version_ | 1797590460803317760 |
|---|---|
| author | Zishuo Xia Zhiquan Feng Xiaohui Yang Dehui Kong Hong Cui |
| author_facet | Zishuo Xia Zhiquan Feng Xiaohui Yang Dehui Kong Hong Cui |
| author_sort | Zishuo Xia |
| collection | DOAJ |
| description | The current virtual system for secondary school experiments poses several issues, such as limited methods of operation for students and an inability of the system to comprehend the users’ operational intentions, resulting in a greater operational burden for students and hindering the goal of the experimental practice. However, many traditional multimodal fusion algorithms rely solely on individual modalities for the analysis of users’ experimental intentions, failing to fully utilize the intention information for each modality. To rectify these issues, we present a new multimodal fusion algorithm, MFIRA, which intersects and blends intention probabilities between channels by executing parallel processing of multimodal information at the intention layer. Additionally, we developed an augmented reality (AR) virtual experiment platform based on the Hololens 2, which enables students to conduct experiments using speech, gestures, and vision. Employing the MFIRA algorithm, the system captures users’ experimental intent and navigates or rectifies errors to guide students through their experiments. The experimental results indicate that the MFIRA algorithm boasts a 97.3% accuracy rate in terms of interpreting users’ experimental intent. Compared to existing experimental platforms, this system is considerably more interactive and immersive for students and is highly applicable in secondary school experimental chemistry classrooms. |
| first_indexed | 2024-03-11T01:20:48Z |
| format | Article |
| id | doaj.art-96d2e954014d40b6b357024e8091bc8f |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-11T01:20:48Z |
| publishDate | 2023-07-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-96d2e954014d40b6b357024e8091bc8f2023-11-18T18:09:41ZengMDPI AGApplied Sciences2076-34172023-07-011314820010.3390/app13148200MFIRA: Multimodal Fusion Intent Recognition Algorithm for AR Chemistry ExperimentsZishuo Xia0Zhiquan Feng1Xiaohui Yang2Dehui Kong3Hong Cui4School of Information Science and Engineering, University of Jinan, Jinan 250022, ChinaSchool of Information Science and Engineering, University of Jinan, Jinan 250022, ChinaSchool of Information Science and Engineering, University of Jinan, Jinan 250022, ChinaSchool of Information Science and Engineering, University of Jinan, Jinan 250022, ChinaSchool of Information Science and Engineering, University of Jinan, Jinan 250022, ChinaThe current virtual system for secondary school experiments poses several issues, such as limited methods of operation for students and an inability of the system to comprehend the users’ operational intentions, resulting in a greater operational burden for students and hindering the goal of the experimental practice. However, many traditional multimodal fusion algorithms rely solely on individual modalities for the analysis of users’ experimental intentions, failing to fully utilize the intention information for each modality. To rectify these issues, we present a new multimodal fusion algorithm, MFIRA, which intersects and blends intention probabilities between channels by executing parallel processing of multimodal information at the intention layer. Additionally, we developed an augmented reality (AR) virtual experiment platform based on the Hololens 2, which enables students to conduct experiments using speech, gestures, and vision. Employing the MFIRA algorithm, the system captures users’ experimental intent and navigates or rectifies errors to guide students through their experiments. The experimental results indicate that the MFIRA algorithm boasts a 97.3% accuracy rate in terms of interpreting users’ experimental intent. Compared to existing experimental platforms, this system is considerably more interactive and immersive for students and is highly applicable in secondary school experimental chemistry classrooms.https://www.mdpi.com/2076-3417/13/14/8200multimodal fusionvirtual experimentintelligent teachinghuman-computer interactionintention understanding |
| spellingShingle | Zishuo Xia Zhiquan Feng Xiaohui Yang Dehui Kong Hong Cui MFIRA: Multimodal Fusion Intent Recognition Algorithm for AR Chemistry Experiments Applied Sciences multimodal fusion virtual experiment intelligent teaching human-computer interaction intention understanding |
| title | MFIRA: Multimodal Fusion Intent Recognition Algorithm for AR Chemistry Experiments |
| title_full | MFIRA: Multimodal Fusion Intent Recognition Algorithm for AR Chemistry Experiments |
| title_fullStr | MFIRA: Multimodal Fusion Intent Recognition Algorithm for AR Chemistry Experiments |
| title_full_unstemmed | MFIRA: Multimodal Fusion Intent Recognition Algorithm for AR Chemistry Experiments |
| title_short | MFIRA: Multimodal Fusion Intent Recognition Algorithm for AR Chemistry Experiments |
| title_sort | mfira multimodal fusion intent recognition algorithm for ar chemistry experiments |
| topic | multimodal fusion virtual experiment intelligent teaching human-computer interaction intention understanding |
| url | https://www.mdpi.com/2076-3417/13/14/8200 |
| work_keys_str_mv | AT zishuoxia mfiramultimodalfusionintentrecognitionalgorithmforarchemistryexperiments AT zhiquanfeng mfiramultimodalfusionintentrecognitionalgorithmforarchemistryexperiments AT xiaohuiyang mfiramultimodalfusionintentrecognitionalgorithmforarchemistryexperiments AT dehuikong mfiramultimodalfusionintentrecognitionalgorithmforarchemistryexperiments AT hongcui mfiramultimodalfusionintentrecognitionalgorithmforarchemistryexperiments |