Robust-EQA: robust learning for embodied question answering with noisy labels
Embodied question answering (EQA) is a recently emerged research field in which an agent is asked to answer the user's questions by exploring the environment and collecting visual information. Plenty of researchers turn their attention to the EQA field due to its broad potential application are...
| Main Authors: | , , , , , |
|---|---|
| Other Authors: | |
| Format: | Journal Article |
| Language: | English |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/170567 |
| _version_ | 1826122215528071168 |
|---|---|
| author | Luo, Haonan Lin, Guosheng Shen, Fumin Huang, Xingguo Yao, Yazhou Shen, Hengtao |
| author2 | School of Computer Science and Engineering |
| author_facet | School of Computer Science and Engineering Luo, Haonan Lin, Guosheng Shen, Fumin Huang, Xingguo Yao, Yazhou Shen, Hengtao |
| author_sort | Luo, Haonan |
| collection | NTU |
| description | Embodied question answering (EQA) is a recently emerged research field in which an agent is asked to answer the user's questions by exploring the environment and collecting visual information. Plenty of researchers turn their attention to the EQA field due to its broad potential application areas, such as in-home robots, self-driven mobile, and personal assistants. High-level visual tasks, such as EQA, are susceptible to noisy inputs, because they have complex reasoning processes. Before the profits of the EQA field can be applied to practical applications, good robustness against label noise needs to be equipped. To tackle this problem, we propose a novel label noise-robust learning algorithm for the EQA task. First, a joint training co-regularization noise-robust learning method is proposed for noisy filtering of the visual question answering (VQA) module, which trains two parallel network branches by one loss function. Then, a two-stage hierarchical robust learning algorithm is proposed to filter out noisy navigation labels in both trajectory level and action level. Finally, by taking purified labels as inputs, a joint robust learning mechanism is given to coordinate the work of the whole EQA system. Empirical results demonstrate that, under extremely noisy environments (45% of noisy labels) and low-level noisy environments (20% of noisy labels), the robustness of deep learning models trained by our algorithm is superior to the existing EQA models in noisy environments. |
| first_indexed | 2024-10-01T05:44:51Z |
| format | Journal Article |
| id | ntu-10356/170567 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T05:44:51Z |
| publishDate | 2023 |
| record_format | dspace |
| spelling | ntu-10356/1705672023-09-19T06:30:08Z Robust-EQA: robust learning for embodied question answering with noisy labels Luo, Haonan Lin, Guosheng Shen, Fumin Huang, Xingguo Yao, Yazhou Shen, Hengtao School of Computer Science and Engineering Engineering::Computer science and engineering Task Analysis Noise Measurement Embodied question answering (EQA) is a recently emerged research field in which an agent is asked to answer the user's questions by exploring the environment and collecting visual information. Plenty of researchers turn their attention to the EQA field due to its broad potential application areas, such as in-home robots, self-driven mobile, and personal assistants. High-level visual tasks, such as EQA, are susceptible to noisy inputs, because they have complex reasoning processes. Before the profits of the EQA field can be applied to practical applications, good robustness against label noise needs to be equipped. To tackle this problem, we propose a novel label noise-robust learning algorithm for the EQA task. First, a joint training co-regularization noise-robust learning method is proposed for noisy filtering of the visual question answering (VQA) module, which trains two parallel network branches by one loss function. Then, a two-stage hierarchical robust learning algorithm is proposed to filter out noisy navigation labels in both trajectory level and action level. Finally, by taking purified labels as inputs, a joint robust learning mechanism is given to coordinate the work of the whole EQA system. Empirical results demonstrate that, under extremely noisy environments (45% of noisy labels) and low-level noisy environments (20% of noisy labels), the robustness of deep learning models trained by our algorithm is superior to the existing EQA models in noisy environments. Ministry of Education (MOE) National Research Foundation (NRF) This work was supported in part by the National Research Foundation Singapore through its AI Singapore Program under Grant AISGRP-2018-003, in part by the Ministry of Education Singapore (MOE) Tier-1 Research under Grant RG95/20, and in part by the China Postdoctoral Science Foundation under Grant 2022M722630. 2023-09-19T06:30:08Z 2023-09-19T06:30:08Z 2023 Journal Article Luo, H., Lin, G., Shen, F., Huang, X., Yao, Y. & Shen, H. (2023). Robust-EQA: robust learning for embodied question answering with noisy labels. IEEE Transactions On Neural Networks and Learning Systems. https://dx.doi.org/10.1109/TNNLS.2023.3251984 2162-237X https://hdl.handle.net/10356/170567 10.1109/TNNLS.2023.3251984 37028297 2-s2.0-85151383509 en AISGRP-2018-003 RG95/20 IEEE Transactions on Neural Networks and Learning Systems © 2023 IEEE. All rights reserved. |
| spellingShingle | Engineering::Computer science and engineering Task Analysis Noise Measurement Luo, Haonan Lin, Guosheng Shen, Fumin Huang, Xingguo Yao, Yazhou Shen, Hengtao Robust-EQA: robust learning for embodied question answering with noisy labels |
| title | Robust-EQA: robust learning for embodied question answering with noisy labels |
| title_full | Robust-EQA: robust learning for embodied question answering with noisy labels |
| title_fullStr | Robust-EQA: robust learning for embodied question answering with noisy labels |
| title_full_unstemmed | Robust-EQA: robust learning for embodied question answering with noisy labels |
| title_short | Robust-EQA: robust learning for embodied question answering with noisy labels |
| title_sort | robust eqa robust learning for embodied question answering with noisy labels |
| topic | Engineering::Computer science and engineering Task Analysis Noise Measurement |
| url | https://hdl.handle.net/10356/170567 |
| work_keys_str_mv | AT luohaonan robusteqarobustlearningforembodiedquestionansweringwithnoisylabels AT linguosheng robusteqarobustlearningforembodiedquestionansweringwithnoisylabels AT shenfumin robusteqarobustlearningforembodiedquestionansweringwithnoisylabels AT huangxingguo robusteqarobustlearningforembodiedquestionansweringwithnoisylabels AT yaoyazhou robusteqarobustlearningforembodiedquestionansweringwithnoisylabels AT shenhengtao robusteqarobustlearningforembodiedquestionansweringwithnoisylabels |