EKTVQA: Generalized Use of External Knowledge to Empower Scene Text in Text-VQA
The open-ended question answering task of Text-VQA often requires reading and reasoning about <italic>rarely seen or completely unseen</italic> scene text content of an image. We address this zero-shot nature of the task by proposing the generalized use of external knowledge to augment o...
Main Authors: | , , |
---|---|
Format: | Article |
Language: | English |
Published: |
IEEE
2022-01-01
|
Series: | IEEE Access |
Subjects: | |
Online Access: | https://ieeexplore.ieee.org/document/9807310/ |
_version_ | 1818521811349929984 |
---|---|
author | Arka Ujjal Dey Ernest Valveny Gaurav Harit |
author_facet | Arka Ujjal Dey Ernest Valveny Gaurav Harit |
author_sort | Arka Ujjal Dey |
collection | DOAJ |
description | The open-ended question answering task of Text-VQA often requires reading and reasoning about <italic>rarely seen or completely unseen</italic> scene text content of an image. We address this zero-shot nature of the task by proposing the generalized use of external knowledge to augment our understanding of the scene text. We design a framework to extract, validate, and reason with knowledge using a standard multimodal transformer for vision language understanding tasks. Through empirical evidence and qualitative results, we demonstrate how external knowledge can highlight instance-only cues and thus help deal with training data bias, improve answer entity type correctness, and detect multiword named entities. We generate results comparable to the state-of-the-art on three publicly available datasets under the constraints of similar upstream OCR systems and training data. |
first_indexed | 2024-12-11T01:56:20Z |
format | Article |
id | doaj.art-0aa82099de6941e4ad0f990d0ff5084f |
institution | Directory Open Access Journal |
issn | 2169-3536 |
language | English |
last_indexed | 2024-12-11T01:56:20Z |
publishDate | 2022-01-01 |
publisher | IEEE |
record_format | Article |
series | IEEE Access |
spelling | doaj.art-0aa82099de6941e4ad0f990d0ff5084f2022-12-22T01:24:37ZengIEEEIEEE Access2169-35362022-01-0110720927210610.1109/ACCESS.2022.31864719807310EKTVQA: Generalized Use of External Knowledge to Empower Scene Text in Text-VQAArka Ujjal Dey0https://orcid.org/0000-0001-8392-1574Ernest Valveny1Gaurav Harit2IIT Jodhpur, Rajasthan, IndiaComputer Vision Center, Universitat Autònoma de Barcelona, Bellaterra, Barcelona, SpainIIT Jodhpur, Rajasthan, IndiaThe open-ended question answering task of Text-VQA often requires reading and reasoning about <italic>rarely seen or completely unseen</italic> scene text content of an image. We address this zero-shot nature of the task by proposing the generalized use of external knowledge to augment our understanding of the scene text. We design a framework to extract, validate, and reason with knowledge using a standard multimodal transformer for vision language understanding tasks. Through empirical evidence and qualitative results, we demonstrate how external knowledge can highlight instance-only cues and thus help deal with training data bias, improve answer entity type correctness, and detect multiword named entities. We generate results comparable to the state-of-the-art on three publicly available datasets under the constraints of similar upstream OCR systems and training data.https://ieeexplore.ieee.org/document/9807310/External knowledgelanguage and visionscene textvisual semantics |
spellingShingle | Arka Ujjal Dey Ernest Valveny Gaurav Harit EKTVQA: Generalized Use of External Knowledge to Empower Scene Text in Text-VQA IEEE Access External knowledge language and vision scene text visual semantics |
title | EKTVQA: Generalized Use of External Knowledge to Empower Scene Text in Text-VQA |
title_full | EKTVQA: Generalized Use of External Knowledge to Empower Scene Text in Text-VQA |
title_fullStr | EKTVQA: Generalized Use of External Knowledge to Empower Scene Text in Text-VQA |
title_full_unstemmed | EKTVQA: Generalized Use of External Knowledge to Empower Scene Text in Text-VQA |
title_short | EKTVQA: Generalized Use of External Knowledge to Empower Scene Text in Text-VQA |
title_sort | ektvqa generalized use of external knowledge to empower scene text in text vqa |
topic | External knowledge language and vision scene text visual semantics |
url | https://ieeexplore.ieee.org/document/9807310/ |
work_keys_str_mv | AT arkaujjaldey ektvqageneralizeduseofexternalknowledgetoempowerscenetextintextvqa AT ernestvalveny ektvqageneralizeduseofexternalknowledgetoempowerscenetextintextvqa AT gauravharit ektvqageneralizeduseofexternalknowledgetoempowerscenetextintextvqa |