EKTVQA: Generalized Use of External Knowledge to Empower Scene Text in Text-VQA

The open-ended question answering task of Text-VQA often requires reading and reasoning about <italic>rarely seen or completely unseen</italic> scene text content of an image. We address this zero-shot nature of the task by proposing the generalized use of external knowledge to augment o...

Full description

Bibliographic Details
Main Authors: Arka Ujjal Dey, Ernest Valveny, Gaurav Harit
Format: Article
Language:English
Published: IEEE 2022-01-01
Series:IEEE Access
Subjects:
Online Access:https://ieeexplore.ieee.org/document/9807310/
_version_ 1818521811349929984
author Arka Ujjal Dey
Ernest Valveny
Gaurav Harit
author_facet Arka Ujjal Dey
Ernest Valveny
Gaurav Harit
author_sort Arka Ujjal Dey
collection DOAJ
description The open-ended question answering task of Text-VQA often requires reading and reasoning about <italic>rarely seen or completely unseen</italic> scene text content of an image. We address this zero-shot nature of the task by proposing the generalized use of external knowledge to augment our understanding of the scene text. We design a framework to extract, validate, and reason with knowledge using a standard multimodal transformer for vision language understanding tasks. Through empirical evidence and qualitative results, we demonstrate how external knowledge can highlight instance-only cues and thus help deal with training data bias, improve answer entity type correctness, and detect multiword named entities. We generate results comparable to the state-of-the-art on three publicly available datasets under the constraints of similar upstream OCR systems and training data.
first_indexed 2024-12-11T01:56:20Z
format Article
id doaj.art-0aa82099de6941e4ad0f990d0ff5084f
institution Directory Open Access Journal
issn 2169-3536
language English
last_indexed 2024-12-11T01:56:20Z
publishDate 2022-01-01
publisher IEEE
record_format Article
series IEEE Access
spelling doaj.art-0aa82099de6941e4ad0f990d0ff5084f2022-12-22T01:24:37ZengIEEEIEEE Access2169-35362022-01-0110720927210610.1109/ACCESS.2022.31864719807310EKTVQA: Generalized Use of External Knowledge to Empower Scene Text in Text-VQAArka Ujjal Dey0https://orcid.org/0000-0001-8392-1574Ernest Valveny1Gaurav Harit2IIT Jodhpur, Rajasthan, IndiaComputer Vision Center, Universitat Aut&#x00F2;noma de Barcelona, Bellaterra, Barcelona, SpainIIT Jodhpur, Rajasthan, IndiaThe open-ended question answering task of Text-VQA often requires reading and reasoning about <italic>rarely seen or completely unseen</italic> scene text content of an image. We address this zero-shot nature of the task by proposing the generalized use of external knowledge to augment our understanding of the scene text. We design a framework to extract, validate, and reason with knowledge using a standard multimodal transformer for vision language understanding tasks. Through empirical evidence and qualitative results, we demonstrate how external knowledge can highlight instance-only cues and thus help deal with training data bias, improve answer entity type correctness, and detect multiword named entities. We generate results comparable to the state-of-the-art on three publicly available datasets under the constraints of similar upstream OCR systems and training data.https://ieeexplore.ieee.org/document/9807310/External knowledgelanguage and visionscene textvisual semantics
spellingShingle Arka Ujjal Dey
Ernest Valveny
Gaurav Harit
EKTVQA: Generalized Use of External Knowledge to Empower Scene Text in Text-VQA
IEEE Access
External knowledge
language and vision
scene text
visual semantics
title EKTVQA: Generalized Use of External Knowledge to Empower Scene Text in Text-VQA
title_full EKTVQA: Generalized Use of External Knowledge to Empower Scene Text in Text-VQA
title_fullStr EKTVQA: Generalized Use of External Knowledge to Empower Scene Text in Text-VQA
title_full_unstemmed EKTVQA: Generalized Use of External Knowledge to Empower Scene Text in Text-VQA
title_short EKTVQA: Generalized Use of External Knowledge to Empower Scene Text in Text-VQA
title_sort ektvqa generalized use of external knowledge to empower scene text in text vqa
topic External knowledge
language and vision
scene text
visual semantics
url https://ieeexplore.ieee.org/document/9807310/
work_keys_str_mv AT arkaujjaldey ektvqageneralizeduseofexternalknowledgetoempowerscenetextintextvqa
AT ernestvalveny ektvqageneralizeduseofexternalknowledgetoempowerscenetextintextvqa
AT gauravharit ektvqageneralizeduseofexternalknowledgetoempowerscenetextintextvqa