VisText: A Benchmark for Semantically Rich Chart Captioning
Captions that describe or explain charts help improve recall and comprehension of the depicted data and provide a more accessible medium for people with visual disabilities. However, current approaches for automatically generating such captions struggle to articulate the perceptual or cognitive feat...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Published: |
Massachusetts Institute of Technology
2023
|
| Online Access: | https://hdl.handle.net/1721.1/151207 |
| _version_ | 1826216575354535936 |
|---|---|
| author | Tang, Ben Jun-Hong |
| author2 | Satyanarayan, Arvind |
| author_facet | Satyanarayan, Arvind Tang, Ben Jun-Hong |
| author_sort | Tang, Ben Jun-Hong |
| collection | MIT |
| description | Captions that describe or explain charts help improve recall and comprehension of the depicted data and provide a more accessible medium for people with visual disabilities. However, current approaches for automatically generating such captions struggle to articulate the perceptual or cognitive features that are the hallmark of charts (e.g., complex trends and patterns). In response, we introduce VisText: a dataset of 12,441 pairs of charts and captions that describe the charts’ construction, report key statistics, and identify perceptual and cognitive phenomena. In VisText, a chart is available as three representations: a rasterized image, a backing data table, and a scene graph — a hierarchical representation of a chart’s visual elements akin to a web page’s Document Object Model (DOM). To evaluate the impact of VisText, we fine-tune state-of-the-art language models on our chart captioning task and apply prefix-tuning to produce captions that vary the semantic content they convey. Our models generate coherent, semantically rich captions and perform on par with state-of-the-art chart captioning models across machine translation and text generation metrics. Through qualitative analysis, we identify six broad categories of errors that our models make that can inform future work. |
| first_indexed | 2024-09-23T16:49:45Z |
| format | Thesis |
| id | mit-1721.1/151207 |
| institution | Massachusetts Institute of Technology |
| last_indexed | 2024-09-23T16:49:45Z |
| publishDate | 2023 |
| publisher | Massachusetts Institute of Technology |
| record_format | dspace |
| spelling | mit-1721.1/1512072023-08-01T03:02:48Z VisText: A Benchmark for Semantically Rich Chart Captioning Tang, Ben Jun-Hong Satyanarayan, Arvind Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science System Design and Management Program. Captions that describe or explain charts help improve recall and comprehension of the depicted data and provide a more accessible medium for people with visual disabilities. However, current approaches for automatically generating such captions struggle to articulate the perceptual or cognitive features that are the hallmark of charts (e.g., complex trends and patterns). In response, we introduce VisText: a dataset of 12,441 pairs of charts and captions that describe the charts’ construction, report key statistics, and identify perceptual and cognitive phenomena. In VisText, a chart is available as three representations: a rasterized image, a backing data table, and a scene graph — a hierarchical representation of a chart’s visual elements akin to a web page’s Document Object Model (DOM). To evaluate the impact of VisText, we fine-tune state-of-the-art language models on our chart captioning task and apply prefix-tuning to produce captions that vary the semantic content they convey. Our models generate coherent, semantically rich captions and perform on par with state-of-the-art chart captioning models across machine translation and text generation metrics. Through qualitative analysis, we identify six broad categories of errors that our models make that can inform future work. S.M. S.M. 2023-07-31T19:22:39Z 2023-07-31T19:22:39Z 2023-06 2023-06-23T19:56:58.487Z Thesis https://hdl.handle.net/1721.1/151207 0000-0001-9907-8008 In Copyright - Educational Use Permitted Copyright retained by author(s) https://rightsstatements.org/page/InC-EDU/1.0/ application/pdf Massachusetts Institute of Technology |
| spellingShingle | Tang, Ben Jun-Hong VisText: A Benchmark for Semantically Rich Chart Captioning |
| title | VisText: A Benchmark for Semantically Rich Chart Captioning |
| title_full | VisText: A Benchmark for Semantically Rich Chart Captioning |
| title_fullStr | VisText: A Benchmark for Semantically Rich Chart Captioning |
| title_full_unstemmed | VisText: A Benchmark for Semantically Rich Chart Captioning |
| title_short | VisText: A Benchmark for Semantically Rich Chart Captioning |
| title_sort | vistext a benchmark for semantically rich chart captioning |
| url | https://hdl.handle.net/1721.1/151207 |
| work_keys_str_mv | AT tangbenjunhong vistextabenchmarkforsemanticallyrichchartcaptioning |