Papers
arxiv:2407.04172

ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild

Published on Jul 4
Β· Submitted by akhaliq on Jul 8
Authors:
,

Abstract

Given the ubiquity of charts as a data analysis, visualization, and decision-making tool across industries and sciences, there has been a growing interest in developing pre-trained foundation models as well as general purpose instruction-tuned models for chart understanding and reasoning. However, existing methods suffer crucial drawbacks across two critical axes affecting the performance of chart representation models: they are trained on data generated from underlying data tables of the charts, ignoring the visual trends and patterns in chart images, and use weakly aligned vision-language backbone models for domain-specific training, limiting their generalizability when encountering charts in the wild. We address these important drawbacks and introduce ChartGemma, a novel chart understanding and reasoning model developed over PaliGemma. Rather than relying on underlying data tables, ChartGemma is trained on instruction-tuning data generated directly from chart images, thus capturing both high-level trends and low-level visual information from a diverse set of charts. Our simple approach achieves state-of-the-art results across 5 benchmarks spanning chart summarization, question answering, and fact-checking, and our elaborate qualitative studies on real-world charts show that ChartGemma generates more realistic and factually correct summaries compared to its contemporaries. We release the code, model checkpoints, dataset, and demos at https://github.com/vis-nlp/ChartGemma.

Community

Paper submitter

Hi @ahmed-masry feel free to claim this paper so that it appears on your HF profile :)

Also really awesome to see you used my notebook as inspiration :) would you be able to also link your dataset to this paper?

Β·
Paper author

Your tutorials are really awesome! I learned a lot from them. We will format and release the dataset this week on HF as promised in the paper.

Hi @ahmed-masry thank you for your paper!
On the paper, it is said that the dataset will be public.
However, I don't find it on the github.
Would it be possible to upload it on Hugging Face?
Thanks!

Β·

Hello! We would be releasing it within this week :)

Paper author

We have compiled and released the dataset. You can access it here: https://huggingface.co/datasets/ahmed-masry/ChartGemma

Sign up or log in to comment

Models citing this paper 1

Datasets citing this paper 1

Spaces citing this paper 8

Collections including this paper 9