ahmed-masry commited on
Commit
fadd609
1 Parent(s): 2ec6397

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +76 -3
README.md CHANGED
@@ -1,3 +1,76 @@
1
- ---
2
- license: mit
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ language:
4
+ - en
5
+ ---
6
+
7
+ # ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoning
8
+
9
+ Venue: **ACL 2024 (Findings)**
10
+
11
+ Paper Link: https://arxiv.org/abs/2403.09028
12
+
13
+ The abstract of the paper states that:
14
+ > Charts provide visual representations of data and are widely used for analyzing information, addressing queries, and conveying insights to others. Various chart-related downstream tasks have emerged recently, such as question-answering and summarization. A common strategy to solve these tasks is to fine-tune various models originally trained on vision tasks language. However, such task-specific models are not capable of solving a wide range of chart-related tasks, constraining their real-world applicability. To overcome these challenges, we introduce ChartInstruct: a novel chart-specific vision-language Instruction following dataset comprising 191K instructions generated with 71K charts. We then present two distinct systems for instruction tuning on such datasets: (1) an end-to-end model that connects a vision encoder for chart understanding with a LLM; and (2) a pipeline model that employs a two-step approach to extract chart data tables and input them into the LLM. In experiments on four downstream tasks, we first show the effectiveness of our model--achieving a new set of state-of-the-art results. Further evaluation shows that our instruction-tuning approach supports a wide array of real-world chart comprehension and reasoning scenarios, thereby expanding the scope and applicability of our models to new kinds of tasks.
15
+ # Web Demo
16
+ If you wish to quickly try our model, you can access our public web demo hosted on the Hugging Face Spaces platform with a friendly interface!
17
+
18
+ [ChartInstruct-FlanT5-XL Web Demo](https://huggingface.co/spaces/ahmed-masry/ChartInstruct-LLama2)
19
+
20
+ # Inference
21
+ You can easily use our models for inference with the huggingface library!
22
+ You just need to do the following:
23
+ 1. Chage the **_image_path_** to your chart example image path on your system
24
+ 2. Write the **_input_text_**
25
+
26
+ We recommend using beam search with a beam size of 4, but if your machine has low memory, you can remove the num_beams from the generate method.
27
+ ```
28
+ from PIL import Image
29
+ import requests
30
+ from transformers import AutoProcessor, AutoModelForSeq2SeqLM
31
+ import torch
32
+
33
+ torch.hub.download_url_to_file('https://raw.githubusercontent.com/vis-nlp/ChartQA/main/ChartQA%20Dataset/val/png/multi_col_1229.png', 'chart_example_1.png')
34
+
35
+ image_path = "/content/chart_example_1.png"
36
+ input_text = "What is the share of respondants who prefer Whatsapp in the 18-29 age group?"
37
+
38
+ input_prompt = f"<image>\n Question: {input_text} Answer: "
39
+
40
+ model = AutoModelForSeq2SeqLM.from_pretrained("ahmed-masry/ChartInstruct-FlanT5-XL", torch_dtype=torch.float16, trust_remote_code=True)
41
+ processor = AutoProcessor.from_pretrained("ahmed-masry/ChartInstruct-FlanT5-XL")
42
+
43
+ device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
44
+ model.to(device)
45
+
46
+ image = Image.open(image_path).convert('RGB')
47
+
48
+ inputs = processor(text=input_prompt, images=image, return_tensors="pt")
49
+ inputs = {k: v.to(device) for k, v in inputs.items()}
50
+
51
+ # change type if pixel_values in inputs to fp16.
52
+ inputs['pixel_values'] = inputs['pixel_values'].to(torch.float16)
53
+
54
+ # Generate
55
+ generate_ids = model.generate(**inputs, num_beams=4, max_new_tokens=512)
56
+ output_text = processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
57
+ print(output_text)
58
+
59
+ ```
60
+
61
+ # Contact
62
+ If you have any questions about this work, please contact **[Ahmed Masry](https://ahmedmasryku.github.io/)** using the following email addresses: **amasry17@ku.edu.tr** or **ahmed.elmasry24653@gmail.com**.
63
+
64
+ # Reference
65
+ Please cite our paper if you use our model in your research.
66
+
67
+ ```
68
+ @misc{masry2024chartinstruct,
69
+ title={ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoning},
70
+ author={Ahmed Masry and Mehrad Shahmohammadi and Md Rizwan Parvez and Enamul Hoque and Shafiq Joty},
71
+ year={2024},
72
+ eprint={2403.09028},
73
+ archivePrefix={arXiv},
74
+ primaryClass={id='cs.CL' full_name='Computation and Language' is_active=True alt_name='cmp-lg' in_archive='cs' is_general=False description='Covers natural language processing. Roughly includes material in ACM Subject Class I.2.7. Note that work on artificial languages (programming languages, logics, formal systems) that does not explicitly address natural-language issues broadly construed (natural-language processing, computational linguistics, speech, text retrieval, etc.) is not appropriate for this area.'}
75
+ }
76
+ ```