Pathumma-llm-vision-1.0.0

Model Overview

Pathumma-llm-vision-1.0.0 is a multi-modal language model fine-tuned for Visual Question Answering (VQA) and Image Captioning tasks. It contains 8 billion parameters and leverages both image and text processing to understand and generate multi-modal content.

• Model Name: Pathumma-llm-vision-1.0.0
• Base Model: HuggingFaceM4/Idefics3-8B-Llama3
• Architecture: Multi-modal LLM (Visual Language Model)
• Parameters: 8 Billion
• Organization: NECTEC
• License: [Specify License]

Intended Use

• Primary Use Cases:
  • Visual Question Answering (VQA)
  • Image Captioning
• Intended Users: Developers, researchers, and AI practitioners working on multi-modal tasks.
• Possible Applications: Educational tools, accessibility applications, interactive visual content generation.

Model Description

Pathumma-llm-vision-1.0.0 is designed to perform multi-modal tasks by integrating both visual and textual information. The model is fine-tuned with diverse datasets to improve its ability to understand and generate content that aligns with both image and text inputs.

Training Data

The model was fine-tuned on several datasets:

• Thai Image Caption: Data sourced from image captioning competitions on Kaggle.
• Thai Shorthand Dataset: Data related to the Thai language.
• ShareGPT-4o (translated into Thai): Data translated from GPT-4o-mini outputs into Thai.
• Small-Thai-Wikipedia-location: Articles in Thai from Wikipedia about geographic locations.
• Synthetic Data: Additional synthetic data generated to increase dataset diversity.

Dataset Size

• Training Dataset Size: 112,768 examples
• Validation Dataset Size: 9,036 examples

Training Details

• Hardware Used:
  • HPC Cluster: Lanta
  • Number of Nodes: 16 Nodes
  • GPUs per Node: 4 GPUs
  • Total GPUs Used: 64 GPUs
• Fine-tuning Duration: 3 hours, 18 minutes, and 11 seconds (excluding evaluation)

Evaluation Results

Type	Encoder	Decoder	IPU24-dataset (test) (Sentence SacreBLEU)
Idefic3-8B-Llama3	siglip-so400m-patch14-384	Meta-Llama-3.1-8B-Instruct	0.02657
Pathumma-llm-vision-beta-0.0.0	siglip-so400m-patch14-384	Meta-Llama-3.1-8B-Instruct	13.45412
Pathumma-llm-vision-1.0.0	siglip-so400m-patch14-384	Meta-Llama-3.1-8B-Instruct	17.66370
llama-3-typhoon-v1.5-8b-vision-preview	siglip-so400m-patch14-384	Llama-3-Typhoon-1.5-8B-instruct	8.288626

**Note: Other models not target fine-tuned on IPU24-datasets may be less representative of IPU24 performance.

• Accuracy on VQA Tasks with testing a private dataset: 30.34%

Required Libraries

Before you start, ensure you have the following libraries installed:

pip install git+https://github.com/andimarafioti/transformers.git@idefics3

Usage

We provide a inference tutorial. To use the model with the Hugging Face transformers library:

import io
import os
import time
import random
import requests
import shutil
from IPython.display import display, Markdown
from IPython.display import clear_output as cls

import numpy as np
import pandas as pd
from PIL import Image

import torch

import transformers
from transformers import (
    Idefics3ForConditionalGeneration,
    AutoProcessor,
    BitsAndBytesConfig,
)

DEVICE = f"cuda" if torch.cuda.is_available() else 'cpu' if torch.cpu.is_available() else 'mps'
print(DEVICE)
if DEVICE == 'cuda': display(torch.cuda.device_count())

N = 5

revision = "quantized8bit"
processor = AutoProcessor.from_pretrained(
    "nectec/Pathumma-llm-vision-1.0.0",
    revision=revision,                         # Optional
    do_image_splitting=False,
    # size={"longest_edge": N*364},            # Optional
    # size={"height": N*364, "width": N*364},  # Optional
)

model = Idefics3ForConditionalGeneration.from_pretrained(
        "nectec/Pathumma-llm-vision-1.0.0",
        revision=revision,                     # Optional
        torch_dtype=torch.float16,
        device_map=DEVICE
    )

print(processor.image_processor.size)

url_path = None
local_path = "./path/picture.jpg" if not url_path else io.BytesIO(requests.get(url_path).content)
image = Image.open(local_path)

question = "รายละเอียดของรูปภาพนี้"
messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "You are a helpful assistant."},
            {"type": "image"},
            {"type": "text", "text": question}
        ]
    }
]

text = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
)

encoding = processor(
    images=image,
    text=text.strip(),
    # padding='max_length',
    # truncation=True,
    # max_length=,
    return_tensors="pt"
)

encoding = {k: v.to(DEVICE) for k, v in encoding.items()}

# Example: Run inference on text input
start_time = time.time()
model.eval()
with torch.inference_mode():
    # Generate
    generated_ids = model.generate(
        **inputs, 
        max_new_tokens=128, 
        # temperature=.5, 
        # repetition_penalty=1.,
        # # top_k=1.,
        # top_p=1,
    )
    generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
end_time = time.time()

## Get letency_time...
latency_time = end_time - start_time

answer_prompt = generated_text.split('Assistant:')[1].strip()

# Output processing (depends on task requirements)
print(answer_prompt)
print(f"latency_time: {latency_time:.3f} sec.")

# >>> output:
# >>> ลูกฮิปโปแคระกำลังยืนอยู่ข้างแม่ฮิปโปแคระที่กำลังอาบน้ำ
# >>> latency_time: 7.642 sec.

Limitations and Biases

• The model may exhibit biases due to the training data, which might not be fully representative of all contexts.
• Performance may degrade on unfamiliar images or non-standard question formats.

Ethical Considerations

• The model should not be used to generate misleading information or in ways that violate privacy.
• Consider fairness and minimize bias when using the model for language and image processing tasks.

Citation

If you use this model, please cite it as follows:

@misc{PathummaVision,
  author = {Thirawarit Pitiphiphat and NECTEC Team},
  title = {nectec/Pathumma-llm-vision-1.0.0},
  year = {2024},
  url = {https://huggingface.co/nectec/Pathumma-llm-vision-1.0.0}
}

@misc{laurençon2024building,
      title={Building and better understanding vision-language models: insights and future directions.}, 
      author={Hugo Laurençon and Andrés Marafioti and Victor Sanh and Léo Tronchon},
      year={2024},
      eprint={2408.12637},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Contributor Contract

LLM Team
Pakawat Phasook (pakawat.phas@kmutt.ac.th)
Jessada Pranee (jessada.pran@kmutt.ac.th)
Arnon Saeoung (anon.saeoueng@gmail.com)
Kun Kerdthaisong (kun.ker@dome.tu.ac.th)
Kittisak Sukhantharat (kittisak.suk@stu.nida.ac.th)
Chaianun Damrongrat (chaianun.damrongrat@nectec.or.th)
Sarawoot Kongyoung (sarawoot.kongyoung@nectec.or.th)

Audio Team
Pattara Tipaksorn (pattara.tip@ncr.nstda.or.th)
Wayupuk Sommuang (wayupuk.som@dome.tu.ac.th)
Oatsada Chatthong (atsada.cha@dome.tu.ac.th)
Kwanchiva Thangthai (kwanchiva.thangthai@nectec.or.th)

Vision Team
Thirawarit Pitiphiphat (thirawarit.p@gmail.com)
Peerapas Ngokpon (jamesselmon78169@gmail.com)
Theerasit Issaranon (theerasit.issaranon@nectec.or.th)

Contact

For questions or support, please contact https://discord.gg/3WJwJjZt7r.

This formatting provides a clean, structured, and readable Markdown layout for these sections. Let me know if further adjustments are needed!