Papers
arxiv:2412.08687

VisionArena: 230K Real World User-VLM Conversations with Preference Labels

Published on Dec 11
· Submitted by lisabdunlap on Dec 13
Authors:
,
,
,
,
,
,
,

Abstract

With the growing adoption and capabilities of vision-language models (VLMs) comes the need for benchmarks that capture authentic user-VLM interactions. In response, we create VisionArena, a dataset of 230K real-world conversations between users and VLMs. Collected from Chatbot Arena - an open-source platform where users interact with VLMs and submit preference votes - VisionArena spans 73K unique users, 45 VLMs, and 138 languages. Our dataset contains three subsets: VisionArena-Chat, 200k single and multi-turn conversations between a user and a VLM; VisionArena-Battle, 30K conversations comparing two anonymous VLMs with user preference votes; and VisionArena-Bench, an automatic benchmark of 500 diverse user prompts that efficiently approximate the live Chatbot Arena model rankings. Additionally, we highlight the types of question asked by users, the influence of response style on preference, and areas where models often fail. We find open-ended tasks like captioning and humor are highly style-dependent, and current VLMs struggle with spatial reasoning and planning tasks. Lastly, we show finetuning the same base model on VisionArena-Chat outperforms Llava-Instruct-158K, with a 17-point gain on MMMU and a 46-point gain on the WildVision benchmark. Dataset at https://huggingface.co/lmarena-ai

Community

Paper submitter

Collected from Chatbot Arena, VisionArena consists of 230k user-VLM conversations which span 73K users, 45 VLMs, and 138 languages.

A breakdown of Vision Arena's 3 datasets 🧵
1️⃣ VisionArena-Chat: 200K real convos between users & VLMs
2️⃣ VisionArena-Battle: 30K blind A/B tests where users pick their preferred model's response
3️⃣ VisionArena-Bench: 500 carefully selected prompts that predict arena rankings

Checkout these datasets and more on HuggingFace!

·

Great work ! I noticed the dataset isn’t available via the current link. Could you share when it might be released? Looking forward to your updates!

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2412.08687 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2412.08687 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2412.08687 in a Space README.md to link it from this page.

Collections including this paper 1