Model,Open Ended VQA: % Human Rating,Multiple Choice VQA: % Accuracy,Hints-Multiple Choice VQA: % Accuracy ,Attributions-Multiple Choice VQA: % Accuracy ,Refernce Based-Automatic Evaluation: Accuracy of Judge Prediction Compared to Human Ratings,Refernce Free-Automatic Evaluation: Accuracy of Judge Prediction Compared to Human Ratings,Automatic Evaluation: % Auto-Rater Ratings,Hints-Automatic Evaluation: % Auto-Rater Ratings,Attributions-Automatic Evaluation: % Auto-Rater Ratings
Humans,82,*,*,*,*,*,78,*,*
Gemini Pro 1.5,40,38,66,72,87,52,53,62,29
Gemini Pro Vision,30,41,62,*,75,38,34,47,
GPT4,34,45,69,82,86,51,38,61,25
LlaVA-1.6-34B,15,24,30,*,76,43,21,16,*
LlaVA-1.5-7B,13,17,29,*,70,35,19,30,*
InstructBlip,13,*,*,*,*,*,20,28,*
Gemini Pro 1.5 Caption _ Gemini Pro 1.5,23,*,*,*,*,*,*,*,*
Human (Oracle) Caption _ Gemini Pro 1.5,50,*,*,*,*,*,*,*,*
Claude 3.5 Sonnet,*,46,45,*,*,*,39,*,*
GPT4o,*,55,83,*,*,*,50,*,*
Qwen-VL-Max,*,35,53,*,*,*,26,*,*
Molmo-7B,*,34,42,*,*,*,36,*,*