Skip to content

Benchmark

CharXiv Reasoning Questions - Overall

Vision benchmarks measure how well models understand images. Scores are percentages. Higher is better.

Models measured
33
Scores recorded
34
Sources
1
IQ weight
1.68

Score distribution

41.8%
Mean
42.4%
Median
14.5%
Std dev
24.7%
10th percentile
60.2%
90th percentile
16%
Min
16%78.6%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.68 for chat models, measured over 33 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1OpenAI o3high· CharXiv Leaderboard data docs78.6%
2o4 Minihigh· CharXiv Leaderboard data docs72%
3Claude Sonnet 3.7· CharXiv Leaderboard data docs64.2%
4Claude 3.5 Sonnet· CharXiv Leaderboard data docs60.2%
5GPT-4.1 mini· CharXiv Leaderboard data docs56.8%
6GPT 4.1· CharXiv Leaderboard data docs56.7%
7GPT-4.5 Preview· CharXiv Leaderboard data docs55.4%
8o1high· CharXiv Leaderboard data docs55.1%
9o1· CharXiv Leaderboard data docs52.6%
10Qwen2.5 VL 72B Instruct· CharXiv Leaderboard data docs49.7%
11GPT 4o· CharXiv Leaderboard data docs47.1%
12internvl3-38b· CharXiv Leaderboard data docs46.4%
13InternVL3 78B· CharXiv Leaderboard data docs46%
14Gemini 1.5 Pro· CharXiv Leaderboard data docs43.3%
15Qwen2-VL-72B· CharXiv Leaderboard data docs43%
16Qwen2.5-VL 7B Instruct· CharXiv Leaderboard data docs42.5%
17Pixtral 12B· CharXiv Leaderboard data docs42.4%
18GPT-4.1 nano· CharXiv Leaderboard data docs40.5%
19internvl3-8b· CharXiv Leaderboard data docs37.6%
20Molmo 72B· CharXiv Leaderboard data docs36.5%
21GPT-4o mini· CharXiv Leaderboard data docs34.1%
22Gemini 1.5 Flash· CharXiv Leaderboard data docs33.9%
23Phi-3.5 Vision Instruct· CharXiv Leaderboard data docs32.7%
24Claude 3 Sonnet· CharXiv Leaderboard data docs32.2%
25Claude 3 Haiku· CharXiv Leaderboard data docs31.8%
26qwen2.5-vl-3b-instruct· CharXiv Leaderboard data docs31.3%
27Claude 3 Opus· CharXiv Leaderboard data docs30.2%
28Reka Core· CharXiv Leaderboard data docs28.9%
29Reka Flash· CharXiv Leaderboard data docs26.6%
30Molmo 7B-D· CharXiv Leaderboard data docs26.4%
31Qwen-VL Max· CharXiv Leaderboard data docs24.7%
32Reka Edge· CharXiv Leaderboard data docs23.5%
33gemini-1.0-pro· CharXiv Leaderboard data docs22.8%
34Qwen-VL Plus· CharXiv Leaderboard data docs16%

Data credits: CharXiv (Wang et al., NeurIPS 2024), Princeton NLP. https://charxiv.github.io