Skip to content

Benchmark

CharXiv Descriptive Questions - Overall

Vision benchmarks measure how well models understand images. Scores are percentages. Higher is better.

Models measured
31
Scores recorded
32
Sources
1
IQ weight
1.38

Score distribution

71.9%
Mean
73.9%
Median
17.2%
Std dev
52.7%
10th percentile
88.9%
90th percentile
28.9%
Min
28.9%95%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.38 for chat models, measured over 31 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1OpenAI o3high· CharXiv Leaderboard data docs95%
2o4 Minihigh· CharXiv Leaderboard data docs94.3%
3GPT-4.5 Preview· CharXiv Leaderboard data docs90%
4o1high· CharXiv Leaderboard data docs88.9%
5GPT-4.1 mini· CharXiv Leaderboard data docs88.4%
6GPT 4.1· CharXiv Leaderboard data docs87.9%
7o1· CharXiv Leaderboard data docs87.5%
8Qwen2.5 VL 72B Instruct· CharXiv Leaderboard data docs87.4%
9internvl3-38b· CharXiv Leaderboard data docs87.2%
10InternVL3 78B· CharXiv Leaderboard data docs85.1%
11GPT 4o· CharXiv Leaderboard data docs84.5%
12Claude 3.5 Sonnet· CharXiv Leaderboard data docs84.3%
13Qwen2-VL-72B· CharXiv Leaderboard data docs81.3%
14Molmo 72B· CharXiv Leaderboard data docs75.5%
15GPT-4o mini· CharXiv Leaderboard data docs74.9%
16GPT-4.1 nano· CharXiv Leaderboard data docs73.9%
17Qwen2.5-VL 7B Instruct· CharXiv Leaderboard data docs73.9%
18Claude 3 Sonnet· CharXiv Leaderboard data docs73.7%
19internvl3-8b· CharXiv Leaderboard data docs73.6%
20Gemini 1.5 Pro· CharXiv Leaderboard data docs72%
21Claude 3 Opus· CharXiv Leaderboard data docs71.6%
22Pixtral 12B· CharXiv Leaderboard data docs68.1%
23Claude 3 Haiku· CharXiv Leaderboard data docs65.1%
24qwen2.5-vl-3b-instruct· CharXiv Leaderboard data docs58.6%
25Reka Flash· CharXiv Leaderboard data docs56.5%
26Reka Core· CharXiv Leaderboard data docs55.6%
27Phi-3.5 Vision Instruct· CharXiv Leaderboard data docs55%
28gemini-1.0-pro· CharXiv Leaderboard data docs54.4%
29Molmo 7B-D· CharXiv Leaderboard data docs52.7%
30Qwen-VL Max· CharXiv Leaderboard data docs41.5%
31Reka Edge· CharXiv Leaderboard data docs33.6%
32Qwen-VL Plus· CharXiv Leaderboard data docs28.9%

Data credits: CharXiv (Wang et al., NeurIPS 2024), Princeton NLP. https://charxiv.github.io