Skip to content

Benchmark

ARC-Challenge

Scores are percentages. Higher is better.

Models and systems measured
35
Scores recorded
35
Sources
2
IQ weight
0.56

Score distribution

67.7%
Mean
66%
Median
20.8%
Std dev
38.7%
10th percentile
94.8%
90th percentile
27%
Min
27%95.9%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 0.56 for chat models, measured over 34 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#Model or systemScore
1DeepSeek V3· Epoch AI Benchmarking Hub data95.3%
2Llama 3.1 405B· Epoch AI Benchmarking Hub data95.3%
3Llama 3.1 70B Instruct0-shot· Reported by Meta· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.94.8%
4Qwen2.5 72B· Epoch AI Benchmarking Hub data94.5%
5Phi-3 Medium 128K Instruct· Epoch AI Benchmarking Hub data91.6%
6Phi-3 Small 8K Instruct· Epoch AI Benchmarking Hub data90.7%
7GPT-3.5 Turbo· Epoch AI Benchmarking Hub data87.4%
8Mixtral 8X7b V0.1· Epoch AI Benchmarking Hub data87.3%
9Claude Instant 1.2· Epoch AI Benchmarking Hub data86.3%
10Text Davinci 002· Epoch AI Benchmarking Hub data85.2%
11Phi-3 Mini 4K Instruct· Epoch AI Benchmarking Hub data84.9%
12Llama 3 8B Instruct· Epoch AI Benchmarking Hub data82.8%
13Mistral 7B v0.1· Epoch AI Benchmarking Hub data78.6%
14gemma-7b· Epoch AI Benchmarking Hub data78.3%
15phi-2· Epoch AI Benchmarking Hub data75.9%
16Qwen 2.5 Coder 32B· Epoch AI Benchmarking Hub data70.5%
17qwen2p5-coder-14b· Epoch AI Benchmarking Hub data66%
18deepseek-coder-v2-base· Epoch AI Benchmarking Hub data64.3%
19ChatGLM2 6B· Epoch AI Benchmarking Hub data61%
20Qwen2.5-Coder-7B· Epoch AI Benchmarking Hub data60.9%
21Llama 2 13b· Epoch AI Benchmarking Hub data60.3%
22deepseek-coder-v2-lite-base· Epoch AI Benchmarking Hub data57.3%
23text-davinci-001· Epoch AI Benchmarking Hub data53.2%
24qwen2p5-coder-3b· Epoch AI Benchmarking Hub data52.9%
25LLaMA 13B· Epoch AI Benchmarking Hub data52.7%
26Yi-6B· Epoch AI Benchmarking Hub data50.3%
27Starcoder2 15b· Epoch AI Benchmarking Hub data47.2%
28Llama 2 7b· Epoch AI Benchmarking Hub data45.9%
29qwen2p5-coder-1p5b· Epoch AI Benchmarking Hub data45.2%
30gemma-2b· Epoch AI Benchmarking Hub data42.1%
31starcoder2-7b· Epoch AI Benchmarking Hub data38.7%
32qwen2p5-coder-0p5b· Epoch AI Benchmarking Hub data34.4%
33starcoder2-3b· Epoch AI Benchmarking Hub data34.2%
34StableLM Tuned Alpha 7B· Epoch AI Benchmarking Hub data27%

Not ranked

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
MiMo V2 Flash Baseofficial Base table · 25-shot· Reported by Xiaomi MiMo· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.95.9%

Data credits: Epoch AI Benchmarking Hub, CC BY 4.0 (licence) · BenchBoard by SYuan03 (github.com/SYuan03/benchboard). MIT covers its code and data organization; no results licence is inferred. Numeric relays retain the original stating-source links.