Skip to content

Benchmark

U-MATH overall accuracy

Math benchmarks measure how well models solve maths problems. Scores are percentages. Higher is better.

Models measured
30
Scores recorded
90
Sources
1
IQ weight
1.61

Score distribution

44.8%
Mean
43.9%
Median
22.3%
Std dev
16%
10th percentile
80.7%
90th percentile
9%
Min
9%93.1%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.61 for chat models, measured over 30 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1o1u_math, text, gpt-4o-2024-08-06· U-MATH and mu-MATH93.1%
2o3 Miniu_math, text, gpt-4o-2024-08-06· U-MATH and mu-MATH92.8%
3DeepSeek R1u_math, text, gpt-4o-2024-08-06· U-MATH and mu-MATH91.3%
4Gemini 2.0 Flash Experimentalu_math, text, gpt-4o-2024-08-06, thinking· U-MATH and mu-MATH89.2%
5o1u_math, all, gpt-4o-2024-08-06· U-MATH and mu-MATH86.8%
6Gemini 2.0 Flash Experimentalu_math, all, gpt-4o-2024-08-06, thinking· U-MATH and mu-MATH83.6%
7O1 Miniu_math, text, gpt-4o-2024-08-06· U-MATH and mu-MATH82.9%
8QwQ 32B Previewu_math, text, gpt-4o-2024-08-06· U-MATH and mu-MATH82.7%
9o3 Miniu_math, all, gpt-4o-2024-08-06· U-MATH and mu-MATH82.2%
10DeepSeek R1u_math, all, gpt-4o-2024-08-06· U-MATH and mu-MATH80.7%
11O1 Miniu_math, all, gpt-4o-2024-08-06· U-MATH and mu-MATH76.3%
12QwQ 32B Previewu_math, all, gpt-4o-2024-08-06· U-MATH and mu-MATH73.1%
13QVQ 72B Previewu_math, text, gpt-4o-2024-08-06· U-MATH and mu-MATH69.7%
14DeepSeek V3u_math, text, gpt-4o-2024-08-06· U-MATH and mu-MATH69.3%
15Qwen2.5 Math 72B Instructu_math, text, gpt-4o-2024-08-06· U-MATH and mu-MATH68.7%
16QVQ 72B Previewu_math, all, gpt-4o-2024-08-06· U-MATH and mu-MATH65%
17Gemini 1.5 Prou_math, text, gpt-4o-2024-08-06· U-MATH and mu-MATH63.4%
18DeepSeek V3u_math, all, gpt-4o-2024-08-06· U-MATH and mu-MATH62.6%
19Gemini 1.5 Flashu_math, text, gpt-4o-2024-08-06· U-MATH and mu-MATH61.2%
20Qwen2.5 32B Instructu_math, text, gpt-4o-2024-08-06· U-MATH and mu-MATH60.4%
21Gemini 1.5 Prou_math, all, gpt-4o-2024-08-06· U-MATH and mu-MATH60.1%
22Qwen2.5 Math 72B Instructu_math, all, gpt-4o-2024-08-06· U-MATH and mu-MATH59.5%
23Qwen2.5 72B Instructu_math, text, gpt-4o-2024-08-06· U-MATH and mu-MATH58.9%
24Gemini 2.0 Flash Experimentalu_math, visual, gpt-4o-2024-08-06, thinking· U-MATH and mu-MATH58.5%
25o1u_math, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH58.5%
26Phi 4u_math, text, gpt-4o-2024-08-06· U-MATH and mu-MATH58.3%
27Gemini 1.5 Flashu_math, all, gpt-4o-2024-08-06· U-MATH and mu-MATH57.8%
28Mistral Large Instruct 2411u_math, text, gpt-4o-2024-08-06· U-MATH and mu-MATH55.6%
29Phi 4u_math, all, gpt-4o-2024-08-06· U-MATH and mu-MATH54.5%
30GPT 4ou_math, text, gpt-4o-2024-08-06· U-MATH and mu-MATH53.9%
31Qwen2.5 Math 7B Instructu_math, text, gpt-4o-2024-08-06· U-MATH and mu-MATH53%
32Qwen2.5 32B Instructu_math, all, gpt-4o-2024-08-06· U-MATH and mu-MATH52.4%
33Llama 3.3 70B Instructu_math, text, gpt-4o-2024-08-06· U-MATH and mu-MATH51.7%
34Qwen2.5 72B Instructu_math, all, gpt-4o-2024-08-06· U-MATH and mu-MATH51.2%
35Qwen2.5 7B Instructu_math, text, gpt-4o-2024-08-06· U-MATH and mu-MATH50.4%
36GPT 4ou_math, all, gpt-4o-2024-08-06· U-MATH and mu-MATH50.2%
37Nvidia Llama 3.1 Nemotron 70B Instructu_math, text, gpt-4o-2024-08-06· U-MATH and mu-MATH47.7%
38Mistral Large Instruct 2411u_math, all, gpt-4o-2024-08-06· U-MATH and mu-MATH47.6%
39GPT-4o miniu_math, text, gpt-4o-2024-08-06· U-MATH and mu-MATH47.2%
40O1 Miniu_math, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH46.5%
41Qwen2.5 Math 7B Instructu_math, all, gpt-4o-2024-08-06· U-MATH and mu-MATH45.5%
42Gemini 1.5 Prou_math, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH45%
43Llama 3.3 70B Instructu_math, all, gpt-4o-2024-08-06· U-MATH and mu-MATH44.7%
44QVQ 72B Previewu_math, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH44%
45Qwen2-VL (72B) Instructu_math, text, gpt-4o-2024-08-06· U-MATH and mu-MATH43.9%
46GPT-4o miniu_math, all, gpt-4o-2024-08-06· U-MATH and mu-MATH43.4%
47Qwen2.5 7B Instructu_math, all, gpt-4o-2024-08-06· U-MATH and mu-MATH43.3%
48Nvidia Llama 3.1 Nemotron 70B Instructu_math, all, gpt-4o-2024-08-06· U-MATH and mu-MATH42.5%
49Gemini 1.5 Flashu_math, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH42.5%
50Qwen2-VL (72B) Instructu_math, all, gpt-4o-2024-08-06· U-MATH and mu-MATH41.8%

Showing 1–50 of 90 results.

Data credits: Toloka, U-MATH and mu-MATH public leaderboard. MIT Space; generated solution outputs retain underlying model terms and are not ingested. (licence)