Skip to content

Benchmark

mu-MATH MCC

Math benchmarks measure how well models solve maths problems. Higher scores are better.

Models measured
17
Scores recorded
85
Sources
1
IQ weight
1.46

Score distribution

0.748
Mean
0.764
Median
0.095
Std dev
0.598
10th percentile
0.86
90th percentile
0.493
Min
0.4930.932
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.46 for chat models, measured over 17 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1o1Llama-3.1-70B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.932
2o1Gemini-1.5-Pro, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.907
3O1 MiniLlama-3.1-70B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.897
4o1mu_math, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.895
5o1GPT-4o, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.884
6DeepSeek R1Llama-3.1-70B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.882
7QwQ 32B PreviewQwen2.5-72B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.868
8O1 MiniGemini-1.5-Pro, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.862
9Gemini 2.0 Flash ExperimentalGemini-1.5-Pro, MCC, Qwen/Qwen2.5-72B-Instruct, thinking· U-MATH and mu-MATH0.86
10O1 Minimu_math, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.848
11QwQ 32B PreviewLlama-3.1-70B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.848
12o1Qwen2.5-72B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.839
13QwQ 32B Previewmu_math, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.838
14Gemini 1.5 ProLlama-3.1-70B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.837
15Gemini 2.0 Flash ExperimentalLlama-3.1-70B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct, thinking· U-MATH and mu-MATH0.834
16DeepSeek R1mu_math, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.824
17DeepSeek R1Gemini-1.5-Pro, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.823
18GPT 4oLlama-3.1-70B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.82
19Gemini 2.0 Flash Experimentalmu_math, MCC, Qwen/Qwen2.5-72B-Instruct, thinking· U-MATH and mu-MATH0.816
20QwQ 32B PreviewGemini-1.5-Pro, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.814
21O1 MiniGPT-4o, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.813
22Gemini 1.5 FlashLlama-3.1-70B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.811
23Gemini 1.5 Promu_math, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.809
24DeepSeek R1GPT-4o, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.803
25Gemini 1.5 ProGemini-1.5-Pro, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.802
26O1 MiniQwen2.5-72B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.795
27Qwen2.5 72B InstructLlama-3.1-70B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.793
28Claude 3.5 SonnetLlama-3.1-70B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.79
29Mistral Large Instruct 2411Llama-3.1-70B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.786
30QwQ 32B PreviewGPT-4o, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.784
31Gemini 1.5 ProGPT-4o, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.783
32GPT 4omu_math, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.781
33Gemini 1.5 ProQwen2.5-72B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.78
34Qwen2.5 Math 72B InstructLlama-3.1-70B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.779
35GPT 4oGPT-4o, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.778
36DeepSeek R1Qwen2.5-72B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.774
37Gemini 1.5 FlashGemini-1.5-Pro, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.772
38Qwen2.5 Math 72B InstructGemini-1.5-Pro, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.77
39GPT-4o miniLlama-3.1-70B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.768
40Mistral Large Instruct 2411mu_math, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.767
41Claude 3.5 Sonnetmu_math, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.766
42Gemini 2.0 Flash ExperimentalQwen2.5-72B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct, thinking· U-MATH and mu-MATH0.766
43Gemini 2.0 Flash ExperimentalGPT-4o, MCC, Qwen/Qwen2.5-72B-Instruct, thinking· U-MATH and mu-MATH0.764
44Gemini 1.5 Flashmu_math, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.763
45Mistral Large Instruct 2411GPT-4o, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.76
46Claude 3.5 SonnetGemini-1.5-Pro, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.759
47Qwen2.5 72B Instructmu_math, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.756
48Mistral Large Instruct 2411Gemini-1.5-Pro, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.754
49GPT 4oQwen2.5-72B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.752
50GPT 4oGemini-1.5-Pro, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.746

Showing 1–50 of 85 results.

Data credits: Toloka, U-MATH and mu-MATH public leaderboard. MIT Space; generated solution outputs retain underlying model terms and are not ingested. (licence)