Skip to content

Benchmark

mu-MATH MCC

Math benchmarks measure how well models solve maths problems. Higher scores are better.

Models measured
17
Scores recorded
85
Sources
1
IQ weight
1.46

Score distribution

0.748
Mean
0.764
Median
0.095
Std dev
0.598
10th percentile
0.86
90th percentile
0.493
Min
0.4930.932
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.46 for chat models, measured over 17 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
51Qwen2.5 72B InstructGemini-1.5-Pro, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.745
52GPT-4o minimu_math, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.743
53Claude 3.5 SonnetGPT-4o, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.742
54Qwen2.5 Math 72B Instructmu_math, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.742
55Qwen2.5 72B InstructGPT-4o, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.738
56Claude 3.5 SonnetQwen2.5-72B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.734
57GPT-4o miniGemini-1.5-Pro, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.733
58Qwen2.5 7B InstructLlama-3.1-70B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.733
59Gemini 1.5 FlashQwen2.5-72B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.731
60Mistral Large Instruct 2411Qwen2.5-72B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.728
61GPT-4o miniGPT-4o, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.722
62GPT-4o miniQwen2.5-72B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.721
63Gemini 1.5 FlashGPT-4o, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.716
64Qwen2.5 72B InstructQwen2.5-72B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.705
65Qwen2.5 Math 72B InstructQwen2.5-72B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.698
66Qwen2.5 7B Instructmu_math, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.697
67Llama 3.1 70B InstructGPT-4o, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.694
68Qwen2.5 7B InstructGPT-4o, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.693
69Qwen2.5 7B InstructGemini-1.5-Pro, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.692
70Qwen2.5 Math 72B InstructGPT-4o, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.687
71Qwen2.5 Math 7B InstructLlama-3.1-70B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.684
72Qwen2.5 Math 7B InstructGemini-1.5-Pro, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.638
73Qwen2.5 7B InstructQwen2.5-72B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.628
74Qwen2.5 Math 7B Instructmu_math, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.628
75Llama 3.1 70B Instructmu_math, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.61
76Llama 3.1 70B InstructGemini-1.5-Pro, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.603
77Qwen2.5 Math 7B InstructQwen2.5-72B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.598
78Llama 3.1 70B InstructLlama-3.1-70B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.586
79Qwen2.5 Math 7B InstructGPT-4o, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.586
80Llama 3.1 8B InstructGemini-1.5-Pro, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.579
81Llama 3.1 70B InstructQwen2.5-72B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.56
82Llama 3.1 8B Instructmu_math, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.523
83Llama 3.1 8B InstructGPT-4o, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.515
84Llama 3.1 8B InstructLlama-3.1-70B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.497
85Llama 3.1 8B InstructQwen2.5-72B-Instruct, MCC, Qwen/Qwen2.5-72B-Instruct· U-MATH and mu-MATH0.493

Showing 51–85 of 85 results.

Data credits: Toloka, U-MATH and mu-MATH public leaderboard. MIT Space; generated solution outputs retain underlying model terms and are not ingested. (licence)