Skip to content

Benchmark

U-MATH multivariable_calculus accuracy

Math benchmarks measure how well models solve maths problems. Scores are percentages. Higher is better.

Models measured
30
Scores recorded
90
Sources
1
IQ weight
1.46

Score distribution

41.2%
Mean
40.7%
Median
24%
Std dev
10.7%
10th percentile
78.7%
90th percentile
0%
Min
0%92%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.46 for chat models, measured over 30 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1o1multivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH92%
2o1multivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH87.1%
3DeepSeek R1multivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH86.7%
4Gemini 2.0 Flash Experimentalmultivariable_calculus, text, gpt-4o-2024-08-06, thinking· U-MATH and mu-MATH85.3%
5o3 Minimultivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH85.3%
6Gemini 2.0 Flash Experimentalmultivariable_calculus, all, gpt-4o-2024-08-06, thinking· U-MATH and mu-MATH83.7%
7QwQ 32B Previewmultivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH80.7%
8DeepSeek R1multivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH79.8%
9o3 Minimultivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH79.8%
10O1 Minimultivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH78.7%
11Gemini 2.0 Flash Experimentalmultivariable_calculus, visual, gpt-4o-2024-08-06, thinking· U-MATH and mu-MATH75%
12O1 Minimultivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH73%
13QwQ 32B Previewmultivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH73%
14DeepSeek V3multivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH69.3%
15Qwen2.5 Math 72B Instructmultivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH69.3%
16QVQ 72B Previewmultivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH65.3%
17DeepSeek V3multivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH65.2%
18QVQ 72B Previewmultivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH62.9%
19Qwen2.5 Math 72B Instructmultivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH61.8%
20o1multivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH60.7%
21Gemini 1.5 Promultivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH60.7%
22Gemini 1.5 Promultivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH60.1%
23Gemini 1.5 Promultivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH57.1%
24Gemini 1.5 Flashmultivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH55.3%
25Gemini 1.5 Flashmultivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH55.1%
26Phi 4multivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH54.7%
27Gemini 1.5 Flashmultivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH53.6%
28Qwen2.5 72B Instructmultivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH52%
29Phi 4multivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH50.6%
30o3 Minimultivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH50%
31QVQ 72B Previewmultivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH50%
32Qwen2.5 32B Instructmultivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH50%
33GPT 4omultivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH49.3%
34Llama 3.3 70B Instructmultivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH48.7%
35GPT 4omultivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH48.3%
36Qwen2.5 72B Instructmultivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH46.1%
37Mistral Large Instruct 2411multivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH45.3%
38Qwen2.5 32B Instructmultivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH44.9%
39Qwen2.5 Math 7B Instructmultivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH44%
40DeepSeek R1multivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH42.9%
41DeepSeek V3multivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH42.9%
42GPT 4omultivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH42.9%
43O1 Minimultivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH42.9%
44Llama 3.3 70B Instructmultivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH42.7%
45Nvidia Llama 3.1 Nemotron 70B Instructmultivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH40.7%
46Mistral Large Instruct 2411multivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH40.4%
47Claude 3.5 Sonnetmultivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH39.3%
48GPT-4o minimultivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH39.3%
49Qwen2.5 Math 7B Instructmultivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH38.8%
50GPT-4o minimultivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH37.6%

Showing 1–50 of 90 results.

Data credits: Toloka, U-MATH and mu-MATH public leaderboard. MIT Space; generated solution outputs retain underlying model terms and are not ingested. (licence)