Skip to content

Benchmark

U-MATH multivariable_calculus accuracy

Math benchmarks measure how well models solve maths problems. Scores are percentages. Higher is better.

Models measured
30
Scores recorded
90
Sources
1
IQ weight
1.46

Score distribution

41.2%
Mean
40.7%
Median
24%
Std dev
10.7%
10th percentile
78.7%
90th percentile
0%
Min
0%92%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.46 for chat models, measured over 30 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
51GPT-4o minimultivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH37.3%
52Qwen2.5 7B Instructmultivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH36.7%
53Nvidia Llama 3.1 Nemotron 70B Instructmultivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH36.5%
54Claude 3.5 Sonnetmultivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH34.3%
55Claude 3.5 Sonnetmultivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH33.3%
56QwQ 32B Previewmultivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH32.1%
57Qwen2-VL (72B) Instructmultivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH32%
58Qwen2-VL (72B) Instructmultivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH31.5%
59Qwen2.5 7B Instructmultivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH31.5%
60Llama 3.2 90B Vision Instructmultivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH30%
61Phi 4multivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH28.6%
62Qwen2-VL (72B) Instructmultivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH28.6%
63Llama 3.2 90B Vision Instructmultivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH27%
64Llama 3.1 70B Instructmultivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH26%
65Llama 3.1 8B Instructmultivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH23.3%
66Llama 3.1 70B Instructmultivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH23%
67Qwen2.5 Math 72B Instructmultivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH21.4%
68Llama 3.1 8B Instructmultivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH20.2%
69Qwen2 VL 7b Instructmultivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH17.9%
70Qwen2.5 32B Instructmultivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH17.9%
71Qwen2 VL 7b Instructmultivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH14.6%
72Mistral Large Instruct 2411multivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH14.3%
73Nvidia Llama 3.1 Nemotron 70B Instructmultivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH14.3%
74Qwen2.5 72B Instructmultivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH14.3%
75Qwen2 VL 7b Instructmultivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH14%
76LFM-7bmultivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH13.3%
77Llama 3.2 11B Vision Instructmultivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH13.3%
78LFM-7bmultivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH12.9%
79Llama 3.2 11B Vision Instructmultivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH11.2%
80LFM-7bmultivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH10.7%
81Llama 3.2 90B Vision Instructmultivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH10.7%
82Llama 3.3 70B Instructmultivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH10.7%
83Qwen2.5 Math 7B Instructmultivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH10.7%
84Llama 3.1 70B Instructmultivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH7.1%
85Pixtral 12Bmultivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH7.1%
86Pixtral 12Bmultivariable_calculus, all, gpt-4o-2024-08-06· U-MATH and mu-MATH6.7%
87Pixtral 12Bmultivariable_calculus, text, gpt-4o-2024-08-06· U-MATH and mu-MATH6.7%
88Llama 3.1 8B Instructmultivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH3.6%
89Qwen2.5 7B Instructmultivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH3.6%
90Llama 3.2 11B Vision Instructmultivariable_calculus, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH0%

Showing 51–90 of 90 results.

Data credits: Toloka, U-MATH and mu-MATH public leaderboard. MIT Space; generated solution outputs retain underlying model terms and are not ingested. (licence)