Skip to content

Benchmark

U-MATH integral_calc accuracy

Math benchmarks measure how well models solve maths problems. Scores are percentages. Higher is better.

Models measured
30
Scores recorded
90
Sources
1
IQ weight
2.00

Score distribution

32.3%
Mean
27.3%
Median
22.9%
Std dev
9.6%
10th percentile
67.3%
90th percentile
1.3%
Min
1.3%90.7%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 2.00 for chat models, measured over 30 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1o1integral_calc, text, gpt-4o-2024-08-06· U-MATH and mu-MATH90.7%
2o3 Miniintegral_calc, text, gpt-4o-2024-08-06· U-MATH and mu-MATH90.7%
3Gemini 2.0 Flash Experimentalintegral_calc, text, gpt-4o-2024-08-06, thinking· U-MATH and mu-MATH88.7%
4DeepSeek R1integral_calc, text, gpt-4o-2024-08-06· U-MATH and mu-MATH87.3%
5o1integral_calc, all, gpt-4o-2024-08-06· U-MATH and mu-MATH83.2%
6Gemini 2.0 Flash Experimentalintegral_calc, all, gpt-4o-2024-08-06, thinking· U-MATH and mu-MATH82.2%
7o3 Miniintegral_calc, all, gpt-4o-2024-08-06· U-MATH and mu-MATH82.2%
8DeepSeek R1integral_calc, all, gpt-4o-2024-08-06· U-MATH and mu-MATH76.9%
9O1 Miniintegral_calc, text, gpt-4o-2024-08-06· U-MATH and mu-MATH72%
10QwQ 32B Previewintegral_calc, text, gpt-4o-2024-08-06· U-MATH and mu-MATH67.3%
11Gemini 2.0 Flash Experimentalintegral_calc, visual, gpt-4o-2024-08-06, thinking· U-MATH and mu-MATH65.5%
12O1 Miniintegral_calc, all, gpt-4o-2024-08-06· U-MATH and mu-MATH64.9%
13o1integral_calc, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH63.8%
14QwQ 32B Previewintegral_calc, all, gpt-4o-2024-08-06· U-MATH and mu-MATH62.5%
15o3 Miniintegral_calc, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH60.3%
16QVQ 72B Previewintegral_calc, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH55.2%
17DeepSeek R1integral_calc, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH50%
18QwQ 32B Previewintegral_calc, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH50%
19O1 Miniintegral_calc, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH46.6%
20QVQ 72B Previewintegral_calc, all, gpt-4o-2024-08-06· U-MATH and mu-MATH45.2%
21Qwen2.5 Math 72B Instructintegral_calc, text, gpt-4o-2024-08-06· U-MATH and mu-MATH44%
22QVQ 72B Previewintegral_calc, text, gpt-4o-2024-08-06· U-MATH and mu-MATH41.3%
23DeepSeek V3integral_calc, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH39.7%
24DeepSeek V3integral_calc, all, gpt-4o-2024-08-06· U-MATH and mu-MATH38.9%
25Qwen2.5 Math 72B Instructintegral_calc, all, gpt-4o-2024-08-06· U-MATH and mu-MATH38.9%
26DeepSeek V3integral_calc, text, gpt-4o-2024-08-06· U-MATH and mu-MATH38.7%
27Mistral Large Instruct 2411integral_calc, text, gpt-4o-2024-08-06· U-MATH and mu-MATH36.7%
28Phi 4integral_calc, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH36.2%
29Qwen2.5 72B Instructintegral_calc, text, gpt-4o-2024-08-06· U-MATH and mu-MATH35.3%
30Qwen2.5 32B Instructintegral_calc, text, gpt-4o-2024-08-06· U-MATH and mu-MATH34.7%
31Qwen2.5 32B Instructintegral_calc, all, gpt-4o-2024-08-06· U-MATH and mu-MATH32.2%
32Qwen2.5 7B Instructintegral_calc, text, gpt-4o-2024-08-06· U-MATH and mu-MATH32%
33Gemini 1.5 Flashintegral_calc, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH31%
34Gemini 1.5 Flashintegral_calc, all, gpt-4o-2024-08-06· U-MATH and mu-MATH30.8%
35Mistral Large Instruct 2411integral_calc, all, gpt-4o-2024-08-06· U-MATH and mu-MATH30.8%
36Gemini 1.5 Flashintegral_calc, text, gpt-4o-2024-08-06· U-MATH and mu-MATH30.7%
37Qwen2.5 72B Instructintegral_calc, all, gpt-4o-2024-08-06· U-MATH and mu-MATH30.3%
38Phi 4integral_calc, all, gpt-4o-2024-08-06· U-MATH and mu-MATH28.4%
39Qwen2.5 7B Instructintegral_calc, all, gpt-4o-2024-08-06· U-MATH and mu-MATH28.4%
40GPT 4ointegral_calc, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH27.6%
41Qwen2-VL (72B) Instructintegral_calc, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH27.6%
42GPT 4ointegral_calc, all, gpt-4o-2024-08-06· U-MATH and mu-MATH27.4%
43GPT 4ointegral_calc, text, gpt-4o-2024-08-06· U-MATH and mu-MATH27.3%
44Llama 3.3 70B Instructintegral_calc, text, gpt-4o-2024-08-06· U-MATH and mu-MATH27.3%
45Gemini 1.5 Prointegral_calc, text, gpt-4o-2024-08-06· U-MATH and mu-MATH27.3%
46Gemini 1.5 Prointegral_calc, all, gpt-4o-2024-08-06· U-MATH and mu-MATH26.4%
47Qwen2.5 32B Instructintegral_calc, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH25.9%
48Qwen2.5 Math 72B Instructintegral_calc, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH25.9%
49Llama 3.3 70B Instructintegral_calc, all, gpt-4o-2024-08-06· U-MATH and mu-MATH25.5%
50Phi 4integral_calc, text, gpt-4o-2024-08-06· U-MATH and mu-MATH25.3%

Showing 1–50 of 90 results.

Data credits: Toloka, U-MATH and mu-MATH public leaderboard. MIT Space; generated solution outputs retain underlying model terms and are not ingested. (licence)