Skip to content

Benchmark

U-MATH precalculus_review accuracy

Math benchmarks measure how well models solve maths problems. Scores are percentages. Higher is better.

Models measured
30
Scores recorded
90
Sources
1
IQ weight
1.38

Score distribution

56.7%
Mean
65.6%
Median
31.6%
Std dev
10%
10th percentile
93.1%
90th percentile
0%
Min
0%99.3%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.38 for chat models, measured over 30 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1o1precalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH99.3%
2o3 Miniprecalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH99.3%
3DeepSeek R1precalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH98.7%
4QwQ 32B Previewprecalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH97.3%
5O1 Miniprecalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH96.7%
6o1precalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH96.3%
7Gemini 2.0 Flash Experimentalprecalculus_review, text, gpt-4o-2024-08-06, thinking· U-MATH and mu-MATH95.3%
8QVQ 72B Previewprecalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH95.3%
9o3 Miniprecalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH94.4%
10DeepSeek R1precalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH93.1%
11Gemini 2.0 Flash Experimentalprecalculus_review, all, gpt-4o-2024-08-06, thinking· U-MATH and mu-MATH92.5%
12O1 Miniprecalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH92.5%
13QwQ 32B Previewprecalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH92.5%
14QVQ 72B Previewprecalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH91.3%
15DeepSeek V3precalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH90%
16Qwen2.5 Math 72B Instructprecalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH89.3%
17Phi 4precalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH88%
18Gemini 1.5 Proprecalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH87.3%
19DeepSeek V3precalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH86.9%
20Gemini 1.5 Proprecalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH86.3%
21Qwen2.5 32B Instructprecalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH85.3%
22Phi 4precalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH84.4%
23Qwen2.5 Math 72B Instructprecalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH84.4%
24Qwen2.5 72B Instructprecalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH84%
25Gemini 1.5 Flashprecalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH82.7%
26Qwen2.5 Math 7B Instructprecalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH81.3%
27GPT 4oprecalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH80%
28Qwen2.5 32B Instructprecalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH80%
29Gemini 1.5 Flashprecalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH79.4%
30Qwen2.5 72B Instructprecalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH79.4%
31Qwen2.5 7B Instructprecalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH78.7%
32Mistral Large Instruct 2411precalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH78%
33GPT 4oprecalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH76.9%
34Qwen2.5 Math 7B Instructprecalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH76.3%
35GPT-4o miniprecalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH76%
36Qwen2.5 7B Instructprecalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH74.4%
37Mistral Large Instruct 2411precalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH73.1%
38GPT-4o miniprecalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH72.5%
39Gemini 1.5 Proprecalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH70%
40Llama 3.2 90B Vision Instructprecalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH70%
41Llama 3.3 70B Instructprecalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH68.7%
42Llama 3.1 70B Instructprecalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH67.3%
43Nvidia Llama 3.1 Nemotron 70B Instructprecalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH67.3%
44Qwen2-VL (72B) Instructprecalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH66%
45Llama 3.2 90B Vision Instructprecalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH65.6%
46Llama 3.3 70B Instructprecalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH65%
47Nvidia Llama 3.1 Nemotron 70B Instructprecalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH64.4%
48Claude 3.5 Sonnetprecalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH64%
49Llama 3.1 70B Instructprecalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH63.1%
50Qwen2-VL (72B) Instructprecalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH62.5%

Showing 1–50 of 90 results.

Data credits: Toloka, U-MATH and mu-MATH public leaderboard. MIT Space; generated solution outputs retain underlying model terms and are not ingested. (licence)