Skip to content

Benchmark

U-MATH precalculus_review accuracy

Math benchmarks measure how well models solve maths problems. Scores are percentages. Higher is better.

Models measured
30
Scores recorded
90
Sources
1
IQ weight
1.38

Score distribution

56.7%
Mean
65.6%
Median
31.6%
Std dev
10%
10th percentile
93.1%
90th percentile
0%
Min
0%99.3%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.38 for chat models, measured over 30 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
51Claude 3.5 Sonnetprecalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH61.3%
52LFM-7bprecalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH58.7%
53LFM-7bprecalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH56.3%
54Llama 3.1 8B Instructprecalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH50.7%
55Gemini 2.0 Flash Experimentalprecalculus_review, visual, gpt-4o-2024-08-06, thinking· U-MATH and mu-MATH50%
56o1precalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH50%
57Llama 3.1 8B Instructprecalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH48.8%
58Llama 3.2 11B Vision Instructprecalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH44%
59Qwen2 VL 7b Instructprecalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH42.7%
60Llama 3.2 11B Vision Instructprecalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH41.9%
61Qwen2 VL 7b Instructprecalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH40.6%
62DeepSeek V3precalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH40%
63Pixtral 12Bprecalculus_review, text, gpt-4o-2024-08-06· U-MATH and mu-MATH32%
64Gemini 1.5 Flashprecalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH30%
65GPT 4oprecalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH30%
66O1 Miniprecalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH30%
67Phi 4precalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH30%
68Pixtral 12Bprecalculus_review, all, gpt-4o-2024-08-06· U-MATH and mu-MATH30%
69QVQ 72B Previewprecalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH30%
70Claude 3.5 Sonnetprecalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH20%
71GPT-4o miniprecalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH20%
72LFM-7bprecalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH20%
73Llama 3.1 8B Instructprecalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH20%
74Nvidia Llama 3.1 Nemotron 70B Instructprecalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH20%
75o3 Miniprecalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH20%
76QwQ 32B Previewprecalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH20%
77DeepSeek R1precalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH10%
78Llama 3.2 11B Vision Instructprecalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH10%
79Llama 3.3 70B Instructprecalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH10%
80Qwen2 VL 7b Instructprecalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH10%
81Qwen2-VL (72B) Instructprecalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH10%
82Qwen2.5 72B Instructprecalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH10%
83Qwen2.5 7B Instructprecalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH10%
84Qwen2.5 Math 72B Instructprecalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH10%
85Llama 3.1 70B Instructprecalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH0%
86Llama 3.2 90B Vision Instructprecalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH0%
87Mistral Large Instruct 2411precalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH0%
88Pixtral 12Bprecalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH0%
89Qwen2.5 32B Instructprecalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH0%
90Qwen2.5 Math 7B Instructprecalculus_review, visual, gpt-4o-2024-08-06· U-MATH and mu-MATH0%

Showing 51–90 of 90 results.

Data credits: Toloka, U-MATH and mu-MATH public leaderboard. MIT Space; generated solution outputs retain underlying model terms and are not ingested. (licence)