Skip to content

Benchmark

Omni-MATH

Math benchmarks measure how well models solve maths problems. Scores are percentages. Higher is better.

Models measured
10
Scores recorded
98
Sources
2
IQ weight
1.43

Score distribution

35.4%
Mean
30.3%
Median
18.9%
Std dev
15.7%
10th percentile
65.4%
90th percentile
8.9%
Min
8.9%82.2%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.43 for chat models, measured over 10 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
51Claude 3.5 SonnetPrecalculus· Omni-MATH29.6%
52GPT 4oApplied Mathematics· Omni-MATH29.4%
53Qwen2.5 Math 7B InstructDifficulty: 3-5· Omni-MATH29.2%
54Qwen2.5 Math 7B InstructNumber Theory· Omni-MATH28.6%
55Qwen2.5 Math 72B InstructDifficulty: 5-8· Omni-MATH27.8%
56Qwen2.5 Math 7B InstructGeometry· Omni-MATH26.9%
57Claude 3.5 SonnetApplied Mathematics· Omni-MATH26.7%
58Qwen2.5 Math 72B InstructGeometry· Omni-MATH26.6%
59Claude 3.5 Sonnet· Omni-MATH26.2%
60DeepSeek Coder V2 Instruct· Omni-MATH25.8%
61GPT 4oNumber Theory· Omni-MATH25.8%
62GPT 4oDifficulty: 3-5· Omni-MATH25%
63Qwen2.5 Math 7B InstructDifficulty: 5-8· Omni-MATH24.7%
64deepseek-coder-v2-lite-instructAlgebra· Omni-MATH24.6%
65deepseek-coder-v2-lite-instructPrecalculus· Omni-MATH23.9%
66DeepSeek Coder V2 InstructApplied Mathematics· Omni-MATH23.6%
67Qwen2.5 Math 72B InstructDifficulty: 8-10· Omni-MATH22.3%
68GPT 4oCalculus· Omni-MATH21.9%
69GPT 4oDifficulty: 5-8· Omni-MATH21.8%
70GPT 4oGeometry· Omni-MATH21.6%
71DeepSeek Coder V2 InstructNumber Theory· Omni-MATH20.9%
72Qwen2.5 Math 7B InstructDifficulty: 8-10· Omni-MATH20.3%
73deepseek-coder-v2-lite-instruct· Omni-MATH19.7%
74Claude 3.5 SonnetCalculus· Omni-MATH19.5%
75Claude 3.5 SonnetNumber Theory· Omni-MATH19.5%
76Claude 3.5 SonnetDifficulty: 3-5· Omni-MATH18.9%
77DeepSeek Coder V2 InstructDifficulty: 3-5· Omni-MATH18.8%
78Qwen2.5 Math 72B InstructDiscrete Mathematics· Omni-MATH18.3%
79Claude 3.5 SonnetDifficulty: 5-8· Omni-MATH18.3%
80DeepSeek Coder V2 InstructDifficulty: 5-8· Omni-MATH18.1%
81DeepSeek Coder V2 InstructGeometry· Omni-MATH18%
82Claude 3.5 SonnetGeometry· Omni-MATH17.7%
83Claude 3.5 SonnetDifficulty: 8-10· Omni-MATH17.4%
84Qwen2.5 Math 7B InstructDiscrete Mathematics· Omni-MATH16.9%
85deepseek-coder-v2-lite-instructApplied Mathematics· Omni-MATH16.8%
86deepseek-coder-v2-lite-instructNumber Theory· Omni-MATH15.9%
87GPT 4oDifficulty: 8-10· Omni-MATH15.8%
88Claude 3.5 SonnetDiscrete Mathematics· Omni-MATH15.7%
89GPT 4oDiscrete Mathematics· Omni-MATH15.7%
90DeepSeek Coder V2 InstructCalculus· Omni-MATH15.6%
91DeepSeek Coder V2 InstructDifficulty: 8-10· Omni-MATH14.6%
92deepseek-coder-v2-lite-instructCalculus· Omni-MATH13.3%
93deepseek-coder-v2-lite-instructDifficulty: 3-5· Omni-MATH13.2%
94deepseek-coder-v2-lite-instructGeometry· Omni-MATH13.1%
95deepseek-coder-v2-lite-instructDifficulty: 5-8· Omni-MATH12.9%
96DeepSeek Coder V2 InstructDiscrete Mathematics· Omni-MATH12.7%
97deepseek-coder-v2-lite-instructDifficulty: 8-10· Omni-MATH9.6%
98deepseek-coder-v2-lite-instructDiscrete Mathematics· Omni-MATH8.9%

Showing 51–98 of 98 results.

Data credits: Data by LLM Stats