Skip to content

Benchmark

ChemBench

Science benchmarks measure how well models answer science questions. Higher scores are better.

Models measured
42
Scores recorded
51
Sources
1
IQ weight
1.45

Score distribution

0.512
Mean
0.516
Median
0.135
Std dev
0.413
10th percentile
0.645
90th percentile
0.0402
Min
0.04020.709
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.45 for chat models, measured over 42 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Claude Opus 4.7claude-opus-4-7-tinyeval· ChemBench verified leaderboard data docs0.709
2Claude Opus 4.8claude-opus-4-8-tinyeval· ChemBench verified leaderboard data docs0.673
3Kimi K2.5kimi-k2-5-tinyeval· ChemBench verified leaderboard data docs0.672
4Claude Opus 4.5claude-opus-4-5-tinyeval· ChemBench verified leaderboard data docs0.655
5DeepSeek V3.2deepseek-v3-2-tinyeval· ChemBench verified leaderboard data docs0.653
6Llama 4 Maverickgroq_meta-llama/llama-4-maverick-17b-128e-instruct_e8a077d8· ChemBench verified leaderboard data docs0.645
7O1 Previewo1-preview· ChemBench verified leaderboard data docs0.643
8gemini-2.5-flash-preview-04-17gemini-2.5-flash-preview-04-17-base-1750491557· ChemBench verified leaderboard data docs0.641
9DeepSeek R1deepseek-r1-tinyeval· ChemBench verified leaderboard data docs0.637
10GPT-5 nanoGPT-5-nano-base-1764459351· ChemBench verified leaderboard data docs0.634
11GPT OSS 120Bgpt-oss-120b-base-1760293341· ChemBench verified leaderboard data docs0.631
12Claude 3.5 Sonnetclaude3.5· ChemBench verified leaderboard data docs0.626
13Claude 3.5 Sonnetclaude3.5-react· ChemBench verified leaderboard data docs0.625
14GPT OSS 20bgpt-oss-20b-base-1760359009· ChemBench verified leaderboard data docs0.612
15GPT 4ogpt-4o· ChemBench verified leaderboard data docs0.611
16Qwen3 235B A22B Instruct 2507qwen3-235b-a22b-2507-tinyeval· ChemBench verified leaderboard data docs0.611
17Qwen3 Next 80B A3Bqwen3-next-80b-a3b-tinyeval· ChemBench verified leaderboard data docs0.61
18Kimi K2 Instructkimi-k2-instruct-0905-base-1760364223· ChemBench verified leaderboard data docs0.597
19GLM 5glm-5-tinyeval· ChemBench verified leaderboard data docs0.593
20Llama 3.1 405B Instructllama3.1-405b-instruct· ChemBench verified leaderboard data docs0.579
21Mistral Large 2mistral-large-2-123b· ChemBench verified leaderboard data docs0.57
22Nemotron Super 3 120bnemotron-super-3-120b-tinyeval· ChemBench verified leaderboard data docs0.553
23Llama 3.1 70B Instructllama3.1-70b-instruct· ChemBench verified leaderboard data docs0.534
24Qwen2.5 32BQwen-2.5-32B-base-1740920072· ChemBench verified leaderboard data docs0.529
25Llama 3 70B Instructllama3-70b-instruct· ChemBench verified leaderboard data docs0.518
26Llama 3 70B Instructllama3-70b-instruct-T-one· ChemBench verified leaderboard data docs0.516
27Llama 3.1 70B Instructllama3.1-70b-instruct-T-one· ChemBench verified leaderboard data docs0.511
28GPT 4ogpt-4o-react· ChemBench verified leaderboard data docs0.505
29GPT-4o miniGPT-4o-mini-base-1764283510· ChemBench verified leaderboard data docs0.503
30Claude 3 Haikuclaude-3-haiku-tinyeval· ChemBench verified leaderboard data docs0.501
31Gemma 2 9B ITgemma-2-9b-it· ChemBench verified leaderboard data docs0.482
32Gemma 2 9B ITgemma-2-9b-it-T-one· ChemBench verified leaderboard data docs0.48
33Phi-3 Medium 4K Instructphi-3-medium-4k-instruct· ChemBench verified leaderboard data docs0.475
34Claude 2claude2· ChemBench verified leaderboard data docs0.473
35Nemotron Nano 3 30Bnemotron-nano-3-30b-tinyeval· ChemBench verified leaderboard data docs0.472
36Llama 3.1 8B Instructllama3.1-8b-instruct· ChemBench verified leaderboard data docs0.472
37GPT-3.5 Turbogpt-3.5-turbo· ChemBench verified leaderboard data docs0.466
38Llama 3.1 8B Instructllama3.1-8b-instruct-T-one· ChemBench verified leaderboard data docs0.461
39Llama 3 8B Instructllama3-8b-instruct-T-one· ChemBench verified leaderboard data docs0.457
40Llama 3 8B Instructllama3-8b-instruct· ChemBench verified leaderboard data docs0.456
41Gemini Progemini-pro· ChemBench verified leaderboard data docs0.453
42Command R Pluscommand-r+· ChemBench verified leaderboard data docs0.448
43MiniMax M2.5minimax-m2-5-tinyeval· ChemBench verified leaderboard data docs0.434
44Mixtral 8x7B Instructmixtral-8x7b-instruct· ChemBench verified leaderboard data docs0.424
45Mixtral 8x7B Instructmixtral-8x7b-instruct-T-one· ChemBench verified leaderboard data docs0.418
46GPT 4gpt-4· ChemBench verified leaderboard data docs0.413
47Llama 2 70B Chatllama2-70b-chat· ChemBench verified leaderboard data docs0.266
48Llama 2 13B Chatllama2-13b-chat· ChemBench verified leaderboard data docs0.255
49Gemma 1.1 7B ITgemma-1-1-7b-it· ChemBench verified leaderboard data docs0.192
50Gemma 1.1 7B ITgemma-1-1-7b-it-T-one· ChemBench verified leaderboard data docs0.191

Showing 1–50 of 51 results.

Data credits: ChemBench leaderboard, Jablonka group / lamalab-org. Public verified numeric result summaries; results licence not separately stated.