Skip to content

Benchmark

ChemBench Chemical Preference

Science benchmarks measure how well models answer science questions. Higher scores are better.

Models measured
42
Scores recorded
51
Sources
1
IQ weight
1.46

Score distribution

0.505
Mean
0.535
Median
0.153
Std dev
0.421
10th percentile
0.611
90th percentile
0.003
Min
0.0030.659
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.46 for chat models, measured over 42 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Claude Opus 4.7claude-opus-4-7-tinyeval· ChemBench verified leaderboard data docs0.659
2Claude Opus 4.5claude-opus-4-5-tinyeval· ChemBench verified leaderboard data docs0.642
3Claude Opus 4.8claude-opus-4-8-tinyeval· ChemBench verified leaderboard data docs0.639
4GLM 5glm-5-tinyeval· ChemBench verified leaderboard data docs0.624
5Kimi K2.5kimi-k2-5-tinyeval· ChemBench verified leaderboard data docs0.614
6Qwen3 Next 80B A3Bqwen3-next-80b-a3b-tinyeval· ChemBench verified leaderboard data docs0.611
7Claude 3.5 Sonnetclaude3.5-react· ChemBench verified leaderboard data docs0.599
8GPT OSS 120Bgpt-oss-120b-base-1760293341· ChemBench verified leaderboard data docs0.595
9Llama 4 Maverickgroq_meta-llama/llama-4-maverick-17b-128e-instruct_e8a077d8· ChemBench verified leaderboard data docs0.594
10gemini-2.5-flash-preview-04-17gemini-2.5-flash-preview-04-17-base-1750491557· ChemBench verified leaderboard data docs0.593
11Qwen3 235B A22B Instruct 2507qwen3-235b-a22b-2507-tinyeval· ChemBench verified leaderboard data docs0.592
12GPT 4ogpt-4o· ChemBench verified leaderboard data docs0.589
13GPT OSS 20bgpt-oss-20b-base-1760359009· ChemBench verified leaderboard data docs0.588
14DeepSeek V3.2deepseek-v3-2-tinyeval· ChemBench verified leaderboard data docs0.585
15Claude 3.5 Sonnetclaude3.5· ChemBench verified leaderboard data docs0.584
16GPT-5 nanoGPT-5-nano-base-1764459351· ChemBench verified leaderboard data docs0.569
17O1 Previewo1-preview· ChemBench verified leaderboard data docs0.563
18DeepSeek R1deepseek-r1-tinyeval· ChemBench verified leaderboard data docs0.563
19Gemma 2 9B ITgemma-2-9b-it-T-one· ChemBench verified leaderboard data docs0.557
20Gemma 2 9B ITgemma-2-9b-it· ChemBench verified leaderboard data docs0.551
21Mistral Large 2mistral-large-2-123b· ChemBench verified leaderboard data docs0.546
22Claude 3 Haikuclaude-3-haiku-tinyeval· ChemBench verified leaderboard data docs0.544
23Llama 3.1 405B Instructllama3.1-405b-instruct· ChemBench verified leaderboard data docs0.54
24Nemotron Super 3 120bnemotron-super-3-120b-tinyeval· ChemBench verified leaderboard data docs0.537
25Llama 3.1 70B Instructllama3.1-70b-instruct-T-one· ChemBench verified leaderboard data docs0.535
26Mixtral 8x7B Instructmixtral-8x7b-instruct· ChemBench verified leaderboard data docs0.535
27GPT-3.5 Turbogpt-3.5-turbo· ChemBench verified leaderboard data docs0.534
28Llama 3 70B Instructllama3-70b-instruct· ChemBench verified leaderboard data docs0.532
29Phi-3 Medium 4K Instructphi-3-medium-4k-instruct· ChemBench verified leaderboard data docs0.532
30Llama 3 70B Instructllama3-70b-instruct-T-one· ChemBench verified leaderboard data docs0.53
31Qwen2.5 32BQwen-2.5-32B-base-1740920072· ChemBench verified leaderboard data docs0.529
32Llama 3.1 8B Instructllama3.1-8b-instruct· ChemBench verified leaderboard data docs0.527
33Nemotron Nano 3 30Bnemotron-nano-3-30b-tinyeval· ChemBench verified leaderboard data docs0.526
34Llama 3.1 8B Instructllama3.1-8b-instruct-T-one· ChemBench verified leaderboard data docs0.523
35Mixtral 8x7B Instructmixtral-8x7b-instruct-T-one· ChemBench verified leaderboard data docs0.522
36Llama 3 8B Instructllama3-8b-instruct-T-one· ChemBench verified leaderboard data docs0.52
37Llama 3.1 70B Instructllama3.1-70b-instruct· ChemBench verified leaderboard data docs0.519
38Kimi K2 Instructkimi-k2-instruct-0905-base-1760364223· ChemBench verified leaderboard data docs0.518
39Llama 3 8B Instructllama3-8b-instruct· ChemBench verified leaderboard data docs0.515
40Command R Pluscommand-r+· ChemBench verified leaderboard data docs0.513
41Claude 2claude2· ChemBench verified leaderboard data docs0.511
42GPT-4o miniGPT-4o-mini-base-1764283510· ChemBench verified leaderboard data docs0.51
43Gemini Progemini-pro· ChemBench verified leaderboard data docs0.5
44Llama 2 70B Chatllama2-70b-chat· ChemBench verified leaderboard data docs0.488
45Llama 2 13B Chatllama2-13b-chat· ChemBench verified leaderboard data docs0.485
46GPT 4ogpt-4o-react· ChemBench verified leaderboard data docs0.421
47GPT 4gpt-4· ChemBench verified leaderboard data docs0.164
48MiniMax M2.5minimax-m2-5-tinyeval· ChemBench verified leaderboard data docs0.145
49Gemma 1.1 7B ITgemma-1-1-7b-it-T-one· ChemBench verified leaderboard data docs0.00899
50Gemma 1.1 7B ITgemma-1-1-7b-it· ChemBench verified leaderboard data docs0.005

Showing 1–50 of 51 results.

Data credits: ChemBench leaderboard, Jablonka group / lamalab-org. Public verified numeric result summaries; results licence not separately stated.