Skip to content

Benchmark

NL2Repo-Bench

Scores are percentages. Higher is better.

Models measured
28
Scores recorded
43
Sources
1
IQ weight
1.87

Score distribution

53.6%
Mean
55.9%
Median
10.5%
Std dev
41.1%
10th percentile
68.2%
90th percentile
29.4%
Min
29.4%75.3%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.87 for chat models, measured over 28 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Claude Opus 5DeepSeek Harness· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.75.3%
2Claude Opus 5DeepSeek V4.1 launch chart · exact public value · setting not further specified · launch-page table· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.75.3%
3Claude Opus 4.8setting not named by the source· Reported by Z.ai· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.69.7%
4Claude Opus 4.8Qwen表:Claude Code · 禁止访问目标仓库的 Bash 命令· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.69.4%
5DeepSeek V4.1 Flash 1017DeepSeek 1017 first-party evaluation · reasoning effort, candidate sampling, exact run count, and unspecified harnesses not published · published main score · harness unspecified· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.68.2%
6DeepSeek V4.1 FlashDeepSeek V4.1 launch chart · exact public value · setting not further specified · launch-page table· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.65.4%
7DeepSeek V4.1 FlashDeepSeek Harness· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.64%
8DeepSeek V4 Pro 0813DeepSeek Harness· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.61.5%
9DeepSeek V4 Pro 0813DeepSeek V4.1 launch chart · exact public value · setting not further specified · launch-page table· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.61.5%
10MiniMax M3.1 FlashMiniMax M3.1 Flash official-weights model-card run · reasoning effort, sampling, and run count not stated for this benchmark · Claude Code 2.1.233 · 512K context · reward hacking = 0 · two inconsistent tasks corrected· Reported by MiniMax· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.61.2%
11DeepSeek V4 Pro 0813setting not named by the source· Reported by Z.ai· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.61.1%
12Hy4 previewofficial model-card appendix· Reported by Tencent Hy· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.58.9%
13GLM 5.3setting not named by the source· Reported by Z.ai· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.58%
14GLM 5.3DeepSeek Harness· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.58%
15GLM 5.3DeepSeek V4.1 launch chart · exact public value · setting not further specified · launch-page table· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.58%
16Kimi K3setting not named by the source· Reported by Z.ai· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.58%
17Kimi K3DeepSeek Harness· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.58%
18Kimi K3DeepSeek V4.1 launch chart · exact public value · setting not further specified · launch-page table· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.58%
19GPT-5.6 SolDeepSeek Harness· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.56.8%
20GPT-5.6 SolDeepSeek V4.1 launch chart · exact public value · setting not further specified · launch-page table· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.56.8%
21GLM 5.3 FlashZ.ai Flash设置· Reported by Z.ai· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.56.3%
22Qwen3.8 Maxsetting not named by the source· Reported by Z.ai· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.55.9%
23Qwen3.8 MaxQwen表:Claude Code · 禁止访问目标仓库的 Bash 命令· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.55.9%
24DeepSeek V4 Flash 0731Claude Code · 禁止访问目标仓库的 Bash 命令· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.54.2%
25DeepSeek V4 Flash 0731DeepSeek V4.1 launch chart · exact public value · setting not further specified · launch-page table· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.54.2%
26GLM 5.2400K context· Reported by Z.ai· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.48.9%
27Qwen3.8 Omni FlashClaude Code · 禁止访问目标仓库的 Bash 命令· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.48.9%
28Qwen 3.8 Flash NextQwen3.8-Flash-Next release/model card · Claude Code · repository-download commands disabled· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.48.1%
29Qwen3.8 FlashClaude Code · 禁止访问目标仓库的 Bash 命令· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.48.1%
30Claude Opus 4.6Claude Code · 禁止访问目标仓库的 Bash 命令 · max effort· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.47.6%
31Qwen3.7 MaxQwen3.7-Max release · Claude Code · repository-download commands disabled· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.47.2%
32Qwen3.7 MaxQwen表:Claude Code · 禁止访问目标仓库的 Bash 命令· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.47.2%
33Hy3 (Deep Infra)Hy3 official Benchmark Appendix · highest reasoning tier unless noted · Claude Code · 250 turns · 12,000s· Reported by Tencent Hunyuan· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.45.6%
34GLM 5.1GLM-5.1 official release table· Reported by Z.ai· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.42.7%
35Qwen 3.8 27BClaude Code · 禁止访问目标仓库的 Bash 命令· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.42.3%
36Qwen 3.8 27BQwen3.8-27B model card · published setting· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.42.3%
37MiniMax M3Claude Code · restricted sandbox· Reported by MiniMax Research· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.42.1%
38Qwen3.7 PlusClaude Code · 禁止访问目标仓库的 Bash 命令· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.41.1%
39Qwen3.7 PlusQwen3.7-Plus release · published setting· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.41.1%
40MiniMax M2.7MiniMax M2.7 official release prose· Reported by MiniMax· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.39.8%
41Qwen3.6 PlusQwen official release table · thinking mode unless benchmark requires otherwise · Claude Code · temp 1.0 · top_p .95 · max_turns 900· Reported by Alibaba Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.37.9%
42Qwen3.6 27BQwen official release table · thinking mode unless benchmark requires otherwise · Claude Code · temp 1.0 · top_p .95 · max_turns 900· Reported by Alibaba Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.36.2%
43Qwen3.6 FlashQwen3.6-35B-A3B official model card · hosted API name Qwen3.6-Flash· Reported by Alibaba Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.29.4%

Data credits: BenchBoard by SYuan03 (github.com/SYuan03/benchboard). MIT covers its code and data organization; no results licence is inferred. Numeric relays retain the original stating-source links.