Benchmark
NL2Repo-Bench
Scores are percentages. Higher is better.
Benchmark sourceScores from BenchBoard docs.
- Models measured
- 28
- Scores recorded
- 43
- Sources
- 1
- IQ weight
- 1.87
Score distribution
- 53.6%
- Mean
- 55.9%
- Median
- 10.5%
- Std dev
- 41.1%
- 10th percentile
- 68.2%
- 90th percentile
- 29.4%
- Min
This benchmark has a weight of 1.87 for chat models, measured over 28 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.
Leaderboard
| # | Model | Score |
|---|---|---|
| 1 | Claude Opus 5DeepSeek Harness· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 75.3% |
| 2 | Claude Opus 5DeepSeek V4.1 launch chart · exact public value · setting not further specified · launch-page table· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 75.3% |
| 3 | Claude Opus 4.8setting not named by the source· Reported by Z.ai· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 69.7% |
| 4 | Claude Opus 4.8Qwen表:Claude Code · 禁止访问目标仓库的 Bash 命令· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 69.4% |
| 5 | DeepSeek V4.1 Flash 1017DeepSeek 1017 first-party evaluation · reasoning effort, candidate sampling, exact run count, and unspecified harnesses not published · published main score · harness unspecified· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 68.2% |
| 6 | DeepSeek V4.1 FlashDeepSeek V4.1 launch chart · exact public value · setting not further specified · launch-page table· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 65.4% |
| 7 | DeepSeek V4.1 FlashDeepSeek Harness· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 64% |
| 8 | DeepSeek V4 Pro 0813DeepSeek Harness· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 61.5% |
| 9 | DeepSeek V4 Pro 0813DeepSeek V4.1 launch chart · exact public value · setting not further specified · launch-page table· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 61.5% |
| 10 | MiniMax M3.1 FlashMiniMax M3.1 Flash official-weights model-card run · reasoning effort, sampling, and run count not stated for this benchmark · Claude Code 2.1.233 · 512K context · reward hacking = 0 · two inconsistent tasks corrected· Reported by MiniMax· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 61.2% |
| 11 | DeepSeek V4 Pro 0813setting not named by the source· Reported by Z.ai· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 61.1% |
| 12 | Hy4 previewofficial model-card appendix· Reported by Tencent Hy· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 58.9% |
| 13 | GLM 5.3setting not named by the source· Reported by Z.ai· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 58% |
| 14 | GLM 5.3DeepSeek Harness· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 58% |
| 15 | GLM 5.3DeepSeek V4.1 launch chart · exact public value · setting not further specified · launch-page table· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 58% |
| 16 | Kimi K3setting not named by the source· Reported by Z.ai· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 58% |
| 17 | Kimi K3DeepSeek Harness· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 58% |
| 18 | Kimi K3DeepSeek V4.1 launch chart · exact public value · setting not further specified · launch-page table· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 58% |
| 19 | GPT-5.6 SolDeepSeek Harness· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 56.8% |
| 20 | GPT-5.6 SolDeepSeek V4.1 launch chart · exact public value · setting not further specified · launch-page table· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 56.8% |
| 21 | GLM 5.3 FlashZ.ai Flash设置· Reported by Z.ai· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 56.3% |
| 22 | Qwen3.8 Maxsetting not named by the source· Reported by Z.ai· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 55.9% |
| 23 | Qwen3.8 MaxQwen表:Claude Code · 禁止访问目标仓库的 Bash 命令· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 55.9% |
| 24 | DeepSeek V4 Flash 0731Claude Code · 禁止访问目标仓库的 Bash 命令· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 54.2% |
| 25 | DeepSeek V4 Flash 0731DeepSeek V4.1 launch chart · exact public value · setting not further specified · launch-page table· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 54.2% |
| 26 | GLM 5.2400K context· Reported by Z.ai· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 48.9% |
| 27 | Qwen3.8 Omni FlashClaude Code · 禁止访问目标仓库的 Bash 命令· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 48.9% |
| 28 | Qwen 3.8 Flash NextQwen3.8-Flash-Next release/model card · Claude Code · repository-download commands disabled· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 48.1% |
| 29 | Qwen3.8 FlashClaude Code · 禁止访问目标仓库的 Bash 命令· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 48.1% |
| 30 | Claude Opus 4.6Claude Code · 禁止访问目标仓库的 Bash 命令 · max effort· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 47.6% |
| 31 | Qwen3.7 MaxQwen3.7-Max release · Claude Code · repository-download commands disabled· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 47.2% |
| 32 | Qwen3.7 MaxQwen表:Claude Code · 禁止访问目标仓库的 Bash 命令· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 47.2% |
| 33 | Hy3 (Deep Infra)Hy3 official Benchmark Appendix · highest reasoning tier unless noted · Claude Code · 250 turns · 12,000s· Reported by Tencent Hunyuan· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 45.6% |
| 34 | GLM 5.1GLM-5.1 official release table· Reported by Z.ai· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 42.7% |
| 35 | Qwen 3.8 27BClaude Code · 禁止访问目标仓库的 Bash 命令· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 42.3% |
| 36 | Qwen 3.8 27BQwen3.8-27B model card · published setting· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 42.3% |
| 37 | MiniMax M3Claude Code · restricted sandbox· Reported by MiniMax Research· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 42.1% |
| 38 | Qwen3.7 PlusClaude Code · 禁止访问目标仓库的 Bash 命令· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 41.1% |
| 39 | Qwen3.7 PlusQwen3.7-Plus release · published setting· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 41.1% |
| 40 | MiniMax M2.7MiniMax M2.7 official release prose· Reported by MiniMax· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 39.8% |
| 41 | Qwen3.6 PlusQwen official release table · thinking mode unless benchmark requires otherwise · Claude Code · temp 1.0 · top_p .95 · max_turns 900· Reported by Alibaba Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 37.9% |
| 42 | Qwen3.6 27BQwen official release table · thinking mode unless benchmark requires otherwise · Claude Code · temp 1.0 · top_p .95 · max_turns 900· Reported by Alibaba Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 36.2% |
| 43 | Qwen3.6 FlashQwen3.6-35B-A3B official model card · hosted API name Qwen3.6-Flash· Reported by Alibaba Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 29.4% |