Skip to content

Benchmark

Nejumi 4: AVG_swebench

Coding benchmarks compare models on programming tasks. Higher scores are better.

Models measured
114
Scores recorded
126
Sources
1
IQ weight
1.11

Score distribution

0.374
Mean
0.405
Median
0.206
Std dev
0.088
10th percentile
0.613
90th percentile
0
Min
00.8
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.11 for chat models, measured over 114 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
51Gemini 3.1 Flash Litegoogle/gemini-3.1-flash-lite-preview· Nejumi Leaderboard 4 data docs0.475
52GLM 4.6zai-org/GLM-4.6-FP8: reasoning-enabled· Nejumi Leaderboard 4 data docs0.475
53DeepSeek V3.2deepseek-ai/DeepSeek-V3.2: reasoning-enabled· Nejumi Leaderboard 4 data docs0.463
54Qwen 3 235B A22B Thinking 2507Qwen/Qwen3-235B-A22B-Thinking-2507: reasoning-enabled· Nejumi Leaderboard 4 data docs0.463
55Grok 4.1 Fastx-ai/grok-4-1-fast-reasoning· Nejumi Leaderboard 4 data docs0.438
56Qwen3.6 35B A3BQwen/Qwen3.6-35B-A3B: reasoning-enabled· Nejumi Leaderboard 4 data docs0.438
57o4 Miniopenai/o4-mini-2025-04-16· Nejumi Leaderboard 4 data docs0.425
58OpenAI o3openai/o3-2025-04-16: high-effort· Nejumi Leaderboard 4 data docs0.425
59GPT 5.4openai/gpt-5.4-2026-03-05: xhigh-effort· Nejumi Leaderboard 4 data docs0.413
60o1openai/o1-2024-12-17: high-effort· Nejumi Leaderboard 4 data docs0.413
61Qwen3 Next 80B A3B InstructQwen/Qwen3-Next-80B-A3B-Instruct· Nejumi Leaderboard 4 data docs0.413
62Qwen3.5 27BQwen/Qwen3.5-27B: reasoning-enabled· Nejumi Leaderboard 4 data docs0.413
63GPT 4.1openai/gpt-4-1-2025-04-14· Nejumi Leaderboard 4 data docs0.405
64MiniMax M2.1MiniMaxAI/MiniMax-M2.1: reasoning-enabled· Nejumi Leaderboard 4 data docs0.4
65Qwen3 Next 80B A3BQwen/Qwen3-Next-80B-A3B-Thinking: reasoning-enabled· Nejumi Leaderboard 4 data docs0.388
66Grok 4grok-4· Nejumi Leaderboard 4 data docs0.375
67o3 Miniopenai/o3-mini-2025-01-31· Nejumi Leaderboard 4 data docs0.375
68Qwen3.5 122B-A10BQwen/Qwen3.5-122B-A10B: reasoning-enabled· Nejumi Leaderboard 4 data docs0.375
69Claude 3.5 Sonnetanthropic/claude-3.5-sonnet-20241022· Nejumi Leaderboard 4 data docs0.363
70MiniMax M2MiniMaxAI/MiniMax-M2: reasoning-enabled· Nejumi Leaderboard 4 data docs0.363
71Qwen3 Max PreviewQwen/Qwen3-Max-Preview· Nejumi Leaderboard 4 data docs0.363
72Claude Sonnet 3.7anthropic/claude-3.7-sonnet-20250219: no-thinking· Nejumi Leaderboard 4 data docs0.35
73Qwen3 Next 80B A3BQwen/Qwen3-Next-80B-A3B-Thinking· Nejumi Leaderboard 4 data docs0.35
74GPT 5.1openai/gpt-5.1-2025-11-13: none-effort· Nejumi Leaderboard 4 data docs0.338
75Gemma 4 E4B-itgoogle/gemma-4-E4B-it· Nejumi Leaderboard 4 data docs0.325
76Qwen3.5 35B A3BQwen/Qwen3.5-35B-A3B: reasoning-enabled· Nejumi Leaderboard 4 data docs0.313
77GPT-4.1 miniopenai/gpt-4-1-mini-2025-04-14· Nejumi Leaderboard 4 data docs0.304
78Grok 3grok-3· Nejumi Leaderboard 4 data docs0.3
79GPT OSS 120Bopenai/gpt-oss-120b: reasoning-enabled· Nejumi Leaderboard 4 data docs0.288
80QwQ 32BQwen/QwQ-32B: reasoning-enabled· Nejumi Leaderboard 4 data docs0.288
81DeepSeek V3deepseek-ai/DeepSeek-V3-0324· Nejumi Leaderboard 4 data docs0.275
82Qwen3 30B A3BQwen/Qwen3-30B-A3B-Thinking-2507: reasoning-enabled· Nejumi Leaderboard 4 data docs0.275
83GPT-5 nanoopenai/gpt-5-nano-2025-08-07: high-effort· Nejumi Leaderboard 4 data docs0.263
84Llama 3.1 405B Instructmeta-llama/Llama-3.1-405B-Instruct-FP8· Nejumi Leaderboard 4 data docs0.263
85Mistral Large 3 675b Instructmistralai/Mistral-Large-3-675B-Instruct-2512· Nejumi Leaderboard 4 data docs0.25
86QwQ 32BQwen/QwQ-32B· Nejumi Leaderboard 4 data docs0.25
87Gemini 2.5 Flashgoogle/gemini-2.5-flash· Nejumi Leaderboard 4 data docs0.238
88Llama 4 Maverickmeta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8· Nejumi Leaderboard 4 data docs0.238
89Qwen3 32BQwen/Qwen3-32B: reasoning-enabled· Nejumi Leaderboard 4 data docs0.238
90Gemma 4 E2B-itgoogle/gemma-4-E2B-it· Nejumi Leaderboard 4 data docs0.225
91Qwen3 VL 32B ThinkingQwen/Qwen3-VL-32B-Thinking· Nejumi Leaderboard 4 data docs0.225
92Gemini 3 Pro Previewgoogle/gemini-3-pro-preview· Nejumi Leaderboard 4 data docs0.213
93GPT 4oopenai/gpt-4o-2024-11-20· Nejumi Leaderboard 4 data docs0.211
94Claude Haiku 3.5anthropic/claude-3.5-haiku-20241022· Nejumi Leaderboard 4 data docs0.2
95Grok 3 Minigrok-3-mini· Nejumi Leaderboard 4 data docs0.188
96Nova Pro 1.0us.amazon.nova-pro-v1:0· Nejumi Leaderboard 4 data docs0.188
97Qwen3 14BQwen/Qwen3-14B: reasoning-enabled· Nejumi Leaderboard 4 data docs0.188
98Llama 3.3 70B Instructmeta-llama/Llama-3.3-70B-Instruct· Nejumi Leaderboard 4 data docs0.175
99Qwen3.5 9BQwen/Qwen3.5-9B: reasoning-enabled· Nejumi Leaderboard 4 data docs0.175
100GLM 4.7 Flashzai-org/GLM-4.7-Flash: reasoning-enabled· Nejumi Leaderboard 4 data docs0.15

Showing 51–100 of 126 results.

Data credits: Nejumi Leaderboard 4, Weights & Biases Japan; public publisher-selected numeric evaluation results