Skip to content

Benchmark

tau2-bench airline

Agentic benchmarks measure how well models carry out tasks that take several steps. Scores are percentages. Higher is better.

Models measured
135
Scores recorded
1083
Sources
1
IQ weight
2.65

Score distribution

68.5%
Mean
73.1%
Median
12.8%
Std dev
47%
10th percentile
77%
90th percentile
0.667%
Min
0.667%81.7%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 2.65 for chat models, measured over 134 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#Model
151Muse Glimmer 30BOpenRouter harness, meta/muse-glimmer-30b-20260810, auto-routing, public 90-day aggregate· OpenRouter benchmark exports76.5%
152GLM 5.3OpenRouter harness, z-ai/glm-5.3-20260816, Sail Research, public 90-day aggregate· OpenRouter benchmark exports76.5%
153GLM 5.3OpenRouter harness, z-ai/glm-5.3-20260816, InferenceNet, public 90-day aggregate· OpenRouter benchmark exports76.5%
154DeepSeek V4 ProOpenRouter harness, deepseek/deepseek-v4-pro-20260423, Baidu, public 90-day aggregate· OpenRouter benchmark exports76.4%
155Kimi K3.1 ProOpenRouter harness, moonshotai/kimi-k3.1-pro-20261023, Moonshot AI, public 90-day aggregate· OpenRouter benchmark exports76.4%
156Qwen3.8 FlashOpenRouter harness, qwen/qwen3.8-flash-20260826, Alibaba, public 90-day aggregate· OpenRouter benchmark exports76.4%
157Gemma 4 31B ITOpenRouter harness, google/gemma-4-31b-it-20260402, SambaNova, public 90-day aggregate· OpenRouter benchmark exports76.4%
158Hy4 previewOpenRouter harness, tencent/hy4-preview-20260827, Novita, public 90-day aggregate· OpenRouter benchmark exports76.4%
159Qwen3.5 397B A17BOpenRouter harness, qwen/qwen3.5-397b-a17b-20260216, DeepInfra, public 90-day aggregate· OpenRouter benchmark exports76.4%
160GLM 5.3OpenRouter harness, z-ai/glm-5.3-20260816, SiliconFlow, public 90-day aggregate· OpenRouter benchmark exports76.4%
161Claude Opus 4.7OpenRouter harness, anthropic/claude-4.7-opus-20260416, Amazon Bedrock, public 90-day aggregate· OpenRouter benchmark exports76.4%
162DeepSeek V4 Flash 0731OpenRouter harness, deepseek/deepseek-v4-flash-20260731, SiliconFlow, public 90-day aggregate· OpenRouter benchmark exports76.4%
163Qwen3.5 122B-A10BOpenRouter harness, qwen/qwen3.5-122b-a10b-20260224, Alibaba, public 90-day aggregate· OpenRouter benchmark exports76.3%
164Gemini 3.6 FlashOpenRouter harness, google/gemini-3.6-flash-20260721, Google, public 90-day aggregate· OpenRouter benchmark exports76.3%
165DeepSeek V4.1 Flash 1017OpenRouter harness, deepseek/deepseek-v4.1-flash-20261017, BaseTen, public 90-day aggregate· OpenRouter benchmark exports76.3%
166DeepSeek V4.1 Flash 1017OpenRouter harness, deepseek/deepseek-v4.1-flash-20261017, CoreWeave, public 90-day aggregate· OpenRouter benchmark exports76.3%
167Qwen3.5 35B A3BOpenRouter harness, qwen/qwen3.5-35b-a3b-20260224, Darkbloom, public 90-day aggregate· OpenRouter benchmark exports76.3%
168Qwen 3.8 27BOpenRouter harness, qwen/qwen3.8-27b-20260814, Cloudflare, public 90-day aggregate· OpenRouter benchmark exports76.3%
169Claude Opus 4.7OpenRouter harness, anthropic/claude-4.7-opus-20260416, auto-routing, public 90-day aggregate· OpenRouter benchmark exports76.3%
170Claude Opus 4.7OpenRouter harness, anthropic/claude-4.7-opus-20260416, Google, public 90-day aggregate· OpenRouter benchmark exports76.3%
171DeepSeek V4 Pro 0813OpenRouter harness, deepseek/deepseek-v4-pro-20260813, NextBit, public 90-day aggregate· OpenRouter benchmark exports76.3%
172Gemma 4 31B ITOpenRouter harness, google/gemma-4-31b-it-20260402, Friendli, public 90-day aggregate· OpenRouter benchmark exports76.3%
173MiMo V2.6 ProOpenRouter harness, xiaomi/mimo-v2.6-pro-20260921, GMICloud, public 90-day aggregate· OpenRouter benchmark exports76.3%
174Muse Glimmer 30BOpenRouter harness, meta/muse-glimmer-30b-20260810, Phala, public 90-day aggregate· OpenRouter benchmark exports76.3%
175GLM 5.3 FlashOpenRouter harness, z-ai/glm-5.3-flash-20260826, Z.AI, public 90-day aggregate· OpenRouter benchmark exports76.3%
176GLM 5.3OpenRouter harness, z-ai/glm-5.3-20260816, PrimeIntellect, public 90-day aggregate· OpenRouter benchmark exports76.3%
177DeepSeek V4.1 FlashOpenRouter harness, deepseek/deepseek-v4.1-flash-20260910, Crusoe, public 90-day aggregate· OpenRouter benchmark exports76.3%
178Kimi K3OpenRouter harness, moonshotai/kimi-k3-20260715, BaseTen, public 90-day aggregate· OpenRouter benchmark exports76.2%
179MiMo V2.6 FlashOpenRouter harness, xiaomi/mimo-v2.6-flash-20260921, Io Net, public 90-day aggregate· OpenRouter benchmark exports76.2%
180DeepSeek V4 Flash Vision ExpOpenRouter harness, deepseek/deepseek-v4-flash-vision-exp-20260821, DeepInfra, public 90-day aggregate· OpenRouter benchmark exports76.2%
181DeepSeek V4 ProOpenRouter harness, deepseek/deepseek-v4-pro-20260423, Novita, public 90-day aggregate· OpenRouter benchmark exports76.2%
182Qwen3.5 35B A3BOpenRouter harness, qwen/qwen3.5-35b-a3b-20260224, Parasail, public 90-day aggregate· OpenRouter benchmark exports76.2%
183Qwen3.6 35B A3BOpenRouter harness, qwen/qwen3.6-35b-a3b-20260415, DekaLLM, public 90-day aggregate· OpenRouter benchmark exports76.2%
184Qwen3.5 9BOpenRouter harness, qwen/qwen3.5-9b-20260310, DeepInfra, public 90-day aggregate· OpenRouter benchmark exports76.2%
185GLM 5.3OpenRouter harness, z-ai/glm-5.3-20260816, DekaLLM, public 90-day aggregate· OpenRouter benchmark exports76.2%
186Qwen3.6 35B A3BOpenRouter harness, qwen/qwen3.6-35b-a3b-20260415, Phala, public 90-day aggregate· OpenRouter benchmark exports76.1%
187Claude Opus 4.7OpenRouter harness, anthropic/claude-4.7-opus-20260416, Claude Platform on AWS, public 90-day aggregate· OpenRouter benchmark exports76.1%
188Qwen 3.8 27BOpenRouter harness, qwen/qwen3.8-27b-20260814, Chutes, public 90-day aggregate· OpenRouter benchmark exports76.1%
189DeepSeek V4 Flash 0731OpenRouter harness, deepseek/deepseek-v4-flash-20260731, GMICloud, public 90-day aggregate· OpenRouter benchmark exports76.1%
190Gemini 3.5 Flash LiteOpenRouter harness, google/gemini-3.5-flash-lite-20260721, Google, public 90-day aggregate· OpenRouter benchmark exports76.1%
191MiniMax M3OpenRouter harness, minimax/minimax-m3-20260531, Together, public 90-day aggregate· OpenRouter benchmark exports76.1%
192GLM 5.3 FlashOpenRouter harness, z-ai/glm-5.3-flash-20260826, Relace, public 90-day aggregate· OpenRouter benchmark exports76.1%
193Claude Opus 4.8OpenRouter harness, anthropic/claude-4.8-opus-20260528, Amazon Bedrock, public 90-day aggregate· OpenRouter benchmark exports76.1%
194DeepSeek V4 Flash 0731OpenRouter harness, deepseek/deepseek-v4-flash-20260731, Sail Research, public 90-day aggregate· OpenRouter benchmark exports76.1%
195DeepSeek V4 Flash 0731OpenRouter harness, deepseek/deepseek-v4-flash-20260731, CoreWeave, public 90-day aggregate· OpenRouter benchmark exports76.1%
196Qwen 3.8 27BOpenRouter harness, qwen/qwen3.8-27b-20260814, Reka, public 90-day aggregate· OpenRouter benchmark exports76.1%
197Kimi K3OpenRouter harness, moonshotai/kimi-k3-20260715, AkashML, public 90-day aggregate· OpenRouter benchmark exports76%
198GLM 5.3OpenRouter harness, z-ai/glm-5.3-20260816, Morph, public 90-day aggregate· OpenRouter benchmark exports76%
199Qwen 3.8 27BOpenRouter harness, qwen/qwen3.8-27b-20260814, Parasail, public 90-day aggregate· OpenRouter benchmark exports76%
200Qwen 3.8 27BOpenRouter harness, qwen/qwen3.8-27b-20260814, Mancer 2, public 90-day aggregate· OpenRouter benchmark exports76%

Showing 151–200 of 1,083 results.

Tokens the model generated per task, as measured by the source. Comparable within this benchmark only.

Data credits: OpenRouter public benchmark leaderboard and search-lane exports, CC BY 4.0. Fractional source scores converted to percentages; immutable dated snapshots. (licence)