Skip to content

Benchmark

tau2-bench airline

Agentic benchmarks measure how well models carry out tasks that take several steps. Scores are percentages. Higher is better.

Models measured
135
Scores recorded
1083
Sources
1
IQ weight
2.65

Score distribution

68.5%
Mean
73.1%
Median
12.8%
Std dev
47%
10th percentile
77%
90th percentile
0.667%
Min
0.667%81.7%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 2.65 for chat models, measured over 134 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#Model
851GPT-6 SolOpenRouter harness, openai/gpt-6-sol-20260922, Azure, public 90-day aggregate· OpenRouter benchmark exports67.1%
852DeepSeek V3.2OpenRouter harness, deepseek/deepseek-v3.2-20251201, Google, public 90-day aggregate· OpenRouter benchmark exports67%
853Nemotron 3.5 LightningOpenRouter harness, nvidia/nemotron-3.5-lightning-20260807, Darkbloom, public 90-day aggregate· OpenRouter benchmark exports67%
854GPT-5.4 miniOpenRouter harness, openai/gpt-5.4-mini-20260317, Azure, public 90-day aggregate· OpenRouter benchmark exports66.8%
855MiniMax M3.1 FlashOpenRouter harness, minimax/minimax-m3.1-flash-20261019, GMICloud, public 90-day aggregate· OpenRouter benchmark exports66.8%
856Kimi K2.7 CodeOpenRouter harness, moonshotai/kimi-k2.7-code-20260612, GMICloud, public 90-day aggregate· OpenRouter benchmark exports66.7%
857GPT-6 LunaOpenRouter harness, openai/gpt-6-luna-pro-20260922, auto-routing, public 90-day aggregate· OpenRouter benchmark exports66.7%
858Kimi K2.5OpenRouter harness, moonshotai/kimi-k2.5-0127, DigitalOcean, public 90-day aggregate· OpenRouter benchmark exports66.7%
859Kimi K2.5OpenRouter harness, moonshotai/kimi-k2.5-0127, StreamLake, public 90-day aggregate· OpenRouter benchmark exports66.7%
860Nemotron 3.5 LightningOpenRouter harness, nvidia/nemotron-3.5-lightning-20260807, ModelRun, public 90-day aggregate· OpenRouter benchmark exports66.7%
861Qwen3.5 9BOpenRouter harness, qwen/qwen3.5-9b-20260310, PrimeIntellect, public 90-day aggregate· OpenRouter benchmark exports66.7%
862GPT-5.4 miniOpenRouter harness, openai/gpt-5.4-mini-20260317, auto-routing, public 90-day aggregate· OpenRouter benchmark exports66.7%
863MiniMax M3.1 FlashOpenRouter harness, minimax/minimax-m3.1-flash-20261019, auto-routing, public 90-day aggregate· OpenRouter benchmark exports66.7%
864GPT OSS 120BOpenRouter harness, openai/gpt-oss-120b, Near AI, public 90-day aggregate· OpenRouter benchmark exports66.6%
865Qwen3.5 9BOpenRouter harness, qwen/qwen3.5-9b-20260310, SiliconFlow, public 90-day aggregate· OpenRouter benchmark exports66.5%
866GPT OSS 120BOpenRouter harness, openai/gpt-oss-120b, auto-routing, public 90-day aggregate· OpenRouter benchmark exports66.4%
867Qwen3.5 9BOpenRouter harness, qwen/qwen3.5-9b-20260310, Venice, public 90-day aggregate· OpenRouter benchmark exports66.2%
868GPT OSS 120BOpenRouter harness, openai/gpt-oss-120b, DekaLLM, public 90-day aggregate· OpenRouter benchmark exports66%
869GLM 4.7 FlashOpenRouter harness, z-ai/glm-4.7-flash-20260119, Cloudflare, public 90-day aggregate· OpenRouter benchmark exports66%
870Inkling FP4OpenRouter harness, thinkingmachines/inkling-20260715, auto-routing, public 90-day aggregate· OpenRouter benchmark exports66%
871MiniMax M2.5OpenRouter harness, minimax/minimax-m2.5-20260211, DigitalOcean, public 90-day aggregate· OpenRouter benchmark exports66%
872MiniMax M2.5OpenRouter harness, minimax/minimax-m2.5-20260211, Friendli, public 90-day aggregate· OpenRouter benchmark exports66%
873MiniMax M3.1 FlashOpenRouter harness, minimax/minimax-m3.1-flash-20261019, Makora, public 90-day aggregate· OpenRouter benchmark exports66%
874GLM 4.7 FlashOpenRouter harness, z-ai/glm-4.7-flash-20260119, DeepInfra, public 90-day aggregate· OpenRouter benchmark exports66%
875Claude Haiku 4.5OpenRouter harness, anthropic/claude-4.5-haiku-20251001, Amazon Bedrock, public 90-day aggregate· OpenRouter benchmark exports66%
876Gemma 4 26B A4BOpenRouter harness, google/gemma-4-26b-a4b-it-20260403, Darkbloom, public 90-day aggregate· OpenRouter benchmark exports65.9%
877Claude Haiku 4.5OpenRouter harness, anthropic/claude-4.5-haiku-20251001, auto-routing, public 90-day aggregate· OpenRouter benchmark exports65.7%
878GPT OSS 120BOpenRouter harness, openai/gpt-oss-120b, AkashML, public 90-day aggregate· OpenRouter benchmark exports65.7%
879Nemotron 3.5 LightningOpenRouter harness, nvidia/nemotron-3.5-lightning-20260807, auto-routing, public 90-day aggregate· OpenRouter benchmark exports65.6%
880Nemotron 3.5 LightningOpenRouter harness, nvidia/nemotron-3.5-lightning-20260807, Phala, public 90-day aggregate· OpenRouter benchmark exports65.6%
881GPT 5.4 NanoOpenRouter harness, openai/gpt-5.4-nano-20260317, OpenAI, public 90-day aggregate· OpenRouter benchmark exports65.6%
882Nemotron 3.5 LightningOpenRouter harness, nvidia/nemotron-3.5-lightning-20260807, DeepInfra, public 90-day aggregate· OpenRouter benchmark exports65.6%
883GPT OSS 120BOpenRouter harness, openai/gpt-oss-120b, DeepInfra, public 90-day aggregate· OpenRouter benchmark exports65.5%
884GPT OSS 120BOpenRouter harness, openai/gpt-oss-120b, Parasail, public 90-day aggregate· OpenRouter benchmark exports65.5%
885Claude Haiku 4.5OpenRouter harness, anthropic/claude-4.5-haiku-20251001, Google, public 90-day aggregate· OpenRouter benchmark exports65.5%
886Claude Haiku 4.5OpenRouter harness, anthropic/claude-4.5-haiku-20251001, Anthropic, public 90-day aggregate· OpenRouter benchmark exports65.5%
887GPT OSS 120BOpenRouter harness, openai/gpt-oss-120b, Crusoe, public 90-day aggregate· OpenRouter benchmark exports65.5%
888Nemotron 3.5 LightningOpenRouter harness, nvidia/nemotron-3.5-lightning-20260807, Io Net, public 90-day aggregate· OpenRouter benchmark exports65.4%
889GPT OSS 120BOpenRouter harness, openai/gpt-oss-120b, Sail Research, public 90-day aggregate· OpenRouter benchmark exports65.3%
890Ling 3.0 Flash FinOpenRouter harness, inclusionai/ling-3.0-flash-fin-20260827, Novita, public 90-day aggregate· OpenRouter benchmark exports65.3%
891Claude Sonnet 4OpenRouter harness, anthropic/claude-4-sonnet-20250522, Google, public 90-day aggregate· OpenRouter benchmark exports65.3%
892GLM 4.7 FlashOpenRouter harness, z-ai/glm-4.7-flash-20260119, auto-routing, public 90-day aggregate· OpenRouter benchmark exports65.3%
893Inkling 2 SmallOpenRouter harness, thinkingmachines/inkling-2-small-20261107, Nebius, public 90-day aggregate· OpenRouter benchmark exports65.3%
894Claude Haiku 4.5OpenRouter harness, anthropic/claude-4.5-haiku-20251001, Azure, public 90-day aggregate· OpenRouter benchmark exports65.3%
895Gemma 4 26B A4BOpenRouter harness, google/gemma-4-26b-a4b-it-20260403, AkashML, public 90-day aggregate· OpenRouter benchmark exports65.2%
896MiMo V2.5 ProOpenRouter harness, xiaomi/mimo-v2.5-pro-20260422, DigitalOcean, public 90-day aggregate· OpenRouter benchmark exports65%
897GPT OSS 120BOpenRouter harness, openai/gpt-oss-120b, Groq, public 90-day aggregate· OpenRouter benchmark exports64.9%
898GPT-5.4 miniOpenRouter harness, openai/gpt-5.4-mini-20260317, OpenAI, public 90-day aggregate· OpenRouter benchmark exports64.9%
899GPT OSS 120BOpenRouter harness, openai/gpt-oss-120b, Amazon Bedrock, public 90-day aggregate· OpenRouter benchmark exports64.7%
900MiniMax M2.5OpenRouter harness, minimax/minimax-m2.5-20260211, auto-routing, public 90-day aggregate· OpenRouter benchmark exports64.7%

Showing 851–900 of 1,083 results.

Tokens the model generated per task, as measured by the source. Comparable within this benchmark only.

Data credits: OpenRouter public benchmark leaderboard and search-lane exports, CC BY 4.0. Fractional source scores converted to percentages; immutable dated snapshots. (licence)