Skip to content

Benchmark

tau2-bench airline

Agentic benchmarks measure how well models carry out tasks that take several steps. Scores are percentages. Higher is better.

Models measured
135
Scores recorded
1083
Sources
1
IQ weight
2.65

Score distribution

68.5%
Mean
73.1%
Median
12.8%
Std dev
47%
10th percentile
77%
90th percentile
0.667%
Min
0.667%81.7%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 2.65 for chat models, measured over 134 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#Model
1051Qwen3 CoderOpenRouter harness, qwen/qwen3-coder-480b-a35b-07-25, Novita, public 90-day aggregate· OpenRouter benchmark exports35%
1052Llama 3.3 70B InstructOpenRouter harness, meta-llama/llama-3.3-70b-instruct, CoreWeave, public 90-day aggregate· OpenRouter benchmark exports34.2%
1053Llama 3.3 70B InstructOpenRouter harness, meta-llama/llama-3.3-70b-instruct, auto-routing, public 90-day aggregate· OpenRouter benchmark exports33%
1054Qwen3 235B A22B Instruct 2507OpenRouter harness, qwen/qwen3-235b-a22b-07-25, Alibaba, public 90-day aggregate· OpenRouter benchmark exports33%
1055qwen3-vl-30bOpenRouter harness, qwen/qwen3-vl-30b-a3b-instruct, Alibaba, public 90-day aggregate· OpenRouter benchmark exports32.8%
1056Qwen3 235B A22B Instruct 2507OpenRouter harness, qwen/qwen3-235b-a22b-07-25, AtlasCloud, public 90-day aggregate· OpenRouter benchmark exports32.7%
1057Qwen3 VL 8B InstructOpenRouter harness, qwen/qwen3-vl-8b-instruct, PrimeIntellect, public 90-day aggregate· OpenRouter benchmark exports32.7%
1058Llama 3.3 70B InstructOpenRouter harness, meta-llama/llama-3.3-70b-instruct, PrimeIntellect, public 90-day aggregate· OpenRouter benchmark exports31.3%
1059GPT-4o miniOpenRouter harness, openai/gpt-4o-mini, auto-routing, public 90-day aggregate· OpenRouter benchmark exports30.7%
1060DeepSeek V3.1OpenRouter harness, deepseek/deepseek-chat-v3.1, CoreWeave, public 90-day aggregate· OpenRouter benchmark exports30%
1061GPT-4o miniOpenRouter harness, openai/gpt-4o-mini, OpenAI, public 90-day aggregate· OpenRouter benchmark exports28.4%
1062GPT OSS 20bOpenRouter harness, openai/gpt-oss-20b, Amazon Bedrock, public 90-day aggregate· OpenRouter benchmark exports28.2%
1063Llama 3.1 8B InstructOpenRouter harness, meta-llama/llama-3.1-8b-instruct, auto-routing, public 90-day aggregate· OpenRouter benchmark exports27.9%
1064Llama 3.3 70B InstructOpenRouter harness, meta-llama/llama-3.3-70b-instruct, Together, public 90-day aggregate· OpenRouter benchmark exports27.5%
1065Qwen2.5 72B InstructOpenRouter harness, qwen/qwen-2.5-72b-instruct, Novita, public 90-day aggregate· OpenRouter benchmark exports27.3%
1066Qwen3 235B A22B Instruct 2507OpenRouter harness, qwen/qwen3-235b-a22b-07-25, GMICloud, public 90-day aggregate· OpenRouter benchmark exports26%
1067Llama 3.3 70B InstructOpenRouter harness, meta-llama/llama-3.3-70b-instruct, DeepInfra, public 90-day aggregate· OpenRouter benchmark exports25.3%
1068Qwen2.5 72B InstructOpenRouter harness, qwen/qwen-2.5-72b-instruct, DeepInfra, public 90-day aggregate· OpenRouter benchmark exports25%
1069Qwen2.5 72B InstructOpenRouter harness, qwen/qwen-2.5-72b-instruct, auto-routing, public 90-day aggregate· OpenRouter benchmark exports24%
1070Qwen3 Coder NextOpenRouter harness, qwen/qwen3-coder-next-2025-02-03, Novita, public 90-day aggregate· OpenRouter benchmark exports20.3%
1071Qwen3 Coder NextOpenRouter harness, qwen/qwen3-coder-next-2025-02-03, PrimeIntellect, public 90-day aggregate· OpenRouter benchmark exports18.7%
1072Nova MicroOpenRouter harness, amazon/nova-micro-v1, Amazon Bedrock, public 90-day aggregate· OpenRouter benchmark exports17.9%
1073GPT-4.1 nanoOpenRouter harness, openai/gpt-4.1-nano-2025-04-14, Azure, public 90-day aggregate· OpenRouter benchmark exports14%
1074GLM 4.6OpenRouter harness, z-ai/glm-4.6, AtlasCloud, public 90-day aggregate· OpenRouter benchmark exports11.3%
1075GPT-4.1 nanoOpenRouter harness, openai/gpt-4.1-nano-2025-04-14, OpenAI, public 90-day aggregate· OpenRouter benchmark exports11.1%
1076GPT-4.1 nanoOpenRouter harness, openai/gpt-4.1-nano-2025-04-14, auto-routing, public 90-day aggregate· OpenRouter benchmark exports11%
1077GPT OSS 120BOpenRouter harness, openai/gpt-oss-120b, Nebius, public 90-day aggregate· OpenRouter benchmark exports8.1%
1078DeepSeek V3.1 TerminusOpenRouter harness, deepseek/deepseek-v3.1-terminus, AtlasCloud, public 90-day aggregate· OpenRouter benchmark exports7.3%
1079qwen3-vl-30bOpenRouter harness, qwen/qwen3-vl-30b-a3b-instruct, SiliconFlow, public 90-day aggregate· OpenRouter benchmark exports6.8%
1080GPT OSS 20bOpenRouter harness, openai/gpt-oss-20b, CoreWeave, public 90-day aggregate· OpenRouter benchmark exports1.5%
1081GPT OSS 120BOpenRouter harness, openai/gpt-oss-120b, BaseTen, public 90-day aggregate· OpenRouter benchmark exports0.833%
1082Qwen3.6 35B A3BOpenRouter harness, qwen/qwen3.6-35b-a3b-20260415, Venice, public 90-day aggregate· OpenRouter benchmark exports0.762%
1083GPT OSS 120BOpenRouter harness, openai/gpt-oss-120b, CoreWeave, public 90-day aggregate· OpenRouter benchmark exports0.667%

Showing 1,051–1,083 of 1,083 results.

Tokens the model generated per task, as measured by the source. Comparable within this benchmark only.

Data credits: OpenRouter public benchmark leaderboard and search-lane exports, CC BY 4.0. Fractional source scores converted to percentages; immutable dated snapshots. (licence)