Skip to content

Benchmark

tau2-bench airline

Agentic benchmarks measure how well models carry out tasks that take several steps. Scores are percentages. Higher is better.

Models measured
135
Scores recorded
1083
Sources
1
IQ weight
2.65

Score distribution

68.5%
Mean
73.1%
Median
12.8%
Std dev
47%
10th percentile
77%
90th percentile
0.667%
Min
0.667%81.7%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 2.65 for chat models, measured over 134 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#Model
1001GPT 4oOpenRouter harness, openai/gpt-4o, OpenAI, public 90-day aggregate· OpenRouter benchmark exports43.2%
1002GPT 4oOpenRouter harness, openai/gpt-4o-2024-08-06, OpenAI, public 90-day aggregate· OpenRouter benchmark exports43.1%
1003DeepSeek ChatOpenRouter harness, deepseek/deepseek-chat-v3, DeepInfra, public 90-day aggregate· OpenRouter benchmark exports43%
1004DeepSeek V3.1OpenRouter harness, deepseek/deepseek-chat-v3.1, auto-routing, public 90-day aggregate· OpenRouter benchmark exports43%
1005Gemma 4 31B ITOpenRouter harness, google/gemma-4-31b-it-20260402, Chutes, public 90-day aggregate· OpenRouter benchmark exports42.8%
1006GPT 4oOpenRouter harness, openai/gpt-4o-2024-08-06, auto-routing, public 90-day aggregate· OpenRouter benchmark exports42.7%
1007Qwen3 14BOpenRouter harness, qwen/qwen3-14b-04-28, DeepInfra, public 90-day aggregate· OpenRouter benchmark exports42.6%
1008Qwen3 235B A22B Instruct 2507OpenRouter harness, qwen/qwen3-235b-a22b-07-25, Nebius, public 90-day aggregate· OpenRouter benchmark exports42.4%
1009Qwen3 Next 80B A3B InstructOpenRouter harness, qwen/qwen3-next-80b-a3b-instruct-2509, Alibaba, public 90-day aggregate· OpenRouter benchmark exports42.3%
1010Qwen3 235B A22B Instruct 2507OpenRouter harness, qwen/qwen3-235b-a22b-07-25, DeepInfra, public 90-day aggregate· OpenRouter benchmark exports42.3%
1011Gemma 4 31B ITOpenRouter harness, google/gemma-4-31b-it-20260402, Modal, public 90-day aggregate· OpenRouter benchmark exports42.3%
1012Qwen3 30B A3B InstructOpenRouter harness, qwen/qwen3-30b-a3b-instruct-2507, StreamLake, public 90-day aggregate· OpenRouter benchmark exports42.2%
1013Mistral SmallOpenRouter harness, mistralai/mistral-small-2603, Mistral, public 90-day aggregate· OpenRouter benchmark exports41.9%
1014GPT 4oOpenRouter harness, openai/gpt-4o, auto-routing, public 90-day aggregate· OpenRouter benchmark exports41.7%
1015Qwen3 VL 235BOpenRouter harness, qwen/qwen3-vl-235b-a22b-instruct, Novita, public 90-day aggregate· OpenRouter benchmark exports41.7%
1016Qwen3 CoderOpenRouter harness, qwen/qwen3-coder-480b-a35b-07-25, DeepInfra, public 90-day aggregate· OpenRouter benchmark exports41%
1017Qwen3 CoderOpenRouter harness, qwen/qwen3-coder-480b-a35b-07-25, Venice, public 90-day aggregate· OpenRouter benchmark exports41%
1018Qwen3 VL 235BOpenRouter harness, qwen/qwen3-vl-235b-a22b-instruct, Parasail, public 90-day aggregate· OpenRouter benchmark exports41%
1019Llama 4 MaverickOpenRouter harness, meta-llama/llama-4-maverick-17b-128e-instruct, auto-routing, public 90-day aggregate· OpenRouter benchmark exports40.7%
1020Qwen3 235B A22B Instruct 2507OpenRouter harness, qwen/qwen3-235b-a22b-07-25, PrimeIntellect, public 90-day aggregate· OpenRouter benchmark exports40.7%
1021Qwen3 VL 235BOpenRouter harness, qwen/qwen3-vl-235b-a22b-instruct, Alibaba, public 90-day aggregate· OpenRouter benchmark exports40.7%
1022Qwen3 VL 235BOpenRouter harness, qwen/qwen3-vl-235b-a22b-instruct, auto-routing, public 90-day aggregate· OpenRouter benchmark exports40.7%
1023Qwen3.6 35B A3BOpenRouter harness, qwen/qwen3.6-35b-a3b-20260415, Io Net, public 90-day aggregate· OpenRouter benchmark exports40.4%
1024DeepSeek ChatOpenRouter harness, deepseek/deepseek-chat-v3, StreamLake, public 90-day aggregate· OpenRouter benchmark exports40.4%
1025GPT-4.1 miniOpenRouter harness, openai/gpt-4.1-mini-2025-04-14, Azure, public 90-day aggregate· OpenRouter benchmark exports40.3%
1026qwen3-vl-30bOpenRouter harness, qwen/qwen3-vl-30b-a3b-instruct, Novita, public 90-day aggregate· OpenRouter benchmark exports40.3%
1027GPT OSS 20bOpenRouter harness, openai/gpt-oss-20b, AkashML, public 90-day aggregate· OpenRouter benchmark exports40.2%
1028Llama 3.3 70B InstructOpenRouter harness, meta-llama/llama-3.3-70b-instruct, Nebius, public 90-day aggregate· OpenRouter benchmark exports40%
1029Qwen3 235B A22B Instruct 2507OpenRouter harness, qwen/qwen3-235b-a22b-07-25, StreamLake, public 90-day aggregate· OpenRouter benchmark exports40%
1030DeepSeek ChatOpenRouter harness, deepseek/deepseek-chat-v3, auto-routing, public 90-day aggregate· OpenRouter benchmark exports39.8%
1031GPT OSS 20bOpenRouter harness, openai/gpt-oss-20b, PrimeIntellect, public 90-day aggregate· OpenRouter benchmark exports39.8%
1032Qwen3 235B A22B Instruct 2507OpenRouter harness, qwen/qwen3-235b-a22b-07-25, Google, public 90-day aggregate· OpenRouter benchmark exports39.4%
1033Llama 4 MaverickOpenRouter harness, meta-llama/llama-4-maverick-17b-128e-instruct, DigitalOcean, public 90-day aggregate· OpenRouter benchmark exports39.3%
1034Qwen3 30B A3B InstructOpenRouter harness, qwen/qwen3-30b-a3b-instruct-2507, SiliconFlow, public 90-day aggregate· OpenRouter benchmark exports39%
1035Qwen3 VL 235BOpenRouter harness, qwen/qwen3-vl-235b-a22b-instruct, DeepInfra, public 90-day aggregate· OpenRouter benchmark exports39%
1036Qwen3 30B A3B InstructOpenRouter harness, qwen/qwen3-30b-a3b-instruct-2507, Ambient, public 90-day aggregate· OpenRouter benchmark exports38.7%
1037Qwen3 30B A3B InstructOpenRouter harness, qwen/qwen3-30b-a3b-instruct-2507, CoreWeave, public 90-day aggregate· OpenRouter benchmark exports38.7%
1038qwen3-vl-30bOpenRouter harness, qwen/qwen3-vl-30b-a3b-instruct, PrimeIntellect, public 90-day aggregate· OpenRouter benchmark exports38.7%
1039Qwen3 30B A3B InstructOpenRouter harness, qwen/qwen3-30b-a3b-instruct-2507, auto-routing, public 90-day aggregate· OpenRouter benchmark exports38.3%
1040Llama 3.3 70B InstructOpenRouter harness, meta-llama/llama-3.3-70b-instruct, Novita, public 90-day aggregate· OpenRouter benchmark exports38.1%
1041Llama 3.3 70B InstructOpenRouter harness, meta-llama/llama-3.3-70b-instruct, AkashML, public 90-day aggregate· OpenRouter benchmark exports37.4%
1042Llama 4 MaverickOpenRouter harness, meta-llama/llama-4-maverick-17b-128e-instruct, Parasail, public 90-day aggregate· OpenRouter benchmark exports37.3%
1043Qwen3 CoderOpenRouter harness, qwen/qwen3-coder-480b-a35b-07-25, Alibaba, public 90-day aggregate· OpenRouter benchmark exports36.7%
1044qwen3-vl-30bOpenRouter harness, qwen/qwen3-vl-30b-a3b-instruct, DeepInfra, public 90-day aggregate· OpenRouter benchmark exports36.5%
1045DeepSeek V3.1OpenRouter harness, deepseek/deepseek-chat-v3.1, DeepInfra, public 90-day aggregate· OpenRouter benchmark exports36.4%
1046Qwen3 235B A22B Instruct 2507OpenRouter harness, qwen/qwen3-235b-a22b-07-25, auto-routing, public 90-day aggregate· OpenRouter benchmark exports36%
1047Qwen3 30B A3B InstructOpenRouter harness, qwen/qwen3-30b-a3b-instruct-2507, PrimeIntellect, public 90-day aggregate· OpenRouter benchmark exports36%
1048Qwen3 30B A3B InstructOpenRouter harness, qwen/qwen3-30b-a3b-instruct-2507, Nebius, public 90-day aggregate· OpenRouter benchmark exports35.8%
1049qwen3-vl-30bOpenRouter harness, qwen/qwen3-vl-30b-a3b-instruct, auto-routing, public 90-day aggregate· OpenRouter benchmark exports35.2%
1050Qwen3 30B A3B InstructOpenRouter harness, qwen/qwen3-30b-a3b-instruct-2507, Alibaba, public 90-day aggregate· OpenRouter benchmark exports35%

Showing 1,001–1,050 of 1,083 results.

Tokens the model generated per task, as measured by the source. Comparable within this benchmark only.

Data credits: OpenRouter public benchmark leaderboard and search-lane exports, CC BY 4.0. Fractional source scores converted to percentages; immutable dated snapshots. (licence)