Skip to content

Benchmark

tau2-bench v0.1 airline pass^1

Agentic benchmarks measure how well models carry out tasks that take several steps. Scores are percentages. Higher is better.

Models measured
10
Scores recorded
11
Sources
1
IQ weight
0.62

Score distribution

59.5%
Mean
59.5%
Median
6.7%
Std dev
52%
10th percentile
69%
90th percentile
48.7%
Min
48.7%70%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 0.62 for chat models, measured over 10 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Claude Sonnet 4.5submission-claude-sonnet-4-5_anthropic_2025-10-02· Sierra tau-bench Leaderboard data docs70%
2Qwen3 Maxsubmission-qwen3-max_qwen_2026-01-23, usersim-gpt-4.1-2025-04-14· Sierra tau-bench Leaderboard data docs69%
3Claude Sonnet 3.7submission-claude-3-7-sonnet_anthropic_2024-06-20, usersim-gpt-4.1-2025-04-14· Sierra tau-bench Leaderboard data docs64.2%
4DeepSeek V3.2submission-deepseek-v3.2_deepseek_2025-12-01, usersim-deepseek-v3.2· Sierra tau-bench Leaderboard data docs63.8%
5GPT 5submission-gpt-5_sierra_2025-08-09, usersim-gpt-4.1-2025-04-14· Sierra tau-bench Leaderboard data docs62.5%
6Qwen3 Maxsubmission-qwen3-max_qwen_2024_09_23· Sierra tau-bench Leaderboard data docs59.5%
7Kimi K2submission-kimi-k2_moonshot-ai_2025-07-11· Sierra tau-bench Leaderboard data docs56.5%
8GPT 4.1submission-gpt-4-1_openai_2024-06-20, usersim-gpt-4.1-2025-04-14· Sierra tau-bench Leaderboard data docs56%
9o4 Minisubmission-o4-mini_openai_2024-06-20, usersim-gpt-4.1-2025-04-14· Sierra tau-bench Leaderboard data docs52.1%
10OpenAI o3submission-o3_openai_2025-01-15· Sierra tau-bench Leaderboard data docs52%
11GPT-4.1 minisubmission-gpt-4-1-mini_openai_2024-06-20, usersim-gpt-4.1-2025-04-14· Sierra tau-bench Leaderboard data docs48.7%

Data credits: Sierra Research tau2-bench leaderboard submissions (github.com/sierra-research/tau2-bench), MIT (licence)