Skip to content

Benchmark

tau2-bench v0.1 retail pass^1

Agentic benchmarks measure how well models carry out tasks that take several steps. Scores are percentages. Higher is better.

Models measured
10
Scores recorded
11
Sources
1
IQ weight
0.96

Score distribution

74.6%
Mean
73.9%
Median
6.7%
Std dev
68.3%
10th percentile
81.6%
90th percentile
61.4%
Min
61.4%86.2%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 0.96 for chat models, measured over 10 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Claude Sonnet 4.5submission-claude-sonnet-4-5_anthropic_2025-10-02· Sierra tau-bench Leaderboard data docs86.2%
2GPT 5submission-gpt-5_sierra_2025-08-09, usersim-gpt-4.1-2025-04-14· Sierra tau-bench Leaderboard data docs81.6%
3DeepSeek V3.2submission-deepseek-v3.2_deepseek_2025-12-01, usersim-deepseek-v3.2· Sierra tau-bench Leaderboard data docs81.1%
4Qwen3 Maxsubmission-qwen3-max_qwen_2026-01-23, usersim-gpt-4.1-2025-04-14· Sierra tau-bench Leaderboard data docs79.4%
5GPT 4.1submission-gpt-4-1_openai_2024-06-20, usersim-gpt-4.1-2025-04-14· Sierra tau-bench Leaderboard data docs74%
6OpenAI o3submission-o3_openai_2025-01-15· Sierra tau-bench Leaderboard data docs73.9%
7Qwen3 Maxsubmission-qwen3-max_qwen_2024_09_23· Sierra tau-bench Leaderboard data docs72.2%
8Claude Sonnet 3.7submission-claude-3-7-sonnet_anthropic_2024-06-20, usersim-gpt-4.1-2025-04-14· Sierra tau-bench Leaderboard data docs72.1%
9Kimi K2submission-kimi-k2_moonshot-ai_2025-07-11· Sierra tau-bench Leaderboard data docs70.6%
10o4 Minisubmission-o4-mini_openai_2024-06-20, usersim-gpt-4.1-2025-04-14· Sierra tau-bench Leaderboard data docs68.3%
11GPT-4.1 minisubmission-gpt-4-1-mini_openai_2024-06-20, usersim-gpt-4.1-2025-04-14· Sierra tau-bench Leaderboard data docs61.4%

Data credits: Sierra Research tau2-bench leaderboard submissions (github.com/sierra-research/tau2-bench), MIT (licence)