Skip to content

Benchmark

AgentDojo utility

Agentic benchmarks measure how well models carry out tasks that take several steps. Scores are percentages. Higher is better.

Models measured
19
Scores recorded
28
Sources
1
IQ weight
1.27

Score distribution

57.8%
Mean
66%
Median
17.9%
Std dev
35.1%
10th percentile
79.4%
90th percentile
24.7%
Min
24.7%88.7%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.27 for chat models, measured over 19 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Claude Sonnet 3.7provider: anthropic, defense: None, attack: important_instructions, date: 2025-02-24· AgentDojo results data docs88.7%
2GPT 4oprovider: openai, defense: repeat_user_prompt, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs84.5%
3Claude 3.5 Sonnetprovider: anthropic, defense: None, attack: important_instructions, date: 2024-06-20· AgentDojo results data docs79.4%
4Claude 3.5 Sonnetprovider: anthropic, defense: None, attack: important_instructions, date: 2024-11-15· AgentDojo results data docs79.4%
5GPT 4oprovider: openai, defense: spotlighting_with_delimiting, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs72.2%
6GPT 4oprovider: openai, defense: tool_filter, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs72.2%
7GPT 4oprovider: openai, defense: None, attack: direct, date: 2024-06-05· AgentDojo results data docs69.1%
8GPT 4oprovider: openai, defense: None, attack: ignore_previous, date: 2024-06-05· AgentDojo results data docs69.1%
9GPT 4oprovider: openai, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs69.1%
10GPT 4oprovider: openai, defense: None, attack: injecagent, date: 2024-06-05· AgentDojo results data docs69.1%
11GPT 4oprovider: openai, defense: None, attack: tool_knowledge, date: 2024-06-05· AgentDojo results data docs69.1%
12Claude 3 Opusprovider: anthropic, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs68%
13GPT-4o miniprovider: openai, defense: None, attack: important_instructions, date: 2024-07-19· AgentDojo results data docs68%
14GPT-4-0125-previewprovider: openai, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs66%
15GPT-4 Turboprovider: openai, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs65%
16gemini-1.5-pro-002provider: google, defense: None, attack: important_instructions, date: 2024-11-04· AgentDojo results data docs61.9%
17Claude 3 Sonnetprovider: anthropic, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs53.6%
18Gemini 2.0 Flash Experimentalprovider: google, defense: None, attack: important_instructions, date: 2024-12-11· AgentDojo results data docs46.4%
19gemini-1.5-pro-001provider: google, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs46.4%
20Gemini 2.0 Flashprovider: google, defense: None, attack: important_instructions, date: 2025-01-30· AgentDojo results data docs43.3%
21GPT 4oprovider: openai, defense: transformers_pi_detector, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs41.2%
22Claude 3 Haikuprovider: anthropic, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs39.2%
23gemini-1.5-flash-001provider: google, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs38.1%
24gemini-1.5-flash-002provider: google, defense: None, attack: important_instructions, date: 2024-11-05· AgentDojo results data docs38.1%
25GPT-3.5 Turboprovider: openai, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs35.1%
26Llama-3-70b-chat-hfprovider: together-prompting, defense: None, attack: important_instructions, date: 2021-09-05· AgentDojo results data docs34%
27Command R+provider: cohere, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs26.8%
28Command R Plusprovider: cohere, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs24.7%

Data credits: AgentDojo. Copyright (c) 2024 Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. MIT License. Not a leaderboard: attack and defense coverage differs between models. (licence)