Skip to content

Benchmark

AgentDojo utility under attack

Agentic benchmarks measure how well models carry out tasks that take several steps. Scores are percentages. Higher is better.

Models measured
19
Scores recorded
28
Sources
1
IQ weight
1.92

Score distribution

46.7%
Mean
49.9%
Median
16.1%
Std dev
28.9%
10th percentile
67.3%
90th percentile
18.3%
Min
18.3%77.3%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.92 for chat models, measured over 19 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Claude Sonnet 3.7provider: anthropic, defense: None, attack: important_instructions, date: 2025-02-24· AgentDojo results data docs77.3%
2Claude 3.5 Sonnetprovider: anthropic, defense: None, attack: important_instructions, date: 2024-11-15· AgentDojo results data docs72.5%
3GPT 4oprovider: openai, defense: None, attack: injecagent, date: 2024-06-05· AgentDojo results data docs68.5%
4GPT 4oprovider: openai, defense: None, attack: direct, date: 2024-06-05· AgentDojo results data docs67.3%
5GPT 4oprovider: openai, defense: repeat_user_prompt, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs67.3%
6GPT 4oprovider: openai, defense: None, attack: ignore_previous, date: 2024-06-05· AgentDojo results data docs66.8%
7GPT 4oprovider: openai, defense: None, attack: tool_knowledge, date: 2024-06-05· AgentDojo results data docs57.7%
8GPT 4oprovider: openai, defense: tool_filter, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs56.3%
9GPT 4oprovider: openai, defense: spotlighting_with_delimiting, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs55.6%
10GPT-4 Turboprovider: openai, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs54.1%
11Claude 3 Opusprovider: anthropic, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs52.5%
12Claude 3.5 Sonnetprovider: anthropic, defense: None, attack: important_instructions, date: 2024-06-20· AgentDojo results data docs51.2%
13GPT 4oprovider: openai, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs50.1%
14GPT-4o miniprovider: openai, defense: None, attack: important_instructions, date: 2024-07-19· AgentDojo results data docs49.9%
15gemini-1.5-pro-002provider: google, defense: None, attack: important_instructions, date: 2024-11-04· AgentDojo results data docs47.1%
16GPT-4-0125-previewprovider: openai, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs40.7%
17Gemini 2.0 Flash Experimentalprovider: google, defense: None, attack: important_instructions, date: 2024-12-11· AgentDojo results data docs39.9%
18Gemini 2.0 Flashprovider: google, defense: None, attack: important_instructions, date: 2025-01-30· AgentDojo results data docs39.8%
19GPT-3.5 Turboprovider: openai, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs34.7%
20gemini-1.5-flash-001provider: google, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs34.2%
21Claude 3 Haikuprovider: anthropic, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs33.4%
22Claude 3 Sonnetprovider: anthropic, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs33.2%
23gemini-1.5-flash-002provider: google, defense: None, attack: important_instructions, date: 2024-11-05· AgentDojo results data docs32.4%
24Command R+provider: cohere, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs30.8%
25gemini-1.5-pro-001provider: google, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs28.9%
26Command R Plusprovider: cohere, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs25.1%
27GPT 4oprovider: openai, defense: transformers_pi_detector, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs21.1%
28Llama-3-70b-chat-hfprovider: together-prompting, defense: None, attack: important_instructions, date: 2021-09-05· AgentDojo results data docs18.3%

Data credits: AgentDojo. Copyright (c) 2024 Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. MIT License. Not a leaderboard: attack and defense coverage differs between models. (licence)