AI Agent Security
Evaluating open-weight LLMs for their resistance to prompt injection attacks, using ETH Zurich's AgentDojo framework to test models against attack vectors in untrusted environments.
Overview
This project is a security evaluation of open-weight language models when they are used as tool-using agents. Unlike a plain chatbot, an agent can act: it reads files, queries databases, sends messages, and moves money on the user's behalf.
That capability is exactly what makes it useful — and what makes it attackable. Anything an agent reads through a tool can contain instructions, and because a language model processes instructions and data over the same channel, it has no reliable way to tell the two apart. An attacker who controls even a single piece of environment text can try to hijack the agent's behaviour. This is indirect prompt injection (IPI), and it is the threat this project studies.
The practical question: how well do medium-sized, open-weight models resist prompt injection, and does being good at a task mean being safe against it? I built an evaluation around AgentDojo (ETH Zürich) and ran a set of open-weight models against several families of attacks in a simulated banking environment.
At a glance
Motivation
Open-weight models get far less attention
Most public security evaluations focus on large, closed frontier models. But the medium-sized, open-weight models that anyone can use — self-hosted, or served through an inference provider — are far less studied, and their robustness properties are not well understood.
- →The failure mode is subtle — prompt injection exploits the absence of a formal instruction/data boundary, so there is no 100% patch
- →Agents raise the stakes — a manipulated chatbot says something wrong; a manipulated agent does something wrong
- →Capability and security may not track each other — an empirical question worth measuring, not assuming
Method
Task + injection, on one poisoned environment
The injected string replaces a field's legitimate content rather than being appended, so the malicious text looks like ordinary environment data. Each experiment runs a user task and an injection task against the same poisoned environment and checks two independent, state-based outcomes.
- →Three attack families: a direct instruction injection (InjecAgent), a template-based social-engineering injection (important_instructions), and a denial-of-service injection (dos)
- →Ten repeated runs per (model, attack), reported as a mean with a 95% confidence interval
- →Planned extension: a domain-specific insurance environment with an industry partner
Findings
Plausibility, not loudness, is what breaks them
Direct, un-camouflaged instructions are broadly resisted
Telling a model outright to “ignore all previous instructions” was the least effective attack across the board. Raw instruction override is not where the danger lies.
Plausibility is what breaks them
When the same malicious goal was wrapped in a legitimate-looking message that impersonated the user and addressed the model by name, some models fell for it constantly while others barely did at all — a 40% difference in attack success rate despite near-identical task performance. Capability does not predict robustness.
There are (at least) two distinct failure modes
Vulnerability to social engineering and to denial-of-service did not move together: a model easily talked into doing something unwanted was not necessarily the one that could be talked into stopping.
Denial-of-service cuts utility, not just security
The stop-instruction attack reduced how often agents finished their legitimate work — a reminder that prompt injection can be an availability problem too.
One model failed in an unexpected way
A smaller model stopped calling tools altogether under attack, so the run was abandoned — itself a finding about how fragile tool-use can be under adversarial pressure.
Takeaway
Challenges
What the evaluation taught me
Utility and security are in tension
A robust model completes the legitimate task and ignores the injection; a manipulated one may abandon the task, perform the attacker's task, or both. Untangling “failed the task” from “was attacked” required care about what each metric actually captures.
Nothing reproduces exactly
Generation is stochastic, so results are averages over many runs with a confidence interval — and single-run numbers should never be over-interpreted.
The injection mechanism matters
Replacing environment content rather than appending can subtly change how the legitimate task reads — a measurement detail that is easy to overlook.
Baselines first
Choosing no defense as the starting point kept the setup small and interpretable, and made the cross-model comparison meaningful before adding complexity.