A Practical Guide to Adversarial Red-Teaming

What red-teaming is

Every serious building gets tested by someone whose job is to break in. Not a burglar - a professional you hired, who tries the locks, tailgates through the side door, and hands you a report on everything that worked. Red-teaming a model is the same deal. You take the adversary's seat, probe the model for the ways it can be made to misbehave, and write up what got through, so the holes get fixed before a real user - or a real attacker - finds them.

The behaviors you are hunting are the ones a model is meant to refuse: handing over dangerous instructions, producing hateful or harassing content, leaking private data, being talked into helping with something it shouldn't. A model that handles polite questions flawlessly can still fold the moment someone gets creative. Red-teaming is the work of getting creative on purpose, in a controlled setting, so the failures show up on your terms instead of a stranger's.

The attacker's advantage

Here is the uncomfortable math of safety. To ship a safe model, the defender has to close every hole. To break it, an attacker has to find one. That asymmetry is why ordinary testing is not enough - confirming the model does the right thing on a thousand normal prompts tells you nothing about the one strange phrasing that slips past.

It is also why red-teaming is a craft rather than a checklist. The interesting failures live in the corners: an unusual framing, a switch of language, a request buried in a wall of context, a slow build across several messages. You are not verifying that the model works. You are assuming it can be broken and trying to prove it - a different mindset, and honestly a different kind of person.

Anatomy of an attack

Strip a red-team attempt down and it has three parts.

• A goal - the specific bad behavior you are trying to trigger. Not "break the model" but something concrete and testable, like "get it to reveal its hidden system prompt."
• A technique - the method you use to get there. This is the creative part, and most of this guide is about the families of technique that tend to work.
• A success test - a clear line for whether the attempt worked. Without one, red-teaming collapses into arguing over whether a borderline response "counts."

Keeping the three separate is what turns a fun afternoon of poking at a chatbot into data a lab can act on. The goal makes the attempt reproducible, the technique makes it teachable, and the success test makes it countable.

A field taxonomy of attacks

Attacks fall into a handful of families, and knowing them is how you cover the space instead of chasing whatever you thought of last. The industry checklist most teams start from is the OWASP Top 10 for LLM Applications; the families that matter most to a red-teamer are these:

• Jailbreaks - getting the model to ignore its own safety rules and produce content it would normally refuse.
• Prompt injection - smuggling instructions into content the model reads (a web page, a document, a tool result) so it follows the attacker instead of the user. The signature risk of any model that browses or uses tools.
• Harmful-content elicitation - coaxing out dangerous, hateful, or illegal material, often the payload a jailbreak is trying to unlock.
• Data extraction - pulling out what the model should keep private: personal data, another user's information, its own hidden instructions, or memorized training data.
• Capability failures - not a bypass but a flaw: confident falsehoods, biased outputs, unsafe advice offered willingly.

A real program works through all of these on purpose, because a model can be rock-solid against one family and paper-thin against the next.

How jailbreaks actually work

Jailbreaks are the family people picture first, and nearly all of them lean on the same move: change the context around a request until the model stops treating it as dangerous. A few recurring shapes, described as tactics rather than recipes:

• Roleplay and persona - wrapping the request in a character or story so the model answers "as" someone who does not share its rules.
• Hypothetical framing - presenting the ask as fiction, a thought experiment, or an "educational" scenario, so harmful content arrives dressed as something harmless.
• Obfuscation - hiding the real request in another language, an encoding, or unusual characters, so it slips past filters that only scan the surface.
• Authority and misdirection - posing as a developer or a safety tester, or burying the real ask under a mountain of distracting instructions.

The pattern worth internalizing is that these rarely work in one shot. The strongest attacks are multi-turn: the attacker warms the model up over several polite messages, sets a frame, and only escalates once the model is committed - a slow build that single-message safety checks miss entirely. When you red-team, test the conversation, not just the prompt.

By hand or by machine

There are two ways to generate attacks, and a good program uses both.

Manual red-teaming is humans being adversarial, ideally a diverse group - because the blind spots of a homogeneous team become the model's blind spots. People are unmatched at genuinely novel attacks, cultural and linguistic corners, and lateral thinking no script would try. The catch is that it is slow, expensive, and prone to getting repetitive as people tire.

Automated red-teaming hands the job to another model: you prompt or train an attacker model to generate thousands of test cases and score the target's replies with a classifier. It buys enormous coverage and repeatability for cheap, and it never gets bored. The catch is that it tends to rediscover the families a human already knows, and it needs a person to notice when the classifier is wrong. The two are complements, not rivals - humans find the new thing, automation blankets the known space and catches regressions.

Running a session

A session has a rhythm once you have the pieces. Pick a goal from the taxonomy. Try to hit it, working through the technique families and improvising as you go. And - the part beginners skip - log everything, the failures as well as the wins. A prompt that got refused today is the baseline for whether tomorrow's model is more or less robust; throw it away and you have lost half the value.

Each logged attempt should carry enough to reproduce it: the exact input, the full conversation if it ran multi-turn, the model and its settings, the response, and your verdict against the success test. Treat it like lab notes, not a highlight reel. The point of a session is not a dramatic screenshot - it is a clean, structured record a lab can learn from.

Rating what you find

A raw list of "these worked" is not much use, because not every hole matters equally. A model that can be nudged into a mildly rude joke is a very different problem from one that hands over something genuinely dangerous. So you rate each finding, usually on two axes: how bad the outcome is if it happens, and how reachable it is - one obvious sentence, or a twenty-message expert campaign?

The two pull apart, which is the point. A catastrophic outcome that needs an implausible setup can rank below a moderate harm anyone can trigger on the first try. Rating against a shared, written scale - the same rubric discipline from the evaluation guide - is what lets a lab triage: fix the reachable-and-severe first, and tell the difference between a papercut and a fire.

Coverage and reproducibility

Two words separate a red-team program from an afternoon of poking.

Coverage means you worked the whole taxonomy deliberately, not just the attacks you enjoy. It is easy to spend a week on clever jailbreaks and never once test prompt injection. A coverage map - families down one axis, topics across the other - shows the corners you have not touched, and the empty cells are usually where the risk is hiding.

Reproducibility means someone else, handed your log, gets the same result. That is what turns red-teaming from anecdote into measurement, and it is what makes a fix checkable: if you cannot reliably reproduce an attack, you cannot tell whether the next version actually closed it or just got lucky.

Closing the loop

Finding a hole is only useful if it gets fixed, and this is where red-teaming quietly becomes the most valuable data work in the pipeline. A confirmed attack is a labeled example of exactly what the model must not do - the raw material the earlier guides describe. Successful attacks become demonstrations of the right refusal (SFT), or preference pairs where the safe answer beats the harmful one (RLHF), and the model gets tuned to close the gap.

Then you test again. Every confirmed attack becomes a regression test: rerun it against the next version to make sure the fix stuck and did not quietly come undone. A mature program is a loop - probe, find, patch, re-probe - and last quarter's logged attacks are the safety net that keeps old holes from reopening.

Working safely

Red-teaming is the one kind of AI data work where the job is to produce harmful outputs on purpose, and that carries real responsibility. A few ground rules separate professional red-teaming from recklessness.

Work inside an authorized scope. That means a sanctioned test on a system you are allowed to probe, not someone else's live service. Handle what you surface with care. A confirmed dangerous output is sensitive material - log it, route it through proper channels, and do not post it for laughs. Aim for the finding, not the payload. The point is to prove the model can be broken and show how, not to write the most damaging thing you can and set it loose. And take the human cost seriously. Sustained exposure to harmful content wears people down, so real programs rotate people and offer support. The mindset is a security researcher's, not a vandal's. You break things so they can be fixed, and you handle the pieces like they matter - because they do.

The record you build

Red-teaming is where a model's safety stops being a promise and becomes something you have actually tried to break. The work rewards a particular temperament - adversarial, systematic, careful with what it turns up - and it pays off only when it is disciplined: concrete goals, a covered taxonomy, reproducible logs, honest severity, and every finding fed back into a safer model and a regression test. Do it well and you are not just catching failures. You are building the record that proves, attack by attack, that the next version is harder to break than the last.

Additional resources

• Red-Teaming Large Language Models
  A readable introduction to red-teaming: what it is, how attacks work, and why it matters.
• Adversarial Attacks on LLMs
  A thorough, well-organized survey of jailbreaks and adversarial attacks from a leading safety researcher.