Red Teaming

Deliberately attacking your own system, the way a real adversary would, to find weaknesses before someone else does.

Last updated

AI & AutomationCloud & Systems

In plain English

A red team plays the bad guy on purpose. They try to break in, trick the system or make it misbehave, then report what worked so it can be fixed. With AI, that means trying to get a chatbot to say harmful things, leak private information or ignore its rules.

Example

Before launching a customer chatbot, a company has a small team spend two weeks trying to make it reveal other customers' orders and give dangerous advice. Each trick that works gets fixed before launch.

In short

Attack yourself first. Finding the weak spots before someone else does.

Why it matters

Red teaming is how companies find problems that normal testing misses, because it thinks like an attacker instead of a typical user. Major AI labs red team new models before release, and many AI safety rules and guidelines now expect it.

Common confusion

Red teaming vs. AI evals. AI evals measure how well a system does on a fixed set of tests, again and again. Red teaming is open-ended and creative, hunting for failures nobody has thought to test yet. What red teams find often becomes a new eval.

Sources

  1. NIST Computer Security Resource Center: red team - Glossary
  2. IBM: What is Red Teaming?