What is AI red teaming?
AI red teaming is structured adversarial testing of an AI system to find the ways it fails or can be misused before an attacker finds them. For agents it means sending crafted inputs through the channels the agent reads, such as tickets, documents and tool results, and checking which actions follow.
Also called agent red teaming, adversarial testing for AI. Last reviewed .
Why it matters
It is expected practice. NIST's Generative AI Profile includes red teaming among the actions for measuring generative AI risk, and Article 55 of the EU AI Act requires providers of general-purpose models with systemic risk to conduct and document adversarial testing.
MITRE ATLAS gives testers a shared catalogue of adversary tactics and techniques against AI systems, so a finding can be named the same way by the team that found it and the team that fixes it.
Testing a model and testing an agent are different jobs. A model test asks what the model will say. An agent test asks what the agent will do with its tools once it has read the attack.
Related terms
- Prompt injection
- Prompt injection is the attack most agent red-team tests begin with.
- Tool poisoning
- Tool poisoning is tested by changing what a tool tells the agent.
- Confused deputy
- Confused deputy chains are a standard agent test: can a low-privileged request reach high-privileged data.
- AI agent security
- AI agent security uses red teaming to verify that its controls hold.
How Guard0 handles it
Guard0 attacks agents through tickets, documents and tool calls, with tests that change course across turns. It runs against staging, through the API of a business platform, or in CI. Each finding carries the payload, every tool call, and its MITRE ATLAS and CWE references, and the same payload runs again after the fix.
Sources
- AI RMF Generative AI Profile (NIST AI 600-1). NIST.
- Regulation (EU) 2024/1689 (the EU AI Act). EUR-Lex.
- MITRE ATLAS. MITRE.