Skip to content
The State of Exposed AgentsRead the report
Guard0

What is AI red teaming?

AI red teaming is structured adversarial testing of an AI system to find the ways it fails or can be misused before an attacker finds them. For agents it means sending crafted inputs through the channels the agent reads, such as tickets, documents and tool results, and checking which actions follow.

Also called agent red teaming, adversarial testing for AI. Last reviewed .

Why it matters

It is expected practice. NIST's Generative AI Profile includes red teaming among the actions for measuring generative AI risk, and Article 55 of the EU AI Act requires providers of general-purpose models with systemic risk to conduct and document adversarial testing.

MITRE ATLAS gives testers a shared catalogue of adversary tactics and techniques against AI systems, so a finding can be named the same way by the team that found it and the team that fixes it.

Testing a model and testing an agent are different jobs. A model test asks what the model will say. An agent test asks what the agent will do with its tools once it has read the attack.

Prompt injection
Prompt injection is the attack most agent red-team tests begin with.
Tool poisoning
Tool poisoning is tested by changing what a tool tells the agent.
Confused deputy
Confused deputy chains are a standard agent test: can a low-privileged request reach high-privileged data.
AI agent security
AI agent security uses red teaming to verify that its controls hold.

How Guard0 handles it

Guard0 attacks agents through tickets, documents and tool calls, with tests that change course across turns. It runs against staging, through the API of a business platform, or in CI. Each finding carries the payload, every tool call, and its MITRE ATLAS and CWE references, and the same payload runs again after the fix.

AI red teaming with Guard0

Sources

  1. AI RMF Generative AI Profile (NIST AI 600-1). NIST.
  2. Regulation (EU) 2024/1689 (the EU AI Act). EUR-Lex.
  3. MITRE ATLAS. MITRE.

Find the agents in your own company.

Start free: discover the AI on three people's machines, register one AI system and connect one repository or AWS account.