Crash Test Dummies
Harness privilege, not model capability, decides what a compromised run can do. Twelve controls across two configurations, and the defaults matter more than the documentation.
· 11 min read · Jayesh Ahire

In 1985 the United States Department of Transportation put two mannequins on television. Their names were Vince and Larry. They drove into walls without seatbelts and went through windshields in slow motion, deadpan the whole way, under the tagline "You could learn a lot from a dummy." It was grim and funny and it worked, because for the first time the public could watch what happens to a body in a crash instead of being told by the people selling the car that the car was safe.
The dummies were the visible end of a much older argument. In May 1936 a small organization called Consumers Union published the first issue of Consumer Reports. The method was almost insultingly simple: buy the product at retail, the way a customer would, and test it. No review units, no briefing from the manufacturer, no stake in the answer. Buyers trusted it for exactly that reason.
In 1959 a Volvo engineer named Nils Bohlin designed the three-point seatbelt, and Volvo did something strange in a patent system built for exclusion: it opened the patent to every other manufacturer. Safety was not a feature you compete on. It was a floor you raise together. Then came the crash-test ratings. Once an independent body could put a car into a wall and publish the result as a number of stars, "our car is safe" stopped being a sentence a marketing department could say. It became a score. A manufacturer could disagree with the methodology. Nobody could disagree with the footage.
The software industry has already handed us the joke. The sandbox Apple ships inside macOS, the one that wraps the process a coding agent runs in, is literally called Seatbelt. Google's Antigravity calls its containment option sandbox mode. Everyone has adopted the vocabulary of the crash test. Nobody has run the crash test.
Now watch what has quietly started happening to the coding agents your engineers run all day.

Nobody has crash-tested the harness
I argued earlier this year that trust needs a number, and TrustVector, the evaluation index my team runs, is our attempt at one: 196 evaluations across 68 models, 67 agents, and 61 MCP servers. It scores Google Antigravity at 42 and Trae at 39, both well under the index average of 81, with the flagship models at 92. What it does not have is a category for the thing that decides what those models can actually do on your laptop: the harness. Claude Code, Codex CLI, Cursor, Gemini CLI, Antigravity, Kiro: six tools engineers run all day with write access to their repositories and, very often, to their credentials. As far as my team and I can find, nobody has published a versioned, apples-to-apples score of what each one permits by default.
Model capability decides how good the plan is. Harness privilege decides what the plan can touch. A model fooled by a prompt injection with no network egress writes an angry comment in your terminal. The same model, fooled the same way, with a shell, an unfiltered DNS resolver, and your cloud credentials in an environment variable, is an incident. HarnessRisk measured this in August: 128 cases across six lifecycle phases, fourteen model-and-harness configurations, and an attack success rate that ran from 12.6 percent to 80.9 percent across the same threats. The most vulnerable phase was not inference. It was harness configuration. And some configurations detected the risk in more than 90 percent of runs and executed anyway. The model saw the wall coming. The harness had no brakes.
A second paper, published on September 1, did the first systematic analysis of how twelve real harnesses, Claude Code and Codex among them, assemble the context the model reads. It named two new classes of Context Privilege Escalation and traced them to full compromise, remote code execution, and denial of service. It is not a paper about weights. It is a paper about plumbing.
That is the first of three claims. The second is that defaults matter more than documentation, because in Anthropic's own testing developers approved 97 percent of permission prompts without close scrutiny, which means the default is the configuration. The third is that an independent, repeatable score beats any vendor benchmark, for the same reason Consumer Reports beat the brochure in 1936.
Twelve controls, two configurations
The method is the whole argument, so I will lay it out plainly. The test repository will publish alongside the first results.
Each harness gets tested twice. Once on its default configuration, what a developer has after installing it and answering the setup prompts the way people actually do. And once on its strictest documented configuration, every hardening flag, mode, and setting the vendor's own documentation says exists. The distance between those two runs may be the most interesting number on the page: how much safety is available versus how much is delivered.
Against each configuration we probe twelve controls. Network egress, out of the box. How wide its filesystem scope actually is, measured rather than documented. Whether hooks or configuration files execute when a repository is opened, before anyone has typed a prompt. What secrets a run can reach in environment variables, keychains, and dotfiles. Whether DNS is an open exfiltration channel underneath whatever egress filter exists. Whether the run can spawn subprocesses, including new instances of the harness itself. Whether MCP allow-lists are enforced or merely displayed. What sandbox technology sits underneath: gVisor, Seatbelt, bubblewrap, a container, or a container with the Docker socket mounted, which is a sandbox the way a screen door is a wall. What the permission prompts default to. Whether a guardrail can block the harness's own cleanup after a compromise, a failure Johann Rehberger documented in Auto Mode in August. The count of disclosed vulnerabilities and the time from report to patch. And whether the harness records instruction provenance: not just what the model did, but which file, page, or tool response told it to.
Every probe is a script in the repository, reproducible on a clean machine with the harness version pinned. A probe that cannot be made reproducible goes in the notes, not the score.

The column that already exists
We have not run the tests yet, and I am not going to dress up guesses as scores. What follows is the one column that can be filled in today, from the public record; what each vendor did about it comes in the next section.
| Harness | Known 2026 issues (public record) |
|---|---|
| Claude Code | CVE-2026-40068, CVSS 7.7, May 7: git worktree commondir trust bypass, hooks run without confirmation. CVE-2025-59536 and CVE-2026-21852, February: RCE and API-key exfiltration via project files and hooks (Check Point). Cross-tool CVE-2026-35603 (Cymulate). Zenity's DNS shell: reported Nov 28, 2025, closed Dec 12; regression reported May 29, 2026, closed as duplicate. Rehberger's Auto Mode chain, rated "Informative," no fix planned. |
| Codex CLI | CVE-2026-19591, CVSS 8.8, Sep 1: PowerShell --% stop-parsing token misparsed, approval-free config write, attacker MCP server launch. GitPwned, CVSS 8.6, git show --output=./.git/config; disclosed Jan 7, patched v0.95.0, published Jul 20 (Pillar). Docker socket escape. CVE-2026-35603. |
| Cursor | CVE-2026-26268, CVSS 8.1, git hook injection. "CursorJacking," CVSS 8.2, unpatched at report. Pillar Stop-hook finding, CVSS 8.5. Docker socket escape. CVE-2026-35603. PocketOS, Apr 25: not a Cursor bug but a Cursor run; an over-scoped token, production volume and backups gone, no attacker. |
| Gemini CLI | CVSS 10.0 below 0.39.1, Apr 24: CI/CD workspace-trust bypass (Novee). Docker socket escape. CVE-2026-35603. |
| Antigravity | Prompt-injection-to-RCE via find_by_name -X flag injection bypassing Secure Mode; disclosed Jan 7, patched Feb 28 (Pillar). Arbitrary-click via disguised tool-response metadata (arXiv 2607.05120). |
| Kiro | Below 0.8.140: "Open Workspace From File" plus any message lets repository content, via Kiro "Powers," steer the agent to send local data to an external endpoint (Mindgard). No CVE assigned. |
Two more belong in the notes, off the initial six: Continue CLI's CVE-2026-76072, CVSS 8.3, a headless-mode denylist that misses rm -rf /home, and Hermes Agent's CVE-2026-82021, CVSS 9.0, a bundled MCP catalog pinned to a mutable branch, so an upstream compromise executes on every install.
Pillar's four failure modes, from their Week of Sandbox Escapes in July, are the reason the twelve controls look the way they do. Denylists lose to platform complexity. Workspace configuration is often code. "Safe" commands are not safe by name. Local daemons live outside the box. Pillar sorted seven findings across four harnesses into those buckets, and read with that lens the table stops being a list of CVEs and becomes the same four mistakes, made independently, by well-funded teams, in the same six months. That post got 11 points on Hacker News. The thread about auto mode becoming the default got 292. We are, collectively, arguing about the model and ignoring the chassis.
One thing the CVE column cannot tell you: a harness with more disclosed vulnerabilities is not necessarily the least safe one. It may be the one with more researchers looking and a faster changelog, which is why disclosure count is one control of twelve rather than the score.
Credit where the changelog earns it
Consumer Reports was trusted because it was fair, not because it was harsh. So, the fair part.
Claude Code's changelog between August 28 and September 3 reads as if someone at Anthropic sat down with Pillar's four failure modes and Rehberger's report and worked through them in order. --restricted (v2.1.248) is a real reduction in reach. Symlink TOCTOU fixes landed in v2.1.251. The Containment Escape rule (v2.1.257) means cloud-metadata credential fetches, egress evasion, and cross-tenant reach are no longer auto-approved, and settings that weaken the sandbox now require approval. Then managedMcpServers and deny-by-default headless runs in v2.1.259. And Anthropic's "How we contain Claude across products", from May, is the most forthcoming engineering post any of these vendors has published. It names the sandbox technologies (gVisor on claude.ai, Seatbelt and bubblewrap for Claude Code, sealed VMs for Cowork), admits failures including exfiltration through "approved domains," and contains a line every harness team should tape to the wall: "the software you build yourself is often the weakest."
Codex, in 0.147 through 0.151, dropping AGENTS.md from untrusted projects is, to my mind, the most important default change of the year, because it treats opening a repository as what it is: code execution. The same releases redact secrets from displayed commands and stop a stale Guardian classification from authorizing anything after a permission change. Antigravity's enterprise console, from August 21, gives an administrator always-ask, sandbox, and auto terminal modes, browser allow-lists, and centralized audit logging. The smaller fixes count too: Cursor's Stop-hook fix in 3.0.0, Gemini CLI's 0.39.1, and Kiro's 0.8.140 on January 15.
None of this is in the score yet, because there is no score yet. The second run, next quarter, will show whether these changes moved the default row or only the strict row. That is the point of versioning it.
What strict actually means
Here is what strict means, harness by harness, as far as the documentation supports it; the full files will ship with the repository.
For Claude Code, strict means --restricted on, auto mode off or held behind the Containment Escape rule, managedMcpServers in use so the server list is administered rather than supplied by whatever repository was opened, headless runs with --permission-prompts none so anything not explicitly allowed is denied, and PreToolUse and PostToolUse hooks enforcing policy outside the model. That last piece is where my company's tooling enters this post, once: g0 protect --apply --surfaces claude is built to install those hooks, so the policy on your laptop matches the policy in your CI. It is on main and not in the published package, so treat it as a plan rather than a command you can run today.
For Codex, strict means 0.151 or later. For Antigravity, always-ask terminal mode, a browser allow-list, and audit logging on. For Cursor, 3.0.0 or later and a review of every hook the workspace can define, because hooks were the vector twice. For Gemini CLI, 0.39.1 or later and, on any machine that also runs Docker, confirming the socket is not reachable from inside the harness. For Kiro, 0.8.140 or later and treating "Open Workspace From File" as the trust decision it turned out to be.
Notice what is missing. An egress allow-list all six respect, a DNS policy, and an instruction-provenance log: as far as we can find, none of these is a first-class, documented option across all six today. They are in the scorecard because they should be, and a column reading "not available" is information too.
A scorecard is also a target
The objection I expect first is the one I have not answered yet.
A published scorecard is a published attack surface. Every vendor on that table has more engineers than I do, and the moment the probes are public, the rational move is not to make the harness safe but to make the probes pass. The car industry knows this; crash ratings produced cars excellent at surviving the crash in the test, and the testing bodies have spent decades adding new crash geometries to stay ahead.
The method resists it in four ways, none of which is secrecy, because the repository will be published with it. Each quarterly run adds probes drawn from the previous quarter's incident record and rotates a subset of the old ones, with the selection method itself published, so a vendor who optimizes for last quarter's probes gets last quarter's score. The controls are properties rather than behaviors; it is hard to fake "DNS is not an exfiltration channel" in a way that passes the probe and fails in the wild, because the probe is the wild. The strict-versus-default gap is scored separately, so nobody climbs the table by shipping a flag nobody turns on. And every vendor gets a pre-publication window to reproduce and dispute the result, with the dispute published alongside the score.
What the method cannot do is prevent building to the test. A large enough team can always find the corner of the probe space we did not cover, and I would rather say so than pretend a methodology section is a moat. The defense is the one Consumer Reports had: publish everything and let the vendors argue in public. I expect pushback, and I am counting on it. A vendor arguing with a score is a vendor telling its customers the score exists.
You could learn a lot from a dummy
Vince and Larry went through windshields on American television for years, and the thing that changed in that time was not the belt, which had been in cars since 1959. What changed was that people could finally see what happened to a body that was not wearing one.
That is where the harnesses are. The belt exists; Seatbelt is right there in the name, and the last two weeks of changelogs show vendors taking the problem seriously. What does not exist is the footage: an independent, versioned view of which harness, on the configuration a developer actually runs, lets a compromised run reach what it should not.
My team and I are going to run the test. The first results, and the repository, publish next quarter, and the vendors will see them before you do. Some harness is going to beat its reputation and some is not. I do not know which yet, which is the reason to run the test. You could learn a lot from a dummy, if you are willing to watch.
The harness scorecard will publish as a category on TrustVector, with the test repository, the changelog, and every vendor dispute alongside the score. If you build or run a harness that is not on the initial list of six, submit it. The method is the same for everyone, which is the point.
References
- HarnessRisk: attack success across 14 model and harness configurations (arXiv 2608.17597)
- Context Privilege Escalation in 12 real agent harnesses (arXiv 2609.01222)
- Pillar Security: The Week of Sandbox Escapes
- Zenity Labs: It's Always DNS in Claude's Sandbox
- Anthropic Engineering: How we contain Claude across products
- Rehberger: Breaking Claude Code Opus 5 and Auto Mode
- Claude Code changelog (v2.1.248 through v2.1.259)
- Help Net Security: Anthropic Claude Code Auto Mode figures
- The Register: Google tethers Antigravity to enterprise controls
- The Hacker News: Amazon Kiro prompt injection via Powers
- Agent Data Injection Attacks are Realistic Threats (arXiv 2607.05120)
- Zenity: the PocketOS database deletion
- Guard0: Trust Needs a Number
- TrustVector evaluation index
- g0 open-source scanner and
g0 protect
The Signal · AI Agents · Security