AI-generated code security
45% of it failed.
Veracode · 2025 GenAI Code Security Report · 100+ models · 80 tasksWhen Veracode tested code from more than 100 models, 45% of samples failed security tests, and newer, larger models did not close the gap. Your repos are not the exception. The question is how much, where, and what stops it at merge.
How many of last quarter's merged PRs had AI-written code, and how many got a security review someone would sign?
- Four failure layers, not one.
- A merge gate that checks all four.
- A receipt for every AI-authored change.
Already shipped · Open source · Evidence-rated
The merge gate is already written. It is free. Point it at your repo.
AI-generated code security is the practice of measuring and controlling the weaknesses AI coding tools introduce across four layers: the generated code, the dependencies it pulls in, the actions the agent takes while writing it, and the provenance record of who approved the change before it shipped.
Last updated Tested against AI SAFE² v3.1 · safe2 CLI 1.1.0 · Gateway v3.0 · NEXUS v0.6By Vincent Sullivan
- +def get_invoice(request, invoice_id):
- +q = f"SELECT * FROM invoices WHERE id = {invoice_id}"CWE-89 · SQL from input
- +return db.execute(q).fetchone()No ownership check
- +STRIPE_KEY = "sk_live_51H••••••••"Hardcoded secret
- +import fastjwt_verify2Package not on registry
- +# generated by agent session 7c1e, no approver on recordNo provenance
Four layers: every issue stopped or sent to a human before merge.
Scanner only: 2 of 6 issues caught. The rest merge.
Lab replay
Can you trust AI-generated code in production?
Built for engineering and AppSec teams merging AI-written code into production
Disclosure: the lab replay on this page is CSI's own testing on a sample workload, not an independent benchmark.
02 · Where you are
Three questions about AI code in your repos
Tap the answer closest to true today.
Which repos take AI-assisted changes?
Tracked is the hard part. Next is measuring what those changes carry. Use the estimator on those repos only.
Then the published rates apply to most of your codebase. The estimator turns your merge volume into a planning range.
Provenance is layer four. Without it you cannot scope a review. Start with the provenance record.
What runs at merge today?
Human review is the most expensive layer you have. Put automated gates in front of it so reviewers see only what matters.
That covers the code layer. It does not see a package that does not exist or an install the agent ran. See the four layers.
Peer-reviewed testing found Copilot code review frequently missed critical flaws. Use it, don't rely on it. Evidence is in section 04.
Does anyone check that an added package exists?
Good. Check the install path too. The agent can install outside CI. That is test 3.
A plausible name passes an eyeball. An attacker only has to register it first. Read the dependency path.
Then dependency confusion is open. Block unknown packages at install, not at review. See the control matrix.
03 · What changes
Why does AI-generated code need more than a code scanner?
AI-generated code security spans four layers. Traditional AppSec tools were built for the first one.
| Layer | What goes wrong | Traditional tool | What it misses |
|---|---|---|---|
| Generated code | Injection, missing authorization, hardcoded secrets, weak crypto | SAST, secret scanning | Authorization logic that compiles and looks correct |
| Dependencies | A suggested package that does not exist, or a typo of one that does | SCA, after install | A name that attackers registered yesterday |
| Agent execution | The agent installs, downloads or builds outside CI while writing the code | None at merge | Everything that happened on the laptop |
| Provenance | No record of which tool wrote it or which human approved it | Commit author | The author is the developer, not the agent |
04 · The evidence
What the research says about AI code security
of code samples from more than 100 models failed security tests across 80 tasks in Java, Python, C# and JavaScript. Newer and larger models did not improve security.
Veracode · 2025 · 100+ modelsof Python snippets and 24.2% of JavaScript snippets carried security weaknesses, across 733 AI-generated snippets found in real GitHub projects.
Fu et al. · ACM TOSEM 2025 · 733 snippetsof Copilot-completed programs were vulnerable in controlled, security-relevant scenarios in the earlier Pearce et al. study.
Pearce et al. · CACMGitHub Copilot code review frequently failed to flag critical SQL injection, XSS and insecure deserialization, and mostly commented on low-severity issues.
Amro and Alalfi · PMLR 318 · 2026of developers trust AI output, while 84% use or plan to use AI tools, in Stack Overflow's 2025 survey.
Stack Overflow · 2025 survey05 · Abuse paths
How does AI-written code become an incident?
If 30 to 45 percent of that code carries a weakness, where is it right now, and who owns it?
Plausible but injectable
- AssetCustomer records behind an API
- ActorAny caller of the endpoint
- PathA query built from input, approved because the code reads cleanly
- ResultData read across tenants
The package that did not exist
- AssetYour build and every machine that runs it
- ActorWhoever registers the suggested name first
- PathThe model invents a package; the agent or CI installs it
- ResultAttacker code in your build pipeline
The binary nobody scanned
- AssetThe developer endpoint
- ActorA compromised dependency or download
- PathThe agent builds or downloads an executable and runs it locally
- ResultUnknown code executing with the developer's rights
06 · Architecture
What does a four-layer merge gate look like?
Models propose. Infrastructure decides. Evidence proves. Each layer gets its own check, and the gate decides before merge.
- Code checkSAST plus the 64-rule scanner, with SARIF into the PR.
- Dependency checkEvery new package must exist, be pinned and come from an approved index.
- Execution recordWhat the agent ran while writing the change, with verdicts.
- Provenance recordWhich tool authored it, which human approved it, under which policy.
- Policy gate before mergeAllow, deny or escalate.
safe2 gate --fail-undersets the bar.
The layer nobody scans
The binaries your agent builds or downloads are unknown code.
Warden kernel-level containment runs unknown executables in isolation by default, so a bad build artifact never touches the host.
The merge gate itself is a policy gate before merge, written as code with tests.
07 · Controls
Which controls stop AI-generated code risk, mapped to AI SAFE²?
| Job | Control | AI SAFE² anchor | Shipped artifact |
|---|---|---|---|
| Prevent | Block unknown or unpinned packages at install, in CI and in the agent's shell. | P1 Sanitize and Isolate | cursor-sovereign-runtime |
| Constrain | Merge gate with a score threshold. AI-authored changes cannot self-approve. | CP.4 Agentic Control Plane Governance | safe2 gate |
| Detect | Static rules for injection, secrets and agent-specific patterns, reported as SARIF. | P4 Engage and Monitor | scanner |
| Respond | Escalate authorization changes and new external calls to a named reviewer. | CP.10 HEAR Doctrine | developer-workflow |
| Recover | Trace every AI-authored change so a bad pattern can be found and reverted repo-wide. | P3 Fail-Safe and Recovery | Provenance record |
| Prove | Decision record per PR: tool, approver, policy version, gate score. | P2 Audit and Inventory | safe2 evidence |
08 · How to test
How do you test your merge gate against AI-generated code?
Seed known-bad changes and see what merges. With positive and negative cases and kept results, each test rates E3.
1. Seeded injection E3
- Open a PR with a query built from request input.
- Open a second PR with the parameterized version.
Pass
The first is blocked, the second passes.Fail
Both pass, or both are blocked.2. Package that does not exist E3
- Add an import of a package name that is not on your registry.
- Run CI and the agent's install step.
Pass
Install is refused in both places, with the reason recorded.Fail
Either one attempts to fetch the name.3. Unpinned install in the agent session E3
- Ask the agent to install a package from a public index without a version.
Pass
Denied by policy, and the denial appears on the PR record.Fail
The install runs and nothing at merge knows it happened.4. Missing approver E3
- Open an agent-authored PR with no human approver recorded.
Pass
The PR holds until a named human decides.Fail
It merges on green checks alone.5. Secret in the diff E3
- Commit a test key in the format of your payment provider.
Pass
Blocked before push or at merge, and the key is flagged for rotation.Fail
It reaches the default branch.09 · Proof in public
We did the work in public. Check it.
Every number below links to the file that proves it. Fork it, run it, try to break it.
README.md
19MCP profile controlsCP.5.MCP, MCP-1 to MCP-19, aligned to MCP 2026-07-28.00-cross-pillar/cp5_mcp_server_security.md
64scanner detection rulesJSON and SARIF output, fail-under gate in CI.scanner/README.md
22governed examplesClaude Code, Codex, Cursor, LangGraph, CrewAI, AutoGen, OpenClaw and more.examples/README.md
189NEXUS-A2A tests in the merge gateAgent-to-agent governance with OPA policies. Every PR must pass all of them.NEXUS/CONTRIBUTING.md
E0-E5evidence classesApplied to our own claims, in public, under Evidence Assurance v1.0.AISM/EVIDENCE-ASSURANCE.md
32frameworks mappedNIST AI RMF, ISO/IEC 42001, EU AI Act, OWASP and more, in the control dataset.safe2/data/ai-safe2-controls-v3.0.json
$0to read, run, fork or auditMIT and CC-BY-SA. NEXUS ships under Apache 2.0. No gate on the code.LICENSE
They define the risk. We ship the enforcement.
These bodies are the mapping targets, not the competition. AI SAFE² maps to them, then ships the code they leave to you.
| Who | Defines the risk | Maps to compliance | Open enforcement code for named agent stacks | Rates its own evidence in public | Published bypass process |
|---|---|---|---|---|---|
| OWASP GenAI Security Project | Yes | Partial | Not its scope | No | No |
| NIST AI RMF | Yes | Yes, it is the target | Not its scope | No | No |
| MITRE ATLAS | Yes | Partial | Not its scope | No | No |
| CSA MAESTRO and Agentic Control Plane | Yes | Partial | Not its scope | No | No |
| Runtime security vendors | Partial | Partial | Closed, own product only | Vendor reports | Some, through bug bounties |
| AI SAFE² | Yes, 161 controls | Yes, 32 frameworks | Yes, 22 examples, CLI, gateway | Yes, E0-E5 | Yes, Challenge Lab |
As of October 2026 · Got one wrong? Send the link. We correct it the same week.
Challenge Lab · C0 to C5 claim maturity
Break it.
Preregistered scenarios. Explicit invalidation criteria. A valid bypass or a grader defect is a high-value contribution, and it goes into the public record.
Or run it on yours, now
$ pip install "ai-safe2[all]==1.1.0" $ safe2 doctor . $ safe2 gate project . --fail-under 70 $ safe2 scan mcp https://your-host/mcp
Four commands. No sales call. If the gate fails, you just learned something your current vendor did not tell you.
Install line from MIGRATION.md. Commands from the safe2 CLI reference.
10 · Tools
Which tool categories cover which layer?
Buy for coverage, not for features. Every layer needs an owner.
| Tool category | Code | Dependencies | Agent execution | Provenance |
|---|---|---|---|---|
| SAST | Yes | No | No | No |
| SCA and dependency scanning | No | After install | No | No |
| Secret scanning | Secrets only | No | No | No |
| AI code review | Partial, misses critical flaws in testing | No | No | No |
| Coding agent native controls | No | If sandboxed | Per tool | No |
| AI SAFE² scanner, gate and decision record | Yes, 64 rules | Yes, at gate | Yes, with a runtime package | Yes, record starts at hold |
CSI analysis · October 2026
11 · Start here
Estimate your AI code debt
Move the slider to your merge volume. The range uses two published rates.
Free · AI Code Debt Estimator
Low · 29.5%
177
High · 45%
270
Changes per quarter likely carrying a weakness. Low rate: Fu et al., Python snippets in real GitHub projects. High rate: Veracode, 100+ models. Your real number comes from your repos.
Productized review
AI-Generated Code Security Assessment
- Scope
- The repos that take AI-assisted changes, and the workflow around them.
- Deliverables
- An inventory of AI-authored changes, a sampled review across all four layers, a check of your merge-time controls, and a remediation backlog ranked by consequence.
- Timeline
- Agreed in writing at scoping, before work starts.
- Risk reversal
- Sampling method disclosed up front. If the sample cannot support a finding, the report says so.
- AI Code Debt EstimatorFree, result on this page
- Free 8-minute AI governance assessmentScore your AI risk
- Strategic Certainty SessionA fixed-scope working session
- AI-Generated Code Security AssessmentCustom quote
- Secure AI Development ArchitectureEnterprise program
Would the real number for your repos change what you ship next quarter?
The code is already merged. Measuring it costs less than finding it in production.
13 · FAQ
Straight answers on AI-generated code security
Is 45% my number?
It is Veracode's, across more than 100 models on 80 defined tasks. Fu et al. found 29.5% of Python and 24.2% of JavaScript snippets with weaknesses in real GitHub projects. Different methods, both public. Your number comes from your repos, and we disclose the sample we use to get it.
Why a range in the estimator?
Because two credible studies measured different things. A planning range is honest. A single number from a benchmark applied to your code would not be.
Doesn't our SAST cover this?
It covers the code layer. It does not catch a package that does not exist yet, an install the agent ran from an unpinned index, or a change nobody approved. That is three of four layers.
Can AI review the AI's code?
Use it. Don't rely on it. Peer-reviewed testing published in 2026 found Copilot code review frequently missed critical SQL injection, XSS and insecure deserialization flaws.
Is the pull request in the replay real?
It is a lab replay built from the weakness classes in the cited studies. The key and package name are fake. The scanner-only result shows what a code-layer tool sees, and the four-layer result shows the gate described on this page.
What does Warden have to do with code security?
Agents build and download executables while they work. Those binaries are unknown code on the developer endpoint. Warden contains unknown executables at the kernel by default, which covers the one layer a merge gate never sees.
14 · Sources and method
Sources, method and update log
External sources
- Veracode · 2025 GenAI Code Security Report · 100+ models · 80 tasks
- Fu et al. · ACM TOSEM 2025 · 733 snippets
- Pearce et al. · CACM
- Amro and Alalfi · PMLR 318 · 2026
- Stack Overflow · 2025 survey
CSI files cited
Method
External statistics name the publisher, year and sample. CSI claims link the repository file behind them and carry a rating under AI SAFE² Evidence Assurance v1.0: E1 for a specification, E2 for shipped code, E3 for a defined test with positive and negative cases. Vendor behavior comes from each vendor's current documentation, opened on the date shown.
Update log
- : page published. Vendor controls, repository counts and install line verified the same day.
Tested versions
AI SAFE² v3.1 · CP.5.MCP MCP-1 to MCP-19 · safe2 CLI 1.1.0 · Gateway v3.0 · NEXUS v0.6 · Evidence Assurance v1.0
Vincent Sullivan, founder and CEO of Cyber Strategy Institute. Twenty-two years as a U.S. Air Force cyber officer, finishing as Technical Director at U.S. Cyber Command. Author of the AI SAFE² framework.