AI Coding Agents, Code Review, and Testing: Where Verification Moves

Writing code stopped being the bottleneck. Reviewing and integrating it became one.

That inversion is the whole story of AI coding agents in professional software work, and most teams are still running a review process designed for the opposite world. OpenAI reports that weekly pull request volume inside its own repositories more than doubled since Q4 as coding agents took on larger changes over longer horizons. Google's 2025 DORA report, drawn from nearly 5,000 technology professionals, found that 90% of developers now use AI at work and more than 80% say it made them more productive — but AI adoption showed a positive relationship with delivery throughput and a negative one with delivery stability. More output, more breakage.

Both figures are company or research claims, not field-wide law. But they point at the same mechanism, and the mechanism is what matters: generation got cheap, verification did not. Review and test design are now the governing constraint on how fast a team can actually ship.

The Bottleneck Moved From Writing to Verifying

For most of software history, review throughput was implicitly sized to human writing speed. A developer could produce a handful of meaningful changes a day, so a reviewer could keep up by reading carefully and asking questions in the PR thread. The process was never designed for volume. It was designed for scarcity.

Remove the scarcity and the process does not degrade gracefully. It queues.

The reason is an asymmetry that has nothing to do with how smart any model is. Producing a plausible diff is cheap: the agent searches broadly, generates tokens, and stops when the output looks coherent. Falsifying that diff is a different problem. It requires forming a specific hypothesis about how the change could be wrong, then running a check that would actually fail if the hypothesis were true. That is targeted work, and it does not get cheaper just because the code appeared quickly.

So the failure mode of the old model is not that reviewers get lazy. It is that review becomes a queue with no backpressure, and the queue becomes the release schedule.

I have watched this pattern in my own work across roughly twenty years of building software: whenever a step in a pipeline gets ten times faster, the constraint does not disappear. It relocates, usually somewhere less visible. Here it relocates into review, testing, and release control — the parts of the workflow that were always doing the real quality work but were never the headline.

The thesis of this article is blunt: verification is a design problem, not a tooling purchase. Buying an AI reviewer without redesigning what your team checks, who owns test intent, and how releases gate will move the queue, not shorten it.

What Actually Changed in the Diff

To predict where your own process will break, you need to see what is different about an agent-generated change compared to a human one.

Three things shift at once.

Batch size and horizon. Agents take on larger changes and work over longer time windows than a human typically would in one sitting. A single PR may touch a dozen files across several services. The reviewer's mental model, which used to be built incrementally across small diffs, now has to be assembled from a large one.

Missing authorial reasoning. A human diff carries implicit context: the author knows why they chose this approach, what they tried first, and which edge case they deliberately deferred. An agent diff carries the result but not the reasoning, unless someone writes it down. OpenAI's own guidance names this directly — a diff can look completely reasonable and still break an older client or cross a boundary the author did not know about. Someone has to remember that context and share it while the author can still act on it.

Context that used to travel verbally. In a small team, half of review context lives in hallway conversations and shared memory. When changes arrive faster and from a non-human author, that informal channel stops scaling. Context that is not encoded in the repository is effectively lost.

Grant the narrow case: for small, well-scoped changes with low blast radius — a typo fix, a dependency bump, a localized refactor — the old review model still works fine. The problem is not that every change is now hard. It is that the fraction of changes that need real scrutiny has grown faster than the capacity to give it.

Verification Is Cheaper Than Generation

Here is the asymmetry that makes automated review viable at all, and it is worth stating precisely because it is easy to misread as "AI can check AI."

Generating a correct code change often requires broad search: the model has to consider many possible implementations and pick one that satisfies the spec. Falsifying a proposed change usually needs only targeted hypothesis generation and checks. You do not have to search the whole space of correct programs. You have to find one specific way this program is wrong.

OpenAI's alignment team reports the concrete numbers from its deployed code review agent: it comments on 36% of pull requests that were entirely generated by its cloud agent, and 46% of those comments result in the author making a code change. On human-generated PRs, 53% of comments lead to a change. Those are vendor-reported figures from one system, not a benchmark across the field. But the shape is instructive: the reviewer finds something worth acting on in a meaningful fraction of agent-generated changes, and the comments are actionable often enough that authors act on them.

The design priority is the part most teams get wrong. Training-time verification and deployed review look similar and solve opposite problems. Training-time checks want high recall — catch every possible mistake, tolerate false alarms, because a noisy signal is still useful when you are optimizing a model. Deployed review has the inverse priority. OpenAI states it plainly: optimize for signal-to-noise first, and only then push recall without compromising reliability.

Why? Because a defense that is slow or noisy gets bypassed. This is not a soft usability concern. It is a correctness requirement. If your AI reviewer leaves twenty comments on a clean PR, reviewers learn to skim past all of them, and the one real finding dies in the noise. The tool did not fail technically. It failed operationally, and the failure is indistinguishable from not having it.

Which Checks Can Be Automated, and Which Cannot

This is the decision boundary most teams actually need, and it is cleaner than the marketing suggests.

Automatable, with acceptable signal-to-noise:

• Style and lint enforcement
• Static analysis and type checking
• Security scanning
• Dependency and build verification
• Test execution
• Repeated invariant checks that reviewers keep explaining by hand

Human-owned:

• Ambiguous intent — what the change is for, when the spec is incomplete
• Cross-team contracts and compatibility commitments
• Data boundaries and anything touching regulated or sensitive data
• Any decision where a wrong merge is not recoverable by a revert

The rule I use: automate a check when it is repeatable, scoped, and has a clear pass/fail. Keep it human when the check requires intent the diff does not contain. That second clause is doing most of the work. A reviewer can look at a diff and ask "does this match what we promised the payments team?" An automated reviewer cannot, because the promise is not in the diff.

Microsoft's engineering blog describes the same split from the other direction: the goal of its internal AI review assistant was to let AI handle repetitive or easily overlooked aspects of review, freeing human reviewers for higher-level concerns. It flags style inconsistencies, minor bugs, potential null references, inefficient algorithms — categorized so the author can judge impact. That is a reasonable division of labor, and note what it does not claim: it does not replace the human judgment about whether the change should exist.

One warning that gets lost in adoption pressure. An automated reviewer is an additional reviewer, not an enforcement layer. OpenAI says this explicitly about its own tool: tests, branch protections, and required approvals continue to provide hard enforcement. If your team treats an AI review comment as a gate, you have replaced a soft signal with a false sense of a hard one. Keep the hard gates hard.

Encoding Review Context Instead of Repeating It

The most useful shift in agent-assisted review is not better detection. It is moving reviewer knowledge out of individual heads and into the repository, where an agent can find it.

The mechanism is simple. If your team keeps explaining the same invariant in PR after PR — "this field must stay backward compatible with the v1 client," "never log this payload" — that explanation is a rule waiting to be written down. OpenAI's repository-rules interface puts concise, scoped review guidance in an AGENTS.md file that the review agent can apply and cite in a finding. The agent can find and cite local guidance that a default review would miss.

The discipline that makes this work is scope. Broad instructions create noise. OpenAI's own finding: small, scoped rule sets with an explicit safe path helped the agent focus, while broad instructions applied rules to every nearby change and buried the signal.

Three rules I would follow:

Start with one consequential, non-obvious invariant. Not a style preference. Something that would be genuinely costly to miss — a compatibility requirement, a data boundary, a security constraint.

Scope rules to the code they govern. Repository-wide guidance at the root, service-specific guidance nested closer to the code. Narrow scope keeps unrelated instructions from competing for attention and makes ownership clear.

Test each rule before trusting it. Run one change that should trigger it, one safe counterexample, and one unrelated change. The first should produce a useful finding; the other two should not create noise. If removing a rule would not change the review, leave it out.

This is where I think the real skill development is happening. Writing a good repository-scoped review rule is a new engineering artifact, and it is closer to writing a test than to writing documentation. It encodes intent in a form a machine can act on.

Test Ownership When the Author Is an Agent

The question that stalls teams is simple: who owns the tests when the agent wrote the change?

The tempting answer — the agent writes the tests too — has a specific failure mode. Generated tests can pass while encoding the same misunderstanding as the generated code. If the agent misunderstood the contract, it will write a test that asserts the wrong behavior, and the test will pass. Green checkmarks are not evidence of correctness when the same system produced both the code and the check.

So separate tests by the job they do, not by who wrote them:

• Regression tests pin existing behavior so it cannot silently change.
• Contract tests verify the boundary between your code and another system.
• Characterization tests capture what legacy code actually does, so you can detect when a change alters it.

The ownership rule: the human owner of the changed surface owns the test intent, even when the agent writes the test code. The agent can produce the boilerplate. A person has to decide what the test is supposed to prove.

The failure mode to watch for is coverage theater. Coverage metrics rise while the specific test that would have caught the regression does not exist. If your agent is generating tests to hit a coverage target, you have automated the metric and not the assurance.

Debugging and Release Controls Under Agent Volume

Two downstream consequences deserve attention before they become incidents.

Debugging gets harder when the change history contains less human reasoning. When something breaks in production, the first question is usually "why was this changed?" If the answer is a diff with no recorded intent, you are doing archaeology instead of debugging. Commit and PR records need to carry intent — what the change was supposed to accomplish and what constraints it had to respect — not just the diff. This is the same encoding problem as review rules, applied to history.

Rollback and feature-flag design matter more when change volume rises. The DORA finding that AI adoption correlates with lower delivery stability is the relevant signal here, though it is a correlation from one large survey and should be treated as a signal rather than proof. If change volume is up and stability is down, the ability to reverse a change quickly becomes more valuable than the ability to prevent every bad change. Feature flags and clean rollback paths are the mechanism that makes a wrong merge recoverable.

Release controls should scale with blast radius, not with the number of PRs. A change to a logging line and a change to an authentication path should not pass through the same gate. If your review process treats every PR identically, agent volume will force you to either slow everything down or let everything through.

The open question I cannot resolve from available evidence: how much of the reported throughput gain survives contact with a team's actual incident load. Vendor productivity figures measure output, not the downstream cost of integrating and debugging that output. That cost is real, and it is not yet well measured in public data.

A Practical Adoption Path and What to Watch

Here is the sequence I would run, and it is deliberately slow at the start.

1. Measure before you change anything. Record current review latency and revert rate. These are your baselines. Without them, you cannot tell whether the redesign worked or just moved the queue.

2. Add one scoped rule. Pick the invariant your reviewers keep explaining by hand. Write it into the repository where it applies. Test it against one triggering change, one safe counterexample, and one unrelated change.

3. Run agent review in non-blocking mode. Let it comment. Do not let it gate. Compare its findings against what human reviewers caught on the same PRs.

4. Promote only the checks that earned trust. A check earns promotion when it finds real issues and does not generate noise. Everything else stays advisory.

5. Track review cost per merged change, not just PR volume. Volume is the input. The cost of reviewing and integrating is what determines whether you actually shipped faster.

Three signals should change your mind about the whole approach:

• Noise-driven bypass. If reviewers start skimming past agent comments, the tool has failed regardless of its detection rate.
• Rule sprawl. If your review rules grow faster than your ability to test them, you have rebuilt the documentation problem you were trying to solve.
• Governance questions. Where is proprietary code mirrored or executed? CodeRabbit's architecture, for example, runs untrusted code in a sandboxed cloud environment — a design choice that raises real questions about where your code goes and what executes it. Settle that posture deliberately rather than by default.

The forward-looking claims here are scenarios, not predictions. Agent review is improving, and the economics of verification favor automation for a growing set of checks. But the boundary between automatable and human-owned is set by whether a check requires intent the diff does not contain, and that boundary moves slowly.

The Decision Rule

Automate the checks that are repeatable and scoped. Keep humans on the decisions that are not reversible by a revert. Treat review latency and revert rate as the metrics that tell you whether the redesign worked.

That is the whole redesign, compressed. The teams that get this right will not be the ones with the most sophisticated agent tooling. They will be the ones who moved verification into the repository — as rules, as tests that encode intent, as commit records that carry reasoning — so that the checks run without a human in the loop for the parts that do not need one, and the human attention goes to the parts that do.

The next skill worth building is writing repository-scoped review rules and designing tests that encode intent rather than coverage. If you have not yet evaluated which agent you are trusting with your repository in the first place, that evaluation comes first — the review redesign only pays off if the agent's output is worth verifying.