AI Coding Agent Productivity: Measuring Throughput Without Counting Generated Lines

The diff counter climbs every sprint. The release cadence does not move. That gap is not a tooling problem — it is a measurement problem, and most teams are running the wrong model.

Picture the pattern I keep running into: agents enabled for two quarters, pull requests up sharply, generated lines tracked on a dashboard, and a release train still leaving the station on last year's schedule. Nobody is lying about the numbers. The numbers are simply measuring the wrong stage of the pipeline.

Why Generated Lines Are the Wrong Unit

Lines of code, commit count, and pull request volume were already weak proxies for delivery before agents existed. They measure activity — motion inside the repository — not value that reached a user or a downstream system. When generation was expensive, volume at least correlated loosely with effort, and effort correlated loosely with progress. That chain is now broken at the first link.

The mechanism is straightforward. Agents lower the marginal cost of producing candidate code. When something gets cheaper, you get more of it by default, whether or not you wanted more of it. Verification does not get cheaper at the same rate. Reading a diff, deciding whether it matches intent, running the test suite, checking security boundaries, and approving a release all still consume human attention, and attention does not scale with compute.

Anthropic's internal research describes this pattern: a net decrease in time spent per task category alongside a much larger net increase in output volume. The productivity shows up as more features, more fixes, more experiments — not as the same work done faster. That distinction tells you where to look. If the gain is volume, the constraint moves downstream.

Anthropic's social-science survey data offers a cleaner illustration. Coding agent users started roughly a quarter more projects and posted around half a working paper more than non-users — meaningful early-pipeline gains. The same data found no evidence of more journal submissions or faster resubmission. Output rose at the front of the pipeline and did not visibly rise at the end.

Treat that as a signal, not proof. The survey is descriptive, drawn from self-selected respondents who are likely heavier and more optimistic users than average, and it says nothing about quality. What it does show is the shape of the problem: generation is an input, and inputs are cheap now.

Carry one definition through the rest of this article. Verified delivery is a change that passed the checks your team already trusts and reached a state a user or downstream system depends on. Merged is not the same as verified. Deployed is not the same as depended on. The definition is deliberately strict because the loose version is what produces dashboards that report success while nothing ships.

The Two-Stage Model: Generation and Verification

For the measurement problem in this article, the pipeline reduces to two stages. That is a deliberate boundary, not a claim about all of software work. Product intent, specification quality, release operations, incident response, and post-release learning sit outside it — and they can dominate throughput on their own. Inside the boundary, two cost curves govern what you can observe.

Stage one is generation. Candidate diffs, tests, documentation, refactors — produced at low and falling marginal cost. Generation scales with compute, context window, and model quality.

Stage two is verification. Review, test execution, debugging, security checks, release approval. Verification scales with human attention and the number of consequential decisions embedded in a change.

Different cost curves produce a predictable outcome. As generation throughput rises, the review queue becomes the limiting step unless verification capacity changes too. That is the bottleneck to instrument first, before you instrument anything else.

This is not a novel observation about queues. It is the same dynamic that governs any pipeline where one stage gets faster and the next does not: work accumulates in front of the slow stage, and the slow stage's backlog becomes the real measure of system throughput. The difference here is that the fast stage is invisible in most dashboards. Nobody charts "candidate diffs waiting for a human."

Microsoft's Playwright and Copilot coding agent material describes one concrete way teams push part of verification back into the agent itself. The agent opens a browser, interacts with the UI it just modified, observes the result, and confirms the change against real application state — a closed loop of prompt, generate, run, observe, confirm. Read that as a vendor-described workflow, not an independent field result. My interpretation: it moves a specific class of verification work out of the human queue.

It narrows the review surface. It does not remove ownership. A passing browser check is evidence about behavior under the conditions the agent exercised. It is not a decision about whether the change should ship, whether it belongs in this release, or whether the behavior it produces is the behavior the product needs. Those remain human calls, and they are the calls that consume review capacity.

The mechanics of where verification moves — which checks can be automated, how review and testing get redesigned around agent-authored changes — are covered in the neighboring analysis on review and testing. Here the job is narrower: make verification a measurable stage, with its own throughput, its own queue, and its own cost.

Metrics That Survive Contact With a Real Backlog

The two-stage model converts into a small metric set. Keep it small. A dashboard with twenty numbers gets ignored, and an ignored dashboard is worse than none because it creates the appearance of measurement.

Verified throughput. Verified changes merged and released per unit time. Not diffs opened, not PRs created. Define the denominator — team, sprint, release train — before you compare anything to anything, because a throughput number without a stated denominator is a number you can make say whatever you want.

Segmented cycle time. Split total cycle time into time to first candidate diff, time in review, time to green checks, and time to release. Aggregate cycle time hides the answer to the only question that matters: which segment did the agent actually move? If time-to-first-diff collapsed and time-in-review doubled, the aggregate may look flat while the underlying system changed shape entirely.

Review load. Track two numbers, not one: review minutes per merged change, and total review minutes per period. They answer different questions. Per-change cost tells you whether review got harder; total load tells you whether the team is spending more of its week in review. A team that generates three times the candidate code and reviews it at the same rate per change has tripled its total review cost — and that cost lands on senior engineers who have the least slack. The reverse is also possible: agents can produce smaller, more reviewable changes that cut per-change cost even as volume rises. Measure both before you assume which one you have.

Defect and escape rates. Defects found in review, in staging, and in production — normalized per verified change, not per thousand lines. Per-thousand-lines normalization is a trap when line counts are inflated by generation; it will make quality look like it improved when nothing changed.

Rework and revert rate. How often agent-authored changes are reverted, hotfixed, or rewritten within a short window. A high revert rate is a direct tax on the throughput number, and it usually shows up one or two sprints after the throughput gain, which is why short measurement windows flatter agents.

Cost per verified change. Model and tool spend plus human review hours, divided by verified changes. This is the honest denominator for any ROI claim. If you cannot compute it, you do not have an ROI number — you have a subscription invoice and a feeling. Segment it by task class and risk level, because a migration and a copy fix do not belong in the same average.

One failure mode deserves naming explicitly: optimizing any single metric in isolation. Faster cycle time with rising escapes is not productivity. It is deferred cost, and it will be paid by whoever is on call when the deferred cost comes due.

Perceived Versus Measured Gains

Enterprise research on AI coding assistants reports clear perceived improvements in efficiency and quality, concentrated in repetitive tasks, prototyping, debugging, and understanding unfamiliar code. Those are real gains, and they are the categories where the mechanism makes sense: agents remove the opportunity cost of starting, which is exactly the friction that stops people from doing small useful things.

The same body of work notes discrepancies between perceived and measured productivity for experienced developers, and that gains vary by task type, complexity, expertise level, and organizational context. Less experienced and casual developers report higher adoption and larger gains — consistent with agents removing the cost of getting started rather than accelerating expert judgment. An expert's bottleneck is rarely typing speed. It is deciding what to build and whether the result is correct.

Most available studies are short-term and task-scoped. Long-term maintainability, quality, and organizational outcomes are largely unmeasured. That is not a reason to dismiss the evidence; it is a reason to stop extrapolating from it.

My rule: use developer sentiment as a hypothesis generator, not a scoreboard. When perception and instrumentation disagree, do not resolve the disagreement by picking a winner. Instrument the specific task category where they disagree, and let the segment data tell you which one is right for that category. Sentiment is usually correct about something — often about a category you did not think to measure separately.

One boundary to hold: vendor reports and single-organization surveys describe their own populations. Anthropic's survey describes Anthropic's respondents. Microsoft's material describes Microsoft's products and customers. Neither is a field-wide adoption or effect size, and reading them as one is how teams end up surprised by their own numbers.

Durable Capability: What the Dashboard Cannot See

Throughput metrics tell you what happened this quarter. They do not tell you whether the gains will still be there in four quarters. Four second-order measures determine that.

Onboarding time for a new engineer on an agent-heavy codebase, and time for an existing engineer to explain a change they did not write. If either is climbing, you are accumulating a liability that no throughput metric will show you until an incident forces it into view.

Comprehension debt. The gap between code that exists and code the team can reason about. It behaves like technical debt in one important way: it charges interest quietly, and the bill arrives at the worst possible moment. The difference is that technical debt is usually visible in the code. Comprehension debt is visible only in how long it takes someone to answer a question about it.

Concentration risk. If a small number of people hold the context for agent-generated subsystems, review capacity and bus factor degrade together. This is easy to miss because it looks like normal specialization until one of those people takes a vacation.

Verification capacity as a durable asset. Test suites, fixtures, reproducible environments, and clear ownership rules compound. Prompt libraries do not. A prompt that worked last month may not work next month; a test that catches a real regression keeps catching it. This is the asymmetry that should drive where you invest.

Microsoft's own framing is useful here: a coding agent described as an electric saw rather than a robot that builds the cabinet, with production value contingent on review, security, and developer judgment. The saw is genuinely better than the hand saw. It still requires someone who knows what a cabinet should look like.

The moat is not the agent subscription. Anyone can buy the same subscription. The moat is the verification system that gets stronger each time your team converts a failure into a check.

Running a Measurement Pilot Without Fooling Yourself

Do not measure "all development." Pick one task category with a verifiable definition of done — a specific bug class, a test-coverage gap, a migration — and measure that.

Define a comparison window before you start, and define a small task taxonomy: bug fix, test change, migration, feature slice. Report results by category rather than as one blended number. If the baseline period is dominated by small fixes and the agent period is dominated by a hard migration, you have measured the migration, not the agent.

Baseline the segment metrics for that window before enabling the agent. Then repeat the comparison across multiple comparable batches rather than trusting a single before-and-after. Label the result observational. Unless you randomized assignment and controlled for staffing and seasonality, you have a correlation, not a causal effect — and the honest label protects you from over-committing on the strength of one good month.

Instrument review load and escape rate from day one. These are the two metrics most likely to invalidate an early positive read, and they are the two most likely to be omitted because they are harder to collect than diff counts.

Expect and record failure modes as they appear: agent changes that pass tests but miss intent, review fatigue that lowers scrutiny as queue depth grows, and rework that lands in a later sprint and gets attributed to something else. Write them down when you see them. Retroactive attribution is unreliable.

The decision rule I would use: adopt where verified throughput per unit of review cost improves and escape rate does not degrade. Narrow or stop where the gain is confined to generation volume. That second clause is the one people skip, and it is the one that matters most, because generation volume is the metric that will always look good.

State in advance what would change your conclusion. A sustained rise in escapes, a review queue growing faster than merge rate, or onboarding time lengthening as agent-authored code accumulates. If none of those appear across repeated comparable batches, you have a real signal. If they appear, you have a real signal too — just not the one the dashboard was reporting.

What to Watch Next

Four developments would materially change this measurement model.

Whether agent self-verification loops expand from browser and test checks into broader behavioral verification. Each expansion moves cost out of human review, which raises the ceiling on verified throughput. Today this is vendor-described workflow, not settled practice.

Whether shared, team-level agent workspaces become common. Persistent shared sessions change who reviews what and complicate per-developer attribution — a metric that is already shaky becomes genuinely hard to defend. Reporting through late 2025 and into 2026 points in this direction, but the tooling is young and enterprise patterns are not settled. Treat it as a watchpoint, not a market direction.

Whether longer-horizon studies appear that measure maintainability and quality rather than immediate task speed. Today that evidence is thin, and the thinness is the single biggest reason to hold your conclusions loosely.

Whether cost-per-token competition keeps generation cheap. Model releases through 2026 have been pitched explicitly on lower cost for agentic and coding workloads, which would widen the gap between generation and verification capacity rather than close it.

That last point carries the watchpoint for engineering leads: if your review queue is the constraint, a cheaper model does not help you. It makes the constraint tighter. Verification capacity is the thing to invest in.

The Two Capabilities That Decide Whether Any of This Works

The measurement model rests on two capabilities. Skip either and the numbers will mislead you.

Verifiable task definitions. Writing work items with a definition of done that a machine or a reviewer can check is the prerequisite for measuring anything downstream. If you cannot state what "done" means for a task, no metric will tell you whether the agent helped. This is also where evaluation literacy lives: reading a benchmark or vendor claim as a result under stated conditions, and asking which of those conditions your repository violates. Most benchmark results are true statements about a distribution your codebase is not drawn from.

Instrumentation. Segmenting cycle time, normalizing defects per verified change, and building a small internal dashboard rather than trusting a vendor console. The vendor console measures what the vendor can see, which is generation.

Everything else — test design, review triage, prompt craft — matters, but it matters through these two. Tests expand verification capacity for specified behaviors; they do not replace human ownership of intent, risk, and novel cases. Review triage decides which changes need deep judgment; it is a skill, not a policy, and it degrades if nobody owns it.

The next step is small and specific. Pick one task category this week. Define verified delivery for it in writing. Record the baseline before you enable anything.

Then watch two numbers: verified changes per unit of review cost, and escape rate. If the first rises and the second holds across repeated batches, you have something real. If only the diff counter rises, you have a dashboard that will keep reporting success while the review queue quietly becomes the bottleneck — and by the time that shows up in your release cadence, you will have spent a quarter learning what a baseline would have told you in a week.