30% more code, no more closed issues

At the quarterly review the engineering lead put up two charts. The first showed commits climbing from the month coding agents arrived, up about thirty percent by quarter end. The second showed the delivery board, where the line for closed issues lay flat as a ruler. Nobody in the room said it was impossible, because both charts were true.

That silence became common through late 2026. The speedup vendors report is real. So is what engineers feel at their desks. Then the quarter closes and less has shipped. The failure is not in one step of the process. It sits in a distinction nobody made: the speed of writing code, and the speed of putting a feature in front of a user.

A study published in late October, built on real engineering data rather than vendor telemetry, measured that gap. Its conclusion rewrites how the money should be counted, which makes it worth an hour of any technology leader about to renew an AI coding contract.

Two charts, one finding

Fiona Chen and James Stratton, both at Harvard, ran an observational study on team output data from Jellyfish. The sample covers 2021 through March 2026, more than 700 software firms, over 700,000 employees and roughly 300 million work events: commits, pull requests and Jira issue records. They used a difference-in-differences design, comparing firms before and after they adopted coding assistants and coding agents at different points in time.

The code-side numbers are steep. Introducing coding agents raised total lines of code by 30 percent, commits by 20 percent and pull requests by 23 percent on average. Anyone watching a dashboard would read those three curves as a successful rollout.

The delivery-side numbers are flat. The resolution rate for issues and epics tracked in Jira did not change in a statistically significant way. The researchers also tested the obvious alternative explanation, that teams were slicing requirements into smaller pieces to inflate closure counts. They ruled it out: the size and complexity of closed issues showed no compositional shift around adoption.

Downstream numbers got heavier. Code review took longer, pull requests were more likely to need revisions, and reviewers left more comments. The authors put it in one sentence: time saved in the coding phase is absorbed by constraints further down the production process.

Worth stating what the study can and cannot carry. It is observational. Firms chose when to adopt, so adoption is not random, and difference-in-differences handles some confounding but not simultaneous changes such as reorganisation or a shift in the demand mix. Use it to calibrate your own ledger, not to pronounce on the industry. Vendor speedup figures do not contradict it. They measure different things: a vendor measures one task from start to finish, this study measures one organisation across a quarter.

Closing one issue takes three things

To see why writing faster is not shipping more, take a single issue apart. Three conditions have to hold at once before it can be closed.

The first is an acceptance condition: what counts as done, signed off by whom, under what circumstances. It usually sits in the ticket description, and the vaguer it is, the louder the argument at closing time.

The second is evidence: where someone can see that it is right. A passing test, a screen recording, a reconciliation table, a third party receipt. The form of the evidence decides how long verification takes.

The third is accountability: whose name is on the record if it turns out wrong. This one has no technical substitute, because its whole function is to bind consequences to a person.

Agents draft the first two very fast. They read the repository and write acceptance conditions, run tests and produce evidence, fill in context on a pull request. The third they cannot do, because accountability cannot be delegated to something that can be re-run at will.

So what actually grows inside the organisation is the surface waiting to be verified. A pull request goes from 40 lines to 400, the reviewer has ten times as much to read, and the accountable judgment they can render has not sped up at all. That is where the review-time curve comes from. It is not a decline in human efficiency. The area someone must answer for has expanded.

How that surface grows

Break the chain into five steps and it becomes clear that this is structural, not a management failure at one company.

Generation gets cheaper, so output rises. Items pile up in the verification queue. To keep the queue from exploding, sampling rates drop or review time is compressed. Attention per item falls, so missed defects and out-of-policy changes rise. Rework and production fixes eat the hours that were saved, and booked delivery returns to where it started.

The chain has a cruel property. The first two steps are good news and they show up in reports. The last three are bad news and they do not. An organisation therefore sits comfortably in a state that feels like progress until one incident drags step five into the open.

There is an amplifier on top of that. Agent-written code arrives stylistically consistent, commented, with test coverage that looks complete. Those features lower the guard of a reviewer, because attention is allocated by signal. A tidy pull request gets read less carefully than an obviously rough one. Part of the rise in review comments comes from exactly this: the closer someone reads, the more they find, and the cost of finding lands on the same person.

Running the numbers on closure instead of commits

Switching the metric takes two numbers: how many issues closed in a period, and what one issue is worth. Everything else is a cost line.

Monthly incremental value equals closed issues times value per issue. Total cost equals inference and subscription fees, plus review hours at fully loaded labour cost, plus rework hours at the same rate.

A worked example. A ten-person team closes 40 issues a month, each worth 80,000 in business value. After adopting agents, tools and inference run 30,000 a month. Review hours rise from 120 to 210, which at 400 per fully loaded hour is 84,000. Rework adds 60 hours, or 24,000. Total cost: 138,000 a month.

If closures move from 40 to 50, incremental value is 800,000 against 138,000 of cost, and the case is not close. If closures stay at 40, incremental value is zero and the 138,000 is pure addition. Same tool, same people, opposite conclusions, and the only difference is whether anyone counted closures.

The real use of that arithmetic is that it reverses the order of the conversation. Most teams start with a demo, then size a token budget, and ask about delivery last. The correct order starts with closures. If that number does not move, a cheaper tool is still not worth expanding, because you are paying for a step that produces no delivery.

One cost line is easy to miss: review hours get more expensive quietly. Senior engineers spend their days reading output written by others, time that used to go to architecture or mentoring. It never appears on the books because nobody opens a ticket for it. Load it at full cost and several attractive projects turn ordinary on the spot.

When commit volume is the right metric

State the boundary or this arithmetic becomes an argument against every tool. Four kinds of work have commit volume and delivery moving together, and counting commits there is fine.

Prototypes and exploration, where the point is to see what is possible and the output is meant to be thrown away. Internal scripts and one-off data processing, discarded after use, with verification cost near zero. Documentation, where the reader judges correctness on the spot with no evidence chain required. And refactors that do not change behaviour, where a green test suite is the entire proof.

In four other kinds of work commit volume systematically overstates delivery. Regulated production changes, where every line has to trace back to an approval. Cross-team interfaces, where acceptance depends on a schedule owned by another group. Data migrations, where errors do not raise exceptions, they just make numbers wrong. Customer-facing features, where the acceptance standard is set by the market rather than by the code.

The test is one question: how long does acceptance take? If verification costs more than production, making production faster buys nothing. Optimise acceptance, and skip the faster model.

Our own ledger

Our content pipeline is an extreme version of this logic, which makes it useful as a contrast rather than a model.

Agents draft every bilingual article on our site, but a draft becomes published content only after four machine gates: at least 3,300 Chinese characters or 1,500 English words, complete structured definitions and FAQ blocks, a zero-error article validator, and typecheck plus a full build, re-run again in continuous integration. Fail any gate and output is zero, counted nowhere. There are 349 paired articles live today. That number counts what passed, not what was generated. Draft volume runs several times higher, and we do not book it as capacity, because what a reader cannot see has not been delivered.

The internal task queue runs on the same accounting. Cumulative tickets stand at 3,173, with 2,975 closed and 24 still open. In the past thirty days 254 closed, averaging 1.8 days each. We report closures only. Report creations instead and the system learns within a month to produce things nobody asked for, all of it looking like growth on a chart.

This is not a best-practice claim. It is how we keep books after getting it wrong. One side effect is worth naming: because only closures count, drafting is deliberately valued low, so nobody is rewarded for writing drafts that will be rejected.

Our judgment

Our judgment is that the ROI question around coding agents is currently an accounting problem more than a tooling problem. Organisations book capacity at the cheapest, fastest-growing end, which is commits, so every tool upgrade reads as real growth. The binding constraint lives in the definition of done: whether an issue counts as closed depends on acceptance conditions written in advance, evidence that exists, and a named person who signs. Models cannot supply any of the three.

That also explains a pattern that keeps repeating. The same tool doubles delivery at one team and changes nothing at another. The difference is rarely prompt skill. In the first team acceptance conditions are machine-decidable, so the model plugs into a pipeline that can close on its own. In the second, acceptance needs a meeting, so the model plugs into a room where someone has to nod.

Three moves fit in a week. Put four numbers in one table for the last six quarters: commits, pull requests, closed issues, mean time to close. Which one moves is visible at a glance. Write acceptance conditions and an evidence location into every ticket on the pilot team, and remove generated output from the definition of done, because generation is process, not delivery. Load review hours at full cost into the AI budget, or you will book an attractive return alongside a slower organisation.

The question we would leave a buyer with: on your dashboard, which row records what AI produced? If the answer is commits, lines or conversations, the next upgrade will show growth again while the organisation gains nothing.

References

Ars Technica, AI coding agents generate more code, but not more software (October 2026; the 30 percent lines of code, 20 percent commits, 23 percent pull requests and flat issue resolution figures)

InfoQ, Study Finds AI-generated Code Fails to Boost Productivity Due to Downstream Bottlenecks (October 2026; sample size and method)

Definition of done, Wikipedia (the completion concept used here)