Agent speedup is set by task class

Late in an experiment report there is a line that takes up less than two lines. One participant admitted he had not submitted several tasks. The reason: AI could finish those tasks in two hours, and doing them by hand would take him twenty, so he simply did not report them rather than meet them in the arm where AI was forbidden. The sentence sits in the methodology section. It is the most useful thing in the document.

The report came from METR, a group that runs randomized controlled trials on how AI affects developers. In July 2025 they recruited 16 developers who maintain open-source projects, put 246 real issues from repositories they knew well in front of them, and randomly assigned half the tasks to an AI-allowed condition and half to a control. Before starting, the participants predicted AI would make them 24 percent faster. After finishing, they believed it had made them 20 percent faster. The stopwatch recorded them 19 percent slower.

The three numbers only make sense together. The first two describe one thing: people hold a stable bias about their own speed, and finishing the work does not correct it. The second pair describes something else. It is not that the bias points the wrong way. It is that the two measurements sample different stretches of time.

When people judge their own speed, attention lands on the stretch before the keyboard: working out what to write, then writing it. AI does shorten that stretch, visibly, and the feeling is real. The ledger records a longer stretch: reading the existing constraints, waiting for tool output, reading the generated code line by line, noticing it is wrong at one boundary, correcting it, then finding someone to review it. The first stretch shrinks, the second grows, and a feeling captures the change while a ledger captures the sum.

In February 2026 METR published a follow-up using agentic tools such as Claude Code and Codex, with 57 developers, 143 repositories and more than 800 tasks. Returning participants came in at a point estimate of minus 18 percent, with a confidence interval from minus 38 to plus 9. Newly recruited developers came in at minus 4, with an interval from minus 15 to plus 9. Both intervals cross zero. METR described the data as only very weak evidence. The sign convention is easy to misread: in METR notation a negative number is a slowdown, and several secondary write-ups reported the minus 18 as an 18 percent speedup.

Why the signal weakened is written in the same document. Between 30 and 50 percent of developers declined to submit certain tasks, because they did not want to do those tasks without AI. Pay was cut from 150 dollars an hour to 50, which changed who agreed to take part. Running several agents at once also made wall-clock time per task ambiguous. The control arm stopped being a control arm.

An experiment that cannot be completed usually goes down as a failure of method. This one reads more like a deployment report. A refusal rate of 30 to 50 percent means that inside a real backlog, people have already sorted tasks in their heads, and they sorted them more finely than any public benchmark does: which tasks AI wins on, which it loses on, which come back to be redone. The sorting happened. It was never written down.

That unwritten list is valuable because it is free, calibrated to your own codebase, and bounded by real constraints. Take one quarter of tickets and ask every person who touched them one question: which of these would you not do by hand again. The answers will split the work into two piles rather than arrange it on a spectrum from very effective to useless. Two piles is enough. Budgets and staffing decisions only need two piles.

Why the numbers elsewhere point the other way

The split shows up in other datasets. Observational work from DORA found the opposite direction of association: every 25 percent increase in AI adoption came with 1.5 percent lower delivery throughput and 7.2 percent lower stability. That is correlation, not causation, since teams adopting AI are usually changing other things at the same time, but the direction is worth noting. A 2026 evidence review summarises one longitudinal enterprise case covering 802 developers and 196,212 pull requests over two years, with per-capita throughput at 2.09 times the pre-mandate level while the review load per reviewer rose roughly in proportion. A cleaner experiment points the same way as the first: Microsoft Research and GitHub ran a randomized trial in 2022 on a bounded JavaScript task, and the group with Copilot finished it 55.8 percent faster.

These numbers do not contradict each other. Their disagreement is structured. The same tool produces double-digit speedups on tasks with clear boundaries and machine-checkable outcomes, and produces zero or a negative number on tasks whose boundaries are vague and whose acceptance depends on human judgment. So the question of how much AI speeds work up is missing a premise. Without the task class, any average measures the mix of work, not the tool.

This also explains why the same report gets cited by both sides. Vendors cite 55.8 percent to argue the tool works; buyers cite minus 19 to argue it does not. Both cite real numbers, from different task mixes. To decide which is closer to your situation, look first at what share of your tickets falls into the machine-checkable class. That share appears in no report. It appears in your own systems.

There is a second bias worth naming, because it sets the shape of every public number. A task can only become a benchmark if its outcome is machine-checkable. To enter automatic scoring, a task needs a determinate answer or a verification script that runs. That requirement locks out tasks with blurry boundaries, so benchmark task mixes lean systematically toward the work AI is best at. The impressive vendor numbers are not fabricated. They are computed on a filtered task distribution. The distribution in your own backlog never passed through that filter.

Change the unit of account, not the metric

Most companies already have plenty of numbers on a dashboard: build counts, deploy frequency, median merge time. Those numbers keep looking healthy after AI arrives, because they record system events rather than tickets. While a ticket sits in a review queue for two days, builds and deploys can both be fine. So the decision to expand a rollout is often made in front of green charts that were never designed to answer whether expansion is wise.

The number also gets misused once at the funding stage. A business case promises 30 percent faster, the budget is approved against that number, and at the first-year review the target is unmet, so the project is cancelled rather than re-scoped. What got cancelled was not AI but a commitment written in the wrong unit. The same work stated differently survives: which task class is covered, how many tickets it covers, and a rework rate ceiling. Those three can be checked and can be refuted. A 30 percent speedup can be neither.

Our position is that the unit of account has to change. Most enterprises currently measure output per developer, which hides the task mix inside the denominator. When the mix shifts between two quarters, the number moves and nobody can say whether the tool improved or the work got easier. Measure instead per accepted ticket, with three fields: cycle time from start to merge, the review time inside that cycle, and the rework rate within two weeks. The first two describe speed. The third says whether the speed is real.

Rework rate is the only number that can falsify the claim that we got faster, and the hardest one to wave through in a monthly review. Cycle time can be shortened by cutting scope. Review time can be shortened by reviewing less deeply. Only rework gives both back: code that was reverted, a second edit in the same place, a fix shipped after release. A speedup report without a rework rate describes a traffic pattern, not an improvement.

There is also an ordering question that gets skipped. When per-capita throughput doubles and review capacity does not, the extra output does not disappear. It accumulates in the review queue, which lengthens one segment of the cycle while turning review into a formality. So capacity comes first: measure review headroom, then decide how much to open up. Our view here is blunt. Speed without review headroom does not remove risk, it moves risk from the production side to the inspection side.

Review capacity can be calculated rather than felt. Suppose a team can read one hundred changes carefully each week. Doubling per-capita throughput produces two hundred changes while headcount stays flat, so time per change halves. In queueing terms this is insufficient service capacity; in engineering terms it shows up as two things at once, a longer queue and a higher approval rate. The faster people read, the more they let through. Only the two signals together distinguish a capacity shortfall from a falling standard.

How we keep our own books

Our own content pipeline keeps these two accounts separate. Every article passes a set of deterministic gates before publication: body length, title and summary length, field consistency between SQLite and the static JSON, and a style linter. A machine can judge all of them, so they are measurable, accelerable, and safe to hand to an unattended process. The judgment half has no gate. We do not report its speed. We report its rework: which piece was rewritten, which was withdrawn, which had its thesis changed three days after publication. Keeping two sets of books is what stops us from using the measurable half to prove the unmeasurable half also got faster.

Separating them has a side effect worth naming. It turns the question of what to hand to a machine into something discussable rather than a matter of conviction. Length, field consistency and word count go to scripts. Which topic deserves to be written, and whether the argument holds, stay with people. Once that line is drawn, both the places to add machines and the places to add people become visible.

The boundary of this argument

The boundary matters, or the argument becomes a slogan. The negative result in that experiment is tied to three conditions: large repositories the developers already knew, team standards that were never written down, and early-2025 tooling. Change any one of the three and the sign can flip, and 2026 agentic tools do perform better under all three. What holds is not the sign but the mechanism: the size of the speedup is set by how clearly a task is bounded and how checkable its outcome is. A mechanism predicts the next case. A sign does not.

One dimension is easy to miss. The same tool behaves differently for someone three months into the job and for someone maintaining a ten-year-old repository. The first lacks speed in turning an idea into code, which AI supplies. The second lacks command of the existing constraints, which AI does not supply, and AI adds to the material to be read. So the question of how much a team sped up has to start with who is on the team, then move to the shape of the work. When both change, no cross-company average applies.

That gives a prediction which can be overturned. If speedup is set by task class, then within one team over two quarters, cycle time on machine-checkable tasks should fall steadily while the vague class returns to roughly zero, and any blended average should drift with the task mix without improving over time. The test is not hard. Split last quarter of tickets into two columns by whether correctness can be checked automatically, sample twenty from each, and compare cycle time and rework. If the blended number does drift with the mix, the explanation holds. If both classes improve in the same direction, look for another mechanism.

Three things to finish this week

First, pull twenty closed tickets and backfill three numbers for each: cycle time, review time, rework within two weeks. Without those three, every later discussion about expanding the rollout runs on feeling, and feeling was just measured with a 39-point error in the same experiment.

Second, write down the list of tasks nobody wants to do by hand again, per person, and do not merge the lists. Merging erases disagreement, and disagreement is the expensive signal. When two people judge the same task differently, the task boundary was never written down, and a task with an unwritten boundary is exactly where an agent is most likely to fail.

Third, change next quarter target from a speedup percentage to a rework rate on one task class. The first is an outcome, and you can only wait for it. The second is a constraint, and you can design for it.

The question left for buyers is this. Is your speedup number computed per task class, or is it a blend of the checkable and the uncheckable? If it is a blend, the number mostly describes your task mix, and it will start to worsen on the day you genuinely expand usage, because the tasks added next are the ones with the blurrier boundaries.

References

METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (July 10, 2025: 16 developers, 246 tasks, predicted 24 percent faster, felt 20, measured 19 percent slower)

METR, We are Changing our Developer Productivity Experiment Design (February 24, 2026: point estimate minus 18 percent, interval minus 38 to plus 9, 30 to 50 percent of tasks withheld)

DORA, State of DevOps (a 25 percent rise in AI adoption associated with 1.5 percent lower throughput and 7.2 percent lower stability; observational)

He et al., longitudinal enterprise case as summarised in a 2026 evidence review (802 developers, 196,212 pull requests, 2.09 times per-capita throughput, review load rising in proportion; secondary source)

Peng et al., Microsoft Research and GitHub, The Impact of AI on Developer Productivity (2023: 55.8 percent faster on a bounded JavaScript task)

State of AI Programming 2026, evidence review (eleven evidence blocks sorted by study design, including the DORA and vendor telemetry differences)