Your merged PRs are the coding-agent benchmark

Databricks engineers had a purchasing question to answer: which coding agent should become the team default next week. A public leaderboard can rank models. What they needed to know was which combination finished work cheaply on their own multi-million-line codebase, spread across a dozen languages. So they built an internal benchmark out of their own merged changes. Two default assumptions did not survive the run.

This is not a story about what Databricks bought. It is about how their numbers change one thing: the unit you use to price a coding agent. Change the unit and procurement, vendor selection and acceptance criteria all have to change with it.

The model that is 1.7x cheaper per token costs more per task

The first assumption to fall was that a cheaper token means a cheaper job. Sonnet 5 is roughly 1.7x cheaper per token than Opus 4.8, so on a price sheet it should win. On the Databricks task set, Sonnet cost $2.09 per task against Opus’s $1.94, and it completed six points fewer tasks, 81% against 87%.

The gap is not noise. To finish the work, Sonnet read more and ran longer, consuming 1.9x the tokens. Unit price is a multiplier in the bill; the number it multiplies comes from how the model works. A wordier model can be cheaper per token and more expensive per job.

In procurement terms: the price sheet gives you the multiplier, and only your own tasks give you the number being multiplied.

Same model, a different harness, more than double the cost

The second assumption is closer to engineering instinct: that the model decides cost. They ran the cleaner comparison instead, holding the model and the thinking effort fixed and calling it from two different harnesses, Claude Code/Codex and Pi. Cost per task differed by more than 2x. Quality did not measurably differ.

The difference sat in how much context each harness fed the model on every turn. Pi sent about 3x less context per turn, kept a tighter working set and finished the same tasks in fewer runs. Harness here is not a metaphor for taste; it is the decision code around the model. What enters the context each turn, when history gets compacted, how tool output is truncated, how often a failure is retried. Each of those four decisions moves the final number.

Databricks kept one line deliberately modest: no harness is always cheaper, and native harnesses are not automatically worse. They built a swappable layer between models and harnesses for a practical reason. Once a pair is welded together, you lose your only chance to compare.

Two reasons a public benchmark cannot answer this

First, the tasks are public. The well-known suites have been online long enough that solutions seep into training data, which makes scores improve for reasons that have nothing to do with what you are buying now.

Second, the shape does not match. That codebase spans 10+ languages, Scala, Go, Rust, Java and Python among them, built with Bazel and Protobuf, with service boundaries that grew rather than being designed. Public suites measure general capability. You are buying cost on your shape.

Any company with a few years behind it has things in its repository that no outsider has: naming habits, an internal framework, scaffolding, conventions everyone follows and nobody documented. Those are exactly the things that decide how much an agent reads and how many attempts it needs.

There is a free by-product too. Tasks drawn from merged changes arrive with labels attached: a merge means somebody already confirmed the solution belonged in main. You do not need to hire annotators; the version history did it.

Our judgment: the unit of cost is wrong

Together these numbers say one thing. The cost of a coding agent is roughly context per turn, times turns, times unit price. The first two are set by the harness and the shape of the work; only the third is printed on a public price sheet. Negotiating on unit price means negotiating the least important of the three, and picking from a leaderboard means picking something that is not your workload.

We would put it on a number a reviewer can check: cost per task that passes review. That number cannot be bought, only measured, and the raw material for measuring it, a batch of real tasks with known correct answers, is already in most teams’ repositories.

It also explains the shape of the surprise most teams get from their first invoice: until then, unit price was the only number they could see.

Four steps, one week

• Sample. Pull 30 to 50 tasks from changes merged in the last two or three months, covering the work you do most: config flips, defect fixes, schema additions, refactors. Do not pick only the hardest ones; the benchmark has to represent an ordinary week.
• Grade.Treat “merged” as the correct answer and your existing tests as the grader. The grader has to be automatic, or nobody will run it a second time.
• Fix one variable at a time. Fix the harness and vary only the model in one pass; fix the model and vary only the harness in another. Run them separately, because a mixed run produces numbers nobody can attribute, which is the usual reason harness costs get booked against the model.
• Repeat. Run each task three times and use the pass rate rather than a single verdict. That is how the independent Coding Agent Index does it: three attempts per task, averaged across three benchmark suites.

Record the billing shape alongside it: tokens per task, turns per task, and context per turn. Track unit price alone and you will reproduce the 1.7x inversion above.

Where the method stops working

Tasks have to be gradable by machine. Legacy rewrites, known-defect fixes and test coverage have defined answers and belong in a benchmark; greenfield builds and design exploration are scored by people, do not scale, and should never be turned into a percentage promised to the business.

Sample size sets your resolution. Thirty to fifty tasks separate differences of the two-fold kind. They will not resolve 5%, and comparing two close options needs hundreds.

The result holds for you and nobody else. A private benchmark is not a public verdict, but it is a negotiating asset, because it is your data and the other side cannot rebut it with somebody else’s tests.

One more limit that gets missed: keep tools that are still in preview out of a long-lived benchmark. Preview behaviour and preview billing move, and a number measured this month stops being true next month.

Put the benchmark in the contract, not just in your head

With a benchmark in hand, acceptance can move from features delivered to pass rate and cost per task on a fixed task set. That was impossible while nobody could tell automatically whether the work was actually finished; the grader is your own test suite now.

What belongs in the contract is a scheduled re-run: the same task set, every quarter, so cost movement from a model generation or a harness upgrade stays inside your field of view. Otherwise the next odd invoice restarts the investigation from zero.

One question is worth leaving with buyers: whose task set decided your current default tool? If no name comes to mind, that decision was probably made for you by a vendor’s leaderboard.

A second, independent measurement of the same thing

Databricks is not alone in measuring this. The independent Coding Agent Index runs 20 agent-and-model combinations across three suites, 113 DeepSWE tasks, 66 Terminal-Bench tasks and 124 SWE-Atlas-QnA tasks, averages the pass rate over three attempts per task, and publishes cost per task in API dollars. It differs from the Databricks benchmark in two ways, the tasks are public and the shape is generic, but the methodological agreement matters more than the conclusions: count single tasks, refuse to grade on one lucky run, and record how much context each turn consumed.

Both point the same way. The same task set costs wildly different amounts depending on the pair you run it through, and none of that difference is visible on a price sheet.

Run the arithmetic once and you stop needing the conclusion

Take an illustration of the arithmetic itself. Suppose a task takes 20 turns and each turn sends 40,000 tokens of input. At three dollars per million input tokens, input alone for that one task comes to roughly $2.40, and that number is set entirely by how much is sent per turn multiplied by how many turns run. It has nothing to do with how many lines the task touches. Cut context per turn to a quarter and the same task’s input cost falls to $0.60, which saves more than swapping in a model priced at a third of the unit rate.

Real invoices also carry output tokens, cache hits and tool round trips, so the arithmetic above is not a quote and should not be used as one. It shows why unit price was the first assumption to fall: it is one of three variables, and the other two are usually absent from the procurement file.

Do not forget the review side of the cost

Six points lower completion shows up on an invoice as a few cents. The real cost lands on the team: what ships needs reviewing, what fails needs redoing, and the attention that takes never appears on an API bill. That is why we argue for putting “passed review” inside the counting unit. If the merge did not happen, the cost already did.

Record three things, or you cannot explain the numbers in six months

• The version of the model and of the harness, with the run date. A model can be swapped for a new version without telling you, and the score moves with nothing to attribute it to.
• Context per turn and turns per task. Track total cost alone and you cannot tell whether an invoice moved because the work got harder or the harness got chattier.
• The sampling distribution. Two quarters of only the hardest tasks will lower your pass rate for reasons that have nothing to do with tool quality.

What we do in-house

OOMeta runs its own agent pipeline, with tasks coming from operating data rather than demo scripts, which makes us sensitive to waste like a model reading the same files a second time. So we take deterministic steps away from the model and hand them to scripts: deduplication, data pulls, path assembly and file writes need no model judgement, which means the previous turn’s output does not need to be sent again. The model only handles the step that requires a judgement call. That split is not about convenience. We did the arithmetic, and smaller context per turn means a smaller bill rebound the next time a model is swapped.

Why this became practical only now

Three years ago this was not runnable. Turning merged changes into tasks requires reproducing the code state of that moment, executing the tests of that moment, and stripping the answer out of a real change so the task can be handed over. Those capabilities only spread through mainstream tooling in the last two years. The grader depends on the same shift: most repositories now have a test subset that finishes in minutes, which is the precondition for turning “passed or not” into an automatic number.

So the barrier to a private benchmark is not the method. It is whether your repository has a verification path that finishes in minutes, and any team without one should build it first, because they need it before handing changes to an agent anyway.

Three moves at the negotiating table

• Attach the task set and the grading method to the contract, with the sampling window and task count stated, so the vendor is accountable for the measurement and not only for a pass rate.
• Ask for per-task cost detail once a quarter, including tokens and turns, rather than a monthly lump sum.
• Treat a model swap as a scheduled event. Generations turn over quarterly, so any option that needs the integration rewritten to change models will be abandoned at the second swap.

References

• Databricks: Benchmarking Coding Agents on Databricks’ Multi-Million Line Codebase (methodology and all figures, first party)
• Artificial Analysis: Coding Agent Index (cost per task across agent and model combinations)
• Artificial Analysis: coding-agent benchmarking methodology (DeepSWE 113 tasks / Terminal-Bench 66 / SWE-Atlas-QnA 124, pass@1 over three attempts)
• Omnigent (the swappable layer Databricks built for model and harness swaps, official site)