The vendor now owns your eval tool

A compliance lead has a forty-page red-team report in front of her, generated with Promptfoo, with the tool and version printed on the last page. It is going to the audit committee as proof that a newly deployed agent cannot reach customer data outside its scope. While assembling the pack she notices a date: March 9, 2026, the day OpenAI announced it would acquire Promptfoo and fold it into its own enterprise platform.

So she has to answer a question that did not exist a year ago: who employs the referee on this report?

The question is not rhetorical. Audit rests on the independence of the evaluator from the thing evaluated. Once evidence is produced on a platform owned by the audited party and judged by defaults that platform sets, independence becomes a claim rather than a property. The 2026 wave of evaluation acquisitions turned testing from a procurement line item into a governance question.

What the March announcement actually says

The OpenAI post is restrained and the reasoning holds up. It argues that as enterprises put AI coworkers into real workflows, evaluation, security and compliance become foundational requirements. Enterprises need systematic ways to test agent behaviour, detect risk before deployment, and keep clear records supporting oversight, governance and accountability over time. Promptfoo technology will be integrated into OpenAI Frontier, the platform for building and operating AI coworkers.

Several claims in the post can be checked. Promptfoo tools are used by more than 25 percent of Fortune 500 companies, per OpenAI. The open-source command line tool and library will continue to be maintained. Three integration directions are listed: native security testing and red-teaming on the platform, covering prompt injection, jailbreaks, data leaks, tool misuse and out-of-policy behaviour; security and evaluation embedded in development workflows; and reporting with traceability so organisations can document testing, track change over time and meet governance expectations.

Two other moves in the same quarter point the same way. Scale AI expanded SEAL into a standardised evaluation platform, and LMArena closed a 150 million Series A at a 1.7 billion post-money valuation, with annualised consumption past 30 million within four months of commercial launch. Those two figures come from an industry analysis piece and I did not verify them independently, so treat them as market commentary rather than financial reporting. Taken together, the three moves say one thing: evaluation stopped being a research byproduct and became an infrastructure layer worth fighting over.

What changed is ownership, not capability

Be precise about this. The acquisition does not make evaluation worse. Embedding red-teaming in a development platform lowers the barrier, and more teams will actually run tests, which is good in aggregate. What changed is who holds the evidence.

The older pattern had the buyer purchase a neutral tool, write its own cases, set its own pass criteria, and hand results to an auditor. The newer pattern has someone click a few buttons in a platform, run the platform preloaded cases against the platform default thresholds, and export a report in the platform format. The first is self-attestation. The second is a supplier certificate.

In most situations the difference does not matter. In situations requiring independent review it is decisive. When the system you must prove safe belongs to the same company that supplies the referee, a report generated on that platform and judged by those defaults proves what the supplier believes. In an audit that is a statement, not evidence.

This is not an accusation against one company. Any vendor folding evaluation into its own platform runs into the same structural problem, because a platform is built to make customers succeed while an audit is built to separate the judge from the judged. Those goals do not sit on the same line.

Evaluation is three layers, valued very differently

Take evaluation apart and it has three layers with completely different substitutability. Discussed as one thing, it produces the wrong conclusion in both directions: buy everything, or build everything.

The first layer is tooling: the runner that executes tests, the assertion library, the report format, the continuous integration hook. It is highly standardised, cheap to switch and crowded with vendors. It is being consolidated, and consolidation costs you little, because replacing a runner is usually a few days of adaptation.

The second layer is the dataset: your own failure samples. This one has no substitute, because it records how your system has failed inside your business. The same wrong answer is a generic case in a dataset owned by someone else and a real payout, a customer complaint or a regulatory letter in yours. Its value rises with use, and it cannot be bought.

The third layer is the pass criteria: what counts as passing, where thresholds sit, who approves exceptions, whether changes are recorded. This is a governance asset. Outsource it and independent review stops existing in form, because the author of the rule and the object of the judgment have aligned interests.

Once separated, the procurement strategy writes itself: rent the tooling, grow the dataset, sign the criteria. Most enterprises spend nearly all of their budget on the first layer, because it is the only layer that produces an invoice.

Why failure samples appreciate

There is an easily missed economic asymmetry here. Tooling depreciates as it spreads. Failure samples appreciate as they accumulate.

Tooling depreciation is familiar. More products, closer features, prices approaching cost. When evaluation gets absorbed into a platform, the tool stops being a separate budget line at all and becomes a feature inside a subscription.

Samples move the other way. The value of one sample equals the number of similar errors it will catch times the cost of one error. The more samples you hold, the more failure modes you cover, and the higher the regression hit rate on the next model or the next change. More important is exclusivity: your samples come from your customers, your workflow and your definitions, and no vendor dataset contains them.

So the right order of investment is a sample library first, tooling second. Most teams invert it: compare tool features, then decide what to test, and end up with several runners loaded with cases copied from public datasets that measure how a model performs on someone else problems.

Operationalising this takes one rule. Every production incident or serious misjudgment becomes a reproducible sample with input, expected output, actual output and the reasoning for the verdict, merged into the regression set. If it does not enter the regression set, the post-mortem is not closed. The hard part is not technical, it is accountability, since the person writing the sample is often the person who caused it.

How our own gates grew

Our publishing gates have this structure, so they serve as a contrast. Every article on our site passes four gates before it goes live: at least 3,300 Chinese characters or 1,500 English words, complete structured definitions and FAQ blocks, a zero-error article validator, and typecheck plus a full build plus continuous integration in the green. All four checks are scripts we wrote. None depends on a model vendor.

Where they came from matters, because it shows the real cost structure of a self-built gate. The rules were not designed up front. They were added after incidents. An unescaped single quote in a title passed the local build, failed continuous integration, and the article never shipped, so a static check now catches it. An image was replaced without changing its filename, the cache kept serving the old picture, and readers saw the previous version, so filenames now carry a version suffix plus a check that every reference was updated. A draft once shipped with template placeholders in the body, and the validator now fails loudly on them.

There are 349 paired articles live today, every one through the same gates. The gates are not clever and the rules are plain. But they belong to us, so when a client asks why we claim the content is correct, we can show a reproducible path instead of an assurance that the model is reliable.

That is the hidden return on building your own gate. Each incident becomes a rule, and organisational memory moves into code rather than into whoever happens to remember. Bought tooling cannot do this, because vendor rules are designed for every customer and will not gain a check for your particular incident.

When buying is fine

State the boundary, or this reads as an argument to build everything, which is equally wasteful.

Selection comparisons can be bought. Choosing between models needs a comparable result, independence requirements are low, and any credible benchmark works. What matters is recording the conclusion in the decision log.

Mutually agreed acceptance can be bought. For delivery acceptance testing, if the criteria are signed in the contract, running them on a third-party tool is fine, because the criteria are owned by the contract rather than by the tool.

Exploratory red-teaming can be bought. If you want to see the shape of your attack surface, one pass on a vendor platform is the cheapest way to surface obvious problems.

Three categories must be self-controlled: anything going to an audit committee, anything filed with a regulator, and any accuracy or availability commitment written into a customer SLA. In all three the consequence lands on the buyer, so the referee must sit with the buyer. As for runners and report formats, rewriting them has no value at all. That is pure duplicated labour.

Three clauses that carry the work

This has to land in procurement, or the discussion ends where it started. Three items fit into a contract or a renewal addendum.

First, data export. Confirm that test cases, results and verdict records export in full and can be re-run offline. This clause decides whether you can change referees later. If all you can export is a PDF, what you hold is not evidence but a statement.

Second, criteria versioning. Every threshold change and every exception approval gets recorded, with who changed it, why, and which historical conclusions it affects. Without it you cannot answer the simplest question: under which criteria did that test pass three months ago.

Third, an appendix obligation. Any evaluation report a supplier issues must carry an appendix naming the pass criteria and the sample sources. Fail either and the report does not enter your audit pack. This clause requires no product change, only honesty about deliverables.

Our judgment

Our judgment is that this acquisition wave is an ownership event rather than a security event. It moved the question of who may judge from procurement into governance, while most contracts still sit at the procurement stage, negotiating price, seats and feature lists.

The one move buyers should make before renewing is to split the evidence chain into its three layers and confirm ownership of each: rent the runner, grow the samples, sign the criteria. Fail those three and what you bought is not evaluation capacity but a self-assessment hosted by the seller. It costs little, and it persuades about as much as it costs.

One counterintuitive corollary. Consolidation of the tooling layer is good news for buyers, because runners and report formats get cheaper, easier and bundled. What needs investment becomes clearer at the same time: accumulate your own failure samples and turn pass criteria into a document someone can sign. Neither requires a vendor or a budget approval. Both require an owner.

The question we would leave a buyer with: the last AI safety evidence you handed to a board or a regulator, whose tool, whose samples, whose criteria produced it? If two of those three answers point at a supplier, that evidence persuades up to the point where the supplier believes it.

References

OpenAI, OpenAI to acquire Promptfoo (9 March 2026; Frontier integration, continued open-source maintenance, the claim of use by more than 25 percent of the Fortune 500)

AgentMarketCap, The AI Evaluation Acquisition Wave (April 2026, industry analysis; the Scale AI and LMArena figures come from this piece and were not independently verified here)

Promptfoo open-source repository (the evaluation and red-teaming CLI that the announcement says will keep being maintained)