Hallucination gates cut legal agent pass rates

Two numbers appeared side by side on a legal agent leaderboard in early October: 13.9% and 0.3%. Same model, same 120 legal tasks. The only change was a gate. If a deliverable contains one substantive hallucination, the task does not count as passed, and every partially correct answer goes to zero.

One task set, two ways to read it

The benchmark is Harvey LAB-AA, released on 8 October in its v1.1 form by Artificial Analysis together with Harvey’s legal agent team. Its most consequential move is lifting hallucination out of the rubric and turning it into an admission condition: a task passes only when every rubric item passes and the deliverable carries no substantive hallucination. The benchmark’s own example of a substantive error is a wrong contract date, which is enough to mislead a reader on a material point.

How the evaluation is built decides how the numbers read. The dataset is 120 private legal tasks across 24 practice areas including M&A, capital markets, tax, litigation and restructuring. Scoring is all-pass with no partial credit; in the publisher’s words, finding eight of ten risks is not 80% useful, it is substantively incomplete. Three judge models score each item and the results are averaged. Hallucination checking runs in two stages: flag suspect claims against the source, then return to the source and split confirmed errors into substantive and minor.

The benchmark is a joint output from an evaluation firm and a legal AI company. We read the sources directly but did not reproduce the runs, so every figure below is quoted on the publishers’ terms, with framework differences marked separately rather than blended into settled fact.

What the gate does to the leaderboard

Turn the gate off and the picture changes. Sixteen models clear 85.6% to 96.0% of rubric items, a few near a perfect sheet. With the gate on, the top all-pass rate is 9.4% for Grok 4.7 at xhigh, followed by 8.9% for Muse Spark 1.3, 8.6% for GPT-6 Astra, 6.9% for GPT-6.1 Sol, 6.4% for Claude Fable 5.1 and 5.3% for Kimi K3, with Claude Opus 5.5 at 4.2%.

The clearest illustration is Muse Spark 1.3. Ungated it leads outright at 26.7%, yet two thirds of those passes contain substantive hallucinations. Its lead sits mostly in sounding convincing rather than in being usable.

The aggregate reading is blunter: across the models tested, more than 60% of passing results contained substantive hallucinations, and several models fall to zero once the gate applies. GLM-5.3 shows 13.9% before gating and 0.3% after. The publisher flags two qualifications: v1.1 and v1.0 are not directly comparable because both scoring and judges changed, and this round ran on the evaluator’s own Stirrup harness, so scores across frameworks cannot be subtracted from one another.

Two readings that are easy to invert

The first concerns open-weight models. The benchmark reports that open-weight models average at least 2.09 substantive hallucinations per task, and that hallucination rates track the model rather than the difficulty of a legal area. For teams building in house the implication is direct: the licence cost saved by running open weights comes back as review hours, and review is the most expensive segment of this work.

The second concerns time. The same benchmark’s initial version landed in May 2026, when frontier models finished under 10% of tasks end to end and Opus 4.7 led at 7.1%. Five months later the gated leader sits at 9.4%, with mid-range models between 4% and 7%. That comparison needs care, because scoring and judges both changed, so the two sets cannot strictly be subtracted. It still shows one thing: once the standard moves, progress has to be measured as movement inside one standard. Splicing old and new scores together only produces an illusion.

A change of standard, not a regression

Reading the numbers as evidence that legal agents are immature misses the actual change, which sits in where judgement is applied.

An average score is additive: eight of ten items right means 80%, and the reader knows the remaining 20% is a gap. An all-pass rate is multiplicative: one substantive error fails the whole deliverable. In work that gets signed, the second shape is the risk the client actually carries, because a closing checklist does not become usable just because the rest of it is correct.

Push the mechanism one step further and you get a testable prediction. Once buyers accept against a gated standard, the scores vendors quote will generally fall below prior years. That is not model regression; it is the errors that used to hide inside an average becoming visible. The corollary: a model holding a single-digit or low double-digit all-pass rate under gating has a narrow but trustworthy range.

One more mechanism is worth naming: a gate converts evaluation from average capability into failure probability. The same all-pass rate implies a different failure probability depending on task slicing, because longer tasks carry more chances for a single error. That is why a gated standard favours decomposing complex work into independently acceptable segments. It is not a product preference; it falls out of the arithmetic.

Cost makes the spread concrete. Per-task cost runs from $0.22 for GPT-6 Luna to $21.73 for Claude Fable 5.1, close to a hundredfold, with the leader at $9.49. Price order and capability order do not coincide, which is inconvenient at purchase time, because default procurement compares price.

Three changes to write into acceptance

There are three changes a buyer can make this week.

First, write the all-pass rate into acceptance instead of an average. The question is what share of deliverables can go downstream without rework. That number is much lower than an average and much more honest.

Second, replicate how the gate checks. The method is not complicated: flag suspect claims against the source with one model, confirm against the source, then split errors into substantive and minor. Its value lies in defining what a substantive error is before scoring starts. Harvey built more than 75,000 binary criteria for this benchmark, which tells you that definition work, not the algorithm, is the bulk of the effort.

Third, account for cost per task rather than per token. The best performing model is rarely the cheapest, and the cheapest is not automatically sufficient; the pair of all-pass rate and per-task cost is what survives a pricing conversation.

If you want to build your own

The copyable parts are spelled out by the publisher: a private task set of 120 tasks across 24 practice areas, an expert rubric with more than 75,000 binary criteria, three judges averaged, a two-stage hallucination check, all-pass scoring and normalised submission conditions. The benchmark is open source, so a team can assemble its own version instead of designing the method from nothing.

One warning is about sequence. Define what counts as a substantive error first, then pick the task set, and only then run models. Teams that work in reverse collect plenty of scores but cannot answer how much of their output ships, because the criteria were written after the fact and miss the real points of disagreement.

What the gate still does not cover

The publisher lists the next step itself: folding in the usability factors lawyers care about, such as style and tone. That disclosure matters because it marks the boundary of an all-pass rate. It answers whether output can be delivered, not whether anyone wants to deliver it. A legal memo free of substantive errors but written like machine translation does not get used.

Follow that boundary and acceptance splits into two gates. One is admission, where an all-pass rate keeps undeliverable work out. The other is experience, where human sampling judges usability. Collapsing both into a single score loses two things at once: neither the delivery share nor a usability regression stays visible.

An acceptance routine you can copy

Five steps turn the argument into practice. Take 30 to 50 recent real tasks rather than a public set. Write binary criteria starting from disagreements the team has actually had, not from a template. Assign three judges or three roles and average their item scores instead of relying on one reviewer. Run the two-stage hallucination check: flag suspects, then confirm against the source. Finally build a two-dimensional selection table of all-pass rate against per-task cost, and leave the average score in an appendix.

Step two is the one teams skip and the one that costs the most. Coarse criteria inflate an all-pass rate because nobody has defined what an error is. Harvey covered 120 tasks with more than 75,000 criteria, averaging over six hundred per task. That density is the answer: criteria are not decoration on a scorecard, they decide whether an evaluation can be used for acceptance.

Read cost alongside task length

One more misreading to avoid. An all-pass rate is naturally harsher on long tasks with many criteria, so comparing products requires checking task decomposition and criteria counts first. Otherwise a product that slices work finely looks more capable, when it may simply have avoided the hardest segment.

Cost behaves the same way. If a cheaper per-task price comes with finer slicing, finishing the same unit of work takes more tasks and the bill multiplies back. Multiplying per-task cost by the number of tasks a job needs is the comparable figure.

The same standard now applies to sellers

Watch what happens on the selling side. Once customers ask in gated terms, self-reported benchmark numbers invite questions about framework and criteria, and vendor-built benchmarks are usually tied to the vendor’s own harness. That is not a flaw in itself; comparability is the problem. When two readings come from different harnesses, the gap between them cannot support a selection decision. For buyers, the cheapest effective move is writing a re-test requirement into the contract instead of confirming it verbally during evaluation.

Our judgement

On high-stakes work such as legal tasks, evaluation is shifting from how capable a model is to what share of its output can be used as delivered. A hallucination gate is not one more metric; it sets the value of partial correctness to zero, so it changes how systems are bought and accepted rather than who tops a leaderboard.

The same standard constrains our own work. Our rule in research and writing is that every external fact carries a traceable source, and an unverifiable number does not enter a conclusion on the strength of the correct material around it. The difference is scale: our gate is judged by people, while this benchmark pushes the same idea to 75,000 criteria and, in doing so, produces numbers that can be compared.

Here is a standard you can apply tomorrow. If you are choosing an agent for a legal or financial workflow, can you state its all-pass rate? If not, the evaluation you have measures how plausible it looks, not how much of it you can ship.

Where to stop

A few boundaries. A low all-pass rate does not mean a system is useless: the gate is harsh on long tasks, and clearing a dozen criteria inside one task is a high bar, so a low number describes usable share rather than absence of capability. This round ran on the evaluator’s harness, so its figures cannot be subtracted from vendor-reported numbers produced on a different framework, nor can one side’s gated score be compared with another’s ungated score.

The standard can also be gamed: splitting tasks smaller and narrowing criteria both flatter an all-pass rate, so buyers should compare task decomposition as well as scores. And the benchmark covers legal work; extending its conclusions to domains with different criteria requires re-testing rather than inference.

One last note on sourcing. Every figure here comes from the benchmark’s publishers, a joint output that we read at source but that has not been independently reproduced. Where vendor-reported results and evaluator-run results differ, the text marks them separately rather than blending them into a settled fact.

References

Artificial Analysis release note on Harvey LAB-AA v1.1, 8 October 2026 (all-pass rates, gating, cost and comparison readings)

Harvey LAB-AA evaluation page (per-model gated all-pass rates and per-task cost)

Harvey on the legal agent benchmark’s initial results and all-pass scoring standard