September 2026 · 6 min read
Private code is the only honest agent benchmark
Key Definitions
Real-SWE A benchmark released by Specific Labs in September 2026 that evaluates frontier coding models on private, real-world enterprise codebases. Tasks are work engineers actually do; code and solutions are not on the public internet, so models cannot pass by memorization.
Private-domain eval Evaluating an agent on your own codebase, your own real tasks, and your own acceptance criteria. Contrast with public leaderboards (SWE-bench etc.): those measure the model, this measures whether it can do your work under your constraints.
OOD (out-of-distribution) Out of the training distribution. Private production code is natively OOD for frontier models — roughly 99% of enterprise tokens are invisible to them, so scores are naturally lower than on synthetic tasks.
Public coding leaderboards are systematically overstating enterprise performance. In September 2026, Specific Labs released Real-SWE — the first benchmark to evaluate frontier coding models on private, real-world production codebases. The best model-plus-harness combination (Claude Fable 5.1 + Claude Code) resolves only 38.8% of real production tasks, while the same class of models sits above 90% on SWE-bench Verified. The gap is not model regression — it is that the evaluation moved from inside the training distribution to outside it.
The evidence: private code kicks memorization out of the exam
Real-SWE’s sample comes from private production codebases of real companies: an app with 200K+ users and a top-100 App Store ranking; a consumer fintech platform processing 100K+ bank statements; an enterprise AI sales platform supporting complex business workflows. Tasks are the work engineers were actually assigned — fixing invoice billing, migrating regions, migrating customer identities — changes that reach across multiple services (benchmark-reported; sources at the end).
Across 8 model-plus-harness combinations, 10 tasks and 640 scored rollouts, the resolution rates (pass@1) were: Claude Fable 5.1 + Claude Code 38.8%, GPT-6 Astra + Codex CLI 33.8%, Gemini 3.8 Flash + Gemini CLI 31.2%, GLM 5.3 + Claude Code 28.8%, Grok 4.6 and Muse Spark 1.3 tied at 23.8%, Kimi K3 18.8%, GPT-5.6 Sol 16.2%. That is the actual ceiling of today’s best coding agents on real enterprise work.
The most counterintuitive data point: rollouts under 10 minutes failed 71.4% of the time; longer rollouts failed 73.4% — almost identical. More time and more tokens do not rescue these tasks; the bottleneck is understanding existing business logic, company coding patterns, and cross-service coupling. This is an understanding problem, not a time problem.
Our judgment: public leaderboards are a selection illusion — private-domain eval is the only trustworthy signal
Three reasons public leaderboards cannot transfer to enterprise selection. First, distribution: SWE-bench-style tasks come from open-source repos whose code and solutions sit on the public internet; private production code is natively OOD — Real-SWE notes roughly 99% of real-world enterprise tokens are invisible to frontier models. Second, task shape: Real-SWE’s median instruction is 1,742 characters and the reference solution edits a median of 11 files, versus 6 for FrontierCode and DeepSWE — real tasks have larger cross-sections and harder change boundaries. Third, failure mode: short and long rollouts fail at the same rate, which says the problem is reading the company code, not running longer.
The conclusion: signing off a coding-agent purchase on a 90% SWE-bench score is predicting out-of-distribution performance from an in-distribution score. Enterprise selection needs exactly one signal — your tasks inside the eval: your codebase, your tickets, your acceptance criteria. That is private-domain eval.
Action list: how to build a private-domain eval that can make the call
① Take real tickets, not synthetic tasks
Pull 20-30 completed real tasks from your backlog — with explicit acceptance criteria and the actual diff. Synthetic tasks cannot expose company patterns.
② Judge with real code and business rules
Use test suites or executable business rules as the verifier, not an LLM judge. Real-SWE’s verifiers adopt existing codebase tests or use them verbatim — reproducible and auditable.
③ Run N times, read the distribution, not one pass
A single pass@1 success says nothing about reliability. Run each task multiple times and look at the pass-rate distribution — Real-SWE averages 8 runs per task for this reason.
④ Benchmark tools against each other, not against leaderboards
The comparison basis is tool A vs tool B on the same task set, not their SWE-bench scores. Public leaderboards are for screening, not for decisions.
In one line: public leaderboards tell you what the model has seen; private-domain eval tells you whether it can do your work. Enterprises buy the latter. Whoever can stand up a private-domain eval and keep it running is the only one entitled to talk about scaling coding agents.
OOMeta AI
OOMeta’s own operation is a working example of private-domain eval: one human plus a fleet of agent units running on real, constrained, business-consequential tasks — not scored on synthetic leaderboards. When we build agent capability assessments for clients we insist on the same first step: define your tasks and acceptance criteria before arguing about model choice. Real-SWE only makes the principle extreme — private code is the only honest exam room.
Book a diagnostic callReferences: Specific Labs, “Real-SWE Benchmark” (September 2026; private production code, 8 model-harness combos / 10 tasks / 640 rollouts) https://withspecific.com/benchmarks/real-swe · SWE-bench official leaderboards https://www.swebench.com/ · Real-SWE launch discussion (Hacker News, 2026-09-12) https://news.ycombinator.com/item?id=49676820
FAQ
What exactly differs between Real-SWE and SWE-bench?+
SWE-bench tasks come from open-source repos; code and solutions are on the public internet, so models may have seen them in training. Real-SWE tasks come from private production codebases of real companies, licensed to Specific Labs — code and solutions are not on the public internet. Solving them requires actually understanding the codebase and business rules, not memory.
What does the 38.8% figure mean?+
Real-SWE scored 8 model-plus-harness combinations, 10 tasks, 640 scored rollouts. Resolution rate is pass@1, averaged over eight independent runs per task. The best combo (Claude Fable 5.1 + Claude Code) hit 38.8%; GPT-6 Astra + Codex CLI 33.8%; Gemini 3.8 Flash + Gemini CLI 31.2%.
Why do short rollouts fail about as often as long ones?+
Real-SWE measured 71.4% failure for rollouts under 10 minutes vs 73.4% for longer ones — almost no difference. The bottleneck is not time or token budget but understanding existing business logic, company coding patterns, and cross-service coupling. More time does not fix understanding.
Why does the 99% invisible-tokens point matter?+
Roughly 99% of tokens in real enterprises live in private codebases frontier models have never seen, per Real-SWE. Public leaderboards measure capability inside the training distribution; enterprises need performance outside it. The two cannot be inferred from each other.
How should an enterprise select a coding agent?+
Do not decide from a SWE-bench score. Take your own real tickets, your own codebase, and your own acceptance criteria (test suites or business rules); run each task multiple times and look at the distribution, not a single pass. Compare tools on the same private-domain eval — it is the only signal that reflects your task constraints.
Why does OOMeta stress private-domain eval?+
Our own operation is the evidence: value comes from real, constrained, business-consequential tasks, not synthetic leaderboards. A public score cannot answer whether an agent can work in your codebase — only an eval built from your tasks can.
Related Articles
Microsoft’s ThinkingBox: agents demo well but fail in production
65% single-run vs 25% 20-run — evaluation methodology itself inflates agents, the same class of problem Real-SWE exposes.
Agents fail in the harness, not the model: monday.com’s production guardrails
Input boundaries, output validation, feedback loops — production readiness is set by the harness; private-domain eval is the evaluation face of the same logic.
Eval-first: Zepto’s dual-loop prompt engineering
Eval first is an engineering discipline, not a slogan — build the evaluation before touching prompts; Real-SWE pushes it into private code.
2 weeks from idea to prototype: a cross-border fintech’s AI compliance sprint
A production-grade prototype under real business constraints — private-domain validation in a compliance setting.