September 2026 · 6 min read
Private code is the only honest agent benchmark
Key Definitions
Real-SWE A benchmark released by Specific Labs in September 2026 that evaluates frontier coding models on private, real-world enterprise codebases. Tasks are work engineers actually do; code and solutions are not on the public internet, so models cannot pass by memorization.
Private-domain eval Evaluating an agent on your own codebase, your own real tasks, and your own acceptance criteria. Contrast with public leaderboards (SWE-bench etc.): those measure the model, this measures whether it can do your work under your constraints.
OOD (out-of-distribution) Out of the training distribution. Private production code is natively OOD for frontier models — roughly 99% of enterprise tokens are invisible to them, so scores are naturally lower than on synthetic tasks.
Public coding leaderboards are systematically overstating enterprise performance. In September 2026, Specific Labs released Real-SWE — the first benchmark to evaluate frontier coding models on private, real-world production codebases. The best model-plus-harness combination (Claude Fable 5.1 + Claude Code) resolves only 38.8% of real production tasks, while the same class of models sits above 90% on SWE-bench Verified. The gap is not model regression — it is that the evaluation moved from inside the training distribution to outside it.
The evidence: private code kicks memorization out of the exam
Real-SWE’s sample comes from private production codebases of real companies: an app with 200K+ users and a top-100 App Store ranking; a consumer fintech platform processing 100K+ bank statements; an enterprise AI sales platform supporting complex business workflows. Tasks are the work engineers were actually assigned — fixing invoice billing, migrating regions, migrating customer identities — changes that reach across multiple services (benchmark-reported; sources at the end).
Across 8 model-plus-harness combinations, 10 tasks and 640 scored rollouts, the resolution rates (pass@1) were: Claude Fable 5.1 + Claude Code 38.8%, GPT-6 Astra + Codex CLI 33.8%, Gemini 3.8 Flash + Gemini CLI 31.2%, GLM 5.3 + Claude Code 28.8%, Grok 4.6 and Muse Spark 1.3 tied at 23.8%, Kimi K3 18.8%, GPT-5.6 Sol 16.2%. That is the actual ceiling of today’s best coding agents on real enterprise work.
The most counterintuitive data point: rollouts under 10 minutes failed 71.4% of the time; longer rollouts failed 73.4% — almost identical. More time and more tokens do not rescue these tasks; the bottleneck is understanding existing business logic, company coding patterns, and cross-service coupling. This is an understanding problem, not a time problem.
Our judgment: public leaderboards are a selection illusion — private-domain eval is the only trustworthy signal
Three reasons public leaderboards cannot transfer to enterprise selection. First, distribution: SWE-bench-style tasks come from open-source repos whose code and solutions sit on the public internet; private production code is natively OOD — Real-SWE notes roughly 99% of real-world enterprise tokens are invisible to frontier models. Second, task shape: Real-SWE’s median instruction is 1,742 characters and the reference solution edits a median of 11 files, versus 6 for FrontierCode and DeepSWE — real tasks have larger cross-sections and harder change boundaries. Third, failure mode: short and long rollouts fail at the same rate, which says the problem is reading the company code, not running longer.
The conclusion: signing off a coding-agent purchase on a 90% SWE-bench score is predicting out-of-distribution performance from an in-distribution score. Enterprise selection needs exactly one signal — your tasks inside the eval: your codebase, your tickets, your acceptance criteria. That is private-domain eval.
Action list: how to build a private-domain eval that can make the call
1. Take real tickets, not synthetic tasks
Pull 20-30 completed real tasks from your backlog — with explicit acceptance criteria and the actual diff. Synthetic tasks cannot expose company patterns.
2. Judge with real code and business rules
Use test suites or executable business rules as the verifier, not an LLM judge. Real-SWE’s verifiers adopt existing codebase tests or use them verbatim — reproducible and auditable.
3. Run N times, read the distribution, not one pass
A single pass@1 success says nothing about reliability. Run each task multiple times and look at the pass-rate distribution — Real-SWE averages 8 runs per task for this reason.
4. Benchmark tools against each other, not against leaderboards
The comparison basis is tool A vs tool B on the same task set, not their SWE-bench scores. Public leaderboards are for screening, not for decisions.
In one line: public leaderboards tell you what the model has seen; private-domain eval tells you whether it can do your work. Enterprises buy the latter. Whoever can stand up a private-domain eval and keep it running is the only one entitled to talk about scaling coding agents.
OOMeta AI
OOMeta’s own operation is a working example of private-domain eval: one human plus a fleet of agent units running on real, constrained, business-consequential tasks — not scored on synthetic leaderboards. When we build agent capability assessments for clients we insist on the same first step: define your tasks and acceptance criteria before arguing about model choice. Real-SWE only makes the principle extreme — private code is the only honest exam room.
Book a diagnostic callReferences: Specific Labs, “Real-SWE Benchmark” (September 2026; private production code, 8 model-harness combos / 10 tasks / 640 rollouts) withspecific.com/benchmarks/real-swe · SWE-bench official leaderboards www.swebench.com/ · Real-SWE launch discussion (Hacker News, 2026-09-12) news.ycombinator.com/item?id=49676820
FAQ
What exactly differs between Real-SWE and SWE-bench?+
SWE-bench tasks come from open-source repos; code and solutions are on the public internet, so models may have seen them in training. Real-SWE tasks come from private production codebases of real companies, licensed to Specific Labs — code and solutions are not on the public internet. Solving them requires actually understanding the codebase and business rules, not memory.
What does the 38.8% figure mean?+
Real-SWE scored 8 model-plus-harness combinations, 10 tasks, 640 scored rollouts. Resolution rate is pass@1, averaged over eight independent runs per task. The best combo (Claude Fable 5.1 + Claude Code) hit 38.8%; GPT-6 Astra + Codex CLI 33.8%; Gemini 3.8 Flash + Gemini CLI 31.2%.
Why do short rollouts fail about as often as long ones?+
Real-SWE measured 71.4% failure for rollouts under 10 minutes vs 73.4% for longer ones — almost no difference. The bottleneck is not time or token budget but understanding existing business logic, company coding patterns, and cross-service coupling. More time does not fix understanding.
Why does the 99% invisible-tokens point matter?+
Roughly 99% of tokens in real enterprises live in private codebases frontier models have never seen, per Real-SWE. Public leaderboards measure capability inside the training distribution; enterprises need performance outside it. The two cannot be inferred from each other.
How should an enterprise select a coding agent?+
Do not decide from a SWE-bench score. Take your own real tickets, your own codebase, and your own acceptance criteria (test suites or business rules); run each task multiple times and look at the distribution, not a single pass. Compare tools on the same private-domain eval — it is the only signal that reflects your task constraints.
Why does OOMeta stress private-domain eval?+
Our own operation is the evidence: value comes from real, constrained, business-consequential tasks, not synthetic leaderboards. A public score cannot answer whether an agent can work in your codebase — only an eval built from your tasks can.
Related Articles
Microsoft’s ThinkingBox: agents demo well but fail in production
65% single-run vs 25% 20-run — evaluation methodology itself inflates agents, the same class of problem Real-SWE exposes.
Agents fail in the harness, not the model: monday.com’s production guardrails
Input boundaries, output validation, feedback loops — production readiness is set by the harness; private-domain eval is the evaluation face of the same logic.
Eval-first: Zepto’s dual-loop prompt engineering
Eval first is an engineering discipline, not a slogan — build the evaluation before touching prompts; Real-SWE pushes it into private code.
2 weeks from idea to prototype: a cross-border fintech’s AI compliance sprint
A production-grade prototype under real business constraints — private-domain validation in a compliance setting.