Routing savings aren’t bought
ACL’26 benchmark

Commercial routers promise 40–80% savings on inference cost — but until this year, routers were never compared on a fair, unified playing field. ACL’26’s LLMRouterBench built that field with 400K+ instances, and the result contradicts the vendor narrative: most routing methods perform similarly under unified evaluation, and several recent approaches — including commercial routers — fail to reliably beat a simple baseline. The savings are real; they just do not live in the router.

Evidence: routing measured at scale, for the first time

LLMRouterBench (arXiv 2601.07206, Findings@ACL’26, open code and data) is the largest routing evaluation to date: 400K+ instances, 21 datasets, 33 models (from flagship down to 7B-class), 10 representative routing baselines, with metrics for performance-oriented routing and performance-cost trade-off routing, plus latency-aware analysis. Core findings: ① most routing methods perform similarly under unified evaluation — much of the lead claimed in earlier papers came from differences in the evaluation setting; ② several recent approaches — including commercial routers — fail to reliably outperform a simple baseline such as a deterministic per-task-type model choice; ③ the gap to Oracle (the theoretical upper bound of always picking the best model) is driven mainly by model-recall failures — the pool simply lacks a model that can answer the task; ④ backbone embedding models have limited impact; ⑤ larger ensembles show diminishing returns versus carefully curated smaller ones.

Our judgment (1): routing’s real lever is task tiering, not selection magic

Routing saves money through a simple mechanism: send low-difficulty requests to cheap models. That needs task profiling and deterministic rules, not a black-box scorer. Most enterprise production loads — support summarization, document extraction, RAG answers, code completion — have concentrated difficulty distributions: most requests sit in the “small model is enough” tier. For these, fixed mapping by task type plus a difficulty threshold captures most of the routing benefit; that is why the simple baseline is so hard to beat in the paper.

Our judgment: do task tiering first, routers second. A router — especially a learned one — only pays off when request difficulty is highly dispersed and the cost of a wrong selection is quantifiable. Most enterprises never reach that condition.

Our judgment (2): OOMeta’s own stack — deterministic policy routing

This is not armchair advice. OOMeta’s own multi-model stack runs deterministic policy routing: the main agent unit uses deepseek-v4-flash, a specific analysis sub-unit uses gemini-3.5-flash — a fixed mapping by unit and task type, with no learned router. This matches the benchmark’s conclusion: for a fixed task profile, deterministic mapping plus private-domain evaluation beats a generic black-box router, and it stays explainable, auditable, and free of inference overhead.

Our judgment: for most enterprises, a routing policy should be engineering configuration — who, which task, which model, escalate to whom on failure — not a closed component that needs ongoing training and trust.

Buyer checklist: buy a router or build a policy — four tests

Action and the decision question for buyers

Thirty-day move: do not buy a router yet. Spend two weeks profiling tasks — layer production requests by task type and difficulty, measure a small-model baseline per layer (accuracy plus cost per completed task); if rules already capture most of the benefit, the router is not worth buying. If you do buy, demand two sets of numbers: a unified benchmark and your private-domain eval.

The decision question: when commercial routers cannot beat a simple baseline on a 400K-instance unified benchmark, why should you believe a 40% saving on your workload? If the answer depends on the vendor’s own evaluation, you have already spent trust you have not earned — before the money is saved.

References: LLMRouterBench (arXiv 2601.07206, Findings@ACL’26, submitted 2026-01-12) https://arxiv.org/abs/2601.07206 · Full paper and data https://arxiv.org/html/2601.07206v1 · Open code and evaluation data https://github.com/ynulihao/LLMRouterBench · 2026 LLM model routing survey (cross-reference) https://www.digitalapplied.com/blog/llm-model-routing-2026-cost-quality-optimization-engineering-guide