Evaluation-first agents: the quality gate as a cost lever

Most teams treat evaluation as the final check before launch: pick the model, tune the prompt, run one eval round to confirm nothing broke, ship, and shelve the eval. Zepto inverted that order — evaluation is the primary way agents get built, tested, and operated. The results: 80%+ of tickets fully handled by AI, support cost down 65%, payback in under a month. Evaluation-first is not quality perfectionism; it is the most direct cost lever in this generation of support agents.

The numbers on the cost side carry the most signal

Zepto is India’s quick-commerce platform covering 60+ cities with minute-level delivery, and its support operation processes over 100,000 AI tickets per day. With evaluation infrastructure on Databricks and MLflow, Zepto reports: 80%+ of tickets fully managed by AI agents with human oversight; a 65% reduction in support cost or tickets; payback in under one month; CSAT up 20%, accuracy up 8%; development cycles 3x faster and time to resolution 4x faster (source: Databricks official blog case study, vendor–customer joint reporting — https://www.databricks.com/blog/evaluation-first-ai-agents-how-zepto-scales-customer-support-databricks-and-mlflow ). The figures are self-reported, but where “payback under one month” sits is telling: when evaluation is treated as infrastructure, cost control stops being a post-hoc bill review and becomes part of the development loop itself.

Why does “just ship the agent” break at scale? Because agentic systems are multi-step workflows — intent classification, knowledge retrieval, reasoning, transactional tool calls, response generation — and failures can emerge at any step, not only in the final answer. At 100,000 tickets a day, a 1% error rate is thousands of bad outcomes and real revenue leakage daily. That assurance gap is exactly what evaluation infrastructure fills.

A dual loop plus a quality gate turns development and production into one pipeline

The core of the framework is a dual loop: a development loop (design, iterate, evaluate agent versions) and a production loop (monitor live behavior, detect failures), connected by a quality gate. Any change to agent logic, prompts, or models auto-triggers regression evaluation: the new version must do two things — meet thresholds on all pillars (customer experience, operational efficiency, risk and compliance, financial impact, each with numeric gates) and perform at least as well as the current production baseline. Pass and it is promoted; fail and it goes back to development. Every production failure feeds back into development and becomes training material for the next round.

This structure turns “shipping” from an event into a process: agents flow between development and production continuously, and the gate is the only valve. Zepto replaced the gut-feel decision (“this version feels better”) with an evidence decision (“this version beats the baseline on agreed metrics”).

The golden dataset is a 10x multiplier, and the dev-prod gap is its quality signal

The golden dataset is the framework’s single source of truth: normal, edge, and failure cases shaped by all stakeholders, including adversarial examples from the security team. Zepto grew it from 500 to 2,000 to 5,247 examples over six months while the dev-prod accuracy gap fell from 8 to 2 to 0.4 points. Their rule of thumb: every hour spent on dataset quality saves roughly ten hours of production debugging — the golden dataset is a 10x multiplier, and every production failure adds failure traces that make the system more robust for all future versions.

Here is the counterintuitive judgment buyers should record: the dev-prod accuracy gap is not an excuse about “production being complex” — it is a dataset-quality signal. A large gap means the dev dataset does not cover the scenarios production will meet; what is needed is more data, not a stronger model. Treating the gap as a manageable reading rather than a mystery is exactly what becomes visible once evaluation is infrastructure.

The cost math of sampling: stratified beats uniform

Evaluating 100% of traffic is expensive, but a naive uniform 10% sample misses most edge cases. Zepto weights sampling rates by risk: high-value customers, new features or recently changed flows, negative sentiment, high escalation risk, and image- or fraud-based interactions are sampled more densely. The result: an 18–20% effective evaluation sample (about 14,400 traces per day) that captures 45–60% of edge cases and detects issues within 4–6 minutes (source: same Databricks blog — https://www.databricks.com/blog/evaluation-first-ai-agents-how-zepto-scales-customer-support-databricks-and-mlflow ).

This is the most cost-focused part of the whole method: compared with uniform sampling, the stratified approach cut review cost per issue identified by 86% and improved edge-case detection 9x. It also refutes a common engineering excuse — “evaluation is too expensive, so we only sample 10%.” Blind sampling is what is expensive; put the sampling budget where the risk is, and evaluation flips from a cost item into a cost lever.

Our judgment

First, evaluation-first is not quality perfectionism — it is a cost lever; once eval is infrastructure, cost control (model selection, prompt optimization, sampling) is written into the development loop, which is how payback compresses to under a month. Second, the dev-prod accuracy gap is a dataset-quality signal, not a mystery — a large gap means add data until it converges; that is the direct consequence of treating the dataset as an asset with its own KPIs (size, dev-prod gap, edge-case coverage). Third, sampling strategy matters more than sampling rate — put the evaluation budget on the highest-risk traffic and cost drops 86% while detection rises 9x; uniform sampling’s “cheapness” is fake cheapness. Fourth, this is structurally the same discipline OOMeta runs internally: our spec-driven engineering loop executes dual evaluation (a generator and an evaluator) and explicitly verifies that eval scores are real gaps rather than surface compliance — we do not treat “looks like it passed” as “passed.”

Buyer action checklist

First, start from traces, not models: enforce a shared trace schema and log every agent call — without traces, evaluation, the golden dataset, and the gate are all impossible. Second, make evaluation a mandatory CI/CD gate: no deployment unless it beats the baseline, so “shipping” becomes an evidence decision. Third, operate the golden dataset as an asset: give it its own KPIs (size, dev-prod gap, edge-case coverage) and watch whether the gap converges each month. Fourth, build stratified sampling and real-time alerts before launch — retrofitting operational visibility later is the most expensive path. Fifth, never optimize a single metric: when Zepto chased intent accuracy alone, +5 points of intent bought +133% latency and −0.4 points of CSAT; composite multi-objective scoring is the right direction.

The decision question left for you: if the dev-prod accuracy gap went into your team’s monthly operating report and were forced to converge — how many of your current agents would survive that number?

References: ①Databricks official blog case study “Evaluation-First AI Agents: How Zepto Scales Customer Support” (vendor–customer joint reporting, 2026-09-09) — https://www.databricks.com/blog/evaluation-first-ai-agents-how-zepto-scales-customer-support-databricks-and-mlflow