Agent failures live in the harness, not the model

Picture an agent as a bowling ball: the LLM is the lane, and the harness is the bumpers on either side. Without bumpers, one bad roll ends in the gutter — a hallucinated citation, a malformed output, a runaway loop burning budget. monday.com just published the full production case of feedAgent, a deep agent that curates a personalized activity feed for every workspace member. Their conclusion is direct: nearly every failure mode they hit was not in the model. It was in the assumptions wrapped around it — what goes in, what bounds the run, what gets verified before output ships, what gets remembered for next time. The harness decides production readiness; the model leaderboard does not.

Three places things go wrong: input, flow boundaries, feedback

monday.com groups agent production failure surfaces into three classes (source: monday.com engineering blog — https://engineering.monday.com/building-a-robust-harness-for-agent-in-production-feed-agent-case-study/ ): ① data and context — what goes into the model and what it knows about the world it works in; ② the agent flow itself and its boundaries; ③ the feedback loop — how it improves. They built the harness by walking the flow: going over each phase of the execution, and at every turn asking, what can go wrong here, and what does the harness do about it.

The premise is plain: an agent is not a pipeline, it is a complex loop, different for each feed. Treating where things break as a design problem — instead of betting on how strong the model is — is the methodological core of the case.

Input guardrails: the model never sees the raw flood

feedAgent consumes a large, cross-asset set of data: user activity, agent activity, existing feed items. The model only gets what the harness chooses to expose. Three design decisions matter. First, an activity preprocessor deterministically collapses event bursts, keeps important aggregated changes, drops setup noise — and strips all PII before the LLM ever sees a byte. Second, an alias system replaces raw IDs with short aliases, creating a hallucination detection surface: references the model invents rather than retrieves get caught and dropped. Third, prompt caching: the static system prompt is transmitted once at the gateway and subsequent calls within the cache window pay a fraction of the cost — otherwise every single run pays full input-token price for the same static text.

Run boundaries: schema is a contract, limits are insurance

At initialization the harness defines not only what the agent can do but how far it can go. The most underrated piece is the output schema: without one, the model returns free-form text. Because downstream code must consume the output, the agent is forced to emit it through a schema-validated tool call — the difference between a contract and a hope. When the model misses the schema, the validation error is fed back as context and it retries; the model then has what it needs to correct itself.

On boundaries there are two independent caps: a model call limit counts how many times the model is invoked, and a recursion limit caps how many loop iterations the graph can execute. Either one hit stops the run cleanly instead of failing unhandled. Tools follow the same discipline: a must-have tool is not offered as an option but placed later in the flow so it always happens; error messages are written as actionable instructions — a bare failed status is a signal the model must interpret, while a message that says the data is unavailable and to proceed with what exists is an instruction; tools holding connections get a guaranteed close path.

Output verification and the feedback loop: filtering, dedup, memory

On the output side, any citation that does not resolve to a real activity ID is dropped and logged — if the model cited an alias that was never in the original data, that is a hallucination. The same activity cannot appear in two feed items; a set tracking every resolved ID across the full output pass enforces deduplication. On the feedback side, every run fires a summary event with run-level metrics — duration, activity counts, model name, tool-call trajectory, prompt-length breakdown — while LangSmith tracing records each intermediate step. Feed memory closes the loop: behavioral signals flow back into the next run’s input, with explicit user rules at high authority and inferred observations (patterns across dismissals, CTA clicks, dwell times) as soft context that never overrides explicit rules.

Evals are the fuse of the loop: offline evals run on CI with an LLM-as-judge over a dataset continuously enriched with production cases, stopping regressions before deploy; online evals run on real-time production data to detect issues and suggest fixes. Because the agent is non-deterministic, testing only the deterministic parts is not enough — that is why the two layers exist.

Our take

First, failures living outside the model is counterintuitive but correct: the production ceiling of an agent is set by harness design, not by the model card. It explains why the same model performs so differently across companies — the assumptions around it differ. Second, guardrails are layered: input (size and PII), run (schema, dual caps, tools), output (hallucination filtering, dedup), and memory (feedback loop). Lose one layer and production readiness drops a notch, and the missing layer tends to surface at the worst moment. Third, when evaluating an agent platform, do not only ask how strong the model is — ask which harness layers it actually controls. This matches OOMeta’s own production practice: our skill system, eval gates, and task bus are designed to keep failures inside guardrails; the model is just the ball being routed. Fourth, the buyer implication is direct: add harness maturity to the agent platform evaluation checklist, alongside model capability — and arguably ahead of it.

Buyer action list

First, draw a guardrail checklist for every production agent — input/PII, schema and caps, output verification, feedback memory, evals — and treat missing items as known risks in the release review. Second, output must go through schema-validated tool calls; never let the model generate free-form JSON. Third, cap twice, independently: model calls and loop iterations. Fourth, hallucination filtering is not optional: references must resolve to real data, or they get dropped and logged. Fifth, run two-layer evals — offline on CI, online in production — and feed every failure back into the dataset. Guardrails are not static configuration; they are continuous engineering.

The question for you: how many harness layers does your agent project actually control? If a bad ball rolls in today, who notices first?

References: ①monday.com engineering blog (Sep 9): Building a Robust Harness for Agent in Production — feedAgent case study — https://engineering.monday.com/building-a-robust-harness-for-agent-in-production-feed-agent-case-study/