AI agent monitoring: a 1-token ping misses a 429

Yesterday at four in the afternoon our monitoring dashboard was green and four production jobs finished nothing. Both statements were true, because the dashboard and the jobs were measuring different things.

The failure that looks green

Nothing was wrong with the probe. It calls the provider on a fixed interval and gets a 200, all day, without a gap. The jobs do something else: one session sends about nine thousand input tokens, and in that same hour it was refused seventeen times with a 429.

That gap is the whole story. The service was up. The quota was no longer enough to do the work. Our dashboard only answered the first half of the question, because it counts requests and the provider counts tokens.

We are not writing this to complain about a provider. The point is that once agents go to production, whether a service is up and whether it can still do your work become two different questions, and they can have opposite answers. Most dashboards, ours included, still ask the older one.

Quota is metered in tokens. Your probe is not.

This is not one vendor being awkward. The OpenAI API documentation states it plainly: synchronous traffic has separate limits for requests per minute and tokens per minute, and either one triggers a 429 on its own. (Rate limits, OpenAI docs)

We measure availability in requests because that is how services worked for the past decade. A health check returning 200 meant the machine was there. A request returning 429 means the machine is there and is refusing to do the work.

Put differently: one measures reachability, the other measures capacity. Collapse them into a single probe and you get a light that lies. It tells the truth every single time. The truth is just about something your business does not depend on.

We tested it directly. The same key returns 200 for a one-token request and 429 AccountQuotaExceeded for an agent-sized session of roughly nine thousand input tokens, with the reset date landing on the next Monday. That was not a one-off. We reproduced it in two separate incidents.

Here is the part worth sitting with: both requests used the same endpoint, the same credential and the same rate-limit rules. The only difference was payload size. A green probe in that setup is not weak evidence. It is zero evidence.

The same documentation holds a detail few people read. How much of your limit a request consumes is calculated from the larger of your max_tokens setting and the estimated token count of the request text. A probe can therefore set a tiny output ceiling and leave almost no trace on the quota ledger. The documented advice is to align max_tokens with the real completion size. That advice is about saving quota, and it is also about closing a monitoring blind spot.

How to size the probe

This does not need new infrastructure. It needs three numbers pulled from your own logs, and a probe that becomes two probes.

1. The p50 and p90 of input tokens per real session. Do not estimate these. Agent request size has a wide distribution: the same task can differ by two orders of magnitude between a clean first run and one that retried three times.
2. Peak session count. A capacity probe has to run during the busy hour, otherwise it measures idle-time quota.
3. The cost of re-running one job. This sets what you are willing to pay for probing. As long as a day of probing costs clearly less than one job re-run, the trade is easy.

With those three numbers the design is straightforward. Keep a light probe at high frequency to catch network and availability-zone outages, and add a capacity probe at a payload between p50 and p90, running every fifteen minutes to an hour, to test quota and seat validity. Configure their alerts separately. Do not merge them.

One engineering detail matters more than it looks: give the capacity probe fixed content, such as the same long passage repeated, rather than random text. Fixed content can hit the prompt cache, so it costs less and its daily consumption is comparable from one day to the next. Otherwise you will end up staring at a quota curve you cannot explain.

Request size is itself a variable

In a classic service, request size is roughly constant, so a fixed-size probe represents real traffic well. Agent workloads are not like that. The same task might take thirty requests and a few tens of thousands of tokens the first time, and eight hundred thousand tokens once it retries its way through tool failures. Tool definitions, multi-turn context and retries all multiply the per-session token count.

So a fixed small probe does not sample the middle of that distribution. It samples the floor. In our own logs, failed tool calls are bimodal: the median returns an error in 1.6 seconds and the ninetieth percentile runs into the timeout ceiling, with only 31 percent landing in between. Duration and token spend for one task can differ by two orders of magnitude. Probing that with one token is like taking a temperature with a blood pressure cuff.

Which means the fix is payload, not frequency. Running a probe every minute instead of every five only makes a probe that measures the wrong thing run harder.

Three failures, one alarm

We keep a pool of three credentials under one provider for redundancy. In practice two of them work, so nominal redundancy and real redundancy differ by one tier. That is its own topic. What matters here is that the two unusable keys fail in completely different ways.

• One returns 429 AccountQuotaExceeded: quota spent, recovers by itself at the reset date, no action needed.
• The other returns 400 InvalidSubscription: the seat itself is no longer valid, never recovers by waiting, and needs a human to renew or replace it.

In our monitoring, both fell into the same cell, because it only reports reachable or not reachable. The remediation is opposite: one is wait, the other is act now. Our first reading of the logs called the dead seat a spent quota, and we nearly left it sitting until the reset date. Picture the reset passing, quota returning, and the seat still dead. You would file that as a brand new incident.

That is what failure classification buys you, and it has nothing to do with tidy alerting. Different failures run on different clocks: quota exhaustion is measured in weeks, an invalid seat in configuration changes, rate limiting in minutes, upstream faults in hours. A monitor that merges them turns a one-week configuration fix into something that looks like a capacity planning problem.

Why teams keep inheriting the wrong probe

This is not a hard technical problem. It is a split-ownership problem. Availability monitoring usually sits with platform or operations, while the question of whether this month’s quota will finish the batch belongs to procurement and the business owner. The first group asks whether the service is up. The second asks whether delivery will land on time. Each assumes the other is watching, so nobody is.

The result is a probe inherited from the virtual machine era, pointed at token-metered agent traffic. It raises nothing, because from its point of view nothing abnormal happened. It is also not the thing to blame. It was never asked to answer a capacity question.

A successful request is not health evidence

The third trap is quieter, and it makes teams overconfident about their own setup. A credential pool rotates mid-session on a 429: it marks the current entry exhausted, moves to the next, and when the pool runs dry the session falls through to a pay-as-you-go backup channel. At the end of all that, the job completes and returns a normal answer. Nothing in the output says any of it happened.

So the fact that a job finished proves a fallback exists. It does not prove that the credential which was selected is healthy, and it does not even prove which model produced the output. To judge a specific component you have exactly two options: probe it with a workload-sized payload, or read the rotation lines in that session’s runtime record.

There is a second cost to silent fallback, and it is not on this month’s bill. It is comparability. When the backup path runs a different model, the output of the same task is no longer comparable with the primary path. Work ships, and that piece of work can no longer sit in the same table as earlier batches. Nothing alerts you. The incomparability surfaces the next time you compare evaluations or audit a decision, by which point you have already used it.

Our position

Three claims, each of which you can argue with.

First, agent availability needs two layers. Keep the reachability probe and add a capacity probe at workload scale. With only the first, you are monitoring whether a machine exists rather than whether it can work.

Second, failure classes must be reported separately. Quota, seat validity, rate limiting and upstream faults each carry their own remediation and their own clock. Merging them is a voluntary surrender of judgment, and it is the part of this incident that actually cost us time.

Third, fallbacks can stay, but not silently. Every channel switch belongs in the runtime record, and when it changes the model, and therefore the comparability of results, it belongs in the output. We would rather deliver a result labelled as completed on a backup channel than a clean-looking result that cannot be compared with anything before it.

A five-item checklist for the AI app lead

Where this does not apply

Boundaries are more useful stated than implied.

• Pay-as-you-go channels do not refuse requests when tokens run out. They only cost more, so a capacity probe adds little there. Its value sits on plan and seat-based channels, which is how most enterprises buy.
• For small-context work under a thousand tokens, the gap between a one-token probe and a real payload is small, and so is the payoff from rebuilding it.
• Single-channel applications with no pool and no fallback do not need pool member probing, but they still need failure classes, especially invalid seats, which never self-heal.
• A capacity probe consumes the same limit it is testing. When quota is already tight, probing can squeeze out a real job. Give it a separate budget, or drop its frequency while an alert is open.

The decision in front of you

Back to that green light at four in the afternoon. It did not lie. It answered a question we wrote ten years ago. The question that needs an answer is different: if the quota, the seats and the model behind my agents were all replaced today, would my dashboard tell me?

If the answer is no, the next all-green failure is not a question of whether, only of when you notice. You can spend an hour changing the probe payload now, or spend a week later working out which model quietly produced a deliverable you already shipped.

References: OpenAI API documentation, Rate limits — separate limits for requests per minute and tokens per minute, and how a request consumes the limit: platform.openai.com/docs/guides/rate-limits. Probe and credential-pool figures come from OOMeta’s own production records (provider status codes and hourly counts of 429 events), verified twice, on 23 and 24 September 2026.