Orchestration restores state, not side effects

At twenty to three in the morning, an agent handling refunds submits 4,800 yuan to a payment gateway and gets a success receipt back. Its next step is to write that result into its own database. The process is reclaimed between those two steps. Four minutes later the scheduler wakes it on the retry policy, it reads its checkpoint, and the last recorded step is submit refund. So it submits again.

The model was right at every step. What failed was the orchestration underneath it. The checkpoint faithfully recorded what the agent remembered and could not record what had already happened outside. That distinction sounds like terminology. On the ledger it means one refund paid twice, or one order placed twice.

The crash window has two positions, and both leak

Look at an external call and its state record together and there are only two orders to put them in. Write the checkpoint first, then call: a crash between the two leaves the system believing nothing happened, so the replay does it again and the external write is duplicated. Call first, then write the checkpoint: a crash after the call returns and before the record lands leaves the same belief, and the same duplicate.

There is no third arrangement. The local ledger and the remote ledger cannot join one transaction, because a transaction ends at your own disk. Two-phase commit exists only where the other side participates, and payment gateways, mail providers and ERP systems do not open a prepared phase because your orchestration framework supports checkpoints.

So durable execution restores only the part we remember. What happened outside is known to the other system alone. That is close to common sense, yet the questions enterprises asked when buying orchestration in 2026 sit almost entirely in the other half: can it resume after a crash, will state be lost, will retries run forever. All three are real and all three are solvable by a framework. Together they still do not equal no duplicate execution.

One detail gets missed. The risk inside the crash window is not the crash, it is the retry rhythm. To recover quickly, most frameworks schedule the first retry within seconds, which is exactly the span in which the other system has no result for you yet. The shorter the wait, the less chance a human notices, and the more reliably the replay lands the second blow.

The vendors wrote the window into their own documentation

Temporal described it themselves in a post dated November 10: if a process crashes after posting a ledger entry but before initiating a payment, a plain retry becomes a double charge. Their answer is that a workflow keeps a replayable history, the execution layer owns retries and timeouts, and state lives outside the model. The design genuinely fixes continuity across a process restart. What it does not fix is the call that has already left.

Graph runtimes such as LangGraph record a different layer: which node execution reached, what the shared state holds, whether an edge condition was met. A checkpoint preserves execution state, and human-in-the-loop suspension depends on it. Those capabilities are valuable when debugging long flows, and all of them record the position of the graph, not the books of the external system. The Arize framework comparison lays this layer out plainly: the orchestration runtime supplies graphs, events, branches, retries and checkpoints, which is the state layer, not the side-effect layer.

Two products drawing the boundary in the same place is not a coincidence. An orchestration layer can only be accountable for state it owns. The idempotency of an external write has to be defined by both sides of that write, and no framework can supply that definition unilaterally.

Here is the structural change. For two years, buying orchestration meant asking whether a complex process could run end to end. From this year, it means asking whether the process can pick up after something goes wrong. Pick up gets read as restore state, while the buyer fears duplicated side effects. One phrase covers both, and the acceptance checklist drifts with it: teams now ask whether the product supports checkpoints instead of asking how each external write is made idempotent.

Three variants of the same failure

The window is not specific to refunds. Send an email with an attachment and the replay sends two, and the customer concludes the system is broken. Create a record in a ticketing system and the replay creates two, each gets picked up by a different person, and nobody can later say which one is true. Deduct inventory and the replay drives the position negative, which in a report looks like fraud rather than a bug.

All three share one property: nothing errors. The first attempt succeeds, the second succeeds, and both log lines say done. The system is internally consistent and the outside world has an extra copy. That is why these incidents surface as customer complaints rather than alerts. Monitoring watches whether a call succeeded; it cannot see semantic duplication.

The pattern has a name that predates agents: outbox. Write the intended action into a local table first, push it to the external system from that table asynchronously, and mark it complete only after the push succeeds. The value is not the asynchrony, it is that every external action leaves a local record first, so a crash gives you a row to reconcile from. Backend teams have used this for years and it needs explaining again, because orchestration frameworks pulled all eyes to the graph side.

Put a number on the duplication

The arithmetic is worth doing, because the result is usually larger than expected and cheaper to fix than expected. Suppose a month contains a hundred thousand steps that write to an external system, refunds, orders, invoices, notifications included, and three in a thousand land in the sent-but-unrecorded window. That is an assumption, not a measurement, and a reader should substitute their own rate.

Three in a thousand of a hundred thousand is three hundred duplicates. At twenty minutes each to reconcile, reverse and call the customer, at two hundred yuan an hour, that is twenty thousand yuan a month. It excludes what the customer concludes about the company after seeing two refunds, which does not appear in the ledger and does appear in next year renewals.

Three hundred a month has an unpleasant shape: enough to keep a reconciliation team busy, small enough that no alert fires. No error, no timeout, no 5xx, just one extra record sitting in the other system waiting for a customer to notice first.

The cost of the fix sits at the other end. Adding an idempotency key to an external write usually means one insert with a unique constraint, a few milliseconds. The test is crude and usable: if a year of duplication costs more than ten times the implementation, do idempotency before upgrading orchestration. The reverse also holds. A team with single-digit external writes per quarter, each with a human review, gains little from this engineering and can reasonably skip it.

Where our own ledger sits

Our internal task queue is a state machine table: pending, in progress, completed. Claiming is one state update, completing is writing the result and then changing the state. A crash between those two leaves the hardest record to find, the work is done and the record is not. The next morning another session sees a pending item and does the work again.

We have hit the concurrency version of this. Two sessions read the same batch of pending items, both started, and the results overwrote each other. The first instinct was to add a line to the prompt telling the agent to confirm state before starting. That did nothing, because the gap between read and write is not in the model, it is in the database. The real fix was to make claiming a conditional update: whoever succeeds in moving an item from pending to in progress owns it, and the loser takes the next batch.

External actions moved a layer later. The current shape writes an intent row locally with an idempotency key and the target system, executes the call, then marks it done. When a crash lands in the middle, the intent row is picked up by a scan that checks the other system for the truth. This is not clever design. It splits one action into three writes so that every crash leaves a trail someone can follow.

Limits, stated plainly. This is one company internal queue, not a bank core ledger, with small volume and low concurrency, and the worst duplicate incident added three rows. The mechanism is the same: an external write cannot commit atomically with a local record, so an intermediate state must be queryable. Scale only multiplies the number of times the window is hit. It does not change the window.

One part we have not solved. The intermediate state lives in a table with no time-window query exposed to anyone but the person who wrote the SQL. That makes reconciliation available to the author and unavailable to the on-call engineer. A channel that only its author can use does not exist at three in the morning.

Three questions to ask before buying

First: does every external write carry a stable idempotency key, what rule generates it, and where is it stored. If the answer is that the framework retries automatically, the answer does not hold, because automatic retry is the source of duplication. Distinguish retry policy from duplicate avoidance. The first has parameters for count and interval. The second requires that the receiver recognises you.

Second: when a replay happens after a crash, who checks the other system for whether the action ran. Is there a reconciliation channel, can it pull intended-but-uncertain rows for a time window. This question separates two kinds of vendor better than any feature list. One puts the ledger inside its own runtime. The other hands the ledger to the customer database.

Third: does the orchestration layer expose intermediate state to operations, in a table or event stream the customer can query, rather than only in the vendor console. A product that cannot do this leaves you reading logs after an incident, and logs are least complete precisely inside the crash window.

Failing all three does not make a product unbuyable. It makes the system a black box until the first duplicate incident. You can still buy it, provided duplicate detection ships before launch rather than after the incident.

Our judgment

Over the next year the competitive centre of orchestration moves from how many node types and models a product supports to which side holds the ledger. A product that keeps the ledger inside its framework takes the reconciliation capability with it when a customer switches. A product that puts the ledger in the customer database costs less to leave and demands that the customer design the tables and the state machine. That is a real trade with no free middle path.

The judgment makes a testable prediction. If a company changes orchestration frameworks and its duplicate side-effect rate does not move, idempotency lives in the application layer, which is good news. If the rate moves with the framework, the ledger lives inside it, the company is locked in, and does not know.

One step further. Agent reliability is splitting into two problem classes. Whether the judgment is right belongs to the model. Whether an action repeats belongs to distributed systems. The first improves through evaluation and context. The second improves through idempotency and reconciliation. Mixing them produces a common wrong conclusion, that a stronger model will make duplication disappear. It will not, because duplication is not a reasoning error. It is a timing question about when a commit is recorded.

That is also why we do not believe the sentence that choosing the right orchestration framework makes you safe. A framework can store state well, run retries to completion, and resume a long flow three days later. All of that is real and useful. What it cannot decide for you is whether that payment should go out again, because only you and the receiver can settle that.

Three actions you can take today

First, count last month external writes and how many carried an idempotency key. No new platform is needed. Group the call log by tool name. If the share is below half, orchestration upgrades are not the next item on the list.

Second, take one real duplicate incident and measure the window between issuing the call and recording it. It is usually a few hundred milliseconds, and it is hit far more often than anyone estimates, because being hit produces no error.

Third, put that window into a daily scan instead of into a procedure. Procedures depend on people remembering. Scans depend on time passing. The test for the scan is simple: is there any row sitting in an intermediate state for more than ten minutes, and does a person look when there is.

A question for whoever owns the architecture decision: after your agent crashes, which recovers first, its memory or its books. If only one of the two has been designed, the other will find you at night.

References

External material comes from the pages below. Internal figures come from our own task queue and call records.

Temporal, Who needs workflows when you have agents (November 10, 2026, Tao Guo: the post-ledger-pre-payment crash window and the case for state held outside the model)

Arize, AI agent frameworks compared (what the orchestration runtime records: graphs, events, branches, retries, checkpoints)