Never ran, or ran but I didn't hear back?

The replay-to-skip-LLM-calls part is the easy 80% of this problem. A model call is pure from the workflow’s point of view — skipping it and replaying the recorded output is safe, and that alone saves real money and real wall-clock.

The part that actually bites in production is what happens when a replayed step has an external side effect. Not “call the LLM again,” but “send the email,” “charge the card,” “call this third-party API that isn’t idempotent on its own.” If a step does both — call the model and then call the external API with the result — and the crash lands after the external call succeeds but before the workflow records it, replay re-runs that step and double-fires the side effect.

At-least-once activity semantics help, but they do not give you this for free. “Make your activities idempotent” just relocates the problem to you. So here is how I’ve seen the window actually get closed.

Mint the key up front, and make it content-addressed

The key should be deterministic, minted by the SDK so callers can’t forget it — not a random UUID generated at call time. The version that has held up for me is a UUID derived over (scope/channel, step number, canonical content hash). The content hash is the important half: it makes the key describe this specific work, not just “whatever step 7 is this time.”

That gives you two rules, and the second one is the one I’d steal:

• Same key + identical content → replay. Return the original record. A crashed re-drive is free.
• Same key + changed content → an explicit conflict error, never a silent second record.

The conflict rule catches the “retry mutated the payload” class of bugs. If a retry changed what the step is supposed to do, that is a different operation wearing the same name, and you want to hear about it loudly rather than discover it as a duplicate line item.

The ledger is what answers “did step 7 finish?”

For the side-effect gap, I went one further: tools get claimed and completed through an execution ledger keyed by that operation id. “Did step 7’s tool actually finish?” is then queryable independently of the event log, including heartbeats and cancels for long-running calls. A claim is create-or-replay, and the caller branches on a status field rather than guessing:

claim = await client.claim_execution(
    channel_id="run-42",
    key_name="charge:order-4471",     # deterministic, content-addressed
    tool_name="charge_customer",
    params={"order_id": "4471", "amount_cents": 4200},
    dedup_fields=["order_id"],        # ignore volatile fields in the hash
    lease_seconds=120,
)

if claim.status == "replayed":
    result = claim.result              # the work already happened
elif claim.status == "in_progress":
    ...                                # someone else owns the live lease
else:                                  # "claimed" or "reclaimed" — you own it
    result = await charge_customer(**claim.execution.params)
    await client.complete_execution(
        claim.execution.id, claim.claim_token, result=result,
    )

Still not exactly-once against the outside world — nothing is, without a downstream idempotency check. But the duplicate window collapses to “downstream ignored our key,” which is a much smaller and much more testable claim.

The question the network can’t answer

Here is the case that separates a real ledger from a log. Step 7 completes, the ledger records it, and the response never makes it back to the caller — connection drop, timeout, whatever. Does a retry with the same key return the original recorded result, or is the caller stuck unable to tell “never ran” from “ran but I didn’t hear back”?

I hit exactly this on a payment endpoint: settlement succeeds against the facilitator, but the HTTP response back to the buyer’s agent still fails to land. The fix is to make the success path itself replayable, and two paths cover it:

• A duplicate complete on an already-completed execution returns the stored result and emits no new event, so the retry is free.
• Re-claiming the same key when the execution is already completed with a matching request hash returns replayed with the original result payload — not None, not an error.

So the caller branches on the status field: claimed means you own new work, replayed means the work already happened and here is the result. “Never ran” versus “ran but I didn’t hear back” becomes answerable because the ledger is the source of truth, not the network.

The payment case maps directly: settlement already committed on the facilitator, the connection dies, the caller retries with the same key, gets replayed plus the settlement result, and there is no double settle. Reconciling with the facilitator for its side is still your job; the runtime-side duplicate is closed.

Honest caveat: the result body expires

After the retention window, the stored result body gets purged and the same key fails loudly — a 409 result_expired — instead of returning stale data. That is deliberate. It is better to tell the caller to reconcile than to hand back a result that may no longer reflect reality. A tombstone keeps the key identity so you can never silently re-execute an operation whose outcome you threw away.

The other design: signed capability tokens

The other engineer in that thread runs a live x402 seller, and named a genuinely different approach. There, the artifact returned after settlement isn’t a ledger record at all — it’s a signed capability token (product id and expiry baked in, HMAC’d). Re-presenting the same completed payment just re-derives the same token deterministically, so the happy path needs zero ledger lookup; only the failure path does (reject a token whose signature or expiry doesn’t check out).

The tradeoff he named is real: you lose the queryable “did step 7 actually finish” introspection, because there is no durable record of in-flight state — only of completed ones. That is fine for a single external side effect per request. It would fall over fast for anything with a multi-step tool-call shape.

They’re complementary, not competing

The signed-token approach is clean for the happy path: the proof is self-contained and the ledger is unnecessary for anything that doesn’t fail. The introspection cost is correctly scoped for single-step flows.

What the token can’t answer is the question the ledger was built for: between step 1 and step 8, which steps were safe internal work and which ones touched the outside world? That’s the shape where intent-before-effect records earn their keep — not to prove a payment happened, but to know that you didn’t accidentally double-send while proving it.

The token is the external proof. The step-level record is the internal orchestration state. You want both, and they answer different questions.

If you want to see how this is implemented rather than just described, the mechanics live in idempotent tool executions and forking & recovery. Disclosure, as always: I work on Actae, which is the thing implementing the ledger described above.