What actually needs to survive a crash in an AI agent

An agent runs for forty minutes. It gathers sources, writes a draft, and then calls charge_customer. The provider confirms the charge. Two hundred milliseconds later the worker process is killed — OOM, a deploy, a spot instance reclaim. When the agent restarts, nothing in its conversation history tells it whether the charge actually happened.

This is the failure that separates “the model can use tools” from “the agent can be trusted in production.” It is also where most persistence strategies quietly stop being sufficient. The question is not “how do I save the agent’s state?” It is narrower and much less forgiving: which facts must survive a crash for recovery to be safe?

The gap between output history and effect history

Almost every agent stack persists output: the messages, the tool calls, the tool results, the reasoning. That history is enough to reconstruct a conversation. It is not enough to reconstruct the world.

Consider the ambiguous window. A tool call has three moments that matter: it is requested, it takes effect somewhere outside your database, and its result is recorded. A crash can land between any two of them. If the process dies after the effect but before the result is recorded, the durable history contains a request with no outcome. Recovery now has two options, and both are bad:

• Retry the call — which may charge twice, send twice, or write twice.
• Skip the call — which may leave the workflow permanently missing work that actually needs to happen.

The transcript cannot resolve this ambiguity because the transcript only knows about the request. Deciding safely requires a durable record of the operation and its effect status, not just its output.

Three things people persist, and what each one actually saves

It helps to be precise about the difference between an observability system, a checkpoint, and durable execution. They are not three quality tiers of the same idea; they persist different things.

A trace is invaluable for debugging, but it is a witness, not a mechanism: it cannot resume anything, and it cannot fence a stale worker. A checkpoint limits how much you redo, but the instant between snapshots is still a blind spot, and a snapshot of state says nothing about an effect that is in flight. Durable execution closes both gaps by making every step — not every N minutes — a committed boundary and by giving each external operation a durable identity and status.

The unit that has to survive: the operation

The primitive that makes recovery safe is not the state blob. It is a record per logical tool invocation, holding four things:

1. A stable identity chosen by the caller and reused across retries — a key_name, not a fresh UUID per attempt.
2. An input fingerprint that decides whether a retry is the same operation or a different one. Volatile fields (timestamps, nonces, attempt counters) must be excluded, or every retry looks new.
3. A lifecycle status: claimed, in progress, completed, failed.
4. The recorded result, so a completed operation can be replayed without touching the outside world again.

With those four pieces, the ambiguous window becomes decidable. On restart, an operation that is already completed replays its stored result. An operation that is still claimed under a live lease is not stolen; an operation whose lease has expired can be safely reclaimed, because the previous owner can no longer write.

How Actae models the operation

Actae records ordered events and state transitions, and it keeps a dedicated tool-execution ledger for the side-effecting operations. Actae never runs your tool — the caller does. Actae makes the call idempotent and fenced. The claim is create-or-replay:

from actae_client import ActaeClient

async with ActaeClient(endpoint=..., api_key=...) as client:
    claim = await client.claim_execution(
        channel_id="run-7f2a",
        key_name="charge_customer:order-4471",   # stable across retries
        tool_name="charge_customer",
        params={"order_id": "4471", "amount_cents": 4200},
        dedup_fields=["order_id"],               # ignore volatile fields
        lease_seconds=120,
    )

    if claim.status == "replayed":
        # The charge already happened. Use the stored result; do not call again.
        result = claim.result
    else:
        # claim.status is claimed / reclaimed — we own the lease.
        try:
            result = await charge_customer(claim.execution.params)   # the actual effect
            await client.complete_execution(
                claim.execution.id, claim.claim_token, result=result,
            )
        except Exception as exc:
            await client.fail_execution(
                claim.execution.id, str(exc), claim_token=claim.claim_token,
            )
            raise

The idempotency unit is (channel_id, key_name). A retry with the same key and the same input fingerprint returns the persisted result with status == "replayed" and never re-executes. A retry with a different input under the same key is a programming error and fails loudly instead of silently double-charging. While the tool is running, periodic heartbeats extend the lease; when ownership changes, the old token is rejected rather than allowed to overwrite the new owner.

The same idempotency discipline applies one level up. Recording an event, a state transition, or a fork accepts an operation_id; reusing it replays the original result, and reusing it with different content fails with a 409 idempotency_conflict. State transitions go further and commit the event and its snapshot in a single PostgreSQL transaction, guarded by expected_version and expected_cursor so a stale writer cannot silently clobber newer progress.

What you still own

Durable execution removes the ambiguity inside your system. It does not magically make an arbitrary external API idempotent. You still own three decisions:

• Propagating idempotency outward. Where the provider accepts an idempotency key (payments, email, many SaaS APIs), pass your key_name through. That closes the window on the provider’s side, not just yours.
• Compensation for effects that cannot be made idempotent. If an effect is genuinely non-idempotent and non-reversible, model the compensating action as its own durable step.
• Key lifetime. Choose a key granularity — per order, per attempt, per user action — that matches what “the same logical operation” means for your domain.

The payoff beyond safety: not re-running the expensive prefix

Once every step is a durable boundary, the most expensive part of iterating on an agent becomes optional. You can fork at step N and refine step N+1 while inheriting the real accumulated output of steps 1..N — without paying for them again.

That is not a claim we ask you to take on faith. The repository ships a measured token report and an LLM-judged fidelity suite:

• A deterministic dry-run of the flagship workflow reports 17,660 baseline tokens versus 9,400 forked tokens — 8,260 tokens saved (46.8%) by not re-running the first five steps.
• Five unrelated agent scenarios scored by a temperature-0 judge pass fork fidelity at a mean of 8.47 / 10 versus 7.07 / 10 for a full re-run control, with all five scenarios passing the acceptance threshold.

Both artifacts are committed and reproducible. The point is not the exact numbers — it is that “the fork behaves like the original run” is an assertion you can test, not a promise.

A short checklist

Before you trust an agent to run unattended, ask:

• Does every side-effecting call have a stable, caller-chosen identity?
• Is the input fingerprint defined over business fields only?
• Is there a durable status for each operation, or only its output?
• Can a completed operation replay its result without re-executing?
• Can a stale worker be fenced out when a lease lapses?
• Do event and state commit atomically, or can a crash wedge them apart?
• Is completion recorded after the effect, and the effect made idempotent at the provider too?

If those all hold, a crash is an inconvenience. If they do not, a crash is a coin flip between duplicated work and lost work.