All engineering notesENGINEERING NOTES

Checkpointing vs durable execution for AI agents

A checkpoint limits how much work you redo. Durable execution decides whether redoing it is safe at all. For agents, the difference is the whole ballgame.

Actae Engineering10 min read
durable executioncheckpointingreplayAI agents

“Checkpointing” and “durable execution” are used almost interchangeably in agent tooling, which is unfortunate, because they answer different questions. A checkpoint answers how much work do I lose? Durable execution answers is it safe to do that work again? You need both answers, and they come from different mechanisms.

What a checkpoint actually is

A checkpoint is a snapshot: at some interval, serialize the agent’s state and store it. On restart, load the most recent snapshot and continue. The desirable property is that recovery is cheap to implement — read a blob, keep going.

The two costs are easy to underestimate in the agent setting. First, everything that happened after the last snapshot is gone and must be recomputed. Second — and this is the part that bites — a snapshot records state, not effects. It cannot tell you whether an external action the agent requested actually landed.

What durable execution actually is

Durable execution makes every step a committed boundary rather than every N minutes. Two properties follow:

  • The progress record is complete and atomic. A step and the state it produces commit together, so a crash cannot leave you with a half-updated step.
  • Completed steps replay their recorded result instead of re-executing. Recovery is a walk forward over recorded results, not a re-run.

For that second property to hold across side effects, the engine also needs the durability of operation identity described in the previous post: a stable key per logical invocation, a status, and a stored result.

The differences that matter

DimensionCheckpointingDurable execution
Recovery granularityThe last snapshotThe last committed step
Replays completed work?Yes — everything after the snapshotNo — completed steps return recorded results
Handles in-flight side effects?No mechanismOperation ledger with leases and fencing
Atomicity of progress + stateTwo separate writesOne transaction
Deterministic output on re-runAssumed, often false for LLMsNot required — results are replayed, not regenerated

Why LLM agents break the checkpoint assumption

Traditional checkpointing quietly assumes that re-running a step from a snapshot reproduces the same output. For deterministic business logic, that is reasonable. For an agent, it is frequently false: the model is sampled, the world has moved on, and a tool called twice is called twice.

Three properties compound:

  1. Steps are expensive and slow. A single step may be a 30-second reasoning call over a large context. Losing ten of them is minutes of wall clock and real money — the fork and resume workflow exists precisely because re-running a good prefix is wasteful.
  2. Outputs are not reproducible. Re-running a step is not “recovery”; it is a new sample. The trajectory after the checkpoint may diverge, which is acceptable when you intended it and alarming when you did not.
  3. Steps touch the outside world. A checkpoint can make you re-call an API that already succeeded. That is not a performance regression; it is a correctness bug.

Snapshots still matter — for a different job

This is not an argument against snapshots. It is an argument against using a snapshot as the recovery mechanism. Actae stores both:

  • An ordered, gapless event log per channel, where each state transition pins a state version to an event cursor. This is what recovery walks.
  • Immutable, versioned state snapshots, where every save creates a new version instead of overwriting. This is what forking and inspection use.

Recovery is a cursor walk: resume from the last committed cursor, load the state version at or before that point, and replay recorded results forward. Forking is a snapshot operation: copy the state at or before a boundary into a child channel, record the lineage, and continue there. The snapshot is excellent at scoping a fork; it is the wrong tool for deciding whether an effect happened.

# The shapes differ:
#
#   checkpoint recovery   : state = load_last_snapshot(); redo everything after it
#   durable recovery      : state = state_at_or_before(cursor)
#                           for event in events_after(cursor): apply_recorded_result(event)
#
#   fork (fork)         : child_state = snapshot_at_or_before(step_n)
#                           child continues at step n+1 with inherited state

When a checkpoint is enough

Durable execution is not free — it is more infrastructure than a blob store. It is the right tool when at least one of these is true:

  • The run is long enough that re-running the prefix is expensive.
  • Steps have side effects that must not happen twice.
  • Recovery has to be exact, not “close enough.”
  • You need to fork, diff, or audit execution history.

A checkpoint remains perfectly reasonable when the run is short, the steps are pure, re-running is cheap, and a repeated side effect is harmless. The mistake is choosing checkpointing for convenience and then discovering, in production, that one of those assumptions was false.

How Actae implements the durable side

  • Atomic transitions. The event and its cursor-aligned state snapshot commit in one PostgreSQL transaction, guarded by expected_version and expected_cursor.
  • Idempotent record/transition/fork. An operation_id replays the original result; reusing it with different content fails with a conflict rather than overwriting.
  • A tool-execution ledger. Claim, complete, fail, and heartbeat with leases and fencing tokens, so the ambiguous window is explicit.
  • Replay-to-live delivery. A subscriber catches up from a cursor and crosses into live delivery with no application-built handoff.
  • Fork receipts. Each fork records the requested and resolved boundary plus reproducibility metadata, so the lineage is auditable.

The design goal is simple to state: when a worker dies, the system should know exactly which steps are done, which results are recorded, and which effects are still ambiguous — and it should be able to prove it from durable history rather than hope.