“Checkpointing” and “durable execution” are used almost interchangeably in agent tooling, which is unfortunate, because they answer different questions. A checkpoint answers how much work do I lose? Durable execution answers is it safe to do that work again? You need both answers, and they come from different mechanisms.
What a checkpoint actually is
A checkpoint is a snapshot: at some interval, serialize the agent’s state and store it. On restart, load the most recent snapshot and continue. The desirable property is that recovery is cheap to implement — read a blob, keep going.
The two costs are easy to underestimate in the agent setting. First, everything that happened after the last snapshot is gone and must be recomputed. Second — and this is the part that bites — a snapshot records state, not effects. It cannot tell you whether an external action the agent requested actually landed.
What durable execution actually is
Durable execution makes every step a committed boundary rather than every N minutes. Two properties follow:
- The progress record is complete and atomic. A step and the state it produces commit together, so a crash cannot leave you with a half-updated step.
- Completed steps replay their recorded result instead of re-executing. Recovery is a walk forward over recorded results, not a re-run.
For that second property to hold across side effects, the engine also needs the durability of operation identity described in the previous post: a stable key per logical invocation, a status, and a stored result.
The differences that matter
Why LLM agents break the checkpoint assumption
Traditional checkpointing quietly assumes that re-running a step from a snapshot reproduces the same output. For deterministic business logic, that is reasonable. For an agent, it is frequently false: the model is sampled, the world has moved on, and a tool called twice is called twice.
Three properties compound:
- Steps are expensive and slow. A single step may be a 30-second reasoning call over a large context. Losing ten of them is minutes of wall clock and real money — the fork and resume workflow exists precisely because re-running a good prefix is wasteful.
- Outputs are not reproducible. Re-running a step is not “recovery”; it is a new sample. The trajectory after the checkpoint may diverge, which is acceptable when you intended it and alarming when you did not.
- Steps touch the outside world. A checkpoint can make you re-call an API that already succeeded. That is not a performance regression; it is a correctness bug.
Snapshots still matter — for a different job
This is not an argument against snapshots. It is an argument against using a snapshot as the recovery mechanism. Actae stores both:
- An ordered, gapless event log per channel, where each state transition pins a state version to an event cursor. This is what recovery walks.
- Immutable, versioned state snapshots, where every save creates a new version instead of overwriting. This is what forking and inspection use.
Recovery is a cursor walk: resume from the last committed cursor, load the state version at or before that point, and replay recorded results forward. Forking is a snapshot operation: copy the state at or before a boundary into a child channel, record the lineage, and continue there. The snapshot is excellent at scoping a fork; it is the wrong tool for deciding whether an effect happened.
# The shapes differ: # # checkpoint recovery : state = load_last_snapshot(); redo everything after it # durable recovery : state = state_at_or_before(cursor) # for event in events_after(cursor): apply_recorded_result(event) # # fork (fork) : child_state = snapshot_at_or_before(step_n) # child continues at step n+1 with inherited state
When a checkpoint is enough
Durable execution is not free — it is more infrastructure than a blob store. It is the right tool when at least one of these is true:
- The run is long enough that re-running the prefix is expensive.
- Steps have side effects that must not happen twice.
- Recovery has to be exact, not “close enough.”
- You need to fork, diff, or audit execution history.
A checkpoint remains perfectly reasonable when the run is short, the steps are pure, re-running is cheap, and a repeated side effect is harmless. The mistake is choosing checkpointing for convenience and then discovering, in production, that one of those assumptions was false.
How Actae implements the durable side
- Atomic transitions. The event and its cursor-aligned state snapshot commit in one PostgreSQL transaction, guarded by
expected_versionandexpected_cursor. - Idempotent record/transition/fork. An
operation_idreplays the original result; reusing it with different content fails with a conflict rather than overwriting. - A tool-execution ledger. Claim, complete, fail, and heartbeat with leases and fencing tokens, so the ambiguous window is explicit.
- Replay-to-live delivery. A subscriber catches up from a cursor and crosses into live delivery with no application-built handoff.
- Fork receipts. Each fork records the requested and resolved boundary plus reproducibility metadata, so the lineage is auditable.
The design goal is simple to state: when a worker dies, the system should know exactly which steps are done, which results are recorded, and which effects are still ambiguous — and it should be able to prove it from durable history rather than hope.