Everyone making a durable-execution claim says the same thing: replay makes recovery safe. It is easy to say and rarely measured. So we measured the part that actually matters — not “does it resume,” but how many times does an external side effect fire again after a real process death?

The harness is committed in the repository (examples/durability_benchmark.py), the raw report is committed alongside it, and the unit tests for the arithmetic run without a server. Here is what it found, including the result that is unflattering to us.

How the test works

Real crashes, not simulated exceptions: the worker is a separate OS process that dies with os._exit(137) at an injected point, so in-memory progress is genuinely lost. The external system is a local SQLite database, so it is durable and independent of the worker.

6 steps, each with one external effect, keyed step-<n>.
One crash at step 3, at one of two windows: after the effect but before completion is recorded (the ambiguous window), or after completion but before the next step.
Three strategies: naive (no durable progress), checkpoint (a marker written after each step), and durable (claim/complete through Actae’s execution ledger with a stable key).
Two downstreams: non-idempotent (applies every call) and honors_key (dedupes on the caller’s key). Only the ledger strategy has a stable key to hand over; naive/checkpoint don’t, so each retry presents a fresh one.

The numbers

StrategyCrash pointDownstreamDuplicate effects

naiveafter_effectnon_idempotent4

naiveafter_effecthonors_key4

naiveafter_completenon_idempotent4

naiveafter_completehonors_key4

checkpointafter_effectnon_idempotent1

checkpointafter_effecthonors_key1

checkpointafter_completenon_idempotent0

checkpointafter_completehonors_key0

durableafter_effectnon_idempotent1

durableafter_effecthonors_key0

durableafter_completenon_idempotent0

durableafter_completehonors_key0

No strategy lost work — every configuration applied all six effects. The failure mode being measured is duplication, not loss.

The uncomfortable line

Look at durable | after_effect | non_idempotent: 1 duplicate, the same as checkpoint. That is the honest ceiling. If the effect landed on the external system and we crashed before recording completion, no local ledger knows the effect happened. A reclaim re-runs it. We say this plainly because the alternative — implying the ledger is magic — sets you up for a duplicate charge in production.

The line that does change

durable | after_effect | honors_key is 0, while checkpoint | after_effect | honors_key stays at 1. Same downstream, same crash — the difference is that only the ledger has a stable operation key to put on the wire. The checkpoint strategy literally has nothing deterministic to send.

What this means in practice

The stable key is the asset, not the ledger table. A durable record is necessary but not sufficient. What closes the window is a deterministic, content-addressed operation id that you mint before the effect and propagate to the downstream.
Ask the downstream to honor it. Payments, email, and many SaaS APIs accept an idempotency key. Pass yours through. Where a provider genuinely can’t, reconciliation is the only remaining answer.
Checkpointing is fine until it isn’t. For short, pure, cheap steps a checkpoint is enough. The moment a step has a side effect and a crash can land inside it, the checkpoint’s blind spot is a real duplicate.
“Just retry the job” is the worst option. Naive recovery re-ran four already-applied effects in our run. That is the baseline cost of pretending recovery is free.

Reproduce it

# Start Actae (embedded), then:
ACTAE_URL=http://127.0.0.1:8002 ACTAE_API_KEY=sk-dev-0000000000000000000000 \
  python3 examples/durability_benchmark.py \
  --out examples/durability-benchmark-results.json

# Measurement arithmetic, no server:
python3 -m pytest sdks/example-tests/python/test_durability_benchmark.py -v

Methodology, raw numbers, and limitations are committed in the repository under docs/DURABILITY_BENCHMARK.md; the mechanics being measured are idempotent tool executions.

Caveats, because you should be suspicious

Six steps, one crash per run. This isolates the ambiguous window; it is not a throughput or latency benchmark.
SQLite models a durable, non-transactional third party. Real network semantics (timeouts, partial writes) are broader.
The honors_key case presumes the downstream supports idempotency keys. Where it doesn’t, the non_idempotent column is your reality.
We did not compare against Temporal or any other product. That would require running them, and we would rather publish a number we actually produced than a comparison we did not.

Disclosure: I work on Actae, which is the ledger used in the durable configuration above.