Fork a crashed agent run without re-running the prefix

If you have built a long-running agent, you have experienced this: the run reached 90% completion, made one bad decision, and your only options were to read the trace or start over. "Start over" means paying for the prefix again and re-triggering any side effects the prefix caused.

This post is a practical walkthrough that you can execute with the free self-hosted runtime. You will learn how to make a side effect idempotent, fork a run at the exact step where an issue occurred, continue only from the tail, and demonstrate what changed. Every API call mentioned below is real (Python SDK), and we conclude with twofalsifiable measurements a reviewer should request, rather than a number we created.

1. Start the free runtime, install the SDK

The community build is a single file that includes the API, dashboard, and a managed PostgreSQL. It displays your URL and a one-time API key at the end of the startup process.

# from https://actae.dev/download — pick your platform
$ tar -xzf actae-<version>-<platform>.tar.gz && chmod +x ./actae && ./actae
# ...
# Dashboard:   http://127.0.0.1:8002
# API key:     sk-...   (shown once — store it now)

$ pip install actae-client

2. The whole demo, in one runnable file

This is the complete program. It operates without an API key, as the "model" is a deterministic stub, allowing you to verify the execution mechanics independently.

#!/usr/bin/env python3
"""Fork a crashed agent run without re-running the prefix.
    ACTAE_URL=... ACTAE_API_KEY=... python fork_demo.py"""
import asyncio, copy, os, uuid
from actae_client import ActaeClient
from actae_client.session import AgentSession

PIPELINE = ["research", "insights", "risks", "recommendations", "summary", "finalize"]
FORK_AT_STEP = 3   # fork after step 3: the tail is steps 4..6


async def call_model(step, state):
    # Your model call. Deterministic here so the demo needs no API key.
    # This is also where the "intervention" is applied: the same code path
    # reads state["model"], so changing state["model"] changes the decision.
    if step == "recommendations" and state["model"] == "fast":
        return "(fast draft — low quality)"      # the bad decision at step 4
    return f"[{state['model']}] {step} -> ok"


async def charge(client, session, key, amount_cents):
    # A real side effect, made idempotent by the execution ledger.
    claim = await client.claim_execution(
        session.channel_id, key_name=key, tool_name="charge_customer",
        params={"amount_cents": amount_cents},
    )
    if claim.status == "replayed":               # a prior attempt already ran it
        return claim.result                      # -> do NOT charge again
    result = {"invoice": key, "cents": amount_cents, "attempt": uuid.uuid4().hex}
    await client.complete_execution(claim.execution.id, claim.claim_token, result=result)
    return result


async def baseline(client, channel):
    state = {"topic": "durable execution", "model": "fast", "report": None,
             "invoice": None, "status": "running", "error": None}
    session = AgentSession(
        client, channel,
        state_fn=lambda: copy.deepcopy(state),   # snapshot the real state, every step
        snapshot_interval=1,
    )
    async with session:
        for step in PIPELINE:
            if step == "recommendations":
                state["invoice"] = await charge(client, session, "inv-1", 4900)
            state["report"] = await call_model(step, state)
            await session.step(step, output={"report": state["report"]})
        state["status"], state["error"] = "failed", "low_quality_report"
    return session, state


async def fork_and_fix(client, base):
    fork = await base.fork(
        FORK_AT_STEP,                            # exact boundary: after step 3
        name=f"{base.channel_id}-slow",          # channel ids: [A-Za-z0-9._:-], no "/"
        intervention={"model": "slow"},          # opaque; the app reads and applies it
        tool_policies={"charge_customer": "block", "*": "auto"},
    )
    state = copy.deepcopy(fork.inherited_state or {})   # the real state at the boundary
    state["model"] = fork.intervention["model"]         # apply the intervention
    async with fork:                             # continues at step 4; 1..3 not re-run
        for step in PIPELINE[FORK_AT_STEP:]:
            state["report"] = await call_model(step, state)
            await fork.step(step, output={"report": state["report"]})
        state["status"], state["error"] = "ok", None
    return fork, state


async def main():
    client = ActaeClient(endpoint=os.environ["ACTAE_URL"], api_key=os.environ["ACTAE_API_KEY"])
    await client.connect()
    try:
        channel = f"fork-demo-{uuid.uuid4().hex[:8]}"
        base, base_state = await baseline(client, channel)
        base_events_before = await client.replay(base.channel_id, cursor=0, limit=500)

        fork, fork_state = await fork_and_fix(client, base)

        r = fork.fork_receipt
        assert r.restorable and r.requested_cursor == r.resolved_cursor
        print("boundary restorable:", r.restorable, "| sha256:", r.source_state_sha256[:16])

        diff = await client.diff_states(base.channel_id, fork.channel_id)
        print("diverged paths:", sorted(".".join(e["path"]) for e in diff["entries"]))
        print("shared ancestor status:", diff["common"]["state"]["status"])
        print("baseline:", base_state["report"], "/", base_state["status"])
        print("fork:    ", fork_state["report"], "/", fork_state["status"])

        # The parent is immutable: a fork is copy-on-write, not a mutation.
        base_events_after = await client.replay(base.channel_id, cursor=0, limit=500)
        assert len(base_events_after) == len(base_events_before)
    finally:
        await client.disconnect()


asyncio.run(main())

Run it against your instance. You should observe that the fork's state diverges only on tail paths, the shared ancestor is flagged as failed, the baseline report remains unchanged ((fast draft — low quality)) while the fork's is [slow] …, and the parent's event count is identical before and after.

3. The details that matter in production

Effect identity is distinct from execution. The ledger key is (channel_id, key_name), and the request hash is calculated over the params. A replay occurs only when the key matches and the request is equivalent; a different request under the same key produces an idempotency mismatch instead of silently replaying the incorrect result. Volatile fields, such as timestamps, nonces, and attempt ids, can be excluded with dedup_fields, ensuring that a retry is not considered a different request.

Per-tool fork policy is explicit. tool_policies determines what occurs with inherited work on the fork:

• auto (default): an exact inherited call replays the recorded result, while new or changed work runs live.
• replay: an inherited call must replay. A miss or a mismatch raises fork_tool_blocked instead of running.
• block: the tool is completely refused on the fork. Use it for irreversible effects such as payments and emails to ensure a counterfactual can never re-trigger them.
• live: always run (today’s behaviour).

Snapshots are what allow a step to be forkable. The session saves a state snapshot after each step based on what your state_fn returns at that moment. With snapshot_interval=1, every step is restorable, enabling boundary_mode="exact" to succeed. If snapshots are skipped, the strict boundary fails loudly instead of inheriting later state, which is the main purpose of exact.

Multiple writers are fenced. If more than one process can drive a session, construct it with fence_owner. The session claims a per-channel write lease, and a stale owner's step is rejected. This prevents two workers from silently appending interleaved steps.

Resume without holding the object. The process typically ends when you want to fork. Resume by channel:

resumed = await AgentSession.resume(
    client, "research-7f2a",
    fork_at_step=3, name="research-7f2a-slow",   # steps 1..3 inherited
    intervention={"model": "slow"},
)
async with resumed: ...

4. Where this breaks (the honest list)

• Exactly-once requires a cooperative downstream. Actae fences, retries, and replays recorded results, but it cannot ensure an external system operates in exactly-once mode on its own. The downstream must respect your idempotency key or be queryable; otherwise, the window narrows from "any duplicated step" to "the one dependency that ignored your key," which is the smallest honest claim.
• Only ledgered effects receive protection. The ledger includes tool calls that pass through it, either directly or through the tool_policies fork path. Any bare side effect triggered by your code outside the ledger is not tracked. The reliability report clearly outlines this boundary instead of misleadingly suggesting otherwise.
• Fork continuation is managed by the application. The runtime retains the intervention and displays it on the fork; your code needs to read it and adjust its behaviour. It will not select your model or modify your prompt.
• Strict boundaries require snapshots. Not having a snapshot at the step with boundary_mode="exact" set raises an error. Use approximate or lineage_only intentionally, understanding that the inherited state is no longer an exact checkpoint.

5. The evidence a reviewer should ask for

Two claims are significant, and both are measured in the repository, not asserted. This section is where you should hold us accountable. Everything below is provided with its raw JSON downloadable at benchmark evidence, and the harnesses themselves are public and re-runnable at github.com/BViganotti/actae-benchmarks.

Fork fidelity evaluates whether the fork matches a no-fork re-run. Five unrelated agent scenarios are run twice: once by forking at an intermediate step and once from scratch. A DeepSeek judge at temperature 0 scores how close each final output is to the canonical run. Across the five scenarios, the fork path scores 8.47/10 while the full re-run control scores 7.07/10, indicating that skipping the prefix did not lose output fidelity, and re-running a stochastic prefix is a source of drift. The committed report is located at examples/fidelity-reports/latest.json; reproduce it using examples/fork_fidelity_suite.py.

Crash consistency examines whether external effects occur twice. A worker is terminated with os._exit(137) during execution against a real external service that uses an idempotency key and a status query. In the committed 6-step benchmark, a simple retry causes 4 effects to be reapplied. The execution ledger limits this to at most 1 in the genuinely ambiguous window, where the effect completed but the response was lost, and 0 once the completion is recorded. Querying the downstream on that ambiguity with durable_query results in 0 duplicates, even at the drop or transport-failure crash points. The committed report is examples/durability-benchmark-results.json; reproduce using examples/durability_benchmark.py.

One number we advise you not to trust is the token-savings figure printed by the deterministic demo. In the committed fixture, it shows 17,660 tokens for the full run compared to 9,400 for the fork tail, indicating a saving of 8,260 tokens, or 46.8%. This calculation comes from a fixed usage table we selected, not from a measurement, so consider it an illustration of the mechanism. The two reports above serve as the evidence.

Try it, for free

Everything here operates on the free self-hosted community build, which consists of one binary with embedded PostgreSQL. There is no account, no card, and no control plane needed. You can download it at actae.dev/download and follow the quickstart instructions. The fork workflow is documented at forking & recovery, the ledger is covered at coordination, the failure evidence can be found at reliability, and the canonical API is available at the API reference. If you prefer it to be operated for you, Cloud runs the same runtime.

An optional free account allows you to link the instance to the portal (actae enroll <key>) to manage keys and receive release notices. Linking is opt-in; an unlinked instance phones home to nobody.

Disclosure: we build Actae, the runtime in this post. The fork receipt, the execution ledger, the benchmark harnesses, and the committed reports are all under the open-source Python SDK and the repository. Every claim above is something you can verify rather than accept on faith, which is the only version of this post worth writing.