Durable AI Agents: Crash Recovery Without Temporal

A long-running AI agent will be killed mid-run — an OOM kill, a deploy, a kill -9, a spot instance reclaimed. The question is what happens when it comes back. The durable-execution vendors answer it by asking you to adopt a new platform: Temporal, DBOS or Inngest, each wrapping your agent in their runtime. This guide shows the other answer — crash recovery you build yourself in Python on Promptise Foundry's journal, with no extra platform — and it is honest about where that stops. Promptise recovers an agent's *position* after a crash; it does not deduplicate the agent's *side effects*. We prove both with 50 real kill -9 injections, and show the two lines of your own code that stop a crash from issuing a refund twice. Everything here ran against Promptise Foundry 1.3.3 from PyPI.

Not for this. "Durable" conflates two things that are worth separating:

1. Position recovery — after a crash, knowing which step the agent had reached so it can resume instead of starting over. Promptise's journal and ReplayEngine do this out of the box. 2. Exactly-once side effects — making sure a tool that moved money, sent mail or shipped an order does not run twice when the agent resumes. No framework gives you this for free, Temporal included; what durable-execution engines give you is machinery to make it *easier*. The actual guarantee always comes from an idempotency key that your side-effecting tool checks.

Promptise covers the first and hands you the second. That is the right split for most teams: you keep your own stack, you get resume-from-crash, and you add idempotency exactly where money moves. This guide builds a worker that does both.

Before claiming anything, we read the source. The journal (promptise.runtime.journal) records process events to a backend; FileJournal writes JSONL plus a checkpoint file. ReplayEngine.recover(process_id) loads the last checkpoint, replays the context_update and state_transition entries after it, and returns the reconstructed context_state and lifecycle_state. That is position recovery.

What is not there: any idempotency or deduplication of tool calls. ReplayEngine rebuilds *state*; it never re-runs or skips a tool. The MCP server's idempotent_hint is an advisory annotation it advertises to clients — metadata, not enforcement. There is a within-a-single-run ledger that can answer a repeated tool call from an earlier result, but that is for one live run, not for resuming across a crash. So the correct, verified claim is: Promptise resumes your agent; stopping duplicate side effects on resume is your tool's job. The rest of this guide is how to do that job.

The worker processes ten refunds. Each step issues the refund (the side effect — a line in a payment ledger that stands in for the provider), then journals its progress with a context_update. On start it recovers its position and resumes.

import asyncio, json, os, signal
from pathlib import Path
from promptise.runtime.journal import FileJournal, JournalEntry, ReplayEngine

PROCESS_ID = "refund-worker"
ORDERS = [(f"O-{i+1}", 1000 + i * 10) for i in range(10)]
LEDGER = Path(os.environ["LEDGER"])          # the external system of record
IDEMPOTENT = os.environ.get("IDEMPOTENT") == "1"
CRASH_AT = os.environ.get("CRASH_AT", "")    # "<step>:<phase>", injected by the test

def ledger_keys():
    if not LEDGER.exists():
        return []
    return [json.loads(l)["key"] for l in LEDGER.read_text().splitlines() if l]

def refund(order_id, amount, key):
    # With an idempotency key, a refund whose key is already recorded is not issued again.
    if IDEMPOTENT and key in set(ledger_keys()):
        return {"status": "duplicate_suppressed", "key": key}
    with LEDGER.open("a") as f:
        f.write(json.dumps({"order_id": order_id, "amount": amount, "key": key}) + "\n")
    return {"status": "refunded", "key": key}

async def main():
    journal = FileJournal(base_path=os.environ["JOURNAL_DIR"])
    recovered = await ReplayEngine(journal).recover(PROCESS_ID)
    start = recovered["context_state"].get("last_done", -1) + 1     # resume here

    for i in range(start, len(ORDERS)):
        order_id, amount = ORDERS[i]
        refund(order_id, amount, key=order_id)                      # the side effect
        await journal.append(JournalEntry(process_id=PROCESS_ID, entry_type="context_update",
                                          data={"key": "last_done", "value": i}))
    await journal.append(JournalEntry(process_id=PROCESS_ID, entry_type="state_transition",
                                      data={"to_state": "completed"}))

The idempotency key is the order id: stable, so a re-run of the same order presents the same key. That stability is the whole trick — it lets the ledger recognise a repeat.

The crash is a real kill -9: at the injected point the worker sends itself SIGKILL, which cannot be caught, so nothing is flushed or cleaned up — exactly what a hard kill does. We crash it in the danger window: after order 3's refund is written to the ledger, but before that step is journalled. Then we restart it with no crash and read the journal and the ledger. First with no idempotency key:

==== with NO idempotency key ====
run 1 exit code: -9  (-9 means SIGKILL / kill -9)
ReplayEngine.recover(): last_done=2, lifecycle=created, entries_replayed=3
ledger after the crash (4 refunds): ['O-1', 'O-2', 'O-3', 'O-4']
run 2 (resume) exit code: 0
ledger after resume: 11 refunds, 10 unique, 1 duplicate(s)
duplicated order(s): ['O-4']  <-- refunded twice

Read that carefully. The journal's last recorded step was 2 (O-3), because the crash hit before step 3 was written. But the ledger already had O-4 — the refund fired, then the process died in the gap before journalling it. On resume, recover() honestly reports last_done=2, so the worker redoes step 3 and refunds O-4 a second time. Position recovery worked perfectly and still produced a duplicate refund. This is the gap no replay engine can close for you: the window between the side effect and the record of it.

Now the same crash, same window, with the idempotency key:

==== WITH an idempotency key ====
run 1 exit code: -9  (-9 means SIGKILL / kill -9)
ReplayEngine.recover(): last_done=2, lifecycle=created, entries_replayed=3
ledger after the crash (4 refunds): ['O-1', 'O-2', 'O-3', 'O-4']
run 2 (resume) exit code: 0
ledger after resume: 10 refunds, 10 unique, 0 duplicate(s)

Identical recovery, identical re-run of step 3 — but this time refund sees O-4's key already in the ledger and suppresses it. Ten refunds, no duplicate. The key did what the framework cannot.

One demo is an anecdote. We ran 50 trials per variant, each with a fresh journal and ledger and a crash injected at a uniformly random step (0–9) and phase (before the refund, in the danger window, or after journalling), then restarted the worker until it completed.

no idempotency key : resumed 50/50; trials with duplicate refunds 15/50; total duplicate refunds 15; (crashes in the danger window: 15)
with idempotency key: resumed 50/50; trials with duplicate refunds  0/50; total duplicate refunds  0; (crashes in the danger window: 16)

Two findings, both measured. First, Promptise's journal and replay recovered the worker's position in every one of the 100 crashes — resume-from-crash is real and it is free. Second, without an idempotency key, a duplicate refund happened on exactly the 15 trials whose crash fell in the danger window; a one-to-one match, because that window is the only place a duplicate can come from. The key took those 15 (and the idempotent variant's 16 danger-window crashes) to zero. The framework gives you resume; the key gives you exactly-once.

In the demo the ledger is a file and the check is a membership test. In production the same shape holds, you just move it to where the side effect lives:

• A payment provider — pass an Idempotency-Key header (Stripe and most providers support one); they dedupe for you within a window. Your key is the stable business id, the order or invoice id, never a random per-attempt value.

• Your own database — make the side effect a row with a unique constraint on the key, and write it in the same transaction as the work. A duplicate insert fails loudly instead of paying twice.

• An outbound email or message — keep a small table of sent keys and check it before send; it is the file-ledger pattern with durability.

The rule is the same everywhere: derive a stable key from the business fact (what you are refunding), make the downstream system reject or absorb a repeat of that key, and let the agent retry freely. Combined with Promptise's position recovery, a crash becomes a resume, and a resume becomes a no-op for anything already done.

• Promptise recovers position, not side effects. That is the whole reason this guide makes you write the idempotency key. We verified in the source that there is no tool-call dedup across a crash, and the 15/50 duplicate rate without a key is what "no dedup" looks like in practice. If a guide tells you a framework makes your agent "exactly-once" by itself, be sceptical — ours does not, and neither does anyone else's without a key in your tool.

• This worker is deterministic, not model-driven. The experiment isolates the durability mechanics — journal, replay, the side-effect/record gap — so the numbers aren't clouded by model variance. A real agent adds its own journal entries (the runtime's AgentProcess can journal at checkpoint or full level), but the crash-recovery and idempotency story is identical; the LLM does not change where the gap is.

• `FileJournal` is a single-node file backend. It is perfect for one process on one box and for this demonstration. For multiple workers or multiple machines you need a shared, transactional backend; the journal is a Protocol, so you can implement one over Postgres or Redis, and the idempotency check should live in that same store.

• `recover()` returns `lifecycle_state="created"` here because the worker only journals its completed transition at the very end, which the mid-run crash never reaches. Position (context_state["last_done"]) is what we resume from, and it is always correct. If you want lifecycle to track too, journal a running transition at start.

• The danger window is small but not zero. Ordering the journal write *before* the side effect does not help — then a crash in the new gap loses the side effect instead of duplicating it. There is no ordering that removes the gap; only an idempotency key does.

What is a durable AI agent?

An agent that survives a crash: when the process is killed mid-run and restarted, it resumes from where it was instead of starting over, and it does not repeat side effects it already performed. Promptise gives you the resume via its journal and replay engine; you add exactly-once side effects with an idempotency key in your tool.

Do I need Temporal or DBOS for a durable agent?

Not for crash recovery of a single agent. Promptise's FileJournal plus ReplayEngine recovered position in 50/50 crashes in our test with no extra platform. Durable-execution engines are worth it when you need distributed orchestration, long timers, or managed retries across many workers — but even they rely on idempotency keys for exactly-once side effects, so you write that part regardless.

How do I stop my AI agent from doing something twice after a crash?

Give the side-effecting tool a stable idempotency key — the business id of what it is doing — and make the downstream system reject a repeat of that key. In our run this took duplicate refunds from 15/50 to 0/50. The framework resuming your agent is not enough; the key is what makes the action exactly-once.

Where does a crash cause a duplicate side effect?

In the window between performing the side effect and recording that you performed it. A crash there leaves the effect done but unrecorded, so recovery replays the step and does it again. Our 15 duplicates mapped one-to-one onto the crashes that landed in that window. An idempotency key is the only thing that closes it.

How does Promptise recover agent state after a crash?

ReplayEngine.recover(process_id) loads the last checkpoint from the journal, replays the context_update and state_transition entries recorded after it, and returns the reconstructed context_state and lifecycle_state. You read your resume position out of context_state and continue from there.

• MCP Long-Running Tasks: Background Jobs in Python — the queue side of durability.

• Event-Driven AI Agents: Webhook and Cron Triggers — what wakes a long-running agent up.

• Deploy AI Agents in Production — where crashes actually happen.

• Autonomous AI Agent Runtime (slug: autonomous-ai-agent-runtime) — the runtime and processes this journal belongs to.

• Docs: Journal System, the Replay Engine and Agent Processes.