Durable AI Agents: Crash Recovery and Safe Resume

A long-running AI agent will be killed mid-run sooner or later: an out-of-memory kill, a deploy, a reclaimed spot instance, a kill -9. A durable AI agent survives that. It resumes from where it was instead of starting over, and it doesn't repeat the payments, emails or orders it already made. This guide explains how crash recovery works, why resuming alone still causes duplicate side effects, and how to build both halves, a journal and idempotency keys, with Promptise Foundry.

Split durability into two jobs. First, position recovery: record each finished step in a journal, and on restart, replay the journal to find where to continue. Second, exactly-once side effects: give every tool that changes the outside world a stable idempotency key, derived from the business action, and make the tool or the downstream system refuse a second action with the same key. A journal alone resumes the agent, but it can't know whether the step that was running at the crash already sent the money.

• Durability is two problems. Resuming the work, and not repeating side effects.

• The danger window is between a side effect and the record of it. A crash there leaves the action done but unrecorded.

• No ordering of writes removes that window. Only an idempotency key makes the repeat harmless.

• The key must be stable. The same business action must always get the same key, whoever retries it.

• Durable-execution platforms need idempotent steps too. They make recovery easier, not side effects exactly-once.

An agent run is a sequence of steps: model calls, tool calls, and the work between them. A crash can stop it at any point. Recovery needs a durable record that survives the process, and a way to rebuild state from it.

The journal and replay

A journal is an append-only log of what the agent finished: "step 2 done", "state changed to running". Every so often you can also write a checkpoint, a full snapshot of the state. On restart, a replay engine loads the last checkpoint and applies the entries written after it, which rebuilds the state as it was just before the crash. This is event sourcing: the checkpoint is the snapshot, and the journal entries are the events.

The danger window

step N:  ┌──────────────┐        ┌────────────────────┐
         │ side effect  │ ─────▶ │ journal "N done"   │
         │ (refund sent)│   ▲    └────────────────────┘
         └──────────────┘   │
                            └── crash here: the refund happened,
                                the journal says step N is not done

restart: replay says "resume at step N"
         ──▶ step N runs again ──▶ second refund, unless the tool
                                    sees the same idempotency key
                                    and does nothing

Moving the journal write before the side effect doesn't help. A crash in the new gap loses the side effect instead of duplicating it: the journal says "done", but the refund was never sent. There is no order of writes that closes the window, because the side effect and the journal live in different systems. What closes it is making the repeat harmless.

Idempotency keys

An idempotency key is an identifier for one business action, such as "refund order O-3". The tool sends it with the action, and the system that performs the action remembers the keys it has seen. A second request with a known key returns the first result and does nothing else. Payment providers support this directly: Stripe accepts an Idempotency-Key header and returns the saved response for a retry with the same key, as its idempotency docs describe.

Durable-execution platforms record each step's result and replay the workflow after a crash, so finished steps are not run again. They still ask for idempotent steps, because a step interrupted mid-run is retried. The Temporal docs recommend that activities be idempotent because they may be executed more than once, and the DBOS docs note that an interrupted step is retried during recovery. Whichever tool you pick, the idempotency key is your code's job. For a single agent process, a journal plus keys is often enough, with no extra platform to run.

• Python 3.10 or later and Promptise Foundry 1.4.1 or later.

• An OpenAI API key in OPENAI_API_KEY. The example uses openai:gpt-5-mini, and the whole demo costs about a cent.

• A Unix-like system for the kill -9 demo (macOS or Linux).

• About 15 minutes.

pip install "promptise>=1.4.1"

The example is a refund agent. It works through five refund requests. For each one, an AI agent reads the customer's message, calls issue_refund, and replies. The refund is the side effect: a line in a ledger file that stands in for a payment provider. In production it would be an API call that moves money.

"""A durable refund agent: it journals its progress and resumes after a crash."""
import asyncio
import json
import os
import signal
from pathlib import Path

from langchain_core.tools import tool
from promptise import build_agent
from promptise.runtime.journal import FileJournal, JournalEntry, ReplayEngine

PROCESS_ID = "refund-agent"
REQUESTS = [  # (order id, amount in cents, the customer's message)
    ("O-1", 1999, "The mug arrived broken, please refund it."),
    ("O-2", 4500, "I was charged for an order I cancelled."),
    ("O-3", 1250, "Wrong size. I sent it back last week."),
    ("O-4", 899, "The package never arrived."),
    ("O-5", 3200, "I returned it, tracking says delivered to you."),
]
AMOUNTS = {order_id: cents for order_id, cents, _ in REQUESTS}
LEDGER = Path(os.environ.get("LEDGER", "ledger.jsonl"))  # stands in for the payment provider
IDEMPOTENT = os.environ.get("IDEMPOTENT", "1") == "1"
CRASH_AFTER_REFUND = os.environ.get("CRASH_AFTER_REFUND", "")  # test hook: an order id


def issued_keys() -> set[str]:
    if not LEDGER.exists():
        return set()
    return {json.loads(line)["key"] for line in LEDGER.read_text().splitlines() if line}


@tool
def issue_refund(order_id: str) -> str:
    """Refund an order in full. Call it once per order."""
    if order_id not in AMOUNTS:
        return f"Unknown order {order_id}."
    key = f"refund-{order_id}"  # stable: derived from the order, not from the attempt or the model
    if IDEMPOTENT and key in issued_keys():
        return f"The refund for {order_id} was already issued; nothing was charged again."
    with LEDGER.open("a") as f:  # the side effect: money moves here
        f.write(json.dumps({"key": key, "order_id": order_id, "cents": AMOUNTS[order_id]}) + "\n")
    if CRASH_AFTER_REFUND == order_id:
        os.kill(os.getpid(), signal.SIGKILL)  # a real kill -9, after the refund, before the journal write
    return f"Refunded ${AMOUNTS[order_id] / 100:.2f} for {order_id}."


async def main():
    journal = FileJournal(base_path=os.environ.get("JOURNAL_DIR", ".promptise/journal"))
    recovered = await ReplayEngine(journal).recover(PROCESS_ID)
    start = recovered["context_state"].get("last_done", -1) + 1  # resume after the last finished request
    if start:
        print(f"recovered: last_done={start - 1}, resuming at {REQUESTS[start][0]}")

    agent = await build_agent(
        model="openai:gpt-5-mini", servers={}, extra_tools=[issue_refund],
        instructions="You handle refund requests for an online shop. Refund every request in full "
                     "with issue_refund, then reply to the customer in one short sentence.")
    for i in range(start, len(REQUESTS)):
        order_id, _, message = REQUESTS[i]
        result = await agent.ainvoke({"messages": [{"role": "user", "content": f"Order {order_id}: {message}"}]})
        print(f"{order_id}: {result['messages'][-1].content}")
        # Record progress only after the step is complete.
        await journal.append(JournalEntry(process_id=PROCESS_ID, entry_type="context_update",
                                          data={"key": "last_done", "value": i}))
    await journal.append(JournalEntry(process_id=PROCESS_ID, entry_type="state_transition",
                                      data={"to_state": "completed"}))
    await agent.shutdown()


if __name__ == "__main__":
    asyncio.run(main())

Three design choices in issue_refund matter for durability:

• The amount comes from your records, not from the model. The model passes only the order ID. A model can't invent a larger refund if the tool doesn't take an amount.

• The key is derived from the order. refund-O-3 is the same on the first attempt and on every retry, whoever retries.

• The crash is a test hook. CRASH_AFTER_REFUND makes the process send itself SIGKILL right after a refund, before the journal write. That puts the crash exactly in the danger window.

The agent records progress with Promptise's FileJournal, which writes one JSON line per entry. After each request is complete, it appends a context_update that sets last_done to the request's index:

from promptise.runtime.journal import FileJournal, JournalEntry

journal = FileJournal(base_path=".promptise/journal")
await journal.append(JournalEntry(process_id="refund-agent", entry_type="context_update",
                                  data={"key": "last_done", "value": i}))

Write the entry only after the step is finished, including its side effects. An entry that says "done" before the work is done turns a crash into lost work. Keep entries small and about progress: which items are finished, which state the process is in. The journal docs list the entry types.

On start, the agent asks the ReplayEngine for its state and continues after the last finished request:

from promptise.runtime.journal import ReplayEngine

recovered = await ReplayEngine(journal).recover("refund-agent")
start = recovered["context_state"].get("last_done", -1) + 1

recover() loads the last checkpoint, if there is one, replays the context_update and state_transition entries written after it, and returns the rebuilt context_state and lifecycle_state. The replay docs describe the algorithm. Replay rebuilds state. It never runs or skips a tool. Deciding what to do with the recovered state is your code's job.

crash_demo.py runs the agent in a subprocess, kills it with SIGKILL right after the refund for O-3, and restarts it. It does this twice: once with the idempotency check turned off, and once with it on.

"""Kill the refund agent with kill -9 right after it refunds O-3, restart it, and check the ledger."""
import collections
import json
import os
import shutil
import subprocess
import sys
import tempfile
from pathlib import Path

HERE = Path(__file__).parent


def run(env):
    proc = subprocess.run([sys.executable, str(HERE / "refund_agent.py")], env={**os.environ, "PYTHONUNBUFFERED": "1", **env},
                          capture_output=True, text=True)
    print(proc.stdout.rstrip())
    print(f"[exit code {proc.returncode}{' = killed by SIGKILL' if proc.returncode == -9 else ''}]")


def demo(idempotent: bool):
    print(f"==== idempotency key {'ON' if idempotent else 'OFF'} ====")
    work = Path(tempfile.mkdtemp())
    env = {"JOURNAL_DIR": str(work / "journal"), "LEDGER": str(work / "ledger.jsonl"),
           "IDEMPOTENT": "1" if idempotent else "0"}
    run({**env, "CRASH_AFTER_REFUND": "O-3"})  # first run: crashes inside O-3's refund
    print("-- restart --")
    run({**env, "CRASH_AFTER_REFUND": ""})     # second run: recovers and resumes
    ledger = [json.loads(line)["order_id"] for line in (work / "ledger.jsonl").read_text().splitlines()]
    twice = [o for o, n in collections.Counter(ledger).items() if n > 1]
    print(f"ledger: {len(ledger)} refunds for {len(set(ledger))} orders; refunded twice: {twice or 'none'}\n")
    shutil.rmtree(work)


demo(idempotent=False)
demo(idempotent=True)

==== idempotency key OFF ====
O-1: Your refund for order O-1 has been issued in full.
O-2: Your payment for order O-2 has been fully refunded.
[exit code -9 = killed by SIGKILL]
-- restart --
recovered: last_done=1, resuming at O-3
O-3: Your refund for order O-3 has been issued in full.
O-4: I've issued a full refund for order O-4.
O-5: Your refund for order O-5 has been issued in full.
[exit code 0]
ledger: 6 refunds for 5 orders; refunded twice: ['O-3']

==== idempotency key ON ====
O-1: Your refund for order O-1 has been issued.
O-2: I've refunded your payment for order O-2 in full.
[exit code -9 = killed by SIGKILL]
-- restart --
recovered: last_done=1, resuming at O-3
O-3: Your refund for order O-3 has already been issued and should appear on your original payment method within 5–10 business days.
O-4: Your refund for order O-4 has been issued in full.
O-5: Your refund for order O-5 has been processed in full.
[exit code 0]
ledger: 5 refunds for 5 orders; refunded twice: none

Read the first half closely. The agent finished O-1 and O-2, refunded O-3, and was killed before it could journal O-3. On restart, recovery worked exactly as designed: the journal said the last finished request was index 1 (O-2), so the agent resumed at O-3. And because the check was off, it refunded O-3 a second time. Position recovery was correct and still produced a duplicate refund. That is the danger window from the diagram.

In the second half, the same crash and the same resume happened. This time issue_refund found the key refund-O-3 in the ledger and returned "already issued" instead of refunding again. Five orders, five refunds. The agent read that result and told the customer the refund was already issued.

The check is a few lines in the tool:

key = f"refund-{order_id}"
if key in issued_keys():
    return f"The refund for {order_id} was already issued; nothing was charged again."

In this demo the check reads the ledger, because the ledger is the system of record. That is the important part: the check must look where the side effect is recorded, not at the agent's own journal, which by definition doesn't know about the step that crashed. In production, pass the key to the system that performs the action and let it enforce uniqueness:

• A payment provider. Send the key as the provider's idempotency key. Note the provider's retention: Stripe, for example, may prune keys after 24 hours, so a retry days later is a new request.

• Your own database. Make the action a row with a unique constraint on the key, written in the same transaction as the work. A duplicate insert fails instead of paying twice.

• Email, SMS or chat messages. Keep a table of sent keys and check it in the same transaction that records the send, or use the provider's deduplication ID if it has one.

One more detail from the output: the agent's reply for O-3 promised a refund timeline that no tool returned. Durability doesn't fix what the model says. Tool results that state exactly what happened, and reply templates for customer-facing text, do.

durable-refund-agent/
├── refund_agent.py   # the agent, the journal and the idempotent tool (steps 1-3, 5)
└── crash_demo.py     # kill -9 after a refund, restart, check the ledger (step 4)

pip install "promptise>=1.4.1"
export OPENAI_API_KEY=sk-...
python crash_demo.py             # both variants, about a cent
python refund_agent.py           # run the agent on its own; rerun it to see it resume

Running refund_agent.py directly writes .promptise/journal/refund-agent.jsonl and ledger.jsonl in the current directory. Delete both to start over.

Use a shared, transactional journal for many workers

FileJournal suits one process on one machine. It writes with plain file appends and doesn't call fsync, so a process kill is safe, but a power loss can drop the last lines, and the checkpoint file is rewritten in place. For several workers or several machines, implement the journal protocol over a database such as Postgres, and keep the idempotency keys in the same database so the progress record and the key check share a transaction.

Journal agent processes in the runtime

Promptise's AgentProcess can journal its own lifecycle, invocations and checkpoints with ProcessConfig(journal=JournalConfig(level="checkpoint")), or every side effect with level="full". On restart you read the state back with ReplayEngine, as above. restart_policy="on_failure" restarts a process that entered the FAILED state. It doesn't restart a killed operating-system process, so pair it with a supervisor such as systemd, Kubernetes or your platform's restart policy. See agent processes.

Make long jobs asynchronous

A tool that waits 20 minutes for an export keeps the danger window open for 20 minutes. Start the job, return a job ID, and let the completion trigger the next step, as long-running MCP tasks and event-driven agents show. Short steps are cheaper to repeat.

Gate irreversible actions

Idempotency stops repeats. It doesn't stop a wrong first action. Put tools that pay, delete or send behind an approval gate and a budget for irreversible actions. See human-in-the-loop approval and AI agent cost control.

Test crashes on purpose

A recovery path that never runs is a guess. Keep a crash test like crash_demo.py in CI, with a scripted model so it needs no API key, and kill the worker at each phase: before the side effect, between the side effect and the journal write, and after the journal write. Every phase should end with each item done exactly once.

• Promptise recovers position, not side effects. The journal and the replay engine rebuild state. There is no automatic deduplication of tool calls across a crash. The MCP idempotent_hint annotation is advice to clients, not enforcement.

• The agent re-decides the interrupted step. On resume, the model handles the in-flight request again and may decide differently. If that matters, journal the decision before acting and replay it instead of asking the model again.

• `FileJournal` is single-node. Use a shared, transactional backend for several workers.

• Idempotency needs support downstream. A tool that calls an API without idempotency support needs its own record of what it did.

• This is not a workflow engine. For timers that last days, fan-out across services, or managed retries across many workers, a durable-execution platform is the better fit.

What is a durable AI agent?

A durable AI agent survives a crash. When its process is killed mid-run and restarted, it resumes from where it was instead of starting over, and it doesn't repeat side effects it already performed, such as payments or emails. Durability comes from a journal of finished steps for resuming, and idempotency keys on every tool that changes the outside world.

How do I resume an AI agent after a crash?

Record each finished step in a durable journal, and on restart, replay the journal to rebuild the state and find the first unfinished step. In Promptise, FileJournal records the entries and ReplayEngine.recover() returns the rebuilt context_state, from which you read the resume position. Then make the step that was running at the crash safe to repeat.

How do I stop an AI agent from doing something twice after a crash?

Give each side-effecting tool a stable idempotency key, derived from the business action, such as refund-O-3, and make the system that performs the action reject a second request with the same key. A journal can't prevent the repeat on its own, because a crash between the action and the journal write leaves the action done but unrecorded.

Do I need Temporal or DBOS for a durable AI agent?

Not for crash recovery of a single agent process. A journal plus idempotency keys covers resuming and safe repeats. Durable-execution platforms help with many workers, long timers and cross-service workflows, and they record step results so finished steps don't rerun. They still expect idempotent steps, because an interrupted step is retried, so you write the keys either way.

Where does a crash cause a duplicate side effect?

In the window between performing the side effect and recording that it happened. A crash there leaves the action done but unrecorded, so recovery runs the step again. Writing the record first doesn't help, because then a crash loses the action instead. Only an idempotency key, checked where the side effect is recorded, makes the repeat harmless.

What makes a good idempotency key?

It identifies one business action and is the same on every retry: an order ID, an invoice ID, or a request ID created once and stored before the first attempt. Don't use timestamps, attempt counters or text the model wrote, because they change on retry. Avoid personal data in keys, and check how long the downstream system keeps them.

• Start long jobs and finish them later: long-running MCP tasks.

• Wake agents on events and schedules: event-driven AI agents.

• Run agents as long-lived processes: the autonomous AI agent runtime.

• Ship it: deploy AI agents in production.

• See where durability fits in an agent: what is an agent harness?.

• Reference: journal, replay engine and agent processes in the Promptise docs.