AI Agent Auto-Restart: Restart Policies and Timeouts

Count failed runs, and when too many fail in a row, tear the agent down and start it again after a delay that grows with each attempt. Cap the number of attempts so a permanent fault ends in a clear failed state and an alert, not an endless restart loop. In Promptise you set this on the process:

ProcessConfig(
    model="openai:gpt-5-mini",
    max_consecutive_failures=3,   # failed runs in a row before the process is FAILED
    restart_policy="on_failure",  # restart after FAILED ("never" is the default)
    max_restarts=5,               # attempts in a row, reset by a successful run
    restart_backoff=2.0,          # wait 2 s, 4 s, 8 s ... capped at 60 s
    max_invocation_time=120,      # a run that takes longer fails with TimeoutError
)

Key takeaways:

• A failure is a run that raises. Provider errors, timeouts and iteration limits count. A tool that raises does not: the model sees the error and the run goes on.

• Restarts happen in the same Python process. The agent, triggers and MCP connections are rebuilt. Context state and conversation history are kept.

• The restart counter resets after one successful run, so max_restarts limits a streak, not the process's lifetime.

• Alert on the final failure, not every failure. A lifecycle listener can tell when the process will not restart again.

• `restart_policy` does not cover a crashed program. Use Docker or systemd restarts for that.

An autonomous agent runs as a process: something triggers it (a schedule, a webhook, a message), it runs the model and tools, and it waits for the next trigger. Each run either completes or raises. A supervisor around the process decides what a raised run means. One failure is noise. Several in a row usually mean something outside the agent is broken, and running the same failing setup again and again only burns money and fills logs.

Auto-restart turns that judgment into rules. Count consecutive failures, mark the process failed at a threshold, wait, rebuild it from scratch, and try again a limited number of times. Rebuilding matters because some faults live in the agent's own connections: a stale HTTP session, an MCP server that crashed, a half-open socket. Backoff matters because a provider outage doesn't end in one second, and hammering it slows recovery for everyone.

Here is the sequence in Promptise:

run raises ──▶ consecutive_failures += 1
                   │
                   ├─ below max_consecutive_failures ──▶ wait for the next trigger
                   │
                   ▼ reached
          RUNNING ──▶ FAILED            event process.failed (critical)
                   │
                   ├─ restart_policy == "never" ───────────▶ stays FAILED
                   ├─ restart_count == max_restarts ───────▶ journal restart_exhausted,
                   │                                         stays FAILED
                   ▼
          wait restart_backoff × 2^restart_count (max 60 s)
                   │
          journal "restart", event process.restarted (warning)
                   │
          tear down workers, triggers, agent and MCP clients
                   │
          FAILED ──▶ STARTING ──▶ RUNNING    (a failed start is the next failure)
                   │
          first successful run ──▶ restart_count = 0

The terms: a run (or invocation) is one agent call for one trigger event. `consecutive_failures` counts failed runs since the last success and resets on every success and every start. `restart_count` counts restart attempts since the last successful run. A process's states and transitions are covered in depth in AI agent lifecycle states.

What counts as a failure

Two settings sit in front of the counter. trigger_delivery.max_retries (default 0) re-runs a failed event with its own backoff, and only an event that exhausts its retries counts as one consecutive failure. And the provider SDK retries some errors before Promptise sees them at all: the OpenAI client retries a 503 twice by default. Budget limits have their own actions, described in AI agent cost control.

Use restart_policy="on_failure" when failures are likely to be temporary and a fresh start can help:

• Provider outages, rate-limit storms and network blips.

• MCP servers that crash or drop connections. A restart starts stdio servers again and reconnects HTTP ones.

• Long-running agents on schedules or webhooks, where nobody watches the process.

Keep the default "never", or a low max_restarts, when a restart can't fix the cause:

• A wrong API key, a missing environment variable or a bad model name. Every attempt fails the same way.

• Runs whose tools have side effects you can't repeat safely. A restart doesn't replay the failed event, but trigger_delivery.max_retries does, so keep retries off for those.

• Development, where you want the first failure to stop everything so you can read it.

Use restart_policy="always" only if you also want lifetime recycling (step 6). For a crashed Python program, which no in-process policy can restart, use your container or service manager.

• Python 3.10 or later and Promptise Foundry 1.4.3 or later. aiohttp is installed with it.

• No API key. The examples use a small fake model server, so the output is the same every time and costs nothing. Point the same code at a real model by removing two environment variables.

• About 20 minutes.

pip install "promptise>=1.4.3"

You can't test a restart policy by waiting for a real outage. Simulate one. flaky_llm.py is an OpenAI-compatible endpoint with three modes: up answers, down returns HTTP 503 like an overloaded provider, and hang never answers.

"""A fake OpenAI-compatible model server for testing agent processes without an API key.

Point the agent at it with OPENAI_BASE_URL=http://127.0.0.1:8765/v1.
- mode: "up" (answers), "down" (HTTP 503) or "hang" (never answers)
- down_models: model names that get HTTP 503 even when mode is "up"
- delay: seconds before each answer, to simulate a slow model
When the agent has tools, "up" first calls the first tool once, then answers.
"""
import asyncio
import time

from aiohttp import web


class FlakyLLM:
    def __init__(self, port: int = 8765) -> None:
        self.port = port
        self.mode = "up"
        self.down_models: set[str] = set()
        self.delay = 0.0
        self.requests = 0
        self.tool_results: list[str] = []  # what the model was shown as tool results
        self._runner: web.AppRunner | None = None
        self._closing = asyncio.Event()

    async def _chat(self, request: web.Request) -> web.Response:
        self.requests += 1
        body = await request.json()
        if self.mode == "down" or body.get("model") in self.down_models:
            return web.json_response(
                {"error": {"message": "model overloaded", "type": "server_error"}},
                status=503,
                headers={"retry-after-ms": "50"},  # keep the SDK's own retries short
            )
        if self.mode == "hang":
            await self._closing.wait()  # until stop()
        await asyncio.sleep(self.delay)
        tool_results = [m["content"] for m in body["messages"] if m["role"] == "tool"]
        self.tool_results = [str(r) for r in tool_results]
        if body.get("tools") and not tool_results:
            name = body["tools"][0]["function"]["name"]
            message = {"role": "assistant", "content": None, "tool_calls": [{
                "id": "call_1", "type": "function",
                "function": {"name": name, "arguments": "{}"},
            }]}
        else:
            message = {"role": "assistant", "content": "Done."}
        return web.json_response({
            "id": f"chatcmpl-{self.requests}",
            "object": "chat.completion",
            "created": int(time.time()),
            "model": body.get("model", "fake"),
            "choices": [{
                "index": 0,
                "finish_reason": "tool_calls" if message.get("tool_calls") else "stop",
                "message": message,
            }],
            "usage": {"prompt_tokens": 10, "completion_tokens": 3, "total_tokens": 13},
        })

    async def start(self) -> None:
        app = web.Application()
        app.router.add_post("/v1/chat/completions", self._chat)
        self._runner = web.AppRunner(app)
        await self._runner.setup()
        await web.TCPSite(self._runner, "127.0.0.1", self.port).start()

    async def stop(self) -> None:
        self._closing.set()
        if self._runner is not None:
            await self._runner.cleanup()

The agent reaches it through OPENAI_BASE_URL, so the process under test is configured exactly as in production, with model="openai:gpt-5-mini". Keep a copy of this file in your test suite. It lets you check restart behavior in CI without an API key.

Every restart option lives on ProcessConfig. These are the ones that decide when a process fails and how it comes back, with the 1.4.3 defaults:

The demo uses small numbers so you can watch the whole sequence in seconds: a cron trigger that fires every second, two failures to enter FAILED, and at most two restarts.

"""Watch an agent process fail, restart with backoff, and recover (or give up).

    python restart_demo.py recover   # the model comes back after ~6 seconds
    python restart_demo.py exhaust   # the model never comes back
"""
import asyncio
import os
import shutil
import sys
import time

os.environ["OPENAI_BASE_URL"] = "http://127.0.0.1:8765/v1"  # the fake model server
os.environ.setdefault("OPENAI_API_KEY", "sk-fake")

from flaky_llm import FlakyLLM
from promptise.events import CallbackSink, EventNotifier, emit_event
from promptise.runtime import (
    AgentProcess,
    JournalConfig,
    ProcessConfig,
    ProcessState,
    TriggerConfig,
)
from promptise.runtime.journal import FileJournal

T0 = time.monotonic()


def log(message: str) -> None:
    print(f"{time.monotonic() - T0:5.1f}s  {message}")


async def on_event(event) -> None:
    data = event.data
    extra = f" attempt={data['attempt']}" if "attempt" in data else ""
    log(f"event {event.event_type} [{event.severity}]{extra}")


def gave_up(process: AgentProcess) -> bool:
    """True when a FAILED process will not be restarted again."""
    cfg = process.config
    return cfg.restart_policy == "never" or process.status()["restart_count"] >= cfg.max_restarts


async def main(scenario: str) -> None:
    shutil.rmtree("journal", ignore_errors=True)
    llm = FlakyLLM()
    await llm.start()
    llm.mode = "down"

    config = ProcessConfig(
        model="openai:gpt-5-mini",
        instructions="Send the hourly report.",
        triggers=[TriggerConfig(type="cron", cron_expression="* * * * * *")],  # every second
        max_consecutive_failures=2,  # two failed runs in a row -> FAILED
        restart_policy="on_failure",
        max_restarts=2,
        restart_backoff=1.0,  # 1s, then 2s, then 4s ... (max 60s)
        journal=JournalConfig(level="checkpoint", path="journal"),
    )
    notifier = EventNotifier(sinks=[CallbackSink(on_event, events=[
        "process.started", "process.failed", "process.restarted", "process.stopped",
        "process.gave_up",  # our own event, emitted below
    ])])
    process = AgentProcess("reporter", config, event_notifier=notifier)

    def on_transition(t) -> None:
        log(f"{t.from_state.value} -> {t.to_state.value} ({t.reason})")
        if t.to_state is ProcessState.FAILED and gave_up(process):
            emit_event(notifier, "process.gave_up", "critical",
                       {"process_name": process.name, "reason": t.reason}, agent_id=process.name)

    process.lifecycle.add_listener(on_transition)

    await process.start()
    if scenario == "recover":
        await asyncio.sleep(6.5)
        llm.mode = "up"
        log("model is back")
        await asyncio.sleep(2.5)
    else:
        await asyncio.sleep(11)

    status = process.status()
    log(f"status: state={status['state']} invocations={status['invocation_count']} "
        f"restart_count={status['restart_count']} dead_letters={status['dead_letter_count']}")
    reasons = sorted({d["reason"] for d in process.dead_letters})
    log(f"dead-letter reasons: {reasons}")
    await process.stop()
    log(f"after stop(): state={process.state.value}")
    await llm.stop()

    print("\njournal:")
    for entry in await FileJournal("journal").read("reporter"):
        if entry.entry_type in ("restart", "restart_exhausted", "error"):
            data = {k: v for k, v in entry.data.items() if k in ("reason", "attempt", "restarts", "error_type")}
            if "error" in entry.data:  # "Graph 'react' failed at node 'reason': <cause> - {...}"
                data["error"] = entry.data["error"].split(" - ")[0].split(": ", 1)[1]
            print(f"  {entry.entry_type:18} {data}")


asyncio.run(main(sys.argv[1] if len(sys.argv) > 1 else "recover"))

Three parts do the observing. A CallbackSink on an EventNotifier prints the process.* events the runtime emits. A lifecycle listener prints every state transition. And JournalConfig(level="checkpoint") writes a journal that the script reads back at the end. The gave_up() helper is used in step 4.

Start with the model down, and bring it back after about six seconds:

python restart_demo.py recover

  0.0s  created -> starting (start() called)
  0.3s  starting -> running (startup complete)
  0.3s  event process.started [info]
  2.0s  running -> failed (max consecutive failures)
  2.0s  event process.failed [critical]
  3.0s  event process.restarted [warning] attempt=1
  3.0s  failed -> starting (start() called)
  3.0s  starting -> running (startup complete)
  3.0s  event process.started [info]
  5.0s  running -> failed (max consecutive failures)
  5.0s  event process.failed [critical]
  6.8s  model is back
  7.0s  event process.restarted [warning] attempt=2
  7.0s  failed -> starting (start() called)
  7.0s  starting -> running (startup complete)
  7.0s  event process.started [info]
  9.3s  status: state=running invocations=2 restart_count=0 dead_letters=7
  9.3s  dead-letter reasons: ['process failed', 'retries exhausted']
  9.3s  running -> stopping (stop() called)
  9.3s  stopping -> stopped (shutdown complete)
  9.3s  event process.stopped [info]
  9.3s  after stop(): state=stopped

journal:
  error              {'error_type': 'GraphExecutionError', 'error': 'OpenAIAPIError: Error code: 503'}
  error              {'error_type': 'GraphExecutionError', 'error': 'OpenAIAPIError: Error code: 503'}
  restart            {'reason': 'max consecutive failures', 'attempt': 1}
  error              {'error_type': 'GraphExecutionError', 'error': 'OpenAIAPIError: Error code: 503'}
  error              {'error_type': 'GraphExecutionError', 'error': 'OpenAIAPIError: Error code: 503'}
  restart            {'reason': 'max consecutive failures', 'attempt': 2}

Read it from the top:

• Two failures, then `FAILED`. The cron fired twice, both runs got a 503, and the second one reached max_consecutive_failures=2. The runtime emitted process.failed with severity critical.

• Backoff doubles. The first restart came 1 s after the failure (restart_backoff), the second 2 s after. A third would have waited 4 s.

• A restart is a full start. failed -> starting -> running and a new process.started event. Workers, triggers, the agent and its MCP connections were torn down and built again.

• Success resets the count. The model came back, two runs succeeded, and restart_count went back to 0. The next outage gets the full two attempts again.

The dead letters tell you what happened to the trigger events. retries exhausted are the events whose runs failed. process failed are cron events that fired while the process sat in FAILED waiting for its restart: a failed process keeps its triggers, but it refuses their events, and a webhook trigger answers 503. Inspect them with process.dead_letters and put them back on the queue with await process.redeliver_dead_letters().

The journal holds the same story in durable form: one error entry per failed run with the cause, and one restart entry per attempt. It also records every state transition, left out of the printout here.

Now keep the model down:

python restart_demo.py exhaust

  0.0s  created -> starting (start() called)
  0.3s  starting -> running (startup complete)
  0.3s  event process.started [info]
  1.9s  running -> failed (max consecutive failures)
  1.9s  event process.failed [critical]
  2.9s  event process.restarted [warning] attempt=1
  2.9s  failed -> starting (start() called)
  2.9s  starting -> running (startup complete)
  2.9s  event process.started [info]
  4.9s  running -> failed (max consecutive failures)
  4.9s  event process.failed [critical]
  6.9s  event process.restarted [warning] attempt=2
  6.9s  failed -> starting (start() called)
  6.9s  starting -> running (startup complete)
  6.9s  event process.started [info]
  8.9s  running -> failed (max consecutive failures)
  8.9s  event process.failed [critical]
  8.9s  event process.gave_up [critical]
 11.3s  status: state=failed invocations=0 restart_count=2 dead_letters=11
 11.3s  dead-letter reasons: ['process failed', 'retries exhausted']
 11.3s  after stop(): state=failed

journal:
  error              {'error_type': 'GraphExecutionError', 'error': 'OpenAIAPIError: Error code: 503'}
  error              {'error_type': 'GraphExecutionError', 'error': 'OpenAIAPIError: Error code: 503'}
  restart            {'reason': 'max consecutive failures', 'attempt': 1}
  error              {'error_type': 'GraphExecutionError', 'error': 'OpenAIAPIError: Error code: 503'}
  error              {'error_type': 'GraphExecutionError', 'error': 'OpenAIAPIError: Error code: 503'}
  restart            {'reason': 'max consecutive failures', 'attempt': 2}
  error              {'error_type': 'GraphExecutionError', 'error': 'OpenAIAPIError: Error code: 503'}
  error              {'error_type': 'GraphExecutionError', 'error': 'OpenAIAPIError: Error code: 503'}
  restart_exhausted  {'reason': 'max consecutive failures', 'restarts': 2}

After the second restart the process failed a third time. restart_count had reached max_restarts, so the runtime wrote a restart_exhausted journal entry and left the process in FAILED. That is the moment a person needs to know about.

The runtime emits process.failed on every failure, including the ones it is about to recover from. Paging someone for each of those trains them to ignore pages. So the demo adds its own event. The lifecycle listener runs right after each transition, and gave_up() checks whether this FAILED is final: the policy is "never", or the restart count is used up. If it is, the listener calls emit_event() with a custom process.gave_up event, and every sink on the notifier receives it like a built-in one.

To send it to Slack or an incident tool, add a WebhookSink that subscribes to that event only:

from promptise.events import EventNotifier, WebhookSink

notifier = EventNotifier(sinks=[
    WebhookSink("https://hooks.slack.com/services/...", events=["process.gave_up"]),
])

Use the listener rather than a sink to make the decision. Listeners run as part of the transition, so they read the restart count as it was at the failure. A sink's callback runs later, on a delivery task, when the next restart may already have begun. AI agent notifications covers sinks, signing and filtering in detail.

Two details in the output matter. The payload of process.failed has an empty error when the cause is repeated failed runs. The cause is in the journal's error entries and in the dead letters. And stop() on a FAILED process cleans up but leaves the state at failed, with no process.stopped event. To bring it back after you fix the cause, call stop() and then start(), or stop_process() and start_process() on a runtime.

A hung model call or tool never raises, so it never counts as a failure. It just holds a worker forever, and with the default concurrency=1 the process stops doing anything. max_invocation_time turns a hang into a failure. Set the fake model to hang and give runs two seconds:

"""A model call that hangs is cut off by max_invocation_time and counts as a failure."""
import asyncio
import os
import time

os.environ["OPENAI_BASE_URL"] = "http://127.0.0.1:8765/v1"
os.environ.setdefault("OPENAI_API_KEY", "sk-fake")

from flaky_llm import FlakyLLM
from promptise.events import CallbackSink, EventNotifier
from promptise.runtime import AgentProcess, ProcessConfig, TriggerEvent

T0 = time.monotonic()


def log(message: str) -> None:
    print(f"{time.monotonic() - T0:5.1f}s  {message}")


async def on_event(event) -> None:
    data = {k: v for k, v in event.data.items() if k != "process_id"}
    log(f"event {event.event_type} [{event.severity}] {data}")


async def main() -> None:
    llm = FlakyLLM()
    await llm.start()
    llm.mode = "hang"
    config = ProcessConfig(
        model="openai:gpt-5-mini",
        instructions="Summarize the ticket.",
        max_invocation_time=2.0,      # seconds per run, then TimeoutError
        max_consecutive_failures=1,   # one failed run -> FAILED
    )
    notifier = EventNotifier(sinks=[CallbackSink(
        on_event, events=["invocation.timeout", "process.failed"],
    )])
    process = AgentProcess("summarizer", config, event_notifier=notifier)
    await process.start()
    await process.inject(TriggerEvent(trigger_id="manual", trigger_type="manual",
                                      payload={"ticket": "T-1042"}))
    await asyncio.sleep(3)
    letter = process.dead_letters[0]
    log(f"state={process.state.value} dead letter: {letter['reason']} / {letter['error']}")
    await process.stop()
    await llm.stop()


asyncio.run(main())

  2.4s  event invocation.timeout [error] {'timeout_seconds': 2.0}
  2.4s  event process.failed [critical] {'process_name': 'summarizer', 'reason': 'max consecutive failures', 'error': ''}
  3.3s  state=failed dead letter: retries exhausted / TimeoutError: Agent invocation exceeded 2.0s timeout

The run was cancelled at the limit and raised TimeoutError. The agent emitted invocation.timeout, and because max_consecutive_failures=1, the process went to FAILED. With a restart policy it would now restart like any other failure.

Size the limit from real runs: the slowest healthy run plus a margin, often one to five minutes for agents with several tool calls. The timeout covers the whole run, including every model call and tool call in it. A run that is cut off mid-way may already have called some tools, so the same caution about side effects applies.

Some processes degrade slowly. Memory grows, caches go stale, a connection pool leaks. max_lifetime lets you recycle them on a schedule. With restart_policy="always" the process restarts itself; with any other policy it stops.

"""Recycle a process after max_lifetime (restart_policy="always") or stop it ("on_failure").

    python lifetime_demo.py always
    python lifetime_demo.py on_failure
"""
import asyncio
import os
import sys
import time

os.environ["OPENAI_BASE_URL"] = "http://127.0.0.1:8765/v1"
os.environ.setdefault("OPENAI_API_KEY", "sk-fake")

from flaky_llm import FlakyLLM
from promptise.events import CallbackSink, EventNotifier
from promptise.runtime import AgentProcess, ProcessConfig

T0 = time.monotonic()


def log(message: str) -> None:
    print(f"{time.monotonic() - T0:5.1f}s  {message}")


async def on_event(event) -> None:
    log(f"event {event.event_type} attempt={event.data.get('attempt')} reason={event.data.get('reason')!r}")


async def main(policy: str) -> None:
    llm = FlakyLLM()
    await llm.start()
    config = ProcessConfig(
        model="openai:gpt-5-mini",
        instructions="Watch the queue.",
        restart_policy=policy,
        max_lifetime=3.0,       # recycle after 3 seconds (use hours in production)
        heartbeat_interval=1.0,  # lifetime and idle checks run on the heartbeat
    )
    notifier = EventNotifier(sinks=[CallbackSink(on_event, events=["process.restarted"])])
    process = AgentProcess("watcher", config, event_notifier=notifier)
    process.lifecycle.add_listener(
        lambda t: log(f"{t.from_state.value} -> {t.to_state.value} ({t.reason})")
    )
    await process.start()
    await asyncio.sleep(5.5)
    log(f"state={process.state.value} restart_count={process.status()['restart_count']}")
    await process.stop()
    await llm.stop()


asyncio.run(main(sys.argv[1]))

python lifetime_demo.py always

  0.0s  created -> starting (start() called)
  0.3s  starting -> running (startup complete)
  3.3s  running -> stopping (stopping for restart)
  3.3s  event process.restarted attempt=None reason='max lifetime reached'
  3.3s  stopping -> stopped (stopped for restart)
  3.3s  stopped -> starting (start() called)
  3.4s  starting -> running (startup complete)
  5.8s  state=running restart_count=0
  5.8s  running -> stopping (stop() called)
  5.8s  stopping -> stopped (shutdown complete)

python lifetime_demo.py on_failure

  0.0s  created -> starting (start() called)
  0.3s  starting -> running (startup complete)
  3.3s  running -> stopping (stop() called)
  3.3s  stopping -> stopped (shutdown complete)
  5.8s  state=stopped restart_count=0

A recycle goes through STOPPING and STOPPED before it starts again, unlike a failure restart. It emits process.restarted with attempt=None and does not count toward max_restarts. The lifetime clock starts again with each start.

The checks run on the heartbeat. heartbeat_interval sets how often, so the demo set it to 1 s to recycle close to 3 s. With the default of 10 s, a recycle can happen up to 10 s late. The heartbeat is an internal timer only: it sends nothing outside the process, so it is not a liveness probe.

idle_timeout runs on the same heartbeat, but it doesn't restart anything. It suspends a process that has gone idle_timeout seconds without a completed run, and nothing resumes it automatically. That means a process whose trigger fires less often than its idle timeout suspends itself between triggers and then never runs again. Leave it at 0 unless you resume processes yourself. AI agent lifecycle states shows idle suspension and its pitfalls.

auto-restart/
├── flaky_llm.py         # fake OpenAI-compatible model: up, down or hang (step 1)
├── restart_demo.py      # restart with backoff, recovery and giving up (steps 2-4)
├── timeout_demo.py      # max_invocation_time turns a hang into a failure (step 5)
├── lifetime_demo.py     # max_lifetime recycling with "always" (step 6)
├── report_server.py     # an MCP tool that always raises
└── tool_error_demo.py   # a raising tool does not fail the run

pip install "promptise>=1.4.3"
python restart_demo.py recover      # two restarts, then recovery
python restart_demo.py exhaust      # restarts used up, process.gave_up
python timeout_demo.py              # a hung run becomes a TimeoutError
python lifetime_demo.py always      # recycle at max_lifetime
python tool_error_demo.py           # consecutive_failures stays 0

To run against a real model, delete the two os.environ lines and the FlakyLLM calls, and set OPENAI_API_KEY.

Choose the numbers

Start from how long your typical outage lasts. With restart_backoff=2 and max_restarts=5, the waits are 2, 4, 8, 16 and 32 seconds, about one minute in total before the process gives up. With max_restarts=8 the last three waits hit the 60 s cap and the total grows to about four minutes. Add the time to reach FAILED each round: max_consecutive_failures multiplied by the time between triggers. A cron agent that runs every five minutes with max_consecutive_failures=3 needs fifteen minutes to fail once, so it needs fewer restarts than a busy webhook agent.

Restart the program, too

restart_policy restarts a process inside a running Python program. It can't help when the program itself dies: an out-of-memory kill, an unhandled exception in your main(), a host reboot. Run the program under Docker with a restart policy (--restart unless-stopped), Kubernetes, or systemd with Restart=on-failure. The two layers work together: the runtime handles agent failures quickly and keeps the program alive, and the platform handles program crashes. Deploy an AI agent with Docker shows the container setup.

Keep a journal

The journal is off by default. Turn it on in production, at least at level="checkpoint", with the file backend on a persistent volume. The error, restart and restart_exhausted entries are the record you read after an incident, and checkpoints let a restarted program recover its state. Durable AI agents covers recovery from the journal.

Watch restarts, not just failures

A process that restarts every hour and always recovers looks healthy in a status check. Count process.restarted events per process and alert on a rate, for example more than three in an hour. status()["restart_count"] shows the current streak, and the control API exposes the same status over HTTP.

Test the policy in CI

flaky_llm.py makes restart behavior a normal test. Start it, set it to down, start the process with your real config and small timings, and assert on the sequence of transitions your listener recorded. It runs in seconds and needs no API key. A policy you have never seen fire is a guess.

• One program, one machine. Restarts happen inside the Python program. A crash of the program, or of the machine, needs an outside supervisor.

• No replay of the failed event. A restart starts fresh. The events that failed are in the dead-letter list until you redeliver them, and events queued in memory are lost if the program exits.

• No per-error policy. Every raised run counts the same. You can't say "restart on 503, give up at once on 401" in config; a listener that stops the process on certain errors is the workaround.

• No restart event for giving up. The runtime writes restart_exhausted to the journal but emits no event for it, so you add one yourself as in step 4.

• No jitter. The backoff doubles exactly. Many processes that failed together restart together.

How do I make an AI agent restart automatically after a crash?

Set restart_policy="on_failure" on the process. When max_consecutive_failures runs fail in a row, the process enters FAILED, waits restart_backoff seconds (doubling per attempt, up to 60 s) and starts again, up to max_restarts times in a row. For a crash of the whole program, add a restart policy in Docker, Kubernetes or systemd.

What is the difference between restart_policy "always" and "on_failure"?

Both restart a process after it enters FAILED. "always" also restarts a process that reaches max_lifetime, which recycles it on a schedule. With "on_failure", a process that reaches max_lifetime stops and stays stopped. Neither restarts a process you stopped on purpose with stop().

Does max_restarts limit restarts forever?

No. It limits restart attempts in a row. The counter resets after one successful run, so a process that fails and recovers once a day never runs out of restarts. When the counter reaches max_restarts, the process stays FAILED and the journal records restart_exhausted.

Does an AI agent lose its memory when it restarts?

Not with an automatic restart in Promptise. The restart happens in the same Python program, so context state, conversation history, counters and the process ID are kept. The agent, triggers and MCP connections are rebuilt. If the whole program restarts, in-memory state is lost unless you keep a journal and recover from it.

How do I set a timeout for an AI agent run?

Set max_invocation_time in seconds on the process. A run that takes longer is cancelled with TimeoutError, the agent emits invocation.timeout, and the run counts as a failure toward max_consecutive_failures. The limit covers the whole run, including every model and tool call.

How do I get alerted when an agent stops for good?

Add a lifecycle listener that checks, on each transition to FAILED, whether the restart policy is "never" or the restart count has reached max_restarts. If so, emit your own event, such as process.gave_up, and route it to Slack or a pager with a WebhookSink. Alerting on every process.failed also pages for failures the runtime recovers from.

• Your first long-running agent, end to end: deploy AI agents in production.

• All eight process states, suspend and resume: AI agent lifecycle states.

• Many agents in one program, with failures kept apart: run multiple AI agents.

• Catch failures that don't raise: AI agent health monitoring.

• Survive a crash of the whole program: durable AI agents.

• Send alerts where people see them: AI agent notifications.

• Reference: agent processes and restart policies and process lifecycle in the Promptise docs.