Distributed AI Agents: Run Agents Across Many Machines

Run one agent runtime per server and join the runtimes into a cluster. Each node serves a small management API protected by a shared token. One node also hosts a coordinator that keeps the list of nodes and checks their health every few seconds. Nodes register with the coordinator and send their registration again on every heartbeat. Through the API you start and stop agents and send them events on any node. In Promptise Foundry 1.5.0, you switch this on with DistributedConfig in RuntimeConfig. Placement and failover are not included: when a node dies, your code (or Kubernetes) decides where its agents run next.

• One runtime per machine, one coordinator per cluster. coordinator=True on one node, coordinator_url on the others.

• Registration is a heartbeat. A node that stops re-registering for node_timeout seconds turns unhealthy and is dropped.

• Remote control is plain HTTP. POST /processes/{name}/start, /stop and /event on any node, with Authorization: Bearer <token>.

• Nothing moves by itself. The coordinator reports health. Moving work to a healthy node is a loop you write, as in step 6.

A single agent runtime supervises many agent processes in one program on one machine. A distributed runtime runs several of those programs on different machines and adds three things on top: membership, health and remote control.

Membership answers "which nodes exist?". There are two common designs. With static discovery, the coordinator's config lists every node's URL. With registration (a registry), nodes announce themselves to the coordinator and repeat the announcement on a timer. A registration that isn't renewed within its time to live expires, so the list cleans itself up when machines disappear.

Health answers "which nodes work right now?". The coordinator calls each node's health endpoint on a fixed interval. One failed call is usually a network blip, so a good health checker waits until a node has been silent for a timeout before it marks the node unhealthy.

Remote control answers "how do I act on an agent over there?". Each node serves an authenticated API that can start a process, stop it, report its status and push an event into its queue.

                 coordinator node (:9100)
  ┌────────────────────────────────────────────────────────┐
  │ RuntimeTransport    /health /status /processes/...     │
  │                     /cluster/nodes  /cluster/status    │
  │ RuntimeCoordinator  GET /health on every node, every   │
  │                     heartbeat_interval; unhealthy      │
  │                     after node_timeout of silence      │
  │ RegistryDiscovery   registrations expire after         │
  │                     node_timeout                       │
  └──────────▲─────────────────────────────┬───────────────┘
             │ POST /cluster/nodes         │ GET /health
             │ (every heartbeat)           │ POST /processes/{name}/start|stop|event
             │                             ▼
  ┌──────────┴─────────────────────────────────────────────┐
  │ worker-1 (:9101)   RuntimeTransport + AgentRuntime     │
  │                    process: summarizer                 │
  └────────────────────────────────────────────────────────┘

The hard part of any cluster is the fourth thing: placement and failover, deciding where an agent runs and moving it when its node dies. It is hard because a node that stops answering may still be running. If you start a second copy of an agent while the first one is only cut off from the network, both copies act. For agents that send emails or change records, that means duplicate side effects. Promptise leaves this decision to you, on purpose. Step 6 shows a simple policy and its limits.

Distribute when location or capacity forces you to. If you only want high availability, you need a failover policy anyway, and an orchestrator that already has one may be the shorter path. The comparison section below goes through the trade-offs.

• Python 3.10 or later and Promptise Foundry 1.5.0 or later.

• An OpenAI API key in OPENAI_API_KEY. The demo agent makes one short gpt-5-mini call per event, a fraction of a cent each.

• curl and jq, ports 9100 and 9101 free, and about 20 minutes.

pip install "promptise>=1.5.0"

Both nodes run the same agent, a summarizer for support tickets, and print every task they finish. That shared part lives in common.py:

"""Shared by both nodes: the agent they run and a hook that prints each finished task."""
import logging
from datetime import datetime

from promptise.runtime import HookContext, HookEvent, HookManager, ProcessConfig

SUMMARIZER = ProcessConfig(
    model="openai:gpt-5-mini",
    instructions="Summarize the ticket in the payload in one short sentence.",
)


def task_printer(node_id: str) -> HookManager:
    """A shared hook manager that prints every task a process on this node finishes."""
    hooks = HookManager()

    @hooks.on(HookEvent.TASK_COMPLETED)
    async def done(ctx: HookContext) -> None:
        d = ctx.data
        print(f"{datetime.now():%H:%M:%S} [{node_id}] {ctx.process_id}: {d['trigger_type']} event {d['event_id'][:8]} "
              f"{d['outcome']} in {d['duration_ms'] / 1000:.1f}s", flush=True)

    return hooks


def quiet_logs() -> None:
    """Show the cluster's own log lines, hide the rest."""
    logging.basicConfig(level=logging.WARNING, datefmt="%H:%M:%S",
                        format="%(asctime)s %(name)s: %(message)s")
    for name in ("promptise.runtime.distributed.node", "promptise.runtime.distributed.coordinator"):
        logging.getLogger(name).setLevel(logging.INFO)

The coordinator node hosts the cluster coordinator and keeps a standby copy of the summarizer that it starts only if the worker fails:

"""The coordinator node: hosts the cluster coordinator and a standby summarizer."""
import asyncio
import logging
import signal

from common import SUMMARIZER, quiet_logs, task_printer
from failover import keep_one_running
from promptise.runtime import AgentRuntime, DistributedConfig, RuntimeConfig

config = RuntimeConfig(
    distributed=DistributedConfig(
        enabled=True,
        coordinator=True,               # host the coordinator and the /cluster routes
        node_id="coordinator",
        transport_port=9100,
        auth_token="${CLUSTER_TOKEN}",
        heartbeat_interval=2,           # health check every node every 2 s
        node_timeout=6,                 # unhealthy (and dropped) after 6 s of silence
    ),
)


async def main() -> None:
    runtime = AgentRuntime(config, hooks=task_printer("coordinator"))
    await runtime.start_all()                               # starts the coordinator
    await runtime.add_process("summarizer", SUMMARIZER)     # standby: registered, not started
    watcher = asyncio.create_task(
        keep_one_running(runtime.cluster.coordinator, "summarizer",
                         preference=["worker-1", "coordinator"])
    )
    stop = asyncio.Event()
    for sig in (signal.SIGTERM, signal.SIGINT):
        asyncio.get_running_loop().add_signal_handler(sig, stop.set)
    await stop.wait()
    watcher.cancel()
    await runtime.shutdown()


if __name__ == "__main__":
    quiet_logs()
    logging.getLogger("failover").setLevel(logging.INFO)
    asyncio.run(main())

What each DistributedConfig field does here:

• `enabled=True` makes start_all() start a ClusterNode before the processes. If the node can't start (port in use, token variable unset), start_all() raises and no process starts.

• `coordinator=True` adds the coordinator and the /cluster/* routes to this node's transport. The coordinator lists itself as a node.

• `transport_port=9100` is the management API. It binds 127.0.0.1 by default, which is right for this local demo.

• `auth_token="${CLUSTER_TOKEN}"` is the one cluster token. The ${...} reference is resolved when the transport starts, so the token never sits in code.

• `heartbeat_interval=2` is how often the coordinator checks every node's health (the default is 15 seconds; 2 makes the demo fast).

• `node_timeout=6` is how long a node may stay silent before it counts as unhealthy and its registration expires. The default is three heartbeat intervals.

start_all() starts every registered process, so the standby is added after it: add_process() registers a process without starting it. failover.py comes in step 6.

A worker node runs its agents and registers with the coordinator:

"""A worker node: runs the summarizer and registers with the coordinator."""
import asyncio
import os

from common import SUMMARIZER, quiet_logs, task_printer
from promptise.runtime import AgentRuntime, DistributedConfig, RuntimeConfig

NODE_ID = os.environ.get("NODE_ID", "worker-1")
PORT = int(os.environ.get("NODE_PORT", "9101"))

config = RuntimeConfig(
    processes={"summarizer": SUMMARIZER},
    distributed=DistributedConfig(
        enabled=True,
        node_id=NODE_ID,
        transport_port=PORT,                       # this node's management API
        coordinator_url="http://127.0.0.1:9100",   # register here
        auth_token="${CLUSTER_TOKEN}",             # resolved at start
        heartbeat_interval=2,                      # re-register every 2 s
    ),
)

if __name__ == "__main__":
    quiet_logs()
    asyncio.run(AgentRuntime(config, hooks=task_printer(NODE_ID)).run_forever())

coordinator_url makes this node a registering node. It registers when it starts, sends the registration again every heartbeat_interval, and unregisters when it shuts down cleanly. If the coordinator isn't up yet, the node starts anyway, logs the failure and keeps retrying, so you can start nodes in any order. node_id defaults to <hostname>-<port>; set it yourself so logs and failover rules can name nodes.

run_forever() starts everything and waits for SIGTERM or SIGINT, then stops the processes and the cluster node. That matters for the clean shutdown in step 6.

Create the cluster token, start both nodes as separate OS processes, and ask the coordinator what it sees:

export CLUSTER_TOKEN=$(python -c "import secrets; print(secrets.token_urlsafe(32))")
AUTH="Authorization: Bearer $CLUSTER_TOKEN"
nodes() {  # one line per node the coordinator knows
  curl -s -H "$AUTH" http://127.0.0.1:9100/cluster/nodes |
    jq -r '.nodes[] | "  \(.node_id) \(.status) processes=\(.process_count)"'
}

python coordinator.py > coordinator.log 2>&1 & COORD=$!
python worker.py > worker.log 2>&1 & WORKER=$!
sleep 6

curl -s http://127.0.0.1:9100/health; echo
curl -s -o /dev/null -w "without token: %{http_code}\n" http://127.0.0.1:9100/cluster/nodes
nodes

== /health needs no token, /cluster/nodes does
{"status": "healthy", "node_id": "coordinator", "process_count": 1}
without token: 401
  coordinator healthy processes=1
  worker-1 healthy processes=1

GET /health is the only route that answers without the token, so load balancers and probes can use it. Every other route, /cluster/nodes included, answers 401 without Authorization: Bearer <token>. The node list shows the coordinator and worker-1, both healthy. process_count counts registered processes, so the coordinator's stopped standby counts as one.

Every node's transport accepts events for its processes. Send a ticket to the summarizer on worker-1, then read its status:

curl -s -X POST -H "$AUTH" -H 'Content-Type: application/json' \
  -d '{"payload": {"ticket": "Checkout fails for EU cards since 09:00"}}' \
  http://127.0.0.1:9101/processes/summarizer/event; echo
sleep 12
curl -s -H "$AUTH" http://127.0.0.1:9101/processes/summarizer/status |
  jq -c '{state, invocation_count}'

== send work to worker-1
{"status": "injected", "event_id": "fd596786-c2e1-41fc-b611-766a6c4aac42", "process": "summarizer"}
{"state":"running","invocation_count":1}
== worker.log
23:02:51 [worker-1] summarizer: remote event fd596786 success in 2.7s

202 injected means the event is in the process's queue, not that the run is done. The run happened on worker-1 a few seconds later, as its log shows. The event's trigger_type is "remote" unless the body sets another one.

The node API, and the coordinator methods that call it from Python (runtime.cluster.coordinator on the coordinator node):

POST /processes/{name}/start

POST /processes/{name}/stop

POST /processes/{name}/event

A process must already be registered on the node. The transport can start, stop and feed processes, but it can't deploy a new one. To deploy over HTTP, use the control API of that runtime.

Kill the worker without a chance to clean up, as a crash or a lost machine would, and poll the node list every two seconds:

kill -9 $WORKER
for t in 2 4 6 8 10; do sleep 2; echo "after ${t}s:"; nodes; done

== 23:03:00 kill -9 worker-1
./demo.sh: line 31: 79442 Killed: 9               python worker.py > worker.log 2>&1
after 2s:
  coordinator healthy processes=1
  worker-1 healthy processes=1
after 4s:
  coordinator healthy processes=1
  worker-1 healthy processes=1
after 6s:
  coordinator healthy processes=1
  worker-1 unhealthy processes=1
after 8s:
  coordinator healthy processes=1
after 10s:
  coordinator healthy processes=1
== coordinator.log
23:03:07 promptise.runtime.distributed.coordinator: Coordinator: unregistered node worker-1
23:03:07 promptise.runtime.distributed.node: Distributed runtime: node worker-1 expired (no re-registration)

This is the health timeline in practice:

1. 0 to 6 seconds: still healthy. Health checks fail, but a healthy node that misses checks stays healthy until node_timeout has passed since its last good check. One dropped request doesn't flap the cluster view.

2. About 6 seconds: unhealthy. node_timeout has passed without a good check, and the next check marks the node unhealthy.

3. About 7 seconds: dropped. Its registration has expired, so the coordinator's prune loop removes it.

With registry discovery, "unhealthy" and "dropped" come close together, because both use node_timeout. A node listed with static discovery is never dropped; it stays in the list as unhealthy until it answers again.

Now look at what did not happen. The summarizer did not move: it died with its node. Events queued in worker-1's memory are gone, and the coordinator never knew about them. The coordinator only knows that a node it used to see is gone.

The failover policy is a small loop on the coordinator node. Every two seconds it asks the healthy nodes whether they have the summarizer and whether it runs. It keeps exactly one copy running, on the first healthy node in a preference list:

"""Keep one copy of a process running on the first healthy node that has it.

The coordinator only reports health. This loop is the failover policy.
"""
import asyncio
import logging

log = logging.getLogger("failover")


async def running_on(coordinator, name: str) -> dict[str, bool] | None:
    """For each healthy node that has the process: is it running? None if a node didn't answer."""
    found = {}
    for node in coordinator.healthy_nodes:
        try:
            status = await coordinator.get_node_status(node.node_id)
        except Exception:  # an error status (RuntimeError) or no answer (aiohttp, timeout)
            log.info("%s did not answer; waiting for its health check to decide", node.node_id)
            return None
        process = status["processes"].get(name)
        if process is not None:
            found[node.node_id] = process["state"] == "running"
    return found


async def keep_one_running(coordinator, name: str, preference: list[str],
                           every: float = 2.0, grace: float = 6.0) -> None:
    await asyncio.sleep(grace)  # let the nodes register before the first decision
    while True:
        try:
            nodes = await running_on(coordinator, name)
            target = next((n for n in preference if n in (nodes or {})), None)
            if nodes is not None and target is None:
                log.warning("no healthy node has %s registered", name)
            elif nodes is not None:
                if not nodes[target]:
                    await coordinator.start_process_on_node(target, name)
                    log.warning("started %s on %s", name, target)
                for node_id, running in nodes.items():
                    if running and node_id != target:
                        await coordinator.stop_process_on_node(node_id, name)
                        log.warning("stopped %s on %s, it runs on %s", name, node_id, target)
        except Exception as exc:  # a supervisor must survive one bad round
            log.warning("failover round failed: %s", exc)
        await asyncio.sleep(every)

In the demo, after the kill, demo.sh sends the next ticket to the coordinator node (port 9100), waits, and starts worker.py again:

== coordinator.log
23:03:01 failover: worker-1 did not answer; waiting for its health check to decide
23:03:03 failover: worker-1 did not answer; waiting for its health check to decide
23:03:05 failover: worker-1 did not answer; waiting for its health check to decide
23:03:07 promptise.runtime.distributed.coordinator: Coordinator: unregistered node worker-1
23:03:07 promptise.runtime.distributed.node: Distributed runtime: node worker-1 expired (no re-registration)
23:03:08 failover: started summarizer on coordinator
23:03:12 [coordinator] summarizer: remote event d766b75b success in 2.0s
23:03:24 promptise.runtime.distributed.coordinator: Coordinator: registered node worker-1 at http://127.0.0.1:9101
23:03:26 failover: stopped summarizer on coordinator, it runs on worker-1
== worker.log
23:03:24 promptise.runtime.distributed.node: Distributed runtime: node worker-1 registered with http://127.0.0.1:9100

The loop waited while worker-1 was still healthy but silent. Then the coordinator dropped the node and the loop started the standby on the coordinator, which ran the next ticket. When worker-1 came back, run_forever() started its summarizer again, so for a moment two copies ran. The loop saw that and stopped the coordinator's copy: that is the fail back.

Three choices in this loop matter more than the code:

• Wait for the coordinator's verdict. A loop that treats "the status call failed" as "the node is gone" starts the standby one interval after the first missed answer, so a slow network starts a second copy. Let node_timeout decide.

• Catch every error. A node that doesn't answer raises an aiohttp or timeout error, not RuntimeError. A supervisor that crashes on its first dead node is no supervisor.

• Plan for two copies. A node that is cut off, not dead, keeps running its agent. If the agent does irreversible things, make its tools idempotent or cap them; see AI agent irreversible actions.

A clean stop is different from a crash. With SIGTERM, the worker unregisters at once:

kill -TERM $WORKER; wait $WORKER
nodes

== 23:03:30 stop worker-1 cleanly (SIGTERM)
  coordinator healthy processes=1
== coordinator.log
23:03:30 promptise.runtime.distributed.coordinator: Coordinator: unregistered node worker-1

The transport can start, stop and drive every agent on a node, so the config refuses unsafe combinations when you build it:

"""What DistributedConfig refuses, and why (no network needed)."""
import asyncio
import os

from pydantic import ValidationError

from promptise.runtime import AgentRuntime, DistributedConfig, RuntimeConfig

CASES = {
    "public bind, no token": dict(enabled=True, transport_host="0.0.0.0"),
    "wildcard bind, no advertise_url": dict(enabled=True, transport_host="0.0.0.0",
                                            auth_token="t0ken", coordinator=True),
    "coordinator and coordinator_url": dict(enabled=True, coordinator=True,
                                            coordinator_url="http://10.0.0.10:9100",
                                            auth_token="t0ken"),
    "multicast discovery": dict(enabled=True, discovery_method="multicast"),
    "privileged port": dict(enabled=True, transport_port=80),
}

for label, kwargs in CASES.items():
    try:
        DistributedConfig(**kwargs)
    except ValidationError as exc:
        print(f"{label}: {exc.errors()[0]['msg']}")


async def unset_token() -> None:
    os.environ.pop("CLUSTER_TOKEN", None)
    runtime = AgentRuntime(RuntimeConfig(distributed=DistributedConfig(
        enabled=True, coordinator=True, auth_token="${CLUSTER_TOKEN}")))
    try:
        await runtime.start_all()
    except ValueError as exc:
        print(f"unset token variable: {exc}")


asyncio.run(unset_token())

public bind, no token: Value error, transport_host '0.0.0.0' is not a loopback address: set auth_token (the cluster token), so only cluster members can manage this node, or allow_unauthenticated=True if a proxy or private network authenticates callers
wildcard bind, no advertise_url: Value error, transport_host '0.0.0.0' is a wildcard address, so other nodes can't use it: set advertise_url to the URL they reach this node at (e.g. 'http://node-1.internal:9100')
coordinator and coordinator_url: Value error, A coordinator doesn't register with another coordinator: set either coordinator=True or coordinator_url, not both
multicast discovery: Value error, discovery_method='multicast' is not implemented. Use 'registry' (nodes register with the coordinator at coordinator_url) or 'static' (list the nodes in static_nodes on the coordinator).
privileged port: Value error, transport_port 80 is a privileged port; use a port above 1024 (or 0 for a free one)
unset token variable: distributed.auth_token: Environment variable 'CLUSTER_TOKEN' not found (referenced in: distributed.auth_token)

Two more controls apply on the coordinator. The coordinator sends health checks and management calls, with the cluster token, to every URL that registers. Anyone who holds the token could register an internal address and make the coordinator call it. allowed_node_hosts limits the hosts a registering node may name. Static discovery removes registration altogether:

"""Registry vs static coordinators, allowed_node_hosts and a wrong token."""
import asyncio

import aiohttp

from promptise.runtime import AgentRuntime, DistributedConfig, RuntimeConfig

TOKEN = "cluster-token-for-this-demo"


async def post(url: str, body: dict, token: str = TOKEN) -> str:
    async with aiohttp.ClientSession() as s, s.post(
        url, json=body, headers={"Authorization": f"Bearer {token}"}
    ) as r:
        return f"{r.status} {(await r.json()).get('error') or (await r.json())['status']}"


async def main() -> None:
    registry = AgentRuntime(RuntimeConfig(distributed=DistributedConfig(
        enabled=True, coordinator=True, node_id="coord-a", transport_port=9200,
        auth_token=TOKEN, allowed_node_hosts=["127.0.0.1", "*.cluster.internal"],
    )))
    static = AgentRuntime(RuntimeConfig(distributed=DistributedConfig(
        enabled=True, coordinator=True, node_id="coord-b", transport_port=9201,
        auth_token=TOKEN, discovery_method="static", heartbeat_interval=1,
        static_nodes={"worker-9": "http://127.0.0.1:9299"},  # nothing listens there
    )))
    await registry.start_all()
    await static.start_all()
    nodes = "http://127.0.0.1:9200/cluster/nodes"
    print("allowed host:  ", await post(nodes, {"node_id": "w1", "url": "http://w1.cluster.internal:9100"}))
    print("other host:    ", await post(nodes, {"node_id": "w2", "url": "http://169.254.169.254/"}))
    print("wrong token:   ", await post(nodes, {"node_id": "w3", "url": "http://127.0.0.1:9103"}, "guess"))
    print("static coord.: ", await post("http://127.0.0.1:9201/cluster/nodes",
                                        {"node_id": "w4", "url": "http://127.0.0.1:9104"}))
    await asyncio.sleep(2.5)
    status = await static.cluster.coordinator.cluster_status()
    print("static nodes:  ", {n: v["status"] for n, v in status["nodes"].items()})
    await registry.shutdown()
    await static.shutdown()


asyncio.run(main())

allowed host:   200 registered
RuntimeTransport coord-a: refused to register node "'w2'" at http://169.254.169.254/ (host not in allowed_node_hosts)
other host:     403 The node URL's host is not in the coordinator's allowed_node_hosts
wrong token:    401 Unauthorized
static coord.:  409 This coordinator uses static discovery: nodes are listed in its static_nodes, they don't register
static nodes:   {'worker-9': 'unhealthy', 'coord-b': 'healthy'}

The allowed host registered, the metadata address was refused with 403 and a warning, the wrong token got 401, and the static coordinator refused registration with 409. Its listed node that never answered shows as unhealthy instead of disappearing.

distributed-demo/
├── common.py           # the summarizer config and a task_completed printer
├── coordinator.py      # node 1: coordinator + standby summarizer + failover loop
├── worker.py           # node 2: the summarizer, registers with the coordinator
├── failover.py         # keep one copy running on the first healthy node
├── config_errors.py    # what DistributedConfig refuses
├── security_checks.py  # allowed_node_hosts, wrong token, static discovery
└── demo.sh             # the whole run: start, send work, kill, fail over, stop

#!/usr/bin/env bash
# A two-node cluster on one machine: start it, use it, kill a node, bring it back.
set -u
cd "$(dirname "$0")"
export CLUSTER_TOKEN=$(python -c "import secrets; print(secrets.token_urlsafe(32))")
AUTH="Authorization: Bearer $CLUSTER_TOKEN"
nodes() {  # one line per node the coordinator knows
  curl -s -H "$AUTH" http://127.0.0.1:9100/cluster/nodes |
    jq -r '.nodes[] | "  \(.node_id) \(.status) processes=\(.process_count)"'
}

python coordinator.py > coordinator.log 2>&1 & COORD=$!
python worker.py > worker.log 2>&1 & WORKER=$!
sleep 6

echo "== /health needs no token, /cluster/nodes does"
curl -s http://127.0.0.1:9100/health; echo
curl -s -o /dev/null -w "without token: %{http_code}\n" http://127.0.0.1:9100/cluster/nodes
nodes

echo "== send work to worker-1"
curl -s -X POST -H "$AUTH" -H 'Content-Type: application/json' \
  -d '{"payload": {"ticket": "Checkout fails for EU cards since 09:00"}}' \
  http://127.0.0.1:9101/processes/summarizer/event; echo
sleep 12
curl -s -H "$AUTH" http://127.0.0.1:9101/processes/summarizer/status |
  jq -c '{state, invocation_count}'

echo "== $(date +%T) kill -9 worker-1"
kill -9 $WORKER
for t in 2 4 6 8 10; do sleep 2; echo "after ${t}s:"; nodes; done

echo "== work now goes to the coordinator's copy"
curl -s -X POST -H "$AUTH" -H 'Content-Type: application/json' \
  -d '{"payload": {"ticket": "Refund stuck in pending for order 1182"}}' \
  http://127.0.0.1:9100/processes/summarizer/event; echo
sleep 12

echo "== $(date +%T) restart worker-1"
python worker.py >> worker.log 2>&1 & WORKER=$!
sleep 8
nodes

echo "== $(date +%T) stop worker-1 cleanly (SIGTERM)"
kill -TERM $WORKER; wait $WORKER
nodes
kill -TERM $COORD; wait $COORD

echo "== coordinator.log"; cat coordinator.log
echo "== worker.log"; cat worker.log

Protect the token and the wire

Generate the cluster token once, store it in your secret manager, and give it to each node as CLUSTER_TOKEN. One token controls every node, so rotate it like a root credential. The transport speaks plain HTTP. Between machines, bind transport_host="0.0.0.0", put a TLS-terminating proxy in front, and set advertise_url to the https:// address other nodes use. On a registry coordinator, always set allowed_node_hosts, for example ["*.agents.internal"].

Mind the ports

DistributedConfig.transport_port and the control API (OrchestrationAPI) both default to 9100. On a node that runs both, give each its own port, or the second one fails with an address-in-use error.

Treat the coordinator as a single instance

The coordinator isn't replicated. If it dies, the workers keep running their agents and keep trying to register. A restarted coordinator (registry discovery) finds them again within one heartbeat_interval, but its health history starts empty. Run it under a supervisor that restarts it, and keep your failover loop tolerant of an empty node list right after a restart.

Keep state where the agent runs

Each node has its own journal. With a file journal, a process that restarts on the same node recovers its context, conversation and budget counters (crash recovery). A standby on another machine starts fresh unless both nodes use the same journal path on shared storage, and then only one copy may run at a time.

Containers and Kubernetes

Run one runtime per container. The Docker deployment guide covers the image, health probes and SIGTERM. promptise runtime start runs a runtime without a RuntimeConfig, so it can't join a cluster: start nodes with a Python entry point like worker.py. On Kubernetes, give the coordinator a stable Service name and set each worker's advertise_url to an address the coordinator can reach.

Watch the cluster

runtime.status()["cluster"] shows a node's role, URL, registration state and last_registration_error. On the coordinator it also shows how many nodes there are and how many are healthy. GET /cluster/status returns the same summary for all nodes over HTTP. Alert when a node goes unhealthy, not only when an agent fails.

All three keep work running across machines. They solve different parts of the problem, and they combine well.

Kubernetes is the better choice when you mainly want "keep N copies running and reschedule them when a machine dies"; a ReplicaSet's purpose is exactly that. Temporal is the better choice when one long, multi-step job must survive a crash halfway and continue elsewhere with its exact state. Promptise's distributed mode is the better fit when you need agent-level control across machines: send this event to the summarizer in the EU node, stop the agent in the customer VPC, see each agent's budget and status. Common combinations are one Promptise runtime per Kubernetes pod, or a Temporal activity that sends an event to an agent and waits for its result.

• No placement, scheduling or failover. The coordinator monitors health and runs the operations you ask for. It never starts or moves a process on its own.

• The coordinator is a single instance. It holds the registry in memory and isn't replicated.

• No remote deploys over the transport. A process must be registered on a node before the cluster can start it.

• One shared token, plain HTTP. Per-node tokens exist only when you register nodes by hand with RuntimeCoordinator.register_node(..., auth_token=...).

• Queued events live in memory. A crashed node loses the events in its queue. Use idempotent senders and retries for work that must not be lost.

• No cluster-wide event bus. Events reach the node you post them to. For agent-to-agent events, see the multi-agent event bus.

• Multicast discovery isn't implemented. Only registry and static work.

Can I run AI agents across multiple servers with one control point?

Yes. Run one agent runtime per server and let one of them host a coordinator. In Promptise Foundry, DistributedConfig(enabled=True, coordinator=True) on one node and coordinator_url on the others form the cluster. The coordinator lists every node and its health, and each node's API starts, stops and sends events to its agents, with one shared bearer token.

What happens when a node in an agent cluster fails?

The coordinator's health checks start failing. The node stays healthy until node_timeout passes without a good check, then turns unhealthy, and a registered node is dropped once its registration expires. Its agents stop with it, and events queued in its memory are lost. Nothing restarts elsewhere unless your failover code starts a standby on a healthy node.

Does Promptise move agents to another machine automatically?

No. Distributed mode reports health and runs remote operations, but it doesn't place or move processes. You write the policy: watch coordinator.healthy_nodes, call start_process_on_node() on a standby, and stop extra copies when a node returns. That keeps the decision visible, which matters for agents with side effects.

Should I use Kubernetes or a distributed agent runtime?

Use Kubernetes when you want containers restarted and rescheduled automatically when a machine dies. Use a distributed agent runtime when you need to control individual agents across machines, such as sending an event to one agent or stopping the agent in one region. Many teams use both: one agent runtime per pod, with Kubernetes handling restarts.

How is a distributed agent runtime different from Temporal?

Temporal makes workflow code durable: if a worker crashes, the workflow continues on another worker by replaying its history, and activities are retried by default. Promptise's distributed mode doesn't move work; it connects agent runtimes so you can see and control their agents. Use Temporal for long multi-step jobs that must survive crashes mid-way, and an agent runtime for long-running, event-driven agents.

How do the nodes of an agent cluster authenticate?

With one cluster token. Every node's API requires Authorization: Bearer <token> on every route except GET /health. A node uses the same token to register with the coordinator, and the coordinator uses it to call the nodes. Keep it in an environment variable referenced as ${CLUSTER_TOKEN}, and put TLS in front of any node reached over a network.

• Start with one machine: run multiple AI agents in one runtime.

• Package each node as a container: deploy an AI agent with Docker.

• Deploy and manage processes over HTTP: the AI agent control API.

• Keep state across restarts on each node: durable AI agents and crash recovery.

• Learn the bigger picture: what is an agentic runtime? and the autonomous AI agent runtime.

• Reference: distributed mode, the coordinator and discovery and transport in the Promptise docs.