AI Agent Health Monitoring: Catch Silent Failures

Record a few cheap signals for every run and compare them with thresholds: the agent's final reply (is it empty?), whether the run failed, and the sequence of tool calls (is it repeating?). When a signal crosses its threshold, raise an anomaly and act on it: log it, pause the agent, or notify a person. In the Promptise agent runtime this is one config block on the process:

from promptise.runtime import HealthConfig, ProcessConfig

config = ProcessConfig(
    model="openai:gpt-5-mini",
    instructions="Review the order feed and report the orders that need attention.",
    health=HealthConfig(enabled=True, on_anomaly="pause"),
)

Key takeaways:

• Silent failures need behavioral checks. A process can be "running" while its agent produces nothing useful.

• Four detectors cover the common cases: stuck, loop, empty response and high error rate. None of them calls an LLM.

• Empty-response and error-rate detection work across runs, so they catch slow drift that no single run shows.

• The action runs when a run ends. log records it, pause suspends the process, escalate notifies and suspends.

• Tune thresholds to your agent's normal output, or a short but correct reply looks like a failure.

A health monitor is a set of small detectors that compare the agent's behavior with limits. It doesn't judge quality, so it needs no second model. It asks narrower questions: did the agent say anything, did the run fail, is it repeating itself?

 trigger event
      │
      ▼
 run starts ──▶ begin_invocation()      tool history cleared
      │
      ├─ every tool call ──▶ stuck / loop detectors        (within this run)
      │
      ▼
 run ends
      ├─ success ──▶ final reply ──▶ empty-response detector (across runs)
      │         └──▶ error window += ok ─┐
      └─ failure (retries used up) ──────┴▶ error-rate detector (across runs)
      │
      ▼
 new anomalies? ──▶ health.anomaly event ──▶ on_anomaly: log | pause | escalate

The four detectors, with their HealthConfig defaults:

Stuck and loop detection compare tool names and a hash of the arguments. They are covered in depth, including how to stop a loop while it runs, in AI agent stuck in a loop. This guide goes deeper on the other two signals and on the actions.

Empty-response detection reads the agent's final reply at the end of each successful run, after stripping whitespace. Tool results never count: a tool that returns [] is not an empty reply. A reply counts as empty when its length is at most empty_max_chars, so with the default of 10, "OK" (2 characters) and "Done." (5) count. The detector fires when the last empty_threshold replies are all empty.

Error-rate detection keeps a sliding window of the last error_window run outcomes. A run counts as failed only when it raised and used up its retries (trigger_delivery.max_retries). The detector stays silent until the window is full, then fires when failures divided by window size reaches error_rate_threshold.

Each detection is an Anomaly with four fields: anomaly_type (an AnomalyType: STUCK, LOOP, EMPTY_RESPONSE or HIGH_ERROR_RATE), a human-readable description, a UTC timestamp, and details with the evidence, such as the reply lengths or the error count.

• Python 3.10 or later and Promptise Foundry 1.4.3 or later.

• An OpenAI API key in OPENAI_API_KEY for step 4. The examples use openai:gpt-5-mini, and the whole run costs well under a cent. Steps 2 and 3 need no key.

• About 20 minutes.

pip install "promptise>=1.4.3"

Every field of HealthConfig has a working default, so enabled=True is enough to start. Spelling the fields out documents what "unhealthy" means for this agent:

from promptise.runtime import HealthConfig

health = HealthConfig(
    enabled=True,
    stuck_threshold=3,         # identical calls in a row (min 2)
    loop_window=20,            # tool calls searched for repeating sequences (min 4)
    loop_min_repeats=2,        # repeats that make a loop (min 2)
    empty_threshold=3,         # empty final replies in a row (min 2)
    empty_max_chars=10,        # a reply this short or shorter is empty (0 = only "")
    error_window=10,           # recent runs in the error-rate window (min 2)
    error_rate_threshold=0.5,  # failure share that fires (0 < x <= 1)
    on_anomaly="log",          # "log", "pause" or "escalate"
    cooldown=300,              # seconds before the same anomaly is reported again
    escalation=None,           # where "escalate" sends its notification
)

The same fields work in a .agent manifest under health:. The runtime feeds the monitor for you: tool calls through a callback, the final reply and the outcome when a run ends.

Feed a monitor by hand first. HealthMonitor is the class the runtime uses, and it runs without a model:

"""Feed HealthMonitor by hand to see when each detector fires. No model, no API key."""
import asyncio
import logging

from promptise.runtime import HealthConfig, HealthMonitor

logging.disable(logging.WARNING)  # the monitor also logs every anomaly


def show(label, anomaly):
    if anomaly is None:
        print(f"{label:<28} -")
    else:
        print(f"{label:<28} {anomaly.anomaly_type.value}: {anomaly.details}")


async def main():
    # Defaults: empty_threshold=3, empty_max_chars=10, error_window=10,
    # error_rate_threshold=0.5, cooldown=300
    config = HealthConfig(enabled=True)

    print("== empty responses (final replies, across runs)")
    monitor = HealthMonitor(config, process_id="orders-monitor")
    for reply in ["Checked 14 orders, 2 flagged: #4471, #4480.", "OK", "", "Done."]:
        label = f"reply {len(reply):>2} chars"
        show(label, await monitor.record_response(reply))

    print("\n== error rate (runs that failed after their retries)")
    monitor = HealthMonitor(config, process_id="orders-monitor")
    outcomes = "ok fail ok fail ok fail ok ok fail fail".split()
    for run, outcome in enumerate(outcomes, start=1):
        if outcome == "ok":
            await monitor.record_success()
            show(f"run {run:>2} ok", None)
        else:
            show(f"run {run:>2} fail", await monitor.record_error())

    print("\n== cooldown: the same signal again")
    for cooldown in (300, 0):
        monitor = HealthMonitor(HealthConfig(enabled=True, cooldown=cooldown), "orders-monitor")
        reported = [await monitor.record_response("") is not None for _ in range(5)]
        print(f"cooldown={cooldown:<4} five empty replies -> reported {reported}")

    print("\n== reading what the monitor holds")
    anomaly = monitor.latest_anomaly
    print("type:       ", anomaly.anomaly_type, "| value:", anomaly.anomaly_type.value)
    print("description:", anomaly.description)
    print("timestamp:  ", anomaly.timestamp.isoformat(timespec="seconds"))
    counts = monitor.health_status()["anomalies_by_type"]
    print("counts:     ", {kind.value: n for kind, n in counts.items()})


asyncio.run(main())

== empty responses (final replies, across runs)
reply 43 chars               -
reply  2 chars               -
reply  0 chars               -
reply  5 chars               empty_response: {'consecutive_count': 3, 'max_chars': 10, 'lengths': [2, 0, 5]}

== error rate (runs that failed after their retries)
run  1 ok                    -
run  2 fail                  -
run  3 ok                    -
run  4 fail                  -
run  5 ok                    -
run  6 fail                  -
run  7 ok                    -
run  8 ok                    -
run  9 fail                  -
run 10 fail                  high_error_rate: {'error_rate': 0.5, 'error_count': 5, 'window_size': 10, 'threshold': 0.5}

== cooldown: the same signal again
cooldown=300  five empty replies -> reported [False, False, True, False, False]
cooldown=0    five empty replies -> reported [False, False, True, True, True]

== reading what the monitor holds
type:        AnomalyType.EMPTY_RESPONSE | value: empty_response
description: Agent producing empty responses: 3 consecutive responses under 10 characters
timestamp:   2026-10-19T17:46:27+00:00
counts:      {'stuck': 0, 'loop': 0, 'empty_response': 3, 'high_error_rate': 0}

Four things to notice:

• A real report is healthy. The 43-character reply doesn't count. The next three replies (2, 0 and 5 characters) are all at or below 10, so the third one fires. The details carry the lengths as evidence.

• The error-rate detector waits for a full window. Runs 2, 4 and 6 failed, and the rate was already 50% after run 2, but nothing fired until run 10 filled the 10-run window with 5 failures. A brand-new process can't trip it on its first bad run.

• The cooldown hides repeats. With cooldown=300, five empty replies produce one report. With cooldown=0, every reply that continues the streak is reported again.

• `anomaly_type` is a string enum, so anomaly.anomaly_type == "empty_response" and .value both work in your own code.

on_anomaly decides what the runtime does with a new anomaly. Whatever the action, every anomaly is first logged as a warning and, when the process has an event notifier, emitted as a health.anomaly event:

escalate needs an EscalationTarget (a webhook URL or an event type) in escalation. Without one, the runtime only logs the anomaly and the process keeps running. Escalation targets get their own guide; for webhooks and Slack alerts on any runtime event, see AI agent notifications.

The action runs when the run that produced the anomaly ends, whether it succeeded or raised. A high error rate is acted on as soon as the failed run is recorded. Each anomaly is acted on once. To stop a run in the middle, use the in-run guard from the stuck-loop guide.

cooldown keeps one problem from producing a flood of alerts. Two anomalies are "the same" when they share type and evidence: the same stuck call, or the same loop sequence. Empty-response and error-rate anomalies are one signal per process, so after one fires, that type stays silent for cooldown seconds.

Now run a real process. The agent reviews an order feed and replies "OK" when there is nothing new. Each trigger carries an empty feed. The script also starts the management API and reads everything back through OrchestrationClient, as an operator would:

"""A monitored agent process: empty replies pause it, the API shows why, a tuned threshold resumes it."""
import asyncio
import logging
import secrets

from promptise.events import CallbackSink, EventNotifier
from promptise.runtime import (
    AgentRuntime,
    HealthConfig,
    JournalConfig,
    OrchestrationAPI,
    OrchestrationClient,
    ProcessConfig,
    TriggerEvent,
)

logging.disable(logging.WARNING)

INSTRUCTIONS = (
    "You review the order feed in each trigger payload. "
    "If the payload has no new orders, reply with exactly: OK. "
    "Otherwise list the order IDs that need attention."
)


async def on_event(event):
    print(f"  event  {event.event_type}: {event.data.get('anomaly_type', '')}")


async def run_once(process, client, payload):
    before = process.status()["invocation_count"]
    await process.inject(TriggerEvent(trigger_id="feed", trigger_type="manual", payload=payload))
    while process.status()["invocation_count"] == before:
        await asyncio.sleep(0.2)
    await asyncio.sleep(0.5)  # let the post-run health action and events finish
    info = await client.get_process("orders-monitor")
    last = await client.get_journal("orders-monitor", entry_type="invocation_result", limit=1)
    reply = last[0]["data"]["response"]
    print(f"run {info['invocation_count']}: reply={reply!r} state={info['state']} "
          f"anomalies={info['health_anomalies']}")


async def main():
    notifier = EventNotifier([CallbackSink(on_event, events=["health.anomaly", "health.recovered"])])
    runtime = AgentRuntime(event_notifier=notifier)
    process = await runtime.add_process(
        "orders-monitor",
        ProcessConfig(
            model="openai:gpt-5-mini",
            instructions=INSTRUCTIONS,
            journal=JournalConfig(level="checkpoint", backend="memory"),
            health=HealthConfig(enabled=True, on_anomaly="pause"),  # defaults otherwise
        ),
    )
    token = secrets.token_urlsafe(24)
    api = OrchestrationAPI(runtime, host="127.0.0.1", port=9100, auth_token=token)
    await api.start()
    await process.start()

    async with OrchestrationClient("http://127.0.0.1:9100", token=token) as client:
        for _ in range(3):
            await run_once(process, client, {"new_orders": []})

        print("\nanomalies from the API:")
        for anomaly in await client.get_anomalies("orders-monitor"):
            print(" ", anomaly)

        print("\ntune: only a truly empty reply counts, then resume")
        print(" ", await client.update_health("orders-monitor", empty_max_chars=0))
        print(" ", await client.clear_anomalies("orders-monitor"))
        print(" ", await client.resume_process("orders-monitor"))
        await run_once(process, client, {"new_orders": []})

    await process.stop()
    await api.stop()
    await notifier.stop()


asyncio.run(main())

run 1: reply='OK.' state=running anomalies=0
run 2: reply='OK.' state=running anomalies=0
  event  health.anomaly: empty_response
run 3: reply='OK.' state=suspended anomalies=1

anomalies from the API:
  {'type': 'empty_response', 'description': 'Agent producing empty responses: 3 consecutive responses under 10 characters', 'timestamp': '2026-10-19 17:47:36.981927+00:00'}

tune: only a truly empty reply counts, then resume
  {'name': 'orders-monitor', 'health_updated': True, 'effective_from': 'immediate'}
  {'name': 'orders-monitor', 'anomalies_cleared': 1}
  {'name': 'orders-monitor', 'resumed': True}
  event  health.recovered: 
run 4: reply='OK.' state=running anomalies=0

What happened:

• Runs 1 and 2 were fine on their own. Each "OK." is 3 characters, but one or two short replies are below empty_threshold.

• Run 3 completed the streak. The monitor raised empty_response, the health.anomaly event reached the callback sink, and on_anomaly="pause" suspended the process at the end of the run.

• The API showed the anomaly with its type, description and timestamp.

• Was this a real failure? No. For this agent, "OK" is the correct reply to an empty feed, so the threshold was wrong. The script fixes it over the API, the tuning step 6 explains, and run 4 passes. health.recovered marks the first successful run after an anomaly.

You have four ways to see anomalies, depending on where you are:

DELETE /api/v1/processes/{name}/health/anomalies (client.clear_anomalies(name)) clears the list after you have dealt with it. It clears the history only. It doesn't resume a paused process, so call POST /api/v1/processes/{name}/resume as well. The API needs a bearer token whenever it binds to anything other than localhost; see the AI agent control API for the full route list and its security model.

Use events for alerting, because they arrive as the anomaly is handled and carry the details. Use the anomaly list for dashboards and incident reviews. With HealthMonitor on its own, read monitor.anomalies, monitor.latest_anomaly and monitor.health_status().

A health alert that fires on normal behavior trains people to ignore it. Tune each detector against what your agent does on a good day.

Empty responses. Decide what a healthy reply looks like, then pick one of three fixes:

• Change the reply contract. Ask for "No new orders in this batch (0 checked)" instead of "OK". A real empty reply then stands out, and the default limits work.

• Lower `empty_max_chars`. 0 counts only replies that are empty after stripping whitespace. Use this when "OK" or "Done" is a correct answer.

• Raise `empty_threshold` when short replies are common but long streaks of them are not.

Error rate. The default fires when 5 of the last 10 runs failed. That is right for an agent that runs every few minutes. For one that runs twice a day, 10 runs is five days of history, so a smaller error_window such as 4 notices a problem sooner. Raise error_rate_threshold if your agent calls a flaky upstream service and you already retry it with trigger_delivery.max_retries; a run that succeeds on a retry counts as a success.

Change thresholds live, but set windows up front. PATCH /api/v1/processes/{name}/health validates the fields and applies them at once. This is the fix from step 4:

await client.update_health("orders-monitor", empty_max_chars=0)  # PATCH .../health
await client.clear_anomalies("orders-monitor")                   # DELETE .../health/anomalies
await client.resume_process("orders-monitor")                    # POST .../resume

The history windows, though, are sized when the process is created. Raising error_window on a running process leaves the window at its old size, so the error-rate detector never fills it and stops firing. Lowering it works. Set error_window and loop_window in the config you deploy. A restart keeps the same monitor, so to change them, redeploy the process (remove it and add it again).

Stuck and loop. Raise stuck_threshold for agents that legitimately poll, and look at the stuck-loop guide for why changing arguments slip past both detectors.

The project has two files: detectors.py from step 2 (no API key) and monitored_agent.py from step 4.

health-demo/
├── detectors.py         # feed HealthMonitor by hand, see each detector fire
└── monitored_agent.py   # real process: empty replies pause it, the API explains why

pip install "promptise>=1.4.3"
python detectors.py
export OPENAI_API_KEY=sk-...
python monitored_agent.py

To add monitoring to an existing process, add the health block and an event sink for health.anomaly. Start with on_anomaly="log" for a week, tune, then switch to pause or escalate. A production starting point:

from promptise.runtime import EscalationTarget, HealthConfig, ProcessConfig, TriggerDeliveryConfig

config = ProcessConfig(
    model="openai:gpt-5-mini",
    instructions="Review the order feed. Reply with a one-line status, such as: No new orders (0 checked).",
    max_consecutive_failures=3,                           # FAILED after 3 failed runs in a row
    trigger_delivery=TriggerDeliveryConfig(max_retries=2),  # a run fails only after its retries
    health=HealthConfig(
        enabled=True,
        error_window=6,                                   # sized for a few runs per day
        on_anomaly="escalate",
        escalation=EscalationTarget(webhook_url="https://hooks.example.com/agent-alerts"),
    ),
)

Know how health and failures interact

max_consecutive_failures (default 3) moves the process to FAILED after three failed runs in a row, before a 10-run error window is full. The error-rate detector catches intermittent failures, such as every other run, which never form a streak. Keep both. What happens after FAILED is set by the restart policy, covered in AI agent auto-restart.

Treat "recovered" as "a run succeeded"

health.recovered fires on the first successful run after an anomaly. It is recorded before the reply is checked, so it means the run finished without an error, not that the reply was good. Don't auto-close incidents on it alone.

Expect a fresh start after a redeploy

The monitor's history lives in process memory and isn't in the journal checkpoint. Restarting a process keeps it, but a redeployed process or a new OS process starts with empty windows and no anomalies. Forward health.anomaly events to your log pipeline to keep the history. For what the journal does restore, see durable AI agent crash recovery.

Pair health checks with budgets and traces

Health monitoring tells you an agent misbehaved. It doesn't cap what that cost. Set BudgetConfig limits per run and per day as described in AI agent cost control, and record tool calls with the observability collector so you can see what the agent did before the anomaly.

Resume deliberately

A paused process queues its trigger events, up to 1,000, and works through the backlog when you resume. Fix the cause first. Suspend and resume semantics are covered in AI agent lifecycle.

• Shape, not correctness. The detectors see reply length, failures and tool-call patterns. A long, confident, wrong answer is healthy to them. Use evals or a judge for quality.

• Actions run after the run. pause and escalate act when the run ends. To stop a runaway run while it happens, use an in-run guard or a budget.

• One process at a time. Each process has its own monitor. Fleet-wide views, such as "error rate across all billing agents", belong in your metrics system.

• No built-in persistence. Windows and anomaly history reset on a redeploy or a new OS process.

• The API list is thin. It returns type, description and timestamp. The details evidence is on events and on the in-process Anomaly objects.

What is AI agent health monitoring?

AI agent health monitoring checks an agent's behavior, not its infrastructure. It watches signals such as empty final replies, the share of failed runs, repeated identical tool calls and repeating call sequences, and raises an alert when one crosses a threshold. In Promptise Foundry, HealthConfig on a process turns on four detectors that need no extra LLM calls.

How do I detect when an AI agent returns empty responses?

Record the agent's final reply after each run and count consecutive replies below a length limit. Promptise does this with empty_threshold (default 3 in a row) and empty_max_chars (default 10 characters or fewer). Tool results are ignored. If short replies are normal for your agent, set empty_max_chars=0 or ask the agent for a longer status line.

What error rate should trigger an alert for an AI agent?

Start with the default: 50% of the last 10 runs, counting only runs that failed after their retries. Lower error_window for agents that run rarely, so the window covers hours rather than days. Raise the threshold for agents behind flaky APIs that you already retry. Pair it with max_consecutive_failures for runs that fail every time.

What is the difference between pause and escalate in on_anomaly?

pause suspends the process: triggers keep firing, but events queue until someone resumes it. escalate sends a notification to the configured EscalationTarget and then suspends the process. Without an escalation target, escalate only logs. Use log while you tune thresholds, and switch once alerts reflect real problems.

Does health monitoring make extra LLM calls?

No. All four detectors are pattern matching on data the runtime already has: tool names, argument hashes, reply lengths and run outcomes. Health monitoring adds no token cost, so you can enable it on every process. Judging answer quality needs a model, which is what missions with an LLM judge are for.

How do I read health anomalies over HTTP?

Call GET /api/v1/processes/{name}/health/anomalies on the Promptise orchestration API, or OrchestrationClient.get_anomalies(name) from Python. Each entry has a type, description and timestamp. GET /api/v1/processes/{name} shows the count and the process state, and DELETE on the anomalies route clears the list after you have resolved the issue.

• Stop loops while they happen: AI agent stuck in a loop.

• Send alerts to Slack or a webhook: AI agent notifications.

• Let an LLM judge decide when the work is done: goal-driven AI agents with missions.

• Manage paused agents over HTTP: the AI agent control API.

• See every governance feature together: the autonomous AI agent runtime.

• Reference: behavioral health and the orchestration API in the Promptise docs.