Who Decides When an AI Agent Is Done? Checks and Judges

The model decides when to *stop*; your system should decide whether the task is *done*. Put every success criterion you can express in code into a check that reads the real state (records, files, sent messages). Use a separate judge model, with a fresh context and the evidence, only for criteria code can't check, such as tone or policy. Feed what is still missing back to the agent and let it try again, inside a budget of attempts, time and cost. When the budget runs out, a person decides.

Key takeaways:

• "The model stopped" is not "the task is done". A run ends when the model answers; it can end with work left over, honestly reported or not.

• A self-check in the prompt makes the report more honest, but the same model is grading its own work, from its own transcript.

• Code is the best judge where it can judge. A check against the real state is cheap, certain and the same every time.

• A separate judge covers the rest, if it gets the right evidence. Without the facts the code verified, it marks finished work as unfinished.

• Budgets decide when to give up. Attempts, iterations, time and cost limits turn an endless retry into an escalation.

The layers stack. The model's stop is the signal to check; the code check runs first because it is free and certain; the judge runs only when the code has nothing left to object to; the budget sits around the whole loop.

The cheapest layer is a checklist in the agent's instructions: before you answer, check each item and report anything that failed. It works on the *report*. In the run below, the agent said plainly that one email failed instead of claiming the task was complete.

What it can't do is make the model a reliable judge of its own work. It checks from its own transcript, so a tool that returned "sent" for a message that never arrived looks fine to it. It also shares the model's blind spots: if the model misread the task, its checklist is misread the same way. Techniques that verify the *content* of an answer, such as Chain-of-Verification and SelfCheckGPT, are covered in Chain-of-Verification and SelfCheckGPT: Catch Hallucinations; self-critique and retry loops in LLM Self-Correction: Self-Refine, Reflexion and CRITIC. This guide is about the *task*: did the work actually happen?

Use a code check when the criterion is about state you can read: a record exists, a file compiles, tests pass, a payment has a status, every item in a list was processed. Use a judge when it is about meaning: an email is polite, a summary is faithful, a reply follows policy. Most real tasks have both kinds of criteria, so run the code first and ask the judge only about what is left.

Use a judge on its own only when nothing can be checked in code, and then treat its verdict as a probability, not a fact. LLM as a Judge vs a Jury: Making AI Evaluations Reliable covers how to make judges more reliable.

• Python 3.10 or newer and uv pip install "promptise==1.7.0" python-dotenv.

• An OpenAI key in .env (OPENAI_API_KEY); the agent and the judge use gpt-5-mini.

• The example task in crm.py: onboard four customers (a CRM record and a welcome email each), with a mail outage for one customer's domain. missing() is the success criteria written as code.

"""A tiny CRM and mail outbox. Mail to one domain fails until the outage ends."""
from promptise import ToolError, tool

NEW_CUSTOMERS = [
    ("Ana Ruiz", "ana@example.com", "Pro"),
    ("Ben Okafor", "ben@example.com", "Starter"),
    ("Dana Lee", "dana@lee-family.net", "Pro"),
    ("Eli Novak", "eli@example.com", "Team"),
]
CRM: dict[str, dict] = {}
OUTBOX: list[dict] = []
OUTAGE = {"lee-family.net"}  # the mail provider rejects this domain for now


@tool
def create_customer(name: str, email: str, plan: str) -> str:
    """Create a customer record in the CRM."""
    CRM[email] = {"name": name, "plan": plan}
    return f"created {email}"


@tool
def send_email(to: str, subject: str, body: str) -> str:
    """Send an email."""
    if to.split("@")[1] in OUTAGE:
        raise ToolError(f"Mail service unavailable for {to.split('@')[1]}. Try again later.")
    OUTBOX.append({"to": to, "subject": subject, "body": body})
    return f"sent to {to}"


def task() -> str:
    people = "\n".join(f"- {n}, {e}, plan {p}" for n, e, p in NEW_CUSTOMERS)
    return f"Onboard these new customers: create each in the CRM and send each a welcome email.\n{people}"


def missing() -> list[str]:
    """The success criteria, in code: what is not done yet."""
    sent = {m["to"] for m in OUTBOX}
    gaps = [f"no CRM record for {e}" for _, e, _ in NEW_CUSTOMERS if e not in CRM]
    gaps += [f"no welcome email to {e}" for _, e, _ in NEW_CUSTOMERS if e not in sent]
    return gaps

The agent gets a checklist in its instructions and the task. When its run ends, the script prints what it says and what the code check finds:

"""Step 1: the agent decides it is done, with a self-check in its instructions."""
import asyncio

from dotenv import load_dotenv
from promptise import build_agent

import crm

load_dotenv()

INSTRUCTIONS = """You onboard customers.
Before you give your final answer, check your work:
1. Every customer in the task has a CRM record.
2. Every customer got a welcome email that names their plan.
3. If a step failed, say which one and why. Do not report it as done."""


async def main():
    agent = await build_agent(model="openai:gpt-5-mini", instructions=INSTRUCTIONS,
                              extra_tools=[crm.create_customer, crm.send_email])
    try:
        result = await agent.ainvoke({"messages": [{"role": "user", "content": crm.task()}]})
    finally:
        await agent.shutdown()
    calls = [c["name"] for m in result["messages"] for c in getattr(m, "tool_calls", None) or []]
    print(f"tool calls: {len(calls)}")
    print("agent says:", result["messages"][-1].content)
    print("code check:", crm.missing() or "all done")


if __name__ == "__main__":
    asyncio.run(main())

tool calls: 8
agent says: Summary
- CRM records: created for all 4 customers (Ana, Ben, Dana, Eli).
- Welcome emails: sent for 3 customers (Ana, Ben, Eli). Failed for 1 (Dana).
- Failure details: Dana's welcome email failed to send because the mail service for lee-family.net was unavailable (error: "Mail service unavailable for lee-family.net. Try again later." [...]
[per-customer status and next-step options left out]
code check: ['no welcome email to dana@lee-family.net']

(The answer's per-customer details and its offer of next steps are left out above.)

The self-check did its job: the report is honest about Dana's email. The run still *ended* with the task unfinished, because the model stopped and asked what to do. Nothing in the run decides whether to retry, wait or give up. If your code treats "the run returned" as "the task is done", this task is marked complete with a customer who never got their email. The ToolError here used the default retryable=False; marking a transient failure ToolError(..., retryable=True) tells the model it is worth trying again, but the decision still belongs outside the run.

The verification loop runs the agent, checks the real state with crm.missing(), and, if something is missing, runs the agent again with exactly what is missing. MAX_ATTEMPTS is the budget:

"""Step 2: code decides when the task is done; the agent gets the gaps and tries again."""
import asyncio

from dotenv import load_dotenv
from promptise import build_agent

import crm
from self_check import INSTRUCTIONS

load_dotenv()
MAX_ATTEMPTS = 3  # the backstop: stop and escalate instead of retrying forever


async def main():
    agent = await build_agent(model="openai:gpt-5-mini", instructions=INSTRUCTIONS,
                              extra_tools=[crm.create_customer, crm.send_email],
                              max_agent_iterations=15)
    request = crm.task()
    try:
        for attempt in range(1, MAX_ATTEMPTS + 1):
            result = await agent.ainvoke({"messages": [{"role": "user", "content": request}]})
            gaps = crm.missing()
            print(f"attempt {attempt}: agent finished, code check: {gaps or 'all done'}")
            if not gaps:
                break
            crm.OUTAGE.clear()  # in real life: time passes, the mail service recovers
            request = (crm.task() + "\n\nSome of this is already done. Still missing:\n"
                       + "\n".join(f"- {g}" for g in gaps) + "\nDo only what is missing.")
        else:
            print(f"not done after {MAX_ATTEMPTS} attempts: escalate to a person with {gaps}")
    finally:
        await agent.shutdown()
    print("emails sent:", [m["to"] for m in crm.OUTBOX])
    print("last answer:", result["messages"][-1].content[:300])


asyncio.run(main())

attempt 1: agent finished, code check: ['no welcome email to dana@lee-family.net']
attempt 2: agent finished, code check: all done
emails sent: ['ana@example.com', 'ben@example.com', 'eli@example.com', 'dana@lee-family.net']

(The script also prints the start of the agent's last answer, left out above.)

On the first attempt, the code check found the missing email. The outage then ended (the script clears it, standing in for time passing), and the second attempt got a request that listed only the gap. The agent sent the one missing email and the check passed.

Three details make this loop work:

• The check reads the state, not the transcript. missing() looks at the CRM and the outbox, so it can't be fooled by a summary that says "done".

• The feedback is specific. "Still missing: no welcome email to dana@lee-family.net. Do only what is missing." keeps the agent from redoing finished work. Pair it with idempotent tools, so a repeated call can't create a duplicate; see Idempotent Tool Calls: Make AI Agent Retries Safe.

• The loop has an exit. After MAX_ATTEMPTS, it stops and escalates instead of retrying forever.

The task has a criterion code can't check: every welcome email names the customer's plan and promises nothing the plan doesn't include. That needs a reader. Promptise Foundry's mission judge is a separate, tool-less agent that gets the objective, the success criteria and an evidence bundle, and answers with achieved, confidence and reasoning. MissionTracker runs your success_check first and only calls the judge when the check returns None:

"""Step 3: code checks what code can check; a separate judge checks the rest."""
import asyncio
import copy

from dotenv import load_dotenv
from promptise import build_agent
from promptise.runtime import MissionConfig
from promptise.runtime.mission import MissionEvidence, MissionTracker

import crm
from self_check import INSTRUCTIONS

load_dotenv()


def records_check(evidence: MissionEvidence) -> bool | None:
    """Hard criteria in code. None means: the code can't decide, ask the judge."""
    return False if crm.missing() else None


def tracker() -> MissionTracker:
    return MissionTracker(
        MissionConfig(
            enabled=True,
            objective="Onboard the new customers: CRM record and welcome email for each",
            success_criteria=("Every welcome email names the customer's plan and promises nothing "
                              "the plan doesn't include: no discounts, no free months."),
            eval_model="openai:gpt-5-mini",
        ),
        process_id="onboarding",
        success_check=records_check,
    )


def evidence(outbox, verified=None) -> MissionEvidence:
    """What the judge reads: facts the code verified, and one message per email
    (the judge sees 1,000 characters of state and 500 of each message)."""
    return MissionEvidence(state=verified or {}, conversation=[
        {"role": "tool", "content": f"to: {m['to']}\nsubject: {m['subject']}\n{m['body']}"} for m in outbox
    ])


async def main():
    print("before the work:", (await tracker().evaluate(evidence([]), "openai:gpt-5-mini")).reasoning)

    crm.OUTAGE.clear()
    agent = await build_agent(model="openai:gpt-5-mini", instructions=INSTRUCTIONS,
                              extra_tools=[crm.create_customer, crm.send_email])
    try:
        await agent.ainvoke({"messages": [{"role": "user", "content": crm.task()}]})
    finally:
        await agent.shutdown()

    verdict = await tracker().evaluate(evidence(crm.OUTBOX), "openai:gpt-5-mini")
    print(f"emails only:    achieved={verdict.achieved} confidence={verdict.confidence} "
          f"source={verdict.source}\n  {verdict.reasoning}")

    facts = {"verified_by_code": f"CRM records and sent welcome emails exist for all "
                                 f"{len(crm.NEW_CUSTOMERS)} customers"}
    verdict = await tracker().evaluate(evidence(crm.OUTBOX, facts), "openai:gpt-5-mini")
    print(f"with facts:     achieved={verdict.achieved} confidence={verdict.confidence} "
          f"source={verdict.source}\n  {verdict.reasoning}")

    bad = copy.deepcopy(crm.OUTBOX)  # the same outbox with one email that breaks the rule
    bad[1]["body"] += "\nAs a thank-you, your first 3 months are free."
    verdict = await tracker().evaluate(evidence(bad, facts), "openai:gpt-5-mini")
    print(f"one bad email:  achieved={verdict.achieved} confidence={verdict.confidence} "
          f"source={verdict.source}\n  {verdict.reasoning}")


asyncio.run(main())

before the work: Programmatic success check returned False
emails only:    achieved=False confidence=0.9 source=llm
  All four customers were sent welcome emails that name their plan and do not promise discounts or free months, so the email portion meets the success criteria. However, there is no evidence in the conversation that CRM records were created for any of the new customers. The objective requires both a CRM record and a welcome email for each customer, so the mission is not complete.
with facts:     achieved=True confidence=0.95 source=llm
  Verified by code that CRM records and sent welcome emails exist for all 4 customers. Each email explicitly names the customer's plan (Pro, Pro, Starter, Team) in subject/body and only describes plan features; none mention discounts, free months, or other promises outside standard plan features. Therefore the success criteria are met.
one bad email:  achieved=False confidence=0.95 source=llm
  CRM records and welcome emails were created and sent for all four customers (verified_by_code). However one welcome email (to dana@lee-family.net) includes the line "As a thank-you, your first 3 months are free.", which promises free months and therefore violates the success criteria that emails must not promise discounts or free months. The other three emails correctly name the customer's plan and do not make unauthorized promises.

Four evaluations, four lessons:

• Before the work, the code check returned False and no judge was called. Code answers first, for free.

• With only the emails as evidence, the judge said the emails were fine but the mission wasn't achieved, because nothing showed the CRM records existed. A judge evaluates the whole objective with the evidence it has; it can't see what your code already verified.

• With the code's facts in the state, the judge confirmed the emails and the mission completed.

• With one email that promises free months, the judge caught it and named the email.

So give the judge two things: the facts your code verified (in state, which the judge sees up to 1,000 characters) and the material it must assess (in conversation, the last 10 messages, 500 characters each). Keep both short. The judge also sees tool calls only as names and argument hashes, not their results, so anything it needs from a result has to be in the evidence you pass. The judge's verdict is still a model's opinion: confidence helps you decide when to ask a person, and the judge's own reliability is something to test like any other eval.

Every layer above needs a limit around it:

• Per run: build_agent(max_agent_iterations=15) caps model calls in one run; hitting it mid-loop raises GraphIterationLimitError.

• Per task: MAX_ATTEMPTS in the verification loop caps retries.

• For a long-running agent: a mission with max_invocations and timeout_hours, and an autonomy budget on tool calls and turns per run.

For an agent that runs on triggers until its goal is met, the same decision lives in the process configuration. This is the onboarding task as a mission: the code check first, then the judge, a confidence threshold below which the mission pauses and escalates, automatic stop when achieved, and two limits that fail it:

"""The same decision for a long-running process: a mission with a code check, a judge and limits."""
from promptise.runtime import (BudgetConfig, EscalationTarget, MissionConfig, ProcessConfig)
from promptise.runtime.mission import MissionEvidence

import crm


def records_check(evidence: MissionEvidence) -> bool | None:
    return False if crm.missing() else None  # code first; None asks the judge


config = ProcessConfig(
    model="openai:gpt-5-mini",
    instructions="You onboard new customers from the signup queue.",
    mission=MissionConfig(
        enabled=True,
        objective="Onboard every customer in the signup queue",
        success_criteria="Every welcome email names the customer's plan and promises nothing extra.",
        success_check=records_check,     # hard criteria in code, before any judge call
        eval_model="openai:gpt-5-mini",  # the judge: a separate, tool-less agent
        eval_every=1,                    # judge after every successful run
        confidence_threshold=0.7,        # below this (not achieved): pause and escalate
        auto_complete=True,              # achieved: stop the process
        max_invocations=10,              # backstop: fail after 10 runs
        timeout_hours=4,                 # backstop: fail after 4 hours
        escalation=EscalationTarget(webhook_url="https://hooks.example.com/agent-alerts"),
    ),
    budget=BudgetConfig(enabled=True, max_tool_calls_per_run=30, max_llm_turns_per_run=15),
)

if __name__ == "__main__":
    m = config.mission
    print(f"mission: judge={m.eval_model}, threshold={m.confidence_threshold}, "
          f"limits={m.max_invocations} runs / {m.timeout_hours} h, auto_complete={m.auto_complete}")

mission: judge=openai:gpt-5-mini, threshold=0.7, limits=10 runs / 4.0 h, auto_complete=True

When a mission is achieved, auto_complete=True stops the process. A verdict of "not achieved" with confidence below confidence_threshold pauses the mission and notifies the escalation target. max_invocations and timeout_hours fail it with a mission.failed event. If the judge call itself fails, the evaluation counts as not achieved with confidence 0.5, which pauses the mission at the default threshold. Running a mission step by step, with the API to pause, resume or fail it, is covered in Goal-Driven AI Agents: Missions With an LLM Judge.

• Store the verdicts. Log every check and judge result with its reasoning. When a customer says "your agent said it was done", that log is the answer. Foundry journals each mission evaluation.

• Make "not done" visible. A run that ends with gaps should produce a status, not just a polite message. Route it to a queue, a retry or a person, as in AI Agent Escalation: Page a Human When Limits Hit.

• Test the judge. Keep a few known-good and known-bad examples (like the email with free months) and check the judge's verdicts in CI. How to Evaluate AI Agents: Evals, LLM Judges and pass^k shows how.

• Price the judge. Every evaluation is a model call. Run the code check first, raise eval_every for frequent triggers, and use a small model where it is accurate enough.

• Cap the spend. Retries multiply cost; AI Agent Cost Control: Budgets That Stop Runaway Spend puts a ceiling on it.

• Code checks need checkable criteria. Many tasks have a fuzzy part, and the judge for it is a model that can be wrong.

• Foundry's programmatic check is a yes, no or "ask the judge". It returns a boolean (or None), so the specific gaps for the retry prompt come from your own code, as in verify_loop.py.

• The judge sees a bounded slice of evidence (state up to 1,000 characters, 10 messages of 500 characters). Long evidence must be summarized by your code first.

• One small task. The outage, the four customers and the bad email are constructed to show each layer; real tasks have more criteria and messier state.

Who decides when an AI agent is done?

The model decides when its run stops, by answering instead of calling another tool. Whether the task is done should be decided outside the model: by code that checks success criteria against the real state, by a separate judge for criteria code can't check, and by budgets that end retries and hand the task to a person.

Should an agent verify its own work?

Yes, as a first layer: a self-check in the prompt makes the final report more honest about failures. It isn't enough on its own, because the same model grades its work from its own transcript. Add a check that reads the real state, and a separate judge where needed.

What is a self-verification loop for AI agents?

A loop around the agent: run it, check the result against success criteria, and if something is missing, run it again with exactly what is missing, until the check passes or a budget runs out. The check should read the real state, such as records or files, not the agent's answer.

When should I use an LLM judge instead of code to check completion?

When the criterion is about meaning rather than state: tone, faithfulness, policy compliance, quality. Check everything else in code first, and give the judge the facts the code verified, so it only judges what code can't.

How does Promptise Foundry stop an agent automatically when its goal is met?

Give the process a mission with an objective and success criteria. After each run, your success_check runs first, then a separate judge model if needed. When the mission is achieved, auto_complete=True stops the process. A low-confidence "not achieved" pauses it and escalates; max_invocations and timeout_hours fail it.

What threshold should a judge's confidence have?

Foundry's default confidence_threshold is 0.7: below it, a "not achieved" verdict pauses the mission and escalates. Tune it on examples where you know the right answer. A higher threshold sends more cases to people; a lower one lets the agent keep going on weaker evidence.

• Run a mission end to end and control it over the API: Goal-Driven AI Agents: Missions With an LLM Judge.

• Verify the content of answers, not just the task: Chain-of-Verification and SelfCheckGPT: Catch Hallucinations.

• Make judges more reliable: LLM as a Judge vs a Jury: Making AI Evaluations Reliable.

• Understand the loop whose stop you are checking: ReAct Agent Pattern: Tool Execution Loop Explained Simply.

• Hand over cleanly when the budget runs out: AI Agent Escalation: Page a Human When Limits Hit.