What Is an Agent Harness? Build One in Python

An agent harness is everything around the model that turns it into an agent. That includes the loop that calls it, the tools it can use, what goes into its context on each turn, the limits that stop it, and any check on its answer before the answer leaves. The term caught on in 2026 because teams kept finding that two products on the same model behaved very differently, and the difference was the harness. This guide builds a harness in Python one part at a time on Promptise Foundry's graph engine. It then measures what each part adds on a 30-SKU inventory audit: 50 runs on gpt-5-mini and 15 on gpt-4.1-mini. The results are not the tidy "every layer helps" story. On the stronger model, two parts did all the work. On the weaker model, one default part made things much worse, and a third part, a budget, kept that from becoming an expensive incident.

A model on its own takes text and returns text. To act, something has to run it in a loop, execute the tool calls it asks for, feed the results back, and decide when to stop. That something is the harness. Its parts:

• The loop: call the model, run its tool calls, repeat until it answers, with a cap on rounds.

• Tools: the functions it may call, and how their results come back.

• Context management: what the model sees each turn. That can be the whole transcript, or a compacted view once the transcript grows.

• Controls: budgets, loop detection and approval gates that stop or redirect a run.

• Verification: a step that checks the answer before it is returned.

Harness vs framework: a framework is a library you build a harness with. LangGraph, Pydantic AI and Promptise are frameworks, and the harness is what you assemble from them, including the defaults you never changed. Context engineering vs harness engineering: context engineering is one part of harness engineering, the part that decides what the model sees. Context engineering for AI agents covers it in depth.

The agent audits 30 SKUs. It has to page through the SKU list (3 pages), read each SKU's stock record (about 2,400 characters each, mostly warehouse log), and check purchase orders for the SKUs that are below their reorder point. Two SKUs have a purchasing system that never answers. It always says "syncing, try again", which is a trap for an agent that polls forever. The correct answer lists 5 SKUs to reorder and 2 as unknown. A run succeeds only if both lists match exactly. That takes at least 43 tool calls.

The tools are plain LangChain tools, so the harness, not an MCP server, is what is being tested:

"""A fake inventory API with 30 SKUs, exposed as LangChain tools."""
import random

from langchain_core.tools import tool

_rng = random.Random(42)
LOG_LINE = ("{d} cycle count in aisle {a}: {n} units scanned, label check ok, "
            "pallet {p} rotated, humidity {h}% within range. ")
SKUS = {}
for i in range(1, 31):
    sku = f"SKU-{i:02d}"
    reorder_point = _rng.choice([20, 30, 40, 50])
    SKUS[sku] = {"sku": sku, "name": f"Part {i:02d}", "reorder_point": reorder_point,
                 "on_hand": reorder_point + _rng.randint(1, 60)}
# Ten SKUs are below their reorder point.
LOW = ["SKU-03", "SKU-07", "SKU-08", "SKU-12", "SKU-15", "SKU-19", "SKU-22", "SKU-26", "SKU-28", "SKU-30"]
for sku in LOW:
    SKUS[sku]["on_hand"] = SKUS[sku]["reorder_point"] - _rng.randint(1, 15)
OPEN_PO = {"SKU-07": "PO-7781", "SKU-19": "PO-7790", "SKU-26": "PO-7802"}
SYNCING = {"SKU-12", "SKU-28"}  # the purchasing system never answers for these
# The right answer: below reorder point, no open purchase order, PO status known.
NEED_REORDER = sorted(set(LOW) - set(OPEN_PO) - SYNCING)
UNKNOWN = sorted(SYNCING)
CALLS = []  # (tool, args) of every call, for the experiment


def _log(sku):
    r = random.Random(sku)
    return "".join(LOG_LINE.format(d=f"2026-09-{day:02d}", a=r.randint(1, 40), n=r.randint(10, 99),
                                   p=r.randint(100, 999), h=r.randint(30, 60)) for day in range(1, 21))


@tool
def list_skus(page: int = 1) -> dict:
    """List SKUs in the warehouse, 10 per page. Call again with next_page until it is null."""
    CALLS.append(("list_skus", page))
    ids = sorted(SKUS)[(page - 1) * 10:page * 10]
    return {"skus": ids, "next_page": page + 1 if page < 3 else None}


@tool
def get_stock(sku: str) -> dict:
    """Stock record for one SKU: on_hand, reorder_point and the warehouse log."""
    CALLS.append(("get_stock", sku))
    if sku not in SKUS:
        return {"error": f"unknown SKU {sku}"}
    return {**SKUS[sku], "warehouse_log": _log(sku)}


@tool
def get_open_po(sku: str) -> dict:
    """Open purchase order for a SKU, if any."""
    CALLS.append(("get_open_po", sku))
    if sku in SYNCING:
        return {"status": "syncing", "message": "Purchasing system is syncing. Try again."}
    return {"sku": sku, "open_po": OPEN_PO.get(sku)}


TOOLS = [list_skus, get_stock, get_open_po]

build(level) returns a harness with the first level parts switched on. It uses Promptise's graph engine directly (PromptGraphEngine, PromptNode), which is what build_agent() builds for you, so each part is a visible line:

"""An agent harness built up one part at a time on Promptise's graph engine."""
from langchain_openai import ChatOpenAI
from promptise.engine import BudgetHook, ContextCompaction, PromptGraph, PromptGraphEngine, PromptNode
from promptise.runtime import HealthConfig, HealthMonitor

from inventory import TOOLS

TASK = ("Find every SKU that is below its reorder point (on_hand < reorder_point) and has no "
        "open purchase order. Check all SKUs. End with two lines:\n"
        "REORDER: <comma-separated SKUs>\nUNKNOWN: <SKUs whose purchase-order status you could not get>")
SYSTEM = "You are an inventory agent running as a nightly batch job. Work autonomously; never ask questions."
CHECK = ("Check the draft answer above before it is sent. For every SKU in REORDER and UNKNOWN, confirm "
         "it against the tool results (call a tool again if a fact is missing), confirm no SKU below its "
         "reorder point was skipped, fix any mistake, and repeat the final REORDER and UNKNOWN lines.")


class LoopGuard:
    """Blocks a tool call once HealthMonitor sees the agent repeating itself."""

    def __init__(self):
        self.monitor = HealthMonitor(HealthConfig(enabled=True, stuck_threshold=3, cooldown=0),
                                     process_id="inventory-agent")
        self.blocked = 0

    async def pre_tool(self, tool_name, args, state):
        anomaly = await self.monitor.record_tool_call(tool_name, args)
        if anomaly:
            self.blocked += 1
            raise RuntimeError(f"{anomaly.description}. The result will not change in this run: "
                               "stop calling it and report the SKU as UNKNOWN.")
        return args


def build(level: int, model: str = "gpt-5-mini", keep_result_chars: int = 2000,
          max_rounds: int = 60) -> PromptGraphEngine:
    """level 0: model alone, 1: + tools and loop, 2: + compaction, 3: + loop guard, 4: + verification."""
    tools = TOOLS if level >= 1 else []
    work = PromptNode("work", instructions=SYSTEM, tools=tools, context_scope="auto",
                      max_iterations=max_rounds, default_next="check" if level >= 4 else "__end__")
    graph = PromptGraph(name=f"harness-{level}")
    graph.add_node(work)
    if level >= 4:
        graph.add_node(PromptNode("check", instructions=CHECK, tools=tools, context_scope="auto",
                                  max_iterations=6, default_next="__end__"))
    graph.set_entry("work")
    return PromptGraphEngine(
        graph=graph,
        model=ChatOpenAI(model=model),
        compaction=ContextCompaction(enabled=level >= 2, keep_result_chars=keep_result_chars),
        # every level: end a runaway run at 300k tokens instead of draining the account
        hooks=[BudgetHook(max_tokens=300_000)] + ([LoopGuard()] if level >= 3 else []),
        max_node_iterations=60,  # a 30-SKU audit takes 40+ tool rounds when the model calls one tool at a time
        max_iterations=80,
    )


async def run(level: int) -> str:
    engine = build(level)
    result = await engine.ainvoke({"messages": [{"role": "user", "content": TASK}]})
    return result["messages"][-1].content


if __name__ == "__main__":
    import asyncio
    import sys

    print(asyncio.run(run(int(sys.argv[1]))))

$ python harness.py 4
REORDER: SKU-03,SKU-08,SKU-15,SKU-22,SKU-30
UNKNOWN: SKU-12,SKU-28

How each part works:

• Level 1, tools and loop: a PromptNode with tools re-enters itself while the model keeps calling tools, for up to max_rounds rounds. A round is one model call plus all the tool calls it made in parallel.

• Level 2, context compaction: with ContextCompaction(enabled=True) and context_scope="auto", the node switches to a compacted view after 6 tool results. The model then sees the task, its latest exchange, and a deduplicated ledger of older results, each cut to keep_result_chars (2,000 by default). Nothing is summarised by a model. See the context lifecycle docs.

• Level 3, loop guard: a pre_tool hook feeds every call to Promptise's HealthMonitor. It flags the same call three times in a row, or a repeating sequence. When it fires, the hook raises, the tool does not run, and the model gets the message as an error result. Set cooldown=0. The default of 300 seconds suppresses a repeated anomaly of the same type, which suits alerts but not a guard inside one run.

• Level 4, verification node: after the work node answers, a second PromptNode checks the draft against the tool results and can call tools again before it repeats the final lines.

• Every level, a budget: BudgetHook(max_tokens=300_000) ends a run once it has used 300,000 tokens. It is listed under controls but runs at every level, because without it one bad run in the experiment below would have cost more than all the others together.

One complete run of 50 tasks, same task, same tools, max_rounds=60:

What the numbers say:

• The model alone never guessed. In all 10 runs it said it could not reach the inventory and named no SKUs.

• Tools plus a loop did all the work. Every run with tools got both lists exactly right.

• Compaction saved nothing at this size. The full transcript stays near 20,000 tokens, so the compacted view is about as large as the transcript it replaces.

• The loop guard never fired. gpt-5-mini polled each syncing SKU once in 39 of 40 runs and twice in the other, then reported it as unknown. The lower token mean at level 3 is not the guard. Runs fell into two modes: all 63 calls in one parallel round (5 model calls, about 29,000 tokens) or one more round (6 calls, about 51,000 tokens). Eight of ten level-3 runs happened to use the cheaper mode, compared with four of ten at level 1.

• Verification cost one extra call, about 15,000 tokens and 10 s per run, and caught nothing, because there was nothing left to catch.

On a model this capable, the extra parts are insurance you pay for on every run. That can still be the right call, but it is a cost, not a free gain.

The same harness on a weaker, non-reasoning model shows the other side. Five runs per configuration, chosen to keep the cost of runaway runs bounded. Level 2 and 4 runs here used keep_result_chars=4000, so no stock record was truncated:

Without compaction, gpt-4.1-mini was fast and wrong. It answered in about 9 seconds. Three runs skipped 10 to 20 of the 30 stock records, and the two that read all 30 still missed SKU-22 and never checked the two syncing SKUs. Every run got at least one list wrong. With compaction on, it re-read the same records again and again: one run fetched get_stock("SKU-03") five times, and runs made up to 156 tool calls for 63 distinct ones. Four of five runs ran until the budget stopped them. The loop guard fired 5 times in one run, on repeated purchase-order polls, but could not see the re-reads: they come back as batches of 20 to 30 different calls, longer than the patterns HealthMonitor looks for in its 20-call window. The verification node never ran, because the work node never finished.

This is the opposite of what the compacted view is for, and we have logged it as a Promptise issue. Two likely contributors: the ledger tells the model it may call a tool again for "the current value of something that changes over time", and stock levels qualify. And with the default 2,000-character limit, every truncated record carries an invitation to fetch it again in full. In an earlier single trial with that default and no budget, one run made 485 tool calls and used 1.27 million input tokens without answering.

The lesson for any harness, not just this one: a part that helps one model can hurt another, and the budget is the part that turns an unknown failure into a bounded one. Test your harness on the model you will actually run, and give every run a hard budget.

Promptise's default ReAct node allows 15 rounds. In this experiment that was enough: a separate run of level 1 with max_rounds=15 succeeded 10/10, because gpt-5-mini batched its calls into 5 or 6 rounds. In one trial before the experiment, though, the same model called one tool per round and hit the 15-round limit after reading 6 of the 30 records. The engine stopped it with GraphIterationLimitError instead of returning a wrong answer, which is the right failure. But a task that needs 40 sequential steps needs a budget sized for 40.

build_agent() assembles this harness with defaults, so you start at level 2 without writing the graph:

• A ReAct node with tools (from MCP servers or extra_tools=) and a 15-round loop.

• Context compaction on, with context_scope="auto" after 6 tool results. Pass context_compaction=False or a ContextCompaction(...) to change it.

• max_agent_iterations=25 and an optional max_invocation_time.

• Opt-in controls: approval= for human approval of chosen tools, guardrails=, and FallbackChain for model fallback.

• agent_pattern= for other graphs: "verify" (one-pass self-check), "managed" (always compacted), or your own PromptGraph.

Engine hooks such as the loop guard and BudgetHook are not a build_agent() parameter in 1.3.3. Use PromptGraphEngine directly, as above, when you need them. How to build an AI agent in Python shows the build_agent() path end to end, and plan-and-execute agents show a multi-node graph.

• One task, two models, one run each. The results say which parts mattered here, not in general. A task with larger results, more steps or a real polling trap would weigh the parts differently.

• N = 10 and N = 5. 10/10 against 10/10 shows no difference, not a proven absence of one. The gpt-4.1-mini runs use N = 5 because each runaway run cost about $0.10.

• The loop trap did not trigger on `gpt-5-mini`, so this run does not test the loop guard's value on the stronger model. It shows only that the guard did no harm.

• The verification node here is a prompt. It checks the draft with the same model. A deterministic check, such as comparing the answer to the tool results in code, would be stronger where the answer can be checked that way.

• `max_agent_iterations` in `build_agent()` is off by one in 1.3.3 (a limit of N allows N+1 turns). This guide uses the engine's own limits and is not affected.

• The compaction re-read loop is model-dependent. It appeared on gpt-4.1-mini, not on gpt-5-mini. We have not tested other models.

About $2.10 at OpenAI list prices: $0.56 for 65 gpt-5-mini runs (the ablation, the 15-round check, smoke runs and the snippet), and $1.54 for 18 gpt-4.1-mini runs, most of it in the runaway re-read runs.

What is an agent harness in simple terms?

It is the code around a language model that lets it act: the loop that calls it, the tools it uses, what it sees each turn, the limits that stop it, and any checks on its answer. The model decides. The harness runs it.

Is an agent harness the same as an agent framework?

No. A framework is a library. A harness is the specific assembly you run, including every default you kept. Two teams on the same framework and model can have very different harnesses.

What is the difference between context engineering and harness engineering?

Context engineering decides what goes into the model's context on each call. Harness engineering covers that plus the loop, tools, controls and verification. Context is one part of the harness.

Which part of a harness matters most?

In this experiment, tools and the loop took success from 0/10 to 10/10, and a token budget kept the weaker model's runaway runs at about 300,000 tokens each. Compaction, the loop guard and verification added cost without adding success on gpt-5-mini, and compaction hurt gpt-4.1-mini. Measure on your own task and model.

How do I stop an agent from looping forever?

Use three limits: a cap on rounds per node, a total token or cost budget, and a guard that blocks repeated identical calls. The first two always end a run. The third ends a specific kind of loop earlier, with a message the model can act on. Durable AI agents covers what to do when the process itself dies mid-run.

Should I add a verification step to my agent?

Add one when the answer can be wrong in ways a second look can catch, and when that check costs less than the error. Here it added about 15,000 tokens per run and caught nothing, because the model was already right. A check in code is cheaper and stronger where the answer can be checked that way.