AI Agent Observability: Trace Tool Calls, Tokens and Cost

With Promptise Foundry, pass observe=True to build_agent. From then on every model call (tokens, latency, model version, which tools it asked for) and every tool call (arguments, result, latency) is recorded on a timeline. agent.get_stats() returns the totals, and an HTML report is written when the agent shuts down:

from promptise import build_agent
from promptise.config import StdioServerSpec

agent = await build_agent(
    model="openai:gpt-5-mini",
    servers={
        "shop": StdioServerSpec(command=sys.executable, args=["shop_server.py"]),
    },
    instructions="You are a shop assistant. Use your tools to answer. Be brief.",
    observe=True,
)
result = await agent.ainvoke({"messages": [{"role": "user", "content": question}]})
print(json.dumps(agent.get_stats(), indent=2))
print(agent.generate_report("shop-report.html"))

No trace data leaves your machine unless you add a transporter that sends it. The trace lives in memory and in local files, and the code that records it is open source. The rest of this guide shows what the trace contains and how to use it.

When observability is on, build_agent attaches a PromptiseCallbackHandler to every run. It turns each model call and tool call into a timeline event and hands it to an ObservabilityCollector. The collector keeps the events in memory and passes each one to the transporters you chose.

These are the events you'll see from an agent with MCP tools, and what each one carries:

Every event of one request points to that request's agent.input through its parent_id. Events can also carry a user_id and session_id, taken from the CallerContext you pass to ainvoke. That turns out to be the easiest way to pull one request back out of a busy trace.

• Python 3.10 or newer.

• Promptise Foundry 1.3.1 or newer: pip install "promptise>=1.3.1". Everything up to the OpenTelemetry section needs nothing else.

• For OpenTelemetry export: pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-grpc. The docs suggest pip install "promptise[all]", which works too but also installs much more, including PyTorch.

• An API key for a model provider. This guide uses OpenAI, so set OPENAI_API_KEY.

You'll build a shop assistant that answers questions about products and stock, run three real questions through it, and use the trace to find out why one answer is wrong.

The tools live in a small MCP server. check_stock waits 0.8 seconds on purpose, standing in for a slow warehouse API, so you can see latency in the trace. search_products has a bug, the kind that slips into real code: it only matches the start of the product name, like a LIKE 'query%' database index.

import asyncio

from promptise.mcp.server import MCPServer, ToolError

server = MCPServer("shop")

# A stand-in for your product catalog and warehouse system.
PRODUCTS = {
    "KB-200": {"name": "KB-200 mechanical keyboard", "price_eur": 89.0},
    "MS-10": {"name": "MS-10 wireless mouse", "price_eur": 24.0},
    "MS-30": {"name": "MS-30 ergonomic mouse", "price_eur": 49.0},
}
STOCK = {"KB-200": 14, "MS-10": 0, "MS-30": 6}


@server.tool()
async def search_products(query: str) -> list[dict]:
    """Search the catalog by product name. Returns SKU, name and price in EUR.

    Args:
        query: Words to look for in the product name, for example "mouse".
    """
    q = query.lower()
    # Prefix match, like a LIKE 'query%' index in a database.
    return [{"sku": sku, **p} for sku, p in PRODUCTS.items() if p["name"].lower().startswith(q)]


@server.tool()
async def check_stock(sku: str) -> dict:
    """Check how many units of a product are in the warehouse.

    Args:
        sku: The product SKU, for example "KB-200".
    """
    await asyncio.sleep(0.8)  # the warehouse API is slow
    if sku.upper() not in STOCK:
        raise ToolError(f"Unknown SKU {sku}.")
    return {"sku": sku.upper(), "units": STOCK[sku.upper()]}


if __name__ == "__main__":
    server.run()

If MCP servers are new to you, How to Connect MCP Servers to Your AI Agent in Python walks through this part.

observe=True is enough to start. For a trace you can come back to, pass an ObservabilityConfig instead. This one keeps the HTML report and adds the JSON transporter, which streams every event to a file as it happens. The CallerContext tags each question's events with its own session_id.

import asyncio
import json
import sys

from promptise import CallerContext, ObservabilityConfig, TransporterType, build_agent
from promptise.config import StdioServerSpec

QUESTIONS = {
    "q1": "Is the MS-30 in stock?",
    "q2": "How much is the MS-10?",
    "q3": "Do you sell an ergonomic mouse?",
}


async def main():
    agent = await build_agent(
        model="openai:gpt-5-mini",
        servers={
            "shop": StdioServerSpec(command=sys.executable, args=["shop_server.py"]),
        },
        instructions="You are a shop assistant. Use your tools to answer. Be brief.",
        observe=ObservabilityConfig(
            session_name="shop-assistant",
            transporters=[TransporterType.HTML, TransporterType.JSON],
            output_dir="traces",
        ),
    )
    try:
        for qid, question in QUESTIONS.items():
            # Tag every event of this request, so you can pull it out of the trace later.
            caller = CallerContext(user_id="customer-42", metadata={"session_id": qid})
            result = await agent.ainvoke(
                {"messages": [{"role": "user", "content": question}]}, caller=caller
            )
            print(f"[{qid}] {question}")
            print(f"      {result['messages'][-1].content}")

        print(json.dumps(agent.get_stats(), indent=2))
    finally:
        await agent.shutdown()


asyncio.run(main())

python shop_agent.py

[q1] Is the MS-30 in stock?
      Yes — we have 6 units of the MS-30 in stock. Would you like to add one to your cart or need any other details?
[q2] How much is the MS-10?
      The MS-10 (MS-10 wireless mouse) costs €24.00. It’s currently out of stock (0 units available).
[q3] Do you sell an ergonomic mouse?
      Sorry — we don’t currently have any product listed as an ergonomic mouse in our catalogue. 

Would you like me to:
- search again for specific types (vertical, trackball, wireless), 
- recommend models/brands and price ranges, or 
- check whether we can order one for you?
{
  "entry_count": 42,
  "agent_count": 1,
  "total_duration_s": 12.726191759109497,
  "total_prompt_tokens": 2822,
  "total_completion_tokens": 849,
  "total_tokens": 3671,
  "llm_call_count": 9,
  "tool_call_count": 9,
  "error_count": 0,
  "retry_count": 0,
  "latency_p50_ms": 837.755874999857,
  "latency_p95_ms": 3389.6188735961914,
  "latency_p99_ms": 6622.287034988403,
  "tokens_by_agent": {
    "agent": 3671
  },
  "events_by_category": {
    "transparency": 42
  },
  "events_by_type": {
    "agent.input": 3,
    "llm.start": 9,
    "llm.end": 9,
    "tool.call": 9,
    "tool.result": 9,
    "agent.output": 3
  }
}

Three questions took 9 model calls and 9 tool calls, 3,671 tokens in all, and each question also has an agent.input and an agent.output event that mark where the request started and ended. Two details help you read these numbers. The latency percentiles mix model calls and tool calls, because they cover every event that has a duration. And total_duration_s counts from when the agent was built, not just the time spent answering.

The third answer is wrong. The catalog has an ergonomic mouse, the MS-30, and the agent said it didn't. The trace will tell you why.

When the agent shut down, the HTML transporter wrote traces/shop-assistant-report-20261011_121915.html. It's one self-contained file, with no external scripts or styles, that you can open in any browser or attach to a ticket. Rendered in a browser (here, headless Chrome), it shows six stat cards, filter buttons and one row per event:

title: Promptise Agent Report
rendered stat cards: [('42', 'Total Events'), ('3.671', 'Total Tokens'), ('9', 'LLM Calls'), ('9', 'Tool Calls'), ('0', 'Errors'), ('0', 'Cache Hits')]
rendered filter buttons: ['All', 'Agent', 'LLM', 'Tool', 'Error', 'Cache']
rendered rows: 42
   ▶️ agent.input Agent invocation started 12:19:02
   🧠 llm.start LLM call started 12:19:02
   💬 llm.end LLM call completed (288 tokens, 1096.3ms) 1096 ms 12:19:04
   🔧 tool.call Calling tool: check_stock 12:19:04
   ✅ tool.result Tool completed: check_stock 892 ms 12:19:04

Each row shows the event type, a one-line description and the time. The token count and latency of each model call are in its description. The stat cards show the same totals as get_stats() (the token card uses the browser's number format, which on this machine writes 3,671 as 3.671), and the filter buttons narrow the list to agent, model, tool, error or cache events.

You can also write a report at any point with agent.generate_report(path). From the quick example at the top:

print(agent.generate_report("shop-report.html"))

shop-report.html

ls *.html reports/

shop-report.html

reports/:
promptise-report-20261011_122246.html

It wrote exactly the path you gave it. The second file is the automatic report from observe=True, which goes to ./reports unless you set output_dir.

The JSON transporter streamed every event to traces/shop-assistant-events.ndjson, one JSON object per line. Each line has the event type, the session_id you set, a parent_id and a metadata object with the details. This short script turns that file into numbered steps per question, with the token cost of each:

"""Print a saved Promptise trace as numbered steps, one block per request."""

import json
import sys
from collections import defaultdict

# USD per million tokens for openai:gpt-5-mini. Check your provider's current prices.
PRICE_INPUT = 0.25
PRICE_OUTPUT = 2.00

requests = defaultdict(list)
with open(sys.argv[1]) as f:
    for line in f:
        event = json.loads(line)
        requests[event["session_id"]].append(event)

for request_id, events in requests.items():
    print(f"== {request_id}")
    step, prompt_tokens, completion_tokens = 0, 0, 0
    for e in events:
        m = e["metadata"]
        if e["event_type"] == "llm.end":
            step += 1
            prompt_tokens += m["prompt_tokens"]
            completion_tokens += m["completion_tokens"]
            wants = ", ".join(m.get("tool_calls", [])) or "final answer"
            print(f"{step:>2}. model   {m['latency_ms']:>6.0f} ms  {m['total_tokens']:>4} tokens -> {wants}")
        elif e["event_type"] == "tool.call":
            step += 1
            print(f"{step:>2}. call    {m['tool_name']} {m['arguments']}")
        elif e["event_type"] == "tool.result":
            step += 1
            print(f"{step:>2}. result  {m['latency_ms']:>6.0f} ms  {m['result_preview'][:60]}")
    cost = (prompt_tokens * PRICE_INPUT + completion_tokens * PRICE_OUTPUT) / 1_000_000
    print(f"    {prompt_tokens} in + {completion_tokens} out tokens = ${cost:.5f}")

python explain_trace.py traces/shop-assistant-events.ndjson

== q1
 1. model     1096 ms   288 tokens -> check_stock
 2. call    check_stock {'sku': 'MS-30'}
 3. result     892 ms  {"sku": "MS-30", "units": 6}
 4. model      690 ms   340 tokens -> final answer
    570 in + 58 out tokens = $0.00026
== q2
 1. model     1003 ms   288 tokens -> search_products
 2. call    search_products {'query': 'MS-10'}
 3. result      49 ms  [{"sku": "MS-10", "name": "MS-10 wireless mouse", "price_eur
 4. model      662 ms   341 tokens -> check_stock
 5. call    check_stock {'sku': 'MS-10'}
 6. result     822 ms  {"sku": "MS-10", "units": 0}
 7. model      838 ms   397 tokens -> final answer
    951 in + 75 out tokens = $0.00039
== q3
 1. model      840 ms   287 tokens -> search_products
 2. call    search_products {'query': 'ergonomic mouse'}
 3. result       8 ms  []
 4. model     1132 ms   380 tokens -> search_products
 5. call    search_products {'query': 'mouse'}
 6. result       3 ms  []
 7. model     1825 ms   536 tokens -> search_products, search_products, search_products, search_products
 8. call    search_products {'query': 'vertical mouse'}
 9. call    search_products {'query': 'ergonomic'}
10. call    search_products {'query': 'trackball'}
11. call    search_products {'query': 'wireless mouse'}
12. result      32 ms  []
13. result      38 ms  []
14. result      38 ms  []
15. result      38 ms  []
16. model     2760 ms   814 tokens -> final answer
    1301 in + 716 out tokens = $0.00176

Read q3 from the top and the problem is plain:

1. The model picked the right tool and passed a sensible query, ergonomic mouse.

2. The tool returned an empty list. This is the step that went wrong: the catalog has an "MS-30 ergonomic mouse".

3. The model tried again with a broader query, mouse, and got nothing again.

4. It then asked for four searches at once, vertical mouse, ergonomic, trackball and wireless mouse, and all four came back empty.

5. With six empty results, it told the customer the shop has no ergonomic mouse.

The model did everything right with what it was given. Without the trace, you'd probably start rewriting the prompt. With it, you know the fix belongs in search_products, and you can see that the bug also cost extra round trips: q3 used more than three times the tokens of q1.

The same reading works for most agent failures. Look at the first step where what happened differs from what you expected:

• The model called the wrong tool, or none. Improve the tool's description.

• The right tool got the wrong arguments. Describe the parameters better, or validate them in the server.

• A tool returned wrong or empty data. Fix the tool, as here.

• Everything was right, but the answer wasn't. Now it's the prompt or the model.

• The same tool is called over and over. The agent is stuck, and you're paying for every loop.

Make the search match every word anywhere in the name:

    words = query.lower().split()
    # Match every word anywhere in the name, not just at the start.
    return [
        {"sku": sku, **p}
        for sku, p in PRODUCTS.items()
        if all(word in p["name"].lower() for word in words)
    ]

Run shop_agent.py again in a clean folder, and q3 now gets it right:

[q3] Do you sell an ergonomic mouse?
      Yes — we have the MS-30 ergonomic mouse (49.00 EUR). There are 6 units in stock. Would you like to add one to your cart or see more details?

== q3
 1. model     1974 ms   461 tokens -> search_products, search_products, search_products
 2. call    search_products {'query': 'ergonomic mouse'}
 3. call    search_products {'query': 'vertical mouse'}
 4. call    search_products {'query': 'ergonomic'}
 5. result       9 ms  [{"sku": "MS-30", "name": "MS-30 ergonomic mouse", "price_eu
 6. result      10 ms  []
 7. result      10 ms  [{"sku": "MS-30", "name": "MS-30 ergonomic mouse", "price_eu
 8. model     1185 ms   488 tokens -> check_stock
 9. call    check_stock {'sku': 'MS-30'}
10. result     811 ms  {"sku": "MS-30", "units": 6}
11. model      748 ms   484 tokens -> final answer
    1104 in + 329 out tokens = $0.00093

The same query now finds the product, and the agent goes on to check stock on its own. This time the model also tried two other phrasings in its first step, and one of them, ergonomic, found the MS-30 too. Keep the old and new trace side by side when you fix something: it's the quickest proof that the fix worked and didn't break the other questions.

Promptise records tokens, not money. That's less of a gap than it sounds: prices change and differ between providers, so you multiply the tokens by your own rates anyway. explain_trace.py above does it with the OpenAI price list for gpt-5-mini at the time of writing, $0.25 per million input tokens and $2.00 per million output tokens.

Two things to know when you read the result:

• Output tokens include reasoning. gpt-5-mini thinks before it answers, and OpenAI bills those reasoning tokens as output tokens. That's why q3 used 716 output tokens for a few search requests and a short answer.

• Cached input is counted at full price. The trace doesn't record how many input tokens the provider served from its prompt cache, so this estimate is an upper bound.

For the whole session, get_stats() gives you the same totals: 2,822 input and 849 output tokens, about $0.0024 for three questions. Multiply by your daily traffic and you have a budget.

A report on your laptop is fine for debugging. In production you want events where your team already looks. The same config can send them to several places at once:

import asyncio
import sys

from promptise import ObservabilityConfig, ObserveLevel, TransporterType, build_agent
from promptise.config import StdioServerSpec


async def main():
    agent = await build_agent(
        model="openai:gpt-5-mini",
        servers={
            "shop": StdioServerSpec(command=sys.executable, args=["shop_server.py"]),
        },
        instructions="You are a shop assistant. Use your tools to answer. Be brief.",
        observe=ObservabilityConfig(
            level=ObserveLevel.STANDARD,
            session_name="shop-assistant",
            correlation_id="req-7f3a",
            transporters=[
                TransporterType.STRUCTURED_LOG,
                TransporterType.WEBHOOK,
                TransporterType.OTLP,
            ],
            log_file="logs/agent.jsonl",
            webhook_url="http://127.0.0.1:8231/events",
            webhook_headers={"Authorization": "Bearer demo-token"},
            otlp_endpoint="http://127.0.0.1:8230",
        ),
    )
    try:
        result = await agent.ainvoke(
            {"messages": [{"role": "user", "content": "Is the MS-30 in stock?"}]}
        )
        print(result["messages"][-1].content)
    finally:
        await agent.shutdown()


asyncio.run(main())

Structured logs go to logs/agent.jsonl, one JSON line per event, the format log pipelines such as ELK, Datadog or Splunk ingest. Tokens, latency, model and tool name are promoted to top-level fields, so you can filter and chart them without parsing. Here is a tool result from this run:

{"timestamp": "2026-10-11T10:23:40.513256+00:00", "level": "INFO", "service": "promptise", "session": "shop-assistant", "entry_id": "4b243cb5-d254-4da1-9cb2-1f1fcacbd042", "event_type": "tool.result", "category": "transparency", "message": "Tool completed: check_stock", "agent_id": "agent", "duration_ms": 846.6, "parent_id": "f88537a4-5878-41a9-a7b8-0098fd2b0483", "correlation_id": "req-7f3a", "latency_ms": 846.6, "tool_name": "check_stock", "metadata": {"run_id": "01a12a7d-5691-7713-b3ea-f8ebef1521b6", "result_preview": "{\"sku\": \"MS-30\", \"units\": 6}"}}

Webhooks receive each event as an HTTP POST with your headers. A small local endpoint on port 8231 printed what arrived. Each event is posted from its own background thread as it's recorded, so events can arrive out of order; in this run they happened to arrive in order. Sort on timestamp if order matters:

Bearer… agent.input: Agent invocation started
Bearer… llm.start: LLM call started
Bearer… llm.end: LLM call completed (288 tokens, 932.4ms)
Bearer… tool.call: Calling tool: check_stock
Bearer… tool.result: Tool completed: check_stock
Bearer… llm.start: LLM call started
Bearer… llm.end: LLM call completed (339 tokens, 960.7ms)
Bearer… agent.output: Agent completed (627 tokens)

OpenTelemetry export uses OTLP over gRPC. session_name becomes the service name. For this guide, the spans went to a minimal OTLP/gRPC receiver on port 8230, built on the official opentelemetry-proto package, not to a full backend:

service=shop-assistant trace=79271b75 span=c1b72533 parent=50c3b12f chat gpt-5-mini 932.3ms status=UNSET events=[]
service=shop-assistant trace=79271b75 span=454a75da parent=50c3b12f execute_tool check_stock 847.1ms status=UNSET events=[]
service=shop-assistant trace=79271b75 span=561cedd2 parent=50c3b12f chat gpt-5-mini 960.7ms status=UNSET events=[]
service=shop-assistant trace=79271b75 span=50c3b12f parent=- invoke_agent agent 2760.9ms status=UNSET events=[]

Each request is one trace. The root span, invoke_agent, covers the whole request, and every model call (chat) and tool call (execute_tool) is a child span with its real start and end time; the 847 ms check_stock call is easy to spot. The spans carry GenAI semantic-convention attributes such as gen_ai.tool.name and gen_ai.usage.input_tokens, plus Promptise's own under a promptise. prefix, including promptise.correlation_id. A full backend such as Jaeger wasn't tested for this guide, but a tree of parent and child spans is what such backends display.

Point otlp_endpoint at your own collector the same way, for example http://localhost:4317, the default. If the OpenTelemetry packages aren't installed, Promptise logs a warning and the OTLP transporter sends nothing, so check your logs after the first deploy.

Traces are only useful if you can keep them, so know what's in them. This is what 1.3.1 recorded for the same question at each setting, checked event by event:

Prompt and response text follow the level unless you set record_prompts. To turn observability off, use ObserveLevel.OFF or leave out observe.

Promptise's agent observability records the right things. A few things around it are worth knowing before you build on it:

• Redaction is pattern-based. It catches the secrets and identifiers listed above, not names or free text. ObservabilityConfig(redact_sensitive=False) turns it off.

• Webhook events can arrive out of order. Sort on timestamp, or use the JSON or log transporters when order matters.

• Files keep growing. The JSON transporter appends to shop-assistant-events.ndjson on every run, and purge_user doesn't touch files already written.

• The in-memory timeline is capped. It keeps the newest 100,000 events by default (max_entries), and older ones are dropped. Long-running agents should stream to the JSON, log or webhook transporters.

• A failed tool call is an error. It's recorded as tool.error and counted in error_count, so you don't have to search results for error text. Here is a lookup of an SKU that doesn't exist:

tool.call {'sku': 'XX-999'}
tool.error {"tool_name": "check_stock", "error_type": "MCPToolError", "latency_ms": 827.0, "error": "TOOL_ERROR: Unknown SKU XX-999."}
tool.call {'query': 'XX-999'}
tool.result {"result_preview": "[]", "latency_ms": 271.1, "tool_name": "search_products"}
error_count: 1

What is AI agent observability?

It's the record of everything an agent did to produce an answer: each model call with its tokens and latency, each tool call with its arguments and result, and any errors. Logs tell you that something happened; a trace of the steps tells you why the agent decided what it decided, which is what you need to debug it or explain it to someone else.

Does Promptise track the cost of LLM calls?

It tracks tokens per call, split into input and output, plus the exact model version. It doesn't convert them to money. Multiply by your provider's prices, as explain_trace.py does, so the numbers stay right when prices change.

Is there an open source alternative to LangSmith for tracing agents?

If you build with Promptise Foundry, its observability is part of the open source framework (Apache 2.0) and needs no account or hosted service: traces stay in local files unless you send them somewhere. It doesn't give you a hosted trace UI like LangSmith. For a shared view, send events to tools you already run, through structured logs, webhooks or OpenTelemetry.

Will customer data end up in my traces?

Tool arguments and tool results will, up to 2,000 characters each, with the default settings, after Promptise has redacted the secrets and identifiers it recognises, such as email addresses and API keys. Prompts and model responses are only recorded at ObserveLevel.FULL or with record_prompts=True, and record_tool_io=False keeps tool data out. Plan retention and access for trace files and log streams as you would for any store of customer data.

Can I send AI agent traces to OpenTelemetry?

Yes. Add TransporterType.OTLP, set otlp_endpoint, and install the OpenTelemetry SDK and OTLP gRPC exporter. Each request arrives as one trace, with a span for the request and child spans for every model call and tool call, using GenAI semantic-convention attributes.

• Observability: every option of ObservabilityConfig and the transporters.

• Observability API reference: ObservabilityCollector, TimelineEntry and the transporter classes.

• Building Agents: build_agent, CallerContext and the rest of the agent API.

• Events & Notifications: alerts for slow tools and errors, separate from the trace.

• Observability & Monitoring for MCP servers: the other side of every tool call.

• How to Connect MCP Servers to Your AI Agent in Python: the agent setup this guide builds on.

• Build a Production MCP Server in Python: audit logs and server-side limits for the tools your agent calls.