Best AI Agent Observability Tools (2026), Compared

For most teams the shortlist is Langfuse, LangSmith or Arize Phoenix, with Datadog for teams that already run it. Pick by where your data may live and what you already use:

Key takeaways:

• OpenTelemetry is the common ground. LangSmith, Langfuse, Phoenix, Datadog, Braintrust and Weave all ingest OTLP traces with gen_ai.* attributes, so instrument once and choose the backend later.

• Check ownership before you commit. In 2026 ClickHouse acquired Langfuse, Dynatrace acquired Arize, Mintlify put Helicone into maintenance mode, and CoreWeave's docs call Agent Lens the successor to Weave.

• PII redaction is mostly client-side. Most tools give you a masking hook in the SDK; server-side redaction usually needs an enterprise plan.

• Observability is not an audit trail. Traces are sampled and expire. Vendor "audit logs" record admin actions in the tool, not your agent's tool calls.

• We make Promptise. Its tracing is a library inside your agent, not a hosted UI, so it complements the tools above rather than replacing them.

An agent run is a tree. One request fans out into several model calls and tool calls, and the useful questions sit at different levels: which tool failed, which call was expensive, which user drove the cost.

invoke_agent support            user=alice   410 in / 39 out tokens
├── chat gpt-5-mini             167 in / 27 out   the model asks for get_order
├── execute_tool get_order      5 ms              arguments, result or error
└── chat gpt-5-mini             243 in / 12 out   the model answers

The OpenTelemetry GenAI semantic conventions name these spans: invoke_agent for the run, chat <model> for each model call and execute_tool <name> for each tool call, with attributes such as gen_ai.usage.input_tokens and gen_ai.tool.name. Tools that follow the conventions can read each other's data.

Judge a tool on these criteria:

• Tool-call tracing: tool spans with arguments, results and errors, nested under the run.

• Token and cost attribution: per call, run, user and session, with custom prices for your models.

• Evals: LLM-as-judge or code checks on live traces and on datasets.

• Prompt versioning: prompts stored with versions and labels, linked to the traces that used them.

• PII redaction: masking before data leaves your process, or on the server.

• Self-hosting and license: whether you can run it yourself, and under which terms.

• OpenTelemetry: OTLP ingest and export, and which attribute conventions it maps.

• Pricing model: per trace, span, unit, GB or seat, and how long data is kept.

AI agent observability explains how to use a trace to find the step that went wrong.

Every cell comes from the tool's own documentation or pricing page, read on 20 October 2026.

LangSmith

LangSmith is LangChain's hosted platform for tracing, evaluation, prompts and deployment. LangChain and LangGraph are traced with an environment variable, and other code with @traceable or OpenTelemetry. Cost is tracked per trace and per thread with an editable price map, and online evaluators can sample live traffic with a spend limit. Base traces are kept 14 days, and the maximum for extended traces dropped to 180 days for new traces from 14 September 2026. Self-hosting is an Enterprise add-on. Better choice when your agents run on LangChain or LangGraph and you want tracing, evals and deployment from one vendor.

Langfuse

Langfuse is an open-source platform for tracing, prompts and evals. The MIT-licensed core includes tracing, cost tracking, prompt management and LLM-as-judge evals without limits when you self-host; audit logs, server-side masking and SCIM need an Enterprise license. It has dedicated observation types for agents, tools and guardrails, and ingests OTLP over HTTP (not gRPC). ClickHouse acquired Langfuse in January 2026 and said licensing would not change. Better choice when you want to self-host everything, or want prompts, traces and evals in one open-source tool.

Arize Phoenix

Phoenix is built on OpenTelemetry and OpenInference and is free to self-host with no feature limits under the Elastic License 2.0, which forbids offering it as a hosted service. It adds evals, datasets, experiments and a prompt playground, and Arize AX is the hosted enterprise product. Dynatrace completed its acquisition of Arize on 1 October 2026 and said Phoenix continues. Better choice when you want a free, self-hosted tool with strong evals. The example below sends Promptise traces to it.

Datadog LLM Observability

Datadog puts agent traces next to your APM, logs and infrastructure, and its product page now calls it Agent Observability. It estimates cost for more than 800 models, runs managed and custom evaluations, and redacts with Sensitive Data Scanner on the server. Only LLM spans are billed: 40K a month free, then from $160 a month for 100K. It ingests OpenTelemetry spans that follow GenAI conventions 1.37 or later. Better choice when you already run Datadog and want one place for agent and service incidents.

Braintrust

Braintrust starts from evaluation: datasets, scorers, CI runs and online scoring of production traces, with tracing that records tokens and cost per span. All plans include unlimited users; the free tier has 1 GB of data and 10,000 scores, and Pro costs $249 a month. Self-hosting means an Enterprise data plane in your cloud. Better choice when evals drive your release process and you want traces and experiments side by side.

W&B Weave

Weave traces calls, versions prompts as immutable objects, and runs scorers as monitors or inline guardrails. Weights & Biases is now part of CoreWeave, Weave's docs moved to CoreWeave, and they introduce CoreWeave Agent Lens as "the successor to Weights & Biases Weave". PII redaction is a client-side switch that uses Presidio. Better choice when your team already tracks models and experiments in W&B.

OpenLIT

OpenLIT is an Apache-2.0, OpenTelemetry-native stack: an SDK that auto-instruments model and agent libraries with gen_ai.* attributes, a collector, and a self-hosted UI on ClickHouse with cost tracking, evals, guardrails and a prompt hub. Audit logs and RBAC are Enterprise features. Better choice when you want everything on standard OpenTelemetry under a permissive license.

Helicone

Helicone logs through a proxy, so you change a base URL instead of your code, and tags sessions, users and properties with headers. Mintlify acquired it in March 2026 and moved it to maintenance mode with security fixes and new models only; its site now says it is no longer part of Mintlify, with details to come. It has no PII redaction beyond not storing a request. Better choice when you need logging with no code change on an existing deployment. Check its status before a new build.

OpenTelemetry GenAI semantic conventions

The GenAI semantic conventions are not a tool but the format underneath most of the tools above. They define spans for model calls, tool calls and agents, and attributes for tokens, models and tools; message content is opt-in. They moved to their own repository in mid-2026 and are still marked Development, so names can change. Instrumenting with them keeps your traces portable between backends.

Graphsignal: not in this category any more

Graphsignal used to trace LLM calls, which is why "Langfuse vs Graphsignal" comparisons exist. Its current site describes a GPU inference profiler for vLLM, SGLang and TensorRT-LLM. If you serve your own models, it can complement Langfuse; it does not replace it.

Promptise Foundry, which we make, records observability inside the agent. observe=True on build_agent() captures every run, model call, tool call, token count, latency, retry and error, attributed to the calling user and session. Events go to transporters: an HTML report, NDJSON files, structured logs, Prometheus metrics, webhooks or OpenTelemetry spans with GenAI attributes. Credentials, card numbers and email addresses are redacted by default before anything is stored. Its MCP server framework adds a signed, tamper-evident audit log of tool calls.

It is not a hosted backend. There is no shared trace UI, no hosted evals and no prompt management service. Use it to produce clean, attributed telemetry from the agent, and send it to one of the tools above when your team needs dashboards, evals or long retention.

• Python 3.10 or later and an OpenAI API key in OPENAI_API_KEY. The example uses gpt-5-mini.

• Promptise with the OpenTelemetry extras, and Phoenix for the export step, in separate environments:

pip install "promptise[all]==1.4.3"                  # agent, MCP server, OpenTelemetry SDK
pip install "arize-phoenix==20.23.0" && phoenix serve # local OTLP backend on port 6006

• A secret for the audit log in PROMPTISE_AUDIT_SECRET.

The agent's tools live on an MCP server. AuditMiddleware writes one signed line per tool call, separate from the traces. One tool fails for unknown orders, so the trace has an error to show, and the order record holds an email address to redact.

"""Two order tools over stdio. One of them fails for unknown orders."""
from promptise.mcp.server import AuditMiddleware, MCPServer, ToolError

ORDERS = {
    "O-1001": {"order_id": "O-1001", "item": "Desk lamp", "amount": 49.00, "status": "delivered",
               "customer_email": "maria.keller@example.com"},
}
server = MCPServer(name="orders")
# A tamper-evident record of every tool call, separate from the traces.
server.add_middleware(AuditMiddleware(log_path="out/audit.jsonl", signed=True, include_args=True))


@server.tool(read_only_hint=True)
async def get_order(order_id: str) -> dict:
    """Look up an order by ID: item, amount in USD, status and customer email."""
    order = ORDERS.get(order_id.strip().upper())
    if order is None:
        raise ToolError(f"No order {order_id}")
    return order


if __name__ == "__main__":
    server.run()

signed=True chains each entry to the previous one with an HMAC under PROMPTISE_AUDIT_SECRET, and include_args=True records the arguments. Results are left out by default because they often hold personal data.

This program runs two users' questions through one agent, sends the spans to an in-memory OpenTelemetry exporter so you can print them, and prices the tokens per user.

import asyncio
import os

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter

from promptise import build_agent
from promptise.agent import CallerContext
from promptise.config import StdioServerSpec
from promptise.observability_config import ObservabilityConfig, ObserveLevel, TransporterType
from promptise.observability_transporters import OTLPTransporter

PRICES = {"gpt-5-mini": (0.25, 2.00)}  # USD per million input and output tokens (OpenAI, 2026-10-20)
QUESTIONS = [("alice", "What is the status of order O-1001?"),
             ("bob", "Where is my order O-9999?")]


async def main():
    agent = await build_agent(
        model="openai:gpt-5-mini",
        servers={"orders": StdioServerSpec(
            command="python", args=["orders_server.py"],
            env={"PROMPTISE_AUDIT_SECRET": os.environ["PROMPTISE_AUDIT_SECRET"]})},
        instructions="You are an order-support agent. Use the tools. Answer in one sentence.",
        observer_agent_id="support",  # the agent name in every event and span
        observe=ObservabilityConfig(
            level=ObserveLevel.STANDARD,           # one event per model call and tool call
            transporters=[TransporterType.JSON],   # NDJSON event stream on disk
            output_dir="out", session_name="orders",
        ),
    )
    # Send the same runs to OpenTelemetry. In production: an OTLP exporter to your collector.
    spans = InMemorySpanExporter()
    provider = TracerProvider()
    provider.add_span_processor(SimpleSpanProcessor(spans))
    agent.collector.add_transporter(OTLPTransporter(tracer_provider=provider))
    try:
        for user, question in QUESTIONS:
            result = await agent.ainvoke({"messages": [{"role": "user", "content": question}]},
                                         caller=CallerContext(user_id=user))
            print(f"{user}: {result['messages'][-1].content}")
    finally:
        await agent.shutdown()

    print("\nOpenTelemetry spans (GenAI semantic conventions):")
    for span in sorted(spans.get_finished_spans(), key=lambda s: s.start_time):
        a = span.attributes
        indent = "" if span.parent is None or span.name.startswith("invoke_agent") else "  "
        detail = {k: a[k] for k in ("enduser.id", "gen_ai.usage.input_tokens",
                                    "gen_ai.usage.output_tokens", "gen_ai.tool.name") if k in a}
        status = " ERROR" if span.status.status_code.name == "ERROR" else ""
        ms = (span.end_time - span.start_time) / 1e6
        print(f"{indent}{span.name:<26} {ms:7.0f} ms{status}  {detail}")

    print("\nTokens and cost per user:")
    price_in, price_out = PRICES["gpt-5-mini"]
    for user, _ in QUESTIONS:
        calls = [e for e in agent.collector.for_user(user) if e.event_type.value == "llm.end"]
        t_in = sum(e.metadata["prompt_tokens"] for e in calls)
        t_out = sum(e.metadata["completion_tokens"] for e in calls)
        cost = (t_in * price_in + t_out * price_out) / 1e6
        print(f"  {user}: {len(calls)} model calls, {t_in} in / {t_out} out, ${cost:.6f}")

    result = next(e for e in agent.collector.get_timeline() if e.event_type.value == "tool.result")
    print("\nRecorded tool result (redacted):", result.metadata["result_preview"])


asyncio.run(main())

alice: Order O-1001 has been delivered.
bob: I couldn't find order O-9999 in our system—please confirm the order ID or the email used for the purchase and I'll check again.

OpenTelemetry spans (GenAI semantic conventions):
invoke_agent support          3657 ms  {'enduser.id': 'alice', 'gen_ai.usage.input_tokens': 410, 'gen_ai.usage.output_tokens': 39}
  chat gpt-5-mini               2394 ms  {'enduser.id': 'alice', 'gen_ai.usage.input_tokens': 167, 'gen_ai.usage.output_tokens': 27}
  execute_tool get_order           5 ms  {'enduser.id': 'alice', 'gen_ai.tool.name': 'get_order'}
  chat gpt-5-mini               1248 ms  {'enduser.id': 'alice', 'gen_ai.usage.input_tokens': 243, 'gen_ai.usage.output_tokens': 12}
invoke_agent support          5815 ms  {'enduser.id': 'bob', 'gen_ai.usage.input_tokens': 399, 'gen_ai.usage.output_tokens': 385}
  chat gpt-5-mini               2171 ms  {'enduser.id': 'bob', 'gen_ai.usage.input_tokens': 165, 'gen_ai.usage.output_tokens': 91}
  execute_tool get_order           7 ms ERROR  {'enduser.id': 'bob', 'gen_ai.tool.name': 'get_order'}
  chat gpt-5-mini               3630 ms  {'enduser.id': 'bob', 'gen_ai.usage.input_tokens': 234, 'gen_ai.usage.output_tokens': 294}

Tokens and cost per user:
  alice: 2 model calls, 410 in / 39 out, $0.000180
  bob: 2 model calls, 399 in / 385 out, $0.000870

Recorded tool result (redacted): {"order_id": "O-1001", "item": "Desk lamp", "amount": 49.0, "status": "delivered", "customer_email": "[EMAIL]"}

Each run became one trace: an invoke_agent span with two chat spans and one execute_tool span, named and attributed by the GenAI conventions. Bob's tool call failed, so its span has an ERROR status. Every span carries enduser.id from the CallerContext, which is what makes per-user cost possible. Promptise reports tokens, not dollars, so the price table is yours; AI agent cost tracking builds a full ledger with budgets. The last line shows redaction: the order record reached the model unchanged, but the email in the recorded tool result became [EMAIL].

Promptise's built-in OTLP transporter exports over gRPC to port 4317. Langfuse, LangSmith, Braintrust, Weave and Datadog ingest OTLP over HTTP, so pass a TracerProvider with an HTTP exporter instead. This version sends to a local Phoenix:

"""Send the agent's traces to any OTLP/HTTP backend. Here: a local Arize Phoenix on port 6006."""
import asyncio
import os

from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

from promptise import build_agent
from promptise.agent import CallerContext
from promptise.config import StdioServerSpec
from promptise.observability_config import ObservabilityConfig, ObserveLevel
from promptise.observability_transporters import OTLPTransporter

ENDPOINT = os.getenv("OTLP_TRACES_URL", "http://localhost:6006/v1/traces")
HEADERS = {}  # e.g. {"Authorization": "Basic ..."} for Langfuse, {"x-api-key": "..."} for LangSmith

provider = TracerProvider(resource=Resource.create({"service.name": "support-agent"}))
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint=ENDPOINT, headers=HEADERS)))


async def main():
    agent = await build_agent(
        model="openai:gpt-5-mini",
        servers={"orders": StdioServerSpec(
            command="python", args=["orders_server.py"],
            env={"PROMPTISE_AUDIT_SECRET": os.environ["PROMPTISE_AUDIT_SECRET"]})},
        instructions="You are an order-support agent. Use the tools. Answer in one sentence.",
        observer_agent_id="support",
        observe=ObservabilityConfig(level=ObserveLevel.STANDARD, transporters=[]),
    )
    agent.collector.add_transporter(OTLPTransporter(tracer_provider=provider))
    try:
        result = await agent.ainvoke(
            {"messages": [{"role": "user", "content": "What is the status of order O-1001?"}]},
            caller=CallerContext(user_id="alice"))
        print(result["messages"][-1].content)
    finally:
        await agent.shutdown()
        provider.force_flush()


asyncio.run(main())

Order O-1001 has been delivered.

Then read back what Phoenix stored, through its REST API:

"""Read back what Phoenix stored, through its REST API."""
import json
import urllib.request

url = "http://localhost:6006/v1/projects/default/spans?limit=20"
spans = json.load(urllib.request.urlopen(url))["data"]
for span in sorted(spans, key=lambda s: s["start_time"]):
    a = span["attributes"]
    tokens = a.get("llm.token_count.total", "-")
    print(f"{span['span_kind']:<6} {span['name']:<24} tokens={tokens}")

AGENT  invoke_agent support     tokens=513
LLM    chat gpt-5-mini          tokens=258
TOOL   execute_tool get_order   tokens=-
LLM    chat gpt-5-mini          tokens=255

Phoenix mapped the GenAI span names to its own AGENT, LLM and TOOL kinds and computed token totals without any Phoenix-specific code. To switch to another backend, change OTLP_TRACES_URL and the auth headers, for example Langfuse's /api/public/otel endpoint or LangSmith's /otel endpoint. force_flush() matters in short scripts: the batch processor otherwise drops spans that haven't been sent when the process exits.

The audit log from Step 1 now holds every tool call from both programs. Verify the chain:

promptise audit verify audit.jsonl

OK 3 entries, chain continued across 1 restart: intact.
last_hash 890f40f560789ce5b2bbf7eda1088e6804aafc899f41eb7e4eca82cba0d7f574 (seq 2)

"Continued across 1 restart" means two server processes appended to the same chain. Now change one order ID in a copy of the file and verify again:

sed 's/"O-1001"/"O-1002"/' audit.jsonl > tampered.jsonl
promptise audit verify tampered.jsonl

TAMPERED tampered.jsonl:1: modified: the entry was changed after it was signed (its signature no longer matches, the next entry still links to it)
TAMPERED tampered.jsonl:3: bad_signature: the entry's signature does not match: it was modified or forged
FAILED 3 entries: 2 problems; the first is at tampered.jsonl:1.

The edit broke the signatures of the two entries that held that order ID, and the command exits with status 1, so it can run in CI or a nightly job. AI agent audit logging covers key rotation and anchoring.

Searches for "AI agent observability tools with audit trails" mix two different records:

The "audit logs" in observability tools usually record what people did in the tool itself. LangSmith's audit logs are an Enterprise feature that records organization and workspace actions in OCSF format and keeps them up to 400 days. Langfuse's audit logs record who changed what in a project, also on Enterprise. Neither is a signed record of your agent's tool calls. For that, log at the tool boundary, as Step 1 does, or write your own append-only log.

Decide what text leaves your process

Prompts, tool arguments and results often carry personal data. In Promptise, record_prompts and record_tool_io control what is recorded, and redact_sensitive scrubs what is. Pattern redaction can't find names or addresses, so set record_tool_io=False when traces go to a hosted backend you don't control. In other tools, use the SDK's masking hook, which runs before export.

Sample traces, not audit entries

Traces at full volume get expensive: most tools price per trace, span, unit or GB. Sample successful runs and keep every error, but never sample the audit log.

Put a collector in the middle

An OpenTelemetry Collector between the agent and the backend lets you redact attributes, sample, and fan out to two backends while you compare them, without changing agent code.

Watch the retention defaults

Free tiers keep traces for 7 to 30 days, and some paid plans cap retention. If you need traces for incident reviews months later, export them or pay for longer retention.

• Feature cells are from documentation, not tests. Only Promptise and Phoenix were run for this guide. Check the current docs before you buy, because plans and prices change often.

• No benchmark. This guide doesn't compare ingestion speed, UI quality or query performance.

• Promptise is a library, not a backend. It records and exports; it has no shared UI, hosted evals or prompt service. We make it, and we have said where the other tools do more.

What are the best AI agent observability tools?

Langfuse for open-source self-hosting with prompts and evals, LangSmith for LangChain and LangGraph teams, Arize Phoenix for free self-hosting with strong evals, Datadog for teams already on Datadog, Braintrust for eval-driven workflows, and OpenLIT for an OpenTelemetry-native stack. Instrument with the OpenTelemetry GenAI conventions so you can switch later.

What is the best platform for visibility into token usage, tool interactions and failure patterns?

Any tool that records tool spans with errors and tokens per span: Langfuse, LangSmith, Phoenix, Datadog and Braintrust all do. Datadog adds service metrics next to agent traces. Whichever you choose, attribute each run to a user and session, or you can see what failed but not who it affected.

Does LangSmith have an audit trail?

Yes, on the Enterprise plan. LangSmith audit logs record administrative actions in your organization and workspaces, such as membership and settings changes, in OCSF format, retrievable through an API and kept up to 400 days. They are not a record of your agent's tool calls; traces cover those, but traces expire and are not tamper-evident.

Does Langfuse have guardrails?

Not built in. Langfuse's security docs recommend running a guardrail library at runtime, such as LLM Guard, NeMo Guardrails, Lakera or Azure AI Content Safety, and using Langfuse to trace the checks and score them afterward. It has a guardrail observation type for those checks. Best LLM guardrails tools compares the runtime options.

Langfuse vs Graphsignal: which should I use?

They solve different problems in 2026. Langfuse traces LLM and agent calls, with prompts and evals. Graphsignal is now a GPU inference profiler for model servers such as vLLM and SGLang. Use Langfuse for agent behavior, and Graphsignal only if you also serve your own models and need GPU-level profiling.

How does Promptise observability compare to Langfuse or LangSmith?

Promptise records traces inside the agent, with user attribution, default redaction and OpenTelemetry export, plus a signed audit log on its MCP servers. It has no hosted UI, evals or prompt management, which Langfuse and LangSmith provide. Use them together: Promptise produces the telemetry, and the backend stores and analyzes it.

• Use a trace to debug a run: AI agent observability: trace tool calls.

• Turn tokens into a cost ledger with budgets: AI agent cost tracking.

• Build a tamper-evident record of tool calls: AI agent audit logging.

• Monitor the MCP servers themselves: MCP server monitoring.

• Keep personal data out of traces: PII redaction for LLM apps.

• Reference: the Promptise docs for observability.