PromptisePromptise
Why PromptiseDocs
GitHubGet started
Promptise - AI Framework LogoPromptise

The foundation layer for agentic intelligence. Build, secure and operate autonomous AI systems with Promptise Foundry.

pip install promptise

[01] Foundry

  • MCPcast
  • The Promptise Agent
  • Reasoning Engine
  • MCP
  • Agent Runtime
  • Prompt Engineering
  • Execution Engine
  • Agent Identity

[02] Resources

  • Documentation
  • GitHub
  • Guides
  • Learning Paths
  • Questions

[03] Company

  • About
  • Terms of Service
  • Privacy Policy
  • Cookie Policy
  • Subprocessors

© 2026 Promptise by Manser Ventures. All rights reserved.

Open source · Python

← Guides> Observability and debugging

Audit-Grade Agent Traces Without Storing Prompt Content

Audit-grade agent traces without storing prompt content: what to record, keyed digests, hash chains, per-tenant keys, verification and a tamper check.

Level
Advanced
Reading time
18 min
Published
Oct 23, 2026
Last updated
Oct 23, 2026
Tested with
Promptise 1.7.0
By
Promptise Team
  • Audit Logging and Monitoring
  • Observability
  • Multi-Tenancy
  • Guardrails
  • Model Context Protocol (MCP)
Contents · 16 sections↓
  1. 01How do you make audit-grade agent traces without storing prompt content?
  2. 02What should an AI agent audit record contain?
  3. 03How content-free audit trails work
  4. 04When you need this (and when not)
  5. 05What you need
  6. 06Step 1: Build the per-tenant audit trail
  7. 07Step 2: Put it on the document extraction server
  8. 08Step 3: Run the agent with content capture off
  9. 09Step 4: Read what the trail kept
  10. 10Step 5: Verify, check a claim, and tamper with a copy
  11. 11Where content leaks back in
  12. 12Running a content-free audit trail in production
  13. 13Common problems and fixes
  14. 14Limitations
  15. 15FAQ
  16. 16Next steps

Audit-grade agent traces without storing prompt content are possible: record who acted, which tool ran, what was decided and how it ended, and keep arguments and results only as keyed digests. This guide explains what to record, how hash chains make the trail tamper-evident, how to keep each tenant's trail separate and verifiable by that tenant, and how to build it for an AI document extraction agent with Promptise Foundry, with real runs and a tamper check.

[01]

How do you make audit-grade agent traces without storing prompt content?

Record the facts of each action, not its content. For every tool call, write who called (user, tenant, roles), which tool, the outcome (ok, denied, failed) and a request id, and chain the entries with an HMAC so any edit, deletion or reordering shows up. Keep arguments and results out of the record, except identifiers you need in clear; store the rest as keyed digests (HMAC), so you can later check a claimed value without the trail holding it. Give each tenant its own key and files, so a tenant can verify its own trail and nobody else's.

  • Audit at the tool boundary. The MCP server sees every action, whatever the model said. That is where the record belongs.

  • Digests, not hashes. A plain SHA-256 of "1,240.00 EUR" can be guessed by hashing candidates. An HMAC with a key you hold cannot.

  • Chain the entries. Each entry carries the previous entry's MAC, so a verifier finds edits, inserts, deletions and reordering.

  • One key per tenant. Derive it from a master key. The tenant can verify its files; another tenant's key fails.

  • Turn content off everywhere else too. Agent traces, run journals and error messages can hold the content your audit trail leaves out.


[02]

What should an AI agent audit record contain?

An audit record answers "who did what, when, and with what result" well enough that someone who wasn't there can check it. For an agent, that is one record per tool call:

Field

Example

Stored as

Why

Who

dana@acme.example, tenant acme, roles clerk

Clear

Attribution: from the verified token, never from the model

What

save_extraction

Clear

The action itself

Which record

document_id: INV-1001

Clear

Lets you find the affected object without its content

Arguments

total, vendor

Keyed digest

Proves what was sent, without storing it

Decision

ok, error with ACCESS_DENIED

Clear

A refusal is part of the record, not a gap in it

Result

hmac:7e8e…

Keyed digest

Proves what came back

When, how long

timestamp, duration

Clear

Ordering and timing

Integrity

seq, prev_hash, hmac

Clear

Tamper evidence

What stays out: the user's prompt, the model's answer, the system prompt, retrieved documents and the text a tool read. Those are the parts most likely to hold personal or confidential data, and the parts a trail rarely needs. If an investigation needs content, it should come from the system of record (the document store, the CRM) under that system's own access controls, and the digest tells you whether what you were given is what the agent saw.

Whether a particular trail meets a particular legal or contractual requirement is a question for your legal or compliance team; this guide covers the engineering.


[03]

How content-free audit trails work

An agent sends a tool call with the user's token to an MCP server, where it passes auth, the tenant audit trail and the tool. The trail writes chain.jsonl (who, tool, outcome, seq, prev_hash, hmac) and digests.jsonl (document id in clear, other arguments and the result as keyed digests, mac) in audit/acme, both signed with the acme key derived from a master key, so acme verifies its own files and nobody else's.
FigureOne tool call, two signed records per tenant, and no prompt or document text

Three techniques do the work.

A hash chain for tamper evidence. Each entry stores the MAC of the entry before it (prev_hash) and its own MAC over its fields (hmac), computed with a secret key. Change a field and its MAC no longer matches. Delete an entry and the next one points at a MAC that isn't there. Reorder two and the links cross. A forger without the key can't compute valid MACs. The one thing a chain can't show alone is entries cut from the end; for that you record the last hash somewhere else (an anchor). AI Agent Audit Logging: Tamper-Evident Logs in Python walks through all six kinds of tampering, anchors, restarts and log rotation in detail; this guide uses the same chain and focuses on keeping content out.

Keyed digests instead of content. A digest is HMAC(key, canonical value). Given a candidate value, anyone with the key can recompute it and compare. Without the key, the digest reveals nothing useful, and that matters because agent arguments are often short and guessable: amounts, dates, names, yes or no. A plain hash of a guessable value is not a secret.

Per-tenant keys and files. Each tenant's records go to its own folder and are signed with its own key, derived as HMAC(master key, tenant id). You can hand a tenant its key and its files. It can then verify independently that the trail is complete and unaltered, while it can't verify, or learn anything from, another tenant's trail. Rotating the master key rotates every tenant key.


[04]

When you need this (and when not)

You need a content-free trail when:

  • agents act on customer data and someone outside the team may ask what happened: a customer, an auditor, an incident review;

  • one platform serves several tenants and each wants evidence about its own data;

  • the content is the sensitive part, as in document extraction, support, HR or finance workflows;

  • your traces leave your trust boundary, for example to a hosted observability vendor.

You may not need it for a local prototype, for agents that only read public data, or when you already keep full content in a store with stricter controls than your logs. A content-free trail also doesn't replace debugging traces: while developing, you want to see prompts. Keep that in a development environment, not in production.


[05]

What you need

  • Python 3.10 or newer and Promptise Foundry 1.7.0.

  • An OpenAI API key for the demo agent (any model with tool calling works).

  • Two throwaway secrets: one to sign test tokens, one master audit key.

>_Terminal
uv venv && uv pip install "promptise==1.7.0" python-dotenv
export DOCS_JWT_SECRET=$(python -c "import secrets; print(secrets.token_hex(32))")
export AUDIT_MASTER_KEY=$(python -c "import secrets; print(secrets.token_hex(32))")

[06]

Step 1: Build the per-tenant audit trail

The trail is MCP server middleware. For each tenant it keeps one Promptise AuditMiddleware (the HMAC chain, with include_args=False and include_result=False, the defaults) and writes one digest line per call next to it:

Pythontrail.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
"""A content-free, per-tenant audit trail for MCP tool calls.

Each tenant gets two files under audit/<tenant>/:
  chain.jsonl    Promptise AuditMiddleware: who, which tool, outcome, HMAC chain.
                 No arguments, no results.
  digests.jsonl  One line per call: arguments and result as keyed digests,
                 except fields you allow in clear (ids). Each line is signed.
Both files are signed with the tenant's own key, derived from one master key.
"""
import hashlib
import hmac
import json
import os
from pathlib import Path
from typing import Any

from promptise.mcp.server import AuditMiddleware, RequestContext, verify_audit_log

# Argument fields that are safe to keep in clear, per tool. Everything else
# is stored as a keyed digest.
CLEAR_FIELDS = {
    "read_document": {"document_id"},
    "save_extraction": {"document_id", "due_date"},
    "delete_document": {"document_id"},
}


def tenant_key(tenant: str) -> bytes:
    """The tenant's audit key: HMAC(master key, tenant). Hand it to the tenant
    and they can verify their own trail, and nobody else's."""
    master = os.environ["AUDIT_MASTER_KEY"].encode()
    return hmac.new(master, f"audit:{tenant}".encode(), hashlib.sha256).digest()


def digest(key: bytes, value: Any) -> str:
    """Keyed digest of a value. Without the key, nobody can test guesses
    such as "was the total 1240.00?" against it."""
    canonical = json.dumps(value, sort_keys=True, default=str)
    return "hmac:" + hmac.new(key, canonical.encode(), hashlib.sha256).hexdigest()[:32]


def sign(key: bytes, record: dict) -> str:
    return hmac.new(key, json.dumps(record, sort_keys=True).encode(), hashlib.sha256).hexdigest()


class TenantAuditTrail:
    """MCP middleware: one AuditMiddleware chain and one digest file per tenant."""

    def __init__(self, root: str = "audit") -> None:
        self.root = Path(root)
        self._chains: dict[str, AuditMiddleware] = {}

    def _chain(self, tenant: str) -> AuditMiddleware:
        if tenant not in self._chains:
            folder = self.root / tenant
            folder.mkdir(parents=True, exist_ok=True)
            self._chains[tenant] = AuditMiddleware(
                log_path=str(folder / "chain.jsonl"),
                hmac_secret=tenant_key(tenant).hex(),
                include_args=False,    # the default: no argument values in the chain
                include_result=False,  # the default: no result text in the chain
            )
        return self._chains[tenant]

    async def __call__(self, ctx: RequestContext, call_next):
        tenant = (ctx.client.tenant_id if ctx.client else None) or "_no_tenant"
        outcome: dict[str, Any] = {}

        async def run_tool(c: RequestContext):
            outcome["result"] = await call_next(c)
            return outcome["result"]

        try:
            return await self._chain(tenant)(ctx, run_tool)
        finally:
            self._write_digests(tenant, ctx, outcome)

    def _write_digests(self, tenant: str, ctx: RequestContext, outcome: dict) -> None:
        key = tenant_key(tenant)
        clear = CLEAR_FIELDS.get(ctx.tool_name, set())
        args = ctx.state.get("_tool_arguments") or {}
        record = {
            "request_id": ctx.request_id,
            "tool": ctx.tool_name,
            "args": {k: (v if k in clear else digest(key, v)) for k, v in sorted(args.items())},
            "result": digest(key, outcome["result"]) if "result" in outcome else None,
        }
        record["mac"] = sign(key, record)
        with open(self.root / tenant / "digests.jsonl", "a") as f:
            f.write(json.dumps(record) + "\n")


def verify_trail(tenant: str, root: str = "audit") -> list[str]:
    """Check a tenant's chain and digest file together. Returns the problems found."""
    folder = Path(root) / tenant
    key = tenant_key(tenant)
    report = verify_audit_log(str(folder / "chain.jsonl"), key.hex())
    problems = [str(p) for p in report.problems]

    chain_ids = [json.loads(line)["request_id"] for line in open(folder / "chain.jsonl") if line.strip()]
    digest_ids = []
    for n, line in enumerate(open(folder / "digests.jsonl"), start=1):
        record = json.loads(line)
        mac = record.pop("mac", "")
        if not hmac.compare_digest(sign(key, record), mac):
            problems.append(f"digests.jsonl:{n}: signature does not match (edited or forged)")
        digest_ids.append(record["request_id"])
    for rid in sorted(set(chain_ids) - set(digest_ids)):
        problems.append(f"request {rid}: in the chain but its digest line is missing")
    for rid in sorted(set(digest_ids) - set(chain_ids)):
        problems.append(f"request {rid}: digest line with no chain entry")
    return problems

What each part does:

  • `tenant_key()` derives the tenant's key from the master key. The master key never leaves your secret store; a tenant gets only its own derived key.

  • `AuditMiddleware` writes the chain: timestamp, tool, client id, request id, status, duration, the error and error_code on failure, and an identity block with the verified subject, tenant and roles. It never writes the token or the full claim set. With include_args=False and include_result=False it writes no argument values and no result text.

  • `CLEAR_FIELDS` is the policy: which arguments stay readable. Ids and dates you need to find records; everything else becomes hmac:…. A field you forget to list is digested, so the safe failure is "not readable", not "stored in clear".

  • `_write_digests()` reads the validated arguments the server hands middleware (ctx.state["_tool_arguments"]), digests them and the result, and signs the whole line, so a digest can't be swapped for another.

  • `verify_trail()` checks both files: the chain with Promptise's verify_audit_log, each digest line's signature, and that the chain and the digests list the same request ids. The chain makes the set of calls tamper-evident; the request-id cross-check extends that to the digest file.


[07]

Step 2: Put it on the document extraction server

The example is an AI document extraction audit trail. Two companies, Acme and Globex, share one extraction service. Its tools read an invoice, save the extracted fields and, for admins, delete a document:

Pythondocs_server.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
"""Document extraction tools for two tenants, with a content-free audit trail."""
import os

from promptise.mcp.server import (
    AuthMiddleware,
    HasRole,
    JWTAuth,
    MCPServer,
    RequestContext,
    ToolError,
)
from trail import TenantAuditTrail

server = MCPServer("docs", require_tenant=True)
server.add_middleware(AuthMiddleware(JWTAuth(secret=os.environ["DOCS_JWT_SECRET"]), tenant_claim="org"))
server.add_middleware(TenantAuditTrail("audit"))  # after auth: it knows who is calling

DOCUMENTS = {
    "acme": {
        "INV-1001": "Invoice INV-1001 from Northwind Supplies, contact Jane Doe "
                    "(jane.doe@northwind.example). Total due: 1,240.00 EUR. Due date: 2026-11-15.",
    },
    "globex": {
        "INV-2001": "Invoice INV-2001 from Contoso Freight, contact Raj Patel "
                    "(raj@contoso.example). Total due: 18,900.00 USD. Due date: 2026-12-01.",
    },
}
EXTRACTIONS: dict[tuple[str, str], dict] = {}


def tenant_docs(ctx: RequestContext) -> dict:
    return DOCUMENTS.get(ctx.client.tenant_id, {})


@server.tool(read_only_hint=True)
async def read_document(document_id: str, ctx: RequestContext) -> str:
    """Return the text of one of your organisation's documents."""
    text = tenant_docs(ctx).get(document_id)
    if text is None:
        raise ToolError(f"No document {document_id} in your organisation.")
    return text


@server.tool()
async def save_extraction(document_id: str, vendor: str, total: str, due_date: str,
                          ctx: RequestContext) -> dict:
    """Save the fields extracted from a document."""
    if document_id not in tenant_docs(ctx):
        raise ToolError(f"No document {document_id} in your organisation.")
    record = {"vendor": vendor, "total": total, "due_date": due_date}
    EXTRACTIONS[(ctx.client.tenant_id, document_id)] = record
    return {"saved": document_id, **record}


@server.tool(destructive_hint=True, guards=[HasRole("admin")])
async def delete_document(document_id: str, ctx: RequestContext) -> dict:
    """Delete a document. Admins only."""
    tenant_docs(ctx).pop(document_id, None)
    return {"deleted": document_id}


if __name__ == "__main__":
    server.run(transport="http", host="127.0.0.1", port=8781)

The order of middleware matters: AuthMiddleware runs first, so the trail always knows who called and for which tenant (tenant_claim="org" reads the tenant from the token's org claim). require_tenant=True refuses any call whose token names no tenant. The tools filter by ctx.client.tenant_id, so a model asking for another tenant's invoice gets "No document" rather than the document. Multi-Tenant AI Agents: Keep Each Customer's Data Apart explains that server pattern step by step.


[08]

Step 3: Run the agent with content capture off

The agent side has its own trace. By default a Promptise agent's observability records tool arguments and result previews (redacted for credentials and common PII), and at the FULL level also prompt and answer text. For a content-free trace, turn both off:

Pythonagent_demo.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
"""Two tenants use one extraction agent. The trace keeps no prompt or document text."""
import asyncio
import json
import os
import socket
import shutil
import subprocess
import sys
import time

from dotenv import load_dotenv

from promptise import CallerContext, ObservabilityConfig, ObserveLevel, build_agent
from promptise.config import HTTPServerSpec
from promptise.mcp.server import JWTAuth

load_dotenv()  # OPENAI_API_KEY
URL = "http://127.0.0.1:8781/mcp"
issuer = JWTAuth(secret=os.environ["DOCS_JWT_SECRET"])  # stands in for your identity provider


def sign_in(user: str, org: str, roles: list[str]) -> CallerContext:
    token = issuer.create_token({"sub": user, "org": org, "roles": roles})
    return CallerContext(user_id=user, tenant_id=org, bearer_token=token)


def start_server() -> subprocess.Popen:
    shutil.rmtree("audit", ignore_errors=True)
    proc = subprocess.Popen([sys.executable, "docs_server.py"], stdout=subprocess.DEVNULL,
                            stderr=subprocess.DEVNULL)
    for _ in range(50):
        try:
            socket.create_connection(("127.0.0.1", 8781), timeout=1).close()
            break
        except Exception:
            time.sleep(0.2)
    return proc


async def main() -> None:
    server = start_server()
    agent = await build_agent(
        model="openai:gpt-5-mini",
        servers={"docs": HTTPServerSpec(url=URL, bearer_token=issuer.create_token({"sub": "extractor"}))},
        instructions="You process invoices. To extract one, read it, then save vendor, total "
                     "(amount and currency) and due_date (YYYY-MM-DD). For other requests, use the "
                     "matching tool. Reply in one short sentence.",
        observe=ObservabilityConfig(
            level=ObserveLevel.STANDARD,
            record_prompts=False,   # no prompt or answer text in the trace
            record_tool_io=False,   # no tool arguments or results, only their lengths
            redact_sensitive=True,  # the default: credentials and common PII scrubbed anyway
        ),
    )
    dana = sign_in("dana@acme.example", "acme", ["clerk"])
    li = sign_in("li@globex.example", "globex", ["clerk"])
    try:
        for caller, question in [
            (dana, "Extract INV-1001."),
            (li, "Extract INV-2001."),
            (dana, "Delete document INV-1001."),
        ]:
            result = await agent.ainvoke({"messages": [{"role": "user", "content": question}]}, caller=caller)
            print(f"{caller.tenant_id}: {question}\n  -> {result['messages'][-1].content}")

        print("\nTrace events (tool calls):")
        for entry in agent.collector.get_timeline():
            kind = getattr(entry.event_type, "value", entry.event_type)
            if kind in ("tool.call", "tool.result", "tool.error"):
                meta = {k: v for k, v in entry.metadata.items()
                        if k in ("tool_name", "arguments_length", "result_length", "arguments", "result")}
                print(f"  {kind:<11} {json.dumps(meta)}")
    finally:
        await agent.shutdown()
        server.terminate()


asyncio.run(main())

One agent serves both tenants. Each request carries a CallerContext with the user's token, which the agent sends to the MCP server on every tool call, so the server attributes each call to Dana at Acme or Li at Globex. The run:

Output
acme: Extract INV-1001.
  -> Extracted and saved: vendor "Northwind Supplies", total "1,240.00 EUR", due_date "2026-11-15".
globex: Extract INV-2001.
  -> Saved extraction for INV-2001: vendor=Contoso Freight, total=18,900.00 USD, due_date=2026-12-01.
acme: Delete document INV-1001.
  -> I can't delete INV-1001 because I lack the required admin role; please ask an admin to delete it or grant the admin role.

Trace events (tool calls):
  tool.call   {"tool_name": "read_document", "arguments_length": 27}
  tool.result {"result_length": 135, "tool_name": "read_document"}
  tool.call   {"tool_name": "save_extraction", "arguments_length": 110}
  tool.result {"result_length": 104, "tool_name": "save_extraction"}
  tool.call   {"tool_name": "read_document", "arguments_length": 27}
  tool.result {"result_length": 127, "tool_name": "read_document"}
  tool.call   {"tool_name": "save_extraction", "arguments_length": 108}
  tool.result {"result_length": 102, "tool_name": "save_extraction"}
  tool.call   {"tool_name": "delete_document", "arguments_length": 27}
  tool.error  {"tool_name": "delete_document"}

Dana's extraction and Li's extraction ran. Dana's delete was refused by the HasRole("admin") guard. The agent's trace kept the tool names and the lengths of arguments and results, nothing else: no invoice text, no extracted values, no prompt.


[09]

Step 4: Read what the trail kept

Here are two of Acme's chain entries, the extraction and the refused delete:

Output
{"timestamp": 1792740226.2760189, "tool": "save_extraction", "client_id": "dana@acme.example", "request_id": "01de0a84f73d", "status": "ok", "duration_s": 0.0001, "identity": {"subject": "dana@acme.example", "tenant_id": "acme", "roles": ["clerk"]}, "seq": 1, "prev_hash": "9550ccf44ba62dd9402a6b9253cf2bfc91a194a7896176bb36738e3ce05f0325", "hmac": "97f08271a812dffa1d5a347a018a34a39d098867877b925cac9bbe971b56bfab"}
{"timestamp": 1792740231.93259, "tool": "delete_document", "client_id": "dana@acme.example", "request_id": "30014211932a", "status": "error", "duration_s": 0.0001, "error": "AuthenticationError: Requires any of roles [admin], but client has [clerk]", "error_code": "ACCESS_DENIED", "identity": {"subject": "dana@acme.example", "tenant_id": "acme", "roles": ["clerk"]}, "seq": 2, "prev_hash": "97f08271a812dffa1d5a347a018a34a39d098867877b925cac9bbe971b56bfab", "hmac": "f042ef27ddea2db53a129ad5f9cd6ba3e1d8ec5cea0a258246e1271440422c12"}

And the matching digest lines:

Output
{"request_id": "01de0a84f73d", "tool": "save_extraction", "args": {"document_id": "INV-1001", "due_date": "2026-11-15", "total": "hmac:762efd7610f1345145836a58d47a44b1", "vendor": "hmac:05f3ddd77bd43b64687e3486a9a6e20b"}, "result": "hmac:ad2915b3fe953a806a09eaa6ad9b21aa", "mac": "e7fc643821251b5c3782daa8f32b09e99de8fcc7d5fc070b559b2ae59d9c3bff"}
{"request_id": "30014211932a", "tool": "delete_document", "args": {"document_id": "INV-1001"}, "result": null, "mac": "b540d70c447ac7eac718e2f9eee1d02d801bcf964d9751cc426cda8149b44507"}

Together they say: Dana, a clerk at Acme, saved an extraction for INV-1001 due on 2026-11-15, then tried to delete INV-1001 and was refused for lacking the admin role. They don't say who the vendor was or how much the invoice was for, and the chain holds no document text. Notice that the refusal is a full entry with error_code: ACCESS_DENIED. A trail that only records successes can't show that a control worked.


[10]

Step 5: Verify, check a claim, and tamper with a copy

The verification script checks both tenants, runs the same check with the promptise audit verify CLI, tests candidate values against a stored digest, and then edits copies of Acme's trail:

Output
1. Verify both tenants
  acme: intact
  globex: intact

2. The same check with the CLI, as the tenant would run it with their key
  OK 3 entries: intact.
last_hash f042ef27ddea2db53a129ad5f9cd6ba3e1d8ec5cea0a258246e1271440422c12 (seq 2) (exit 0)

3. Which total did the agent save? Test candidate values against the digest
  stored digest: hmac:762efd7610f1345145836a58d47a44b1
  '1,240.00 EUR'  match
  '1240.00 EUR'   no match
  '1240 EUR'      no match

4. Tamper with copies of acme's trail
  turn the denied delete into a success: ["audit-copy/acme/chain.jsonl:3: bad_signature: the entry's signature does not match: it was modified or forged"]
  drop the digest line of the denied delete: ['request 30014211932a: in the chain but its digest line is missing']
  swap the saved total for another digest: ['digests.jsonl:2: signature does not match (edited or forged)']

5. Acme's key does not verify Globex's chain
  ok=False: audit/globex/chain.jsonl:1: wrong_key: no entry verifies with the key given: wrong key

Part by part:

  1. Both trails verify. Every MAC matches, every link holds, and both files list the same request ids.

  2. The tenant can verify alone. The CLI reads the key from an environment variable (never from the command line) and exits with 0 when intact, 1 when tampered. That is what you hand a tenant: its files, its key, and this command. The last_hash it prints is the anchor to store elsewhere.

  3. A digest answers a precise question. The saved total was exactly 1,240.00 EUR. 1240.00 EUR doesn't match, because a digest compares exact bytes. Normalize values before you digest them (amounts as integer cents, dates as ISO 8601) or every check needs the original formatting.

  4. Tampering shows. Turning the refused delete into a success breaks that entry's signature. Dropping its digest line leaves a request in the chain with no digest. Swapping the stored total for another digest breaks that line's signature.

  5. Keys separate tenants. Acme's key verifies none of Globex's entries.

The script's tamper helper is a few lines. It copies the trail and rewrites one file:

Python
1
2
3
4
5
6
7
8
9
def tampered(name: str, edit) -> None:
    shutil.rmtree("audit-copy", ignore_errors=True)
    shutil.copytree("audit", "audit-copy")
    edit()
    print(f"  {name}: {verify_trail('acme', root='audit-copy')}")

tampered("turn the denied delete into a success", lambda: rewrite(
    "audit-copy/acme/chain.jsonl",
    lambda ls: [l.replace('"status": "error"', '"status": "ok"') for l in ls]))

[11]

Where content leaks back in

The trail above holds no content. The rest of the system still can, and these are the usual places:

Place

What it holds by default

Content-free setting

Agent observability (ObservabilityConfig)

Tool arguments and result previews; prompt text at FULL

record_tool_io=False, record_prompts=False; keep redact_sensitive=True

AuditMiddleware

No arguments, no results

Keep include_args=False, include_result=False

StructuredLoggingMiddleware

No arguments

Keep include_args=False (if on, arguments pass through redaction)

Error messages

Whatever the tool put in them

Don't put argument values or content in ToolError messages; the chain stores error text

Runtime journal (JournalConfig)

Conversation in checkpoints; at full, tool arguments and results with secrets redacted

Treat it as a recovery store inside your boundary, not as the audit trail

OpenTelemetry GenAI instrumentation

Message content off by default

Leave OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT unset in production

Model provider

Its own logs and retention

Check the provider's data controls

Two of these deserve a note. The runtime journal exists so a crashed agent can resume where it was, which requires the conversation; it is not content-free and isn't meant to be. And error messages are easy to forget: in the demo, read_document raises No document {document_id}, which is fine because the id is in clear anyway, but ToolError(f"Invalid IBAN {iban}") would put the value into the chain.

Pattern-based redaction (redact_sensitive, the guardrails' PII detectors) catches credentials, emails and card numbers, not names or free text. That is why the trail digests everything not explicitly allowed rather than redacting what looks sensitive. AI-Based PII Detection and Redaction Tools for LLM Apps and Stop Secrets Leaking Into LLM Logs and Traces cover redaction in depth.


[12]

Running a content-free audit trail in production

  • Keep the master key out of the writer's reach where you can. Anyone with the key and write access can rebuild a valid chain from scratch. Keep the key in a secret manager, give the server only what it needs, and ship closed files to storage the server can't rewrite (object storage with a retention lock).

  • Anchor the end. Store each verification's last_hash somewhere the server can't write, such as a ticket, another account or your SIEM. Verifying later with --anchor shows entries cut from the end.

  • One writer per file. Two processes appending to one chain file fork the chain. With several server replicas, give each replica its own file per tenant (for example audit/acme/replica-2.jsonl).

  • Rotate keys deliberately. After rotating, verification needs both keys for the overlapping file (verify_audit_log(path, [new_key, old_key])). Keep old derived keys until the files they signed age out.

  • Normalize before digesting. Agree on one canonical form per field type, or a digest check fails on formatting alone.

  • Verify on a schedule. Run promptise audit verify in CI or a cron job and alert on a non-zero exit. A trail nobody verifies is only a log.

  • Plan deletion. If you must delete one tenant's records, per-tenant files make it a file-level operation that leaves other tenants' chains intact. Deleting single entries from a chain breaks it by design; export and record what you removed instead.

  • Watch the overhead. The middleware adds one HMAC and one appended line per call per file, which is small next to an LLM call. AI Agent Observability: Trace Tool Calls, Tokens and Cost covers the debugging traces you keep alongside it.


[13]

Common problems and fixes

Symptom

Cause

Fix

ValueError: a signed audit log file needs a key you keep

AuditMiddleware(log_path=...) with no hmac_secret and no PROMPTISE_AUDIT_SECRET

Pass the tenant key as hmac_secret, as in trail.py

Entries have no identity or tenant_id

Audit middleware added before AuthMiddleware, or the token has no tenant claim

Add auth first; set tenant_claim to your identity provider's claim; use require_tenant=True

A digest check fails for the right value

Different formatting (1240.00 vs 1,240.00)

Normalize values before digesting, and digest the normalized form

wrong_key for a tenant's own file

Master key changed, or the tenant id differs (Acme vs acme)

Derive with the exact tenant id; keep old keys after rotation

Chain verifies but digests report missing lines

The digest file was edited, or the server crashed between the two writes

Treat as tampering until explained; the server may have crashed between the chain write and the digest write

Content shows up in traces

record_tool_io left at its default, or FULL level

Set record_tool_io=False and record_prompts=False

Values appear in error fields

Tool error messages include arguments

Keep error messages generic; refer to ids, not values


[14]

Limitations

  • A digest can only confirm a value you already have. It can't tell you what the value was. If an investigation will need content, keep it in the system of record under its own controls, or encrypt it under a key the audit writer doesn't hold.

  • The audit sees tool calls, not reasoning. It records what the agent did, not why. The why lives in prompts, which this design deliberately leaves out.

  • Requests rejected before the MCP layer aren't in the chain. A token that fails verification over HTTP is refused with 401 by the transport before middleware runs; it is in the HTTP access log.

  • The digest file is a sidecar of your own. AuditMiddleware records arguments in full or not at all, so the keyed digests in this guide come from your middleware, not from a built-in option.

  • Key custody decides how strong it is. The chain proves the trail is unchanged since someone with the key wrote it. Who holds the key is an operational decision, not something code can settle.


[15]

FAQ

How do you audit an AI agent without storing prompts?

Record each tool call at the tool server: who called (from the verified token), which tool, the outcome and a request id, chained with an HMAC. Store arguments and results as keyed digests, with only identifiers in clear, and turn off prompt and tool input/output capture in the agent's own traces. The trail then shows every action without holding the conversation.

What makes an audit log independently verifiable?

Someone other than the writer can check it with nothing but the files and a key: every entry's MAC verifies, every entry links to the one before it, and the last hash matches an anchor stored elsewhere. Giving each tenant its own derived key lets that tenant verify its own trail without access to anyone else's. Public, keyless verification needs signatures or a transparency log instead of HMACs.

Why use an HMAC instead of a SHA-256 hash for arguments?

Agent arguments are often short and predictable: amounts, dates, names. Anyone can hash candidate values and compare them with a plain SHA-256 digest, which reveals the value. An HMAC needs the key, so only the key holder can test a candidate.

What should an AI document extraction audit trail record?

For each step: the user and tenant, the document id, the tool (read, extract, save, delete), the outcome including refusals, and digests of the extracted fields and of what the tool returned. That lets you show who extracted what from which document and when, and confirm a field's value on request, without the trail holding document text or extracted values.

Is an audit trail the same as observability tracing?

No. Traces are for debugging and performance; they are sampled, change shape often and may hold content in development. An audit trail is complete, append-only, tamper-evident and stable, so someone can rely on it later. Many teams keep both, with different retention and access rules.

Does Promptise store prompt content by default?

Not in the MCP server's AuditMiddleware, which records no arguments or results unless you ask it to. The agent's observability records tool arguments and results by default (with credential and PII redaction) and prompt text only at the FULL level; set record_tool_io=False and record_prompts=False for a content-free trace. The runtime journal stores conversation state for recovery.


[16]

Next steps

  • AI Agent Audit Logging: Tamper-Evident Logs in Python for anchors, restarts, rotation and all six kinds of tampering.

  • Multi-Tenant AI Agents: Keep Each Customer's Data Apart for the tenant-scoped server this trail sits on.

  • MCP Middleware for Servers: Auth, Logging, Rate Limits for middleware order and the other built-ins.

  • AI Agent Identity: Authenticate Agents to MCP Servers for where the "who" in each entry comes from.

  • Best AI Agent Observability Tools (2026), Compared for the debugging side.

  • Reference: MCP server observability and audit logging and agent observability in the Promptise docs.

Contents

  1. 01How do you make audit-grade agent traces without storing prompt content?
  2. 02What should an AI agent audit record contain?
  3. 03How content-free audit trails work
  4. 04When you need this (and when not)
  5. 05What you need
  6. 06Step 1: Build the per-tenant audit trail
  7. 07Step 2: Put it on the document extraction server
  8. 08Step 3: Run the agent with content capture off
  9. 09Step 4: Read what the trail kept
  10. 10Step 5: Verify, check a claim, and tamper with a copy
  11. 11Where content leaks back in
  12. 12Running a content-free audit trail in production
  13. 13Common problems and fixes
  14. 14Limitations
  15. 15FAQ
  16. 16Next steps

Learning paths

Want more structure? Paths put guides in order, like a short course.

See the paths →

> Observability and debugging

Related guides

All guides in this topic →
  • // 01Intermediate · 18 minAI Agent Observability: Trace Tool Calls, Tokens and CostTrace every LLM call and tool call your AI agent makes, with tokens, latency and cost, then use the trace to find the exact step that went wrong.
  • // 02Intermediate · 20 minBest AI Agent Observability Tools (2026), ComparedThe best AI agent observability tools in 2026 compared: LangSmith, Langfuse, Phoenix, Datadog and more on tool tracing, cost, evals, PII, OTel and price.
  • // 03Intermediate · 18 minAI Agent Stuck in a Loop? How to Detect and Stop ItWhy AI agents get stuck in loops and how to stop them: detect repeated tool calls, cap tool calls per run, and write tool results that end the loop.
  • // 04Intermediate · 20 minMCP Server Monitoring: Prometheus, OpenTelemetry and LogsMonitor an MCP server in Python: Prometheus metrics, JSON logs, OpenTelemetry traces and alerts that show which tools are slow, failing or abused.
  • // 05Beginner · 15 minHow to Run, Debug and Deploy an AI Agent in PythonRun, debug and deploy a Python AI agent project with one command: a package.json-style manifest, live traces, a terminal UI, JSON logs and an API.
  • // 06Intermediate · 16 minAI Agent Cost Tracking: Tokens and Cost per Run and UserTrack AI agent token usage and cost per model call, run, user and agent: price tables, cost ledgers, alerts and limits. Runnable code with real output.

Keep going.

Browse every guide by topic and level, or follow a learning path that puts them in order.

All guidesLearning paths