Source Provenance for Research Agents: Track Each Claim

Make claims a data structure instead of prose. Log every source the agent reads, with its URL, time and a content hash. Have the agent submit its answer as a list of claims, each with a source ID and an exact quote from that source. Check each claim in code: the source was read, it's on your list of acceptable sources, the quote is on the page, and the numbers in the claim appear in the quote. Reject the report with reasons if any check fails, and build the final answer from the accepted claims only.

• Citations made after the fact are guesses. Bind each claim to its source when the agent makes it, not by matching text to sources later.

• The read log is the ground truth. A claim may only cite a source the agent fetched in this run.

• Exact quotes are cheap to verify. A substring check catches invented quotes and many changed numbers without another model call.

• Policy belongs in code. In the example, the model was told to skip a source with no origin and cited it anyway. The validator rejected it.

• Store the evidence with the answer so you can re-check it when the pages change.

A research agent reads whatever ranks for its query, and some of what ranks is written to be cited. In October 2026, a researcher at Peec AI described building a site with 50,000 statistics pages whose numbers were invented by a model and never checked (the write-up). Google deindexed most of it, but ChatGPT kept retrieving the pages, and according to the author the invented numbers were cited in an academic paper and in investment research. He calls the effect citation laundering: a made-up number gains authority each time a real publication repeats it.

Agents get citations wrong in three ways:

1. Invented source. The agent cites a URL, paper or page it never read, or that doesn't exist.

2. Real source, wrong claim. The page exists and was read, but it doesn't say what the agent says it does. Numbers drift: 2.4 million becomes 2.5 million, "monthly" becomes "weekly".

3. Real source, no origin. The page says what the agent says, but the page itself has no source for it. A blog citing a statistics site that cites nobody is one source, not two.

A link at the end of an answer protects against none of these. A reader sees a citation and assumes the work was done.

The agent works as usual: it searches, reads and reasons. Two things change. Every read goes into a log, and the answer goes through a validator before anyone sees it.

The validator runs two kinds of checks:

• Structural checks are deterministic and nearly free: was the cited source read in this run, is its domain acceptable, is the quote on the page, do the claim's numbers appear in the quote. They catch invented sources, invented quotes and most number drift.

• Semantic checks ask whether the quote actually supports the claim. A claim can copy a real sentence and still overstate it. This needs a model as judge, costs a call per claim, and is less reliable. Run it on paraphrased claims or on a sample.

When a check fails, the tool returns the reasons to the model, and the model fixes or drops the claim. The model proposes; code decides what reaches the reader.

Build it when:

• the agent's answers feed decisions, reports or published content;

• the answers contain numbers, dates, names or quotes that someone might repeat;

• the agent reads the open web, where pages are written to be picked up by AI systems;

• you need to show later where an answer came from, for review or audit.

You can skip it for answers drawn from one trusted document the user provided, or for brainstorming and drafting, where nobody treats the output as fact. For simple question answering over your own documents, citing a document ID or section may be enough; vectorless RAG shows node-level citations.

• Python 3.10 or later and Promptise Foundry 1.6.0. Foundry has no claim ledger built in; you build it from three tools declared with @tool, a Pydantic model for claims, and ToolError for rejections.

• An OpenAI API key (the example uses gpt-5-mini).

• A search source. The example uses six fictional pages in corpus.json so it runs offline and the results are reproducible: two from a city transport authority, one from a university survey, one news article, one statistics site with no sources, and one blog that repeats the statistics site.

uv venv && uv pip install "promptise==1.6.0"

search_sources returns titles and publishers, not text, so the agent has to read a source before it can use it. read_source returns the full text and writes an entry to the read log: URL, publisher, publication date, retrieval time, a hash of the content, and the source's trust tier.

The tier comes from your own policy, keyed by domain. Here three domains are trusted and everything else can be read but not cited. In production this list is longer and maintained by people; the point is that it lives in code, where the model can't argue with it.

"""Research tools that keep a provenance ledger: what was read, and which source backs each claim."""
import hashlib
import json
import re
from datetime import datetime, timezone
from urllib.parse import urlparse

from pydantic import BaseModel, Field

from promptise import ToolError, tool

SOURCES = {s["id"]: s for s in json.load(open("corpus.json"))}  # stand-in for web search
LEDGER = {"read": {}, "report": None}  # one research run

# Your team's source policy, by domain. Other sites can be read, never cited.
TRUSTED = {
    "lanport-transport.example": "official",
    "lanport-university.example": "research",
    "lanport-courier.example": "news",
}


class Claim(BaseModel):
    text: str = Field(description="One factual statement, as it will appear in the answer.")
    source_id: str = Field(description="ID of the source you read that states it.")
    quote: str = Field(description="The exact sentence from that source that supports the claim.")


def _norm(text: str) -> str:
    return " ".join(text.lower().split())


def _numbers(text: str) -> set[str]:
    return {n.replace(",", "").rstrip(".") for n in re.findall(r"\d[\d,.]*", text)}


@tool
def search_sources(query: str) -> str:
    """Search for sources. Returns IDs, titles, publishers and dates, not full text.

    Args:
        query: Keywords to search for.
    """
    words = set(_norm(query).split())
    hits = sorted(SOURCES.values(), key=lambda s: -len(words & set(_norm(s["title"] + " " + s["text"]).split())))
    return "\n".join(f'{s["id"]}: "{s["title"]}", {s["publisher"]}, {s["published"]}' for s in hits[:6])


@tool
def read_source(source_id: str) -> str:
    """Read a source in full. Only facts from sources you read can be cited.

    Args:
        source_id: An ID from search_sources.
    """
    source = SOURCES.get(source_id)
    if source is None:
        raise ToolError(f"No source {source_id!r}.")
    LEDGER["read"][source_id] = {
        "url": source["url"], "tier": TRUSTED.get(urlparse(source["url"]).hostname, "untrusted"), "publisher": source["publisher"], "published": source["published"],
        "retrieved_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        "sha256": hashlib.sha256(source["text"].encode()).hexdigest()[:16],
    }
    return f'{source["title"]} ({source["publisher"]}, {source["published"]})\n{source["text"]}'


@tool
def submit_report(claims: list[Claim]) -> str:
    """Submit the final answer as a list of claims. Every claim needs a trusted source
    you read and an exact supporting quote.

    Args:
        claims: The claims that make up the answer.
    """
    errors = []
    for i, claim in enumerate(claims, 1):
        if claim.source_id not in LEDGER["read"]:
            errors.append(f"claim {i}: you did not read {claim.source_id!r}; read it or drop the claim")
        elif LEDGER["read"][claim.source_id]["tier"] == "untrusted":
            errors.append(f"claim {i}: {claim.source_id!r} is not a trusted source; "
                          "cite the original or drop the claim")
        elif _norm(claim.quote) not in _norm(SOURCES[claim.source_id]["text"]):
            errors.append(f"claim {i}: the quote is not in {claim.source_id!r}; copy it exactly")
        elif missing := _numbers(claim.text) - _numbers(claim.quote):
            errors.append(f"claim {i}: {sorted(missing)} not in the quote; fix the claim or the quote")
    if errors:
        raise ToolError("Report rejected:\n" + "\n".join(errors))
    LEDGER["report"] = [c.model_dump() for c in claims]
    return f"Report accepted with {len(claims)} claims."

Three design choices matter in this file:

• `search_sources` doesn't return snippets. If it did, the agent could cite a snippet it saw in a result list without reading the page, and the snippet might be out of date or out of context.

• The hash makes the log useful later. A claim is true of a page as it was when the agent read it. The hash tells you whether the page is still the same.

• The claim schema is small. text, source_id and quote. Fields the model fills in about provenance itself, such as "who originally produced this number", turned out to be unreliable, as Step 3 shows.

submit_report is the last tool the agent calls. It takes a list of Claim objects; Foundry turns the Pydantic model into the tool's JSON schema, so the model sees field descriptions and the arguments are validated before the function runs. The function then runs four checks per claim:

1. The source is in the read log.

2. The source's tier isn't "untrusted".

3. The quote appears in the source text, ignoring case and whitespace.

4. Every number in the claim appears in the quote.

If any claim fails, the tool raises ToolError with one line per failure. Foundry sends that to the model as a tool error, and the model can fix the listed claims and submit again. If all pass, the report is stored in the ledger.

The number check is deliberately strict. It treats "2.4 million" in the claim and "2.4 million" in the quote as a match, but rejects a claim that adds a year the quote doesn't contain. That costs some retries, as the run below shows, and it catches the most common drift in research answers: a number changed on the way from source to summary.

The agent gets the three tools and instructions about source quality. The instructions still matter: they tell the model what good research looks like. The validator makes sure the rules that must hold do hold.

"""A research agent that must back every claim with a source it read and an exact quote."""
import asyncio
import json

from promptise import build_agent
from research_tools import LEDGER, read_source, search_sources, submit_report

INSTRUCTIONS = """You research questions using search_sources and read_source.
Rules for sources:
- Prefer the organization that produced a fact (its report, survey or press release)
  over pages that repeat it. Read the original before you cite a number.
- When sources disagree, prefer the newer original and say which date the figure is from.
- A number whose source names no origin is unverified: leave it out.
Finish by calling submit_report with one claim per fact, written in your own words
for the reader; the quote holds the source's exact words. If it is rejected, fix the
claims it lists and submit again. After it is accepted, reply with "Done"."""

QUESTION = "How widely is Lanport's bike share used? Give the key usage numbers."


def render(report: list[dict]) -> str:
    """Number the sources and attach them to the claims, like footnotes."""
    numbers: dict[str, int] = {}
    lines = []
    for claim in report:
        n = numbers.setdefault(claim["source_id"], len(numbers) + 1)
        lines.append(f"- {claim['text']} [{n}]")
    lines.append("\nSources:")
    for source_id, n in numbers.items():
        s = LEDGER["read"][source_id]
        lines.append(f"[{n}] {s['publisher']} ({s['tier']}), {s['published']}, {s['url']}, "
                     f"sha256 {s['sha256']}")
    return "\n".join(lines)


async def main() -> None:
    agent = await build_agent(
        model="openai:gpt-5-mini",
        instructions=INSTRUCTIONS,
        extra_tools=[search_sources, read_source, submit_report],
        max_agent_iterations=20,
    )
    try:
        result = await agent.ainvoke({"messages": [{"role": "user", "content": QUESTION}]})
    finally:
        await agent.shutdown()
    for message in result["messages"]:
        for call in getattr(message, "tool_calls", None) or []:
            args = call["args"] if call["name"] != "submit_report" else f"{len(call['args']['claims'])} claims"
            print(f"-> {call['name']}({args})")
        if message.type == "tool" and message.name == "submit_report":
            print("   " + message.content.replace("\n", "\n   "))
    if LEDGER["report"]:
        print("\n" + render(LEDGER["report"]))
        with open("provenance.json", "w") as f:  # keep the evidence with the answer
            json.dump({"question": QUESTION, **LEDGER}, f, indent=1)


if __name__ == "__main__":
    asyncio.run(main())

One real run:

-> search_sources({'query': "Lanport bike share usage Lanport bike share 'Lanport' 'bike share' usage numbers 'Lanport bike' 'Lanport bike-share'"})
-> read_source({'source_id': 'lta-press-2026-03'})
-> read_source({'source_id': 'lta-report-2025'})
-> read_source({'source_id': 'uni-survey-2026'})
-> read_source({'source_id': 'statsworld'})
-> submit_report(8 claims)
   {
     "error": {
       "code": "TOOL_ERROR",
       "message": "Report rejected:\nclaim 3: ['2025'] not in the quote; fix the claim or the quote\nclaim 4: ['2025'] not in the quote; fix the claim or the quote\nclaim 5: ['1'] not in the quote; fix the claim or the quote\nclaim 6: ['2026'] not in the quote; fix the claim or the quote\nclaim 7: ['2026'] not in the quote; fix the claim or the quote",
       "retryable": false
     }
   }
-> submit_report(10 claims)
   Report accepted with 10 claims.
-> submit_report(12 claims)
   {
     "error": {
       "code": "TOOL_ERROR",
       "message": "Report rejected:\nclaim 10: 'statsworld' is not a trusted source; cite the original or drop the claim\nclaim 11: 'statsworld' is not a trusted source; cite the original or drop the claim\nclaim 12: 'statsworld' is not a trusted source; cite the original or drop the claim",
       "retryable": false
     }
   }

- Riders took 2.4 million trips on Lanport Bike Share in 2025, up from 1.7 million in 2024. [1]
- The average trip was 3.1 km long. [1]
- Electric bikes made up 62% of the fleet at the end of the year. [1]
- Station count rose from 210 to 265. [1]
- As of March 2026, Lanport Bike Share has 4,200 bikes in service, after 600 electric bikes were added in the first quarter. [2]
- The authority plans 40 new stations in 2026. [2]
- We surveyed 1,200 adult residents in March 2026. [3]
- 31% said they had used the bike share at least once in the past month. [3]
- 9% said they use the bike share every week. [3]
- Among weekly users, 58% use it to commute. [3]

Sources:
[1] Lanport Transport Authority (official), 2026-02-14, https://lanport-transport.example/reports/2025-annual-report, sha256 e7e6286bb5d15261
[2] Lanport Transport Authority (official), 2026-03-30, https://lanport-transport.example/news/fleet-update-march-2026, sha256 e8133c8d595a3304
[3] Lanport University, Department of Geography (research), 2026-05-02, https://lanport-university.example/mobility/survey-2026, sha256 55cc50fa76972774

Read the trace from the top:

• The agent read four sources, including the statistics site, and skipped the news article and the blog because it had the originals.

• The first report was rejected for five claims: they added years and other numbers that weren't in their quotes. The agent rewrote them to match their sources and submitted again.

• The second report was accepted with ten claims from the transport authority and the university.

• Then the agent tried to add three claims from the statistics site, the page that gives no source for its numbers (its headline figure is "78% of Lanport commuters use the bike share every week"). The validator rejected all three because the site isn't trusted. The accepted report stayed as it was.

That last step is the reason for putting policy in code. In a first version of this example, the "no origin, leave it out" rule existed only in the instructions, and the claim schema had an origin field for the model to fill in. The model cited the 78% figure anyway and gave "StatsWorld" as its origin. Both the prompt and the model's self-reported provenance failed; the domain check didn't.

Also note what the final answer says: 31% of surveyed residents used the bike share in the past month and 9% use it weekly. That is what the survey found. The 78% figure contradicts it, and a reader of an unchecked answer would have had no way to tell which one to believe.

The answer the reader sees is built by render() from the accepted claims, not written by the model as free text. Each claim gets a footnote number, and each source gets a line with its publisher, tier, date, URL and hash. Nothing reaches the reader that didn't pass the validator.

This trades fluency for traceability. If you want flowing prose, have the model write it from the accepted claims in a second step, then check that every sentence maps to a claim ID. Don't let the model add facts in that step.

agent.py also writes provenance.json: the question, the read log and the accepted claims. Store it next to the answer.

Pages change after an agent reads them. Figures are corrected, articles updated, pages deleted. With the quotes and hashes stored, you can re-check an old answer without running the agent again:

"""Re-check a stored answer later: is every quote still on its page?"""
import hashlib
import json

record = json.load(open("provenance.json"))
pages = {s["url"]: s["text"] for s in json.load(open("corpus.json"))}  # fetch the URLs again
pages["https://lanport-transport.example/news/fleet-update-march-2026"] = (  # simulate an edit
    "As of June 2026, Lanport Bike Share has 4,350 bikes in service.")

for claim in record["report"]:
    source = record["read"][claim["source_id"]]
    text = pages[source["url"]]
    changed = hashlib.sha256(text.encode()).hexdigest()[:16] != source["sha256"]
    still_there = " ".join(claim["quote"].lower().split()) in " ".join(text.lower().split())
    status = "ok" if still_there else "QUOTE GONE"
    print(f"{status:10} {'page changed' if changed else 'unchanged':12} {claim['text'][:60]}")

The script simulates an update to the transport authority's press release, as if the fleet count had been revised in June:

ok         unchanged    Riders took 2.4 million trips on Lanport Bike Share in 2025,
ok         unchanged    The average trip was 3.1 km long.
ok         unchanged    Electric bikes made up 62% of the fleet at the end of the ye
ok         unchanged    Station count rose from 210 to 265.
QUOTE GONE page changed As of March 2026, Lanport Bike Share has 4,200 bikes in serv
QUOTE GONE page changed The authority plans 40 new stations in 2026.
ok         unchanged    We surveyed 1,200 adult residents in March 2026.
ok         unchanged    31% said they had used the bike share at least once in the p
ok         unchanged    9% said they use the bike share every week.
ok         unchanged    Among weekly users, 58% use it to commute.

Both claims from the changed page are flagged, and the others are confirmed unchanged. Run this on a schedule for answers people still rely on, or when someone opens an old report.

The four files in runs/source-provenance-research-agents are the complete example: corpus.json stands in for web search, research_tools.py holds the ledger and the validator, agent.py runs the agent and renders the answer, and recheck.py re-verifies a stored answer. To use real web search, replace search_sources and read_source with your search API and page fetcher, and keep the ledger writes, the hash and the trusted-domain check. Sanitize fetched pages before the agent reads them; hidden text prompt injection shows how.

• Canonicalize URLs before you log them (strip tracking parameters, follow redirects once) so the same page has one ID.

• Store a snapshot, not only the hash. When a page changes or disappears, the stored text shows what the agent saw. A web archive link works too.

• Count origins, not pages. Two pages that repeat the same statistics site are one source. Where a claim needs corroboration, require two different origins.

• Keep the trust list owned by people. Review new domains the agent wants to cite, and log every rejection by domain to see which ones come up.

• Add a semantic check where it pays. For claims that paraphrase their quote, ask a second model whether the quote supports the claim, and sample its verdicts for human review. How to evaluate AI agents covers LLM judges and their limits.

• Show quotes in the interface. Let readers hover over a footnote and see the exact sentence. It's the cheapest way to let a person verify a claim.

• Cap retries. A model that can't satisfy the validator will keep submitting. max_agent_iterations in build_agent limits the loop; return a partial answer with the rejected claims listed, not a silent failure.

• Trace it. The read log and the rejections are part of the agent's trace. AI agent observability shows how to record tool calls.

• A quote can be real and the claim still misleading. Exact matching proves the words exist, not that the claim reads them correctly. Semantic checks help and aren't perfect.

• Trusted sources can be wrong. The validator moves the question from "did the agent invent this" to "do we trust this publisher", which is a better question but still a judgment.

• Exact quotes are hard for some content: tables, charts and PDFs with odd text extraction. Those need page and cell references instead of quotes.

• Strict checks cost retries and therefore tokens. Measure how often reports are rejected, and loosen the checks that reject correct claims.

• The answer reads like a list. Turning verified claims into prose without adding unverified facts takes a second, controlled step.

What is source provenance for AI agents?

It is a record, for each claim in an agent's answer, of which source the claim came from, what that source says word for word, and when the agent read it. It lets a person or a program check each claim instead of trusting the answer as a whole.

How do I stop an AI agent from citing fake statistics?

Make the agent cite each number with a source it read and an exact quote, check the quote in code, and only accept sources from domains you trust or sources that name where the number came from. Pages that repeat a statistic without an origin should not count as evidence.

How do I verify citations in an LLM answer?

First check structure: the source exists, the agent read it, and the quoted text is on the page. Then check meaning: whether the quote supports the claim, using a person or a judge model. Structural checks are cheap and catch most invented citations; semantic checks catch overstatements.

Should the model or the code decide which sources are trusted?

The code, based on a list your team maintains. Instructions help the model choose good sources, but in this guide's example the model cited an untrusted statistics site despite being told not to. A check in the submit tool made the rule hold.

Can Promptise Foundry track sources for each claim?

Not as a built-in feature. You build it with Foundry's @tool functions, a Pydantic schema for claims, and ToolError to send rejections back to the model, as in this guide. Foundry's observability can record the tool calls that make up the read log.

What is citation laundering?

It's when an unsupported or invented claim gains credibility because other sources repeat it. Each repetition looks like independent confirmation, but they all trace back to one origin with no evidence behind it.

• Grounding LLM answers covers giving the model the right context in the first place.

• LLM hallucinations explains why models produce confident, unsupported claims.

• Chain-of-Verification and SelfCheckGPT covers model-based verification loops.

• LLM output validation shows other checks to run before you use a response.

• Structured outputs covers getting reliable JSON from a model.

• How to check AI answers is the version of this guide for people reading AI output.