What Is Agentic RAG? How It Differs from Classic RAG

Agentic RAG is retrieval-augmented generation in which an AI agent controls retrieval. Instead of one fixed search before the answer, the agent decides whether to search, writes its own queries, can search several times or several sources, checks what it found, and answers only when it has enough evidence, citing its sources.

Key takeaways:

• Retrieval becomes a tool. In classic RAG your code always searches first. In agentic RAG the model calls a search tool when it decides it needs one.

• The agent can try again. If the first results miss part of the question, it rewrites the query or searches another source.

• It costs more per question. Every decision is a model call, so agentic RAG is slower and less predictable than one search and one answer.

• Retrieved text is still untrusted. A document can contain instructions aimed at the model, and an agent with tools can act on them.

The term covers a range of designs. The survey *Agentic Retrieval-Augmented Generation* (Singh et al., 2025) describes it as embedding autonomous agents into the RAG pipeline, using reflection, planning, tool use and collaboration between agents. Research systems such as Self-RAG (Asai et al., 2023) trained models to decide when to retrieve and to critique their own output. Most production systems today use the simpler version shown here: a tool-calling agent with a search tool.

Classic RAG, from the original paper by Lewis et al. (2020), is a fixed pipeline. Agentic RAG wraps the same retrieval step in an agent loop.

 classic RAG (fixed pipeline)
   question ──▶ embed + search once ──▶ top-k passages + question ──▶ model ──▶ answer

 agentic RAG (agent loop)
   question ──▶ model ──▶ "search_policies('monitor expense')" ──▶ passages
                  ▲                                                   │
                  └──────────── enough to answer? no: search again ◀──┘
                  │ yes
                  ▼
                answer with citations, or "the documents don't say"

The retrieval index itself doesn't change. You still load documents, split them into chunks, embed the chunks and store them in a vector store. What changes is who calls search. In classic RAG, your code runs it with the user's exact words as the query. In agentic RAG, the model writes the query, which helps with questions that use different words than the documents, combine several topics, or need a fact from one document to know what to look up in another.

An agent can also have more than one source: a vector search over docs, a SQL tool for numbers, a web search for recent events. Picking the right source per question is a large part of what makes the approach useful.

The example indexes five short policy documents with Promptise Foundry's RAG pipeline and gives the agent a search tool that returns the two best passages. The question has two parts, so one search of two passages can't cover it.

import asyncio

from langchain_core.tools import tool
from promptise import (
    Document,
    DocumentLoader,
    Embedder,
    InMemoryVectorStore,
    OpenAIEmbeddingProvider,
    RAGPipeline,
    RecursiveTextChunker,
    build_agent,
)

POLICIES = {
    "travel.md": "Travel. Hotels are reimbursed up to $180 per night in the US and "
                 "EUR 160 in the EU. Purchases made while traveling that are not travel "
                 "costs follow the policy for that kind of purchase.",
    "equipment.md": "Equipment. Laptops are ordered by IT, never expensed. Monitors "
                    "and keyboards can be expensed up to $300 per year.",
    "expenses.md": "Expense reports. Submit receipts within 30 days of purchase.",
    "leave.md": "Leave. Full-time staff get 25 days of paid leave per year.",
    "security.md": "Security. Report a lost device to IT within 24 hours.",
}


class DictLoader(DocumentLoader):
    async def load(self) -> list[Document]:
        return [Document(id=k, text=v, metadata={"source": k}) for k, v in POLICIES.items()]


class OpenAIEmbedder(Embedder):
    def __init__(self):
        self._provider = OpenAIEmbeddingProvider(
            model="text-embedding-3-small", api_key="${OPENAI_API_KEY}"
        )

    async def embed(self, texts: list[str]) -> list[list[float]]:
        return await self._provider.embed(texts)

    @property
    def dimension(self) -> int:
        return 1536


async def main():
    pipeline = RAGPipeline(
        loader=DictLoader(),
        chunker=RecursiveTextChunker(chunk_size=300, overlap=0),
        embedder=OpenAIEmbedder(),
        store=InMemoryVectorStore(dimension=1536),
    )
    await pipeline.index()

    @tool
    async def search_policies(query: str) -> str:
        """Search the company policy documents. Returns the 2 best passages."""
        hits = await pipeline.retrieve(query, limit=2)
        return "\n".join(f"[{h.chunk.metadata['source']}] {h.chunk.text}" for h in hits)

    agent = await build_agent(
        model="openai:gpt-5-mini",
        instructions="Answer from the company policies only. Search as many times as you "
                     "need, answer in two sentences, cite the source files, and say so if the "
                     "policies don't cover it.",
        extra_tools=[search_policies],
    )
    try:
        question = ("I bought a $250 monitor on my work trip to Berlin. Can I expense it, "
                    "and how long do I have to submit the receipt?")
        result = await agent.ainvoke({"messages": [{"role": "user", "content": question}]})
        for msg in result["messages"][1:]:
            for call in getattr(msg, "tool_calls", None) or []:
                print(f"search  {call['args']}")
            if msg.type == "ai" and msg.content:
                print(f"answer  {msg.content}")
    finally:
        await agent.shutdown()


asyncio.run(main())

Real output from one run with gpt-5-mini:

search  {'query': 'expense monitor work trip Berlin equipment purchase expense policy monitor computer peripheral expense allowed'}
search  {'query': 'expense receipt submission timeframe submit receipts within days policy expense report deadline receipts submission timeframe'}
answer  Yes — monitors and keyboards can be expensed up to $300 per year, so your $250 monitor may be expensed (equipment.md; travel.md notes travel purchases follow the relevant purchase policy). Submit the receipt within 30 days of purchase (expenses.md).

The agent split the question into two searches, one for the monitor and one for the receipt deadline, and asked for both in the same turn. A classic RAG pipeline would have embedded the whole question once and hoped the top two passages covered both parts. Each claim in the answer names the document it came from.

Two things to notice. The queries are long, keyword-stuffed strings: models often write queries this way, and with embeddings it usually works, but it is worth testing. And the agent added a reason from travel.md to its answer. That one is correct here, but it shows why you should check citations in your evals, not only the final answer. Promptise also ships rag_to_tool(), which wraps a pipeline as a tool and marks retrieved text as data, not instructions. Agentic RAG in Python builds a full version with metadata filters, freshness and "I don't know" handling.

Agentic RAG is not always better. When the questions are simple and the corpus is clean, classic RAG gives the same answer with one model call instead of two or three. Start with the simple pipeline, as in RAG for beginners, and move to an agent when your evals show multi-part or badly worded questions failing. For memory, see AI agent memory.

"Agentic RAG stops hallucinations." It reduces them when the documents contain the answer, and only if the instructions tell the agent to answer from the documents and to say when they don't cover the question. A model can still mix retrieved facts with its own guesses.

"More searches mean better answers." Each search adds tokens to the context, and long contexts make it harder for the model to use the right passage. Cap the number of searches and the passages per search. Context engineering for AI agents covers how to keep the context small.

"Retrieved documents are safe because they are ours." Anyone who can edit an indexed document, a support ticket or a wiki page can plant instructions for the model. That is indirect prompt injection. Scope what each user can retrieve and treat passages as data. See managing context and retrieval safely.

"You need a special framework for agentic RAG." Any agent that can call tools can do it. The search tool is an ordinary function that queries your index.

What is the difference between RAG and agentic RAG?

Classic RAG runs one search with the user's question and passes the results to the model. Agentic RAG gives the model a search tool, so it decides whether to search, writes its own queries, can search several times or several sources, and answers when it has enough. Agentic RAG handles harder questions at a higher cost per answer.

When should I use agentic RAG?

Use it when questions often combine several topics, use different words than your documents, or need data from more than one source, such as docs plus a database. Stay with classic RAG when questions are simple, latency matters, and one search usually finds the answer.

Is agentic RAG more expensive than RAG?

Usually, yes. Each decision is a model call, so a question that needs two searches costs at least two or three model calls instead of one, and the retrieved passages add input tokens. Limit the number of searches and the passages per search to keep the cost predictable.

Does agentic RAG need a vector database?

No. The search tool can query a vector store, a keyword index, a SQL database or an API. Many agents use several. A vector store is the common choice for unstructured documents, and an in-memory store is enough for small corpora and tests.

How do you evaluate an agentic RAG system?

Test three things separately: whether retrieval returns the right passages for a set of known questions, whether the agent searches when it should and stops when it has enough, and whether each claim in the answer is supported by a cited passage. Evaluate AI agents shows how.

• Build it: Agentic RAG in Python.

• Start simpler: RAG for beginners.

• Keep the context lean: Context engineering for AI agents.

• All guides on the topic: the memory, context and RAG hub.

• Reference: RAG in the Promptise docs.