Grounding LLM Answers: Give the Model the Right Context

Paste the relevant source into the prompt, fenced with clear delimiters such as <policy> tags. Tell the model to answer only from that source, to cite the part it used, and to say plainly when the source doesn't cover the question. Keep the source focused on the question; when you have too many documents to paste, retrieve the relevant passages first (retrieval-augmented generation, or RAG).

Key takeaways:

• Without your facts, the model answers from general knowledge, confidently.

• Delimiters separate data from instructions and make citations easy.

• Ask for a citation (rule number, quote) so a person or code can check the answer.

• Give the model a way out: "if it's not covered, say so". Otherwise it fills gaps.

A model knows what was in its training data, in general terms. It doesn't know your return policy, your product's settings menu or what was said in Tuesday's meeting. When asked anyway, it produces the most plausible answer, which is often a typical policy or a reasonable-sounding feature.

Grounding changes the question from "what do you know?" to "what does this text say?". Models are much better at the second. The approach scales up as retrieval-augmented generation, introduced in Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020): fetch relevant passages, put them in the prompt, generate from them.

Two limits to keep in mind:

• The context window is finite. Every model has a maximum input size, and cost grows with input length. Paste what's relevant, not everything.

• Position matters. Lost in the Middle (Liu et al., 2023) found that models used information at the start and end of long inputs better than information in the middle. Keep the source short, or put the most relevant part first.

A customer asks an outdoor store's assistant:

I bought an Alder rain jacket 40 days ago and wore it once. The zipper broke. Can I return it for a refund? Also, I live in Canada, who pays shipping?

Without context

The output (trimmed):

Short answer: Yes — if the zipper failed after only one wear, you have the right to a refund or replacement. In Canada the seller (or the brand) should cover return shipping for a defective item.
…
3. Ask explicitly for a prepaid return label or in-store return — because it’s a manufacturing defect, the seller should not expect you to pay return postage. If the seller insists you pay, push back …
…
- For online purchases, Canadian retailers are expected to cover return shipping for defective items.

A confident "yes", a promise about shipping, and advice to push back. None of it comes from the store's actual policy, because the model never saw it. If this is the store's own assistant, it just made a commitment the store doesn't offer.

With the policy as context

Answer the customer's question using only the return policy below.
- Quote the numbered rule you rely on, like (rule 2).
- If the policy doesn't cover part of the question, say so plainly and suggest contacting support. Don't fill gaps with general knowledge.
- Under 90 words.

<policy>
Alder Outdoor return policy (updated 2026-09-01)
1. Unworn items with tags can be returned within 60 days of delivery for a full refund.
2. Worn items can be returned within 30 days if they are faulty. We repair or replace faulty items; refunds only if repair or replacement isn't possible.
3. Sale items marked "final sale" can't be returned.
4. Return shipping is free within the US. We email a prepaid label within 1 business day of your request.
5. Refunds go to the original payment method within 5 business days after we receive the item.
</policy>

<question>
…customer question…
</question>

Because this is a worn item with a fault, Alder’s policy says returns for faulty worn items are allowed only within 30 days and they repair or replace them (refunds only if repair/replacement isn’t possible) (rule 2). You bought it 40 days ago, so that rule does not cover a refund. The policy only states free return shipping within the US (rule 4) and does not address Canada — please contact Alder support for Canadian shipping and possible exceptions.

The answer matches the policy, cites rules 2 and 4, and flags the Canada gap instead of inventing a rule. One detail to tighten: the customer counted from purchase, rule 1 counts from delivery, and rule 2 doesn't say. The model treated 40 days as settled. A follow-up line in the prompt, "if dates matter, ask for the delivery date", handles that.

Note also that consumer law may give the customer rights beyond the store policy. A grounded store assistant shouldn't give legal advice either way; "contact support" is the right exit.

Ground the prompt when:

• The answer depends on your documents: policies, product docs, contracts, meeting notes, tickets.

• The facts change: prices, dates, versions, people.

• Being wrong is costly, and you want each answer traceable to a source.

You can skip it for general knowledge and creative tasks, such as explaining what a VAT number is or brainstorming names, as long as someone checks facts that matter.

If you have more source text than fits, or you don't know which document is relevant, you need retrieval. RAG zero to one builds a small version. For agents that gather their own context with tools, see context engineering for AI agents.

• Fence each source with a named tag: <policy>, <email>, <notes>. Use different tags for different sources.

• Label sources if there are several (<doc id="returns-2026-09">) and ask for the ID in citations.

• Put instructions outside the fence. Text inside the tags is data. If a pasted email says "ignore your instructions", it should be treated as part of the email; see prompt injection 101 for why tags alone don't guarantee that.

• Include dates and versions ("updated 2026-09-01") so the model and the reader know how current it is.

• Cut what's irrelevant. Less text means lower cost and less for the model to get lost in.

What is the difference between grounding and RAG?

Grounding is the principle: answer from provided sources. RAG is one way to supply them automatically, by searching a document store and inserting the top passages into the prompt. Pasting a policy by hand is grounding without retrieval.

Does grounding stop hallucinations completely?

No. It reduces them a lot for questions the source covers, but a model can still misread the text, merge two rules or add a plausible detail. Citations and quote checks catch most of this; see hallucination basics.

How much context can I paste?

Up to the model's context window, but more is not better. Paste the sections that answer the question. For long documents, see long-context tactics and prompt compression.

Should the source go in the system prompt or the user message?

Stable reference material used in every turn (a short policy) can go in the system prompt. Material for one question goes in the user message. See system vs user prompts.

Which delimiters work best?

XML-style tags are clear and widely recommended, but the exact choice matters less than consistency. Pick a scheme, name the tags after their content, and refer to them by name in your instructions.

• Previous in the path: how to write clear prompts.

• Next: structured outputs.

• Catch what grounding misses: hallucination basics.

• Retrieve the right passages: RAG zero to one.

• Context for agents: context engineering for AI agents.