A language model does not retrieve an answer and print it. It writes one token at a time, and each token is a choice from a probability distribution. This essay, part of the Mental Models learning path, explains how next-token prediction and sampling work, why the first words of an answer constrain everything after them, and three practical changes to how you order and read prompts. It includes a real example where the model gets the verdict right and invents the reason.
How do LLMs generate text?
An LLM reads your prompt and computes a probability for every token in its vocabulary as the next token. A sampler picks one, the token is appended to the text, and the model runs again on the longer text. The answer is one path through a tree of possible continuations, not a fixed output. Settings such as temperature and top-p change how adventurous the sampler is, and every token chosen narrows what can plausibly come next.
Key takeaways:
Output is sampled. The same prompt can produce different answers, and that is expected behavior.
Early tokens commit the rest. Once a verdict is written, the following text tends to justify it.
The model can't delete. It can only append corrections, which is why you sometimes see "wait" mid-answer.
[02]
Why it works this way
GPT-style models are trained on one objective: predict the next token of real text. At inference time the model produces a score for each candidate token, and a softmax turns the scores into probabilities. Temperature divides the scores before the softmax, so a low temperature sharpens the distribution toward the top choice and a high one flattens it. Top-p, or nucleus sampling, keeps only the smallest set of tokens whose probabilities add up to p and samples from those. It was introduced in The Curious Case of Neural Text Degeneration (Holtzman et al., 2019), which showed that always taking the most likely token (greedy or beam search) tends to produce bland, repetitive text.
Two consequences matter in practice. First, generation is autoregressive: each token is conditioned on everything already written, including the model's own earlier tokens. A wrong early token is not revised; it becomes context. Second, the visible text is the only scratch space the model has, unless it is a reasoning model that writes hidden reasoning tokens before the answer (OpenAI reasoning guide).
"One token at a time" does not mean "no planning". Anthropic's interpretability team found that Claude 3.5 Haiku picks a rhyming word before writing the line that ends in it (On the Biology of a Large Language Model, Lindsey et al., 2025). The model can look ahead internally. It still can't take back what it has already emitted.
[03]
Example: verdict first, reason second
A prompt that forces the verdict into the first token:
Is 8,633 a prime number? Start your reply with Yes or No, then give a one-sentence reason.The model (gpt-5-mini, minimal reasoning effort):
No — 8,633 is divisible by 7 (8,633 = 7 × 1,233)."No" is correct. The reason is not: 7 × 1,233 is 8,631. Another sample of the same prompt shows the path dependence directly:
No — 8,633 = 47 × 183.5? Wait check: 8633/47 = 183.68... That's wrong. Let's recompute: 8,633 = 7 × 1,233.285... hmm. Need correct factorization.
…The model committed to "No", then had to produce a reason that fit, and could only patch its mistakes by writing more. Now put the work before the verdict:
Is 8,633 a prime number? Work it out first: test prime divisors up to the square root and show each multiplication you check. Put the final answer (Yes or No, with the factorization if any) on the last line.…
- 83: 83×104 = 8,632 → close but not equal (83×104 = 8,632) → not divisible.
- 89: 89×97 = 8,633. Check: 89×100 = 8,900; subtract 89×3 = 267 gives 8,900 − 267 = 8,633. So 89 × 97 = 8,633.
Thus 8,633 = 89 × 97, so it is not prime.
Final answer: No — 8,633 = 89 × 97.Same model, same question. The answer now follows from the working instead of the other way round. Results vary by model; stronger reasoning settings make the first prompt safer, but the ordering principle holds.
[04]
What it changes in practice
Order the output so evidence comes before the verdict. Ask for the extracted facts, quotes or calculations first and the label or decision last. This matters most with non-reasoning models or low reasoning effort. If you need the verdict in a fixed place for parsing, use a structured output with a reasoning field before the answer field.
Read a reason that follows a verdict as a justification, not a cause. It was generated to fit the words before it. Check it independently, especially numbers.
Use the controls you have, and know which you don't. On models that accept it, lower temperature for extraction and classification, higher for brainstorming. Some reasoning models accept only the default: gpt-5-mini rejects temperature=0 with "Only the default (1) value is supported." There, the prompt's structure and the reasoning-effort setting are your levers. Even where temperature 0 or a seed is accepted, providers describe determinism as best effort, so don't build on identical outputs.
[05]
Frequently asked questions
What is temperature in an LLM?
Temperature scales the model's token scores before they become probabilities. Low values make the most likely tokens even more likely, which gives more consistent output. High values spread probability to less likely tokens, which gives more varied output and more mistakes. It changes how the model samples, not what it knows.
What is top-p sampling?
Top-p (nucleus) sampling keeps only the most likely tokens whose combined probability reaches p, for example 0.9, and samples from that set. It cuts off the long tail of unlikely tokens while still allowing variety. Holtzman et al. (2019) proposed it as an alternative to greedy decoding.
Why does the same prompt give different answers?
Because each token is sampled from a distribution, and one different early token sends the rest of the answer down another path. Lower temperature reduces this where supported. For tasks with one right answer, compare several samples: disagreement is a sign the model is unsure.
Does the model plan its whole answer in advance?
Partly. Interpretability research shows models can represent upcoming words before writing them, such as a rhyme at the end of a line. But the output is still emitted token by token, and the model can't revise tokens it has already produced.
[06]
Next steps
LLM as a stochastic parrot: the companion essay on what next-token prediction does and doesn't imply.
Why LLMs give similar answers: use sampling on purpose to get variety.
Are LLMs black boxes?: why explanations don't always show what drove an answer.
Chain-of-thought prompting for beginners: when asking for steps helps.
Structured outputs: fix where the verdict goes.
Mental Models: How to Think with LLMs: the full learning path.