How LLMs work: tokens, probabilities and training data

A large language model turns text into tokens, then repeatedly predicts a probability for every possible next token and picks one. Settings such as temperature and top-p decide how adventurous that pick is. Everything the model knows comes from patterns in its training data, compressed into its weights, and later training such as RLHF shapes which answers it prefers.

Many puzzling behaviours follow directly from these mechanics: models struggle to count letters because they see tokens, not characters; they get niche facts wrong because rare facts appear rarely in training; they give similar answers to open questions; and they cannot reliably quote a source word for word. These guides explain each behaviour from the mechanism up and show the prompting or system design that works around it.