04 · How Models Run
Sampling
Choosing the next generated token
The decoding step that turns model probabilities into an output token. A system may choose the top token or sample among candidates using controls such as temperature, top-p, or top-k, so generation is not always “pick the single most likely word.”
Concrete example
Two runs from the same prompt can differ because the decoder sampled different plausible tokens.
Why it matters
Explains why model output can vary even when the prompt and model are unchanged.