AI / A CONCEPT NOTE
Top-k Sampling
limiting choices to the top k tokens
Overview · mechanism
pitfall · examples
01 / THE SHORT VERSION
The idea in a few sentences.
A sampling filter that limits next-token choices to the top 'k' most likely tokens (e.g. k=40), dropping lower-probability choices to prevent the model from generating nonsense.
02 / FOLLOW THE MECHANISM
How token sorting flows
Compute logits
model scores vocabulary tokens probabilities.
Limit sorting
ranks tokens, keeping only the top 40 candidate options.
Scale probability
recalculates percentage weights among these 40 options.
Sample select
samples the next token from this restricted set.
04 / COMMAND NOTES
Read the command, then the result.
Inspect the flags and arguments before trying an example. Snippets can need local setup, replacement values, or resources in your own environment.
query local model with top-k limits
curl -X POST -d '{"model":"llama3","prompt":"hi","options":{"top_k":20}}' http://localhost:11434/api/generate05 / CHECK YOURSELF
Could you explain Top-k Sampling to a teammate?
Try it out loud in two sentences: what it is, and the one detail that changes the picture. If you stall, the gap is the part to reread.
Up next in AI engineeringTop-p Samplingchoosing from cumulative probability p