TOPIC AI
Top-k Sampling limiting choices to the top k tokens
IN 10 SECONDS
A sampling filter that limits next-token choices to the top 'k' most likely tokens (e.g. k=40), dropping lower-probability choices to prevent the model from generating nonsense.
GOTCHA Setting k too low (e.g. k=1) turns sampling into greedy search, causing responses to become repetitive.
HOW TOKEN SORTING FLOWS
01 Compute logits model scores vocabulary tokens probabilities.
02 Limit sorting ranks tokens, keeping only the top 40 candidate options.
03 Scale probability recalculates percentage weights among these 40 options.
04 Sample select samples the next token from this restricted set.
POKE IT YOURSELF
curl -X POST -d '{"model":"llama3","prompt":"hi","options":{"top_k":20}}' http://localhost:11434/api/generate — query local model with top-k limits
Drill this topic →
~65 sec read