How does self-attention decide which tokens contribute to a token representation?
#Show answer
Each token’s hidden state is projected into a query, key and value vector. A query is compared with the keys of tokens it is allowed to attend to; normalized similarity scores become weights on those tokens’ value vectors. The resulting weighted combination is projected and passed to later layers.
In a causal decoder, a mask prevents positions from attending to future tokens. In an encoder, bidirectional attention is typical. Attention does not directly search for human-understandable words: it learns numerical relationships useful for the training objective. Explain the Q/K/V flow and the mask before drawing conclusions about what a particular head means.
Trade-offs
Full attention offers flexible context mixing but standard dense attention has quadratic score work in sequence length.
Common mistake
Assuming each head always corresponds to one fixed linguistic rule.
Follow-up
What changes between a bidirectional encoder attention mask and a causal decoder mask?