Sinusoidal Embeddings

01 · First principlesAttention cannot see order

Attention is a function of sets. Permute the input tokens and the outputs permute identically — nothing in QK^TV depends on where a token sits, only on what it is. "Dog bites man" and "man bites dog" produce the same bag of attention outputs. This is called permutation equivariance, and for language it is a defect: order carries meaning, so order must be injected, because the architecture will never recover it on its own.

The cheapest injection is to add a position-dependent vector p_t to each token embedding. The whole design question is: what should p_t be?

02 · Naive attemptsTwo obvious encodings, two failures

What we actually want is a code where every position is distinct, every component is bounded, and — crucially — the relationship between positions t and t+k looks the same for every t. Adjacency should be a fixed geometric fact.

03 · The fixSinusoids at geometrically spaced frequencies

The Transformer paper's encoding fills the d-dimensional position vector with sine/cosine pairs, each pair ticking at its own frequency:

The right analogy is binary counting. Write integers in binary and the lowest bit flips every step, the next every two steps, the next every four — fast and slow dials together identify the number uniquely. Sinusoids are the smooth version of the same idea: dimension pair 0 spins quickly (distinguishing neighbours), the last pair drifts over thousands of positions (encoding coarse location), and the full vector is a clock with d/2 hands. Bounded in [−1, 1], distinct for every position, no length normalisation anywhere.

Three of the d/2 frequency channels. Reading all hands at once identifies t uniquely, like reading a clock's second, minute and hour hands.

04 · The property that mattersAn offset is a linear map

Here is the reason this encoding was chosen over any other bounded injective code. The angle-addition identities say:

In matrix form: p_t+k = R_k p_t, where R_k is a block-diagonal rotation whose angles depend only on the offset k. "Shift by k" is one fixed linear transformation, the same everywhere in the sequence. A linear attention head can therefore learn to ask "what was 3 tokens back?" as a single matrix — relative position becomes learnable by linear machinery, which is exactly what the model has. This rotation property is the seed that RoPE later promotes from "available if learned" to "enforced by construction".

05 · LimitsWhat it does not buy

• Additive mixing is blunt. Position is summed into the same vector as content, so every downstream layer must disentangle "what" from "where" by itself. Methods that keep position out of the residual stream (relative biases, RoPE) consistently train better.
• Extrapolation is theoretical, not practical. The formula is defined for any t, but models attend poorly at positions longer than anything seen in training; the slow channels enter ranges the network never learned to read.
• Learned absolute embeddings (GPT-2 style) match it in quality at trained lengths and fail harder beyond them — which tells you the sinusoid's elegance was never the binding constraint at short context.