Sequence Models, Attention and Transformers

Text, audio, logs, clicks and time series are sequences. The transformer replaced recurrent networks as the dominant sequence architecture, and attention is now the most frequently asked deep-learning topic. You should be able to explain why recurrent nets struggled, derive scaled dot-product attention, implement it, and describe the architecture and its costs.

1. Representing text

Models consume numbers, so text goes through three steps:

1. Tokenisation: split text into tokens. Modern systems use subword schemes (byte-pair encoding, WordPiece, SentencePiece/unigram) that keep common words whole and break rare words into pieces. This handles unseen words without an enormous vocabulary.
2. Embedding: each token ID indexes a row of a learned matrix, giving a dense vector.
3. Position information: since attention treats its input as a set, positions must be injected (sinusoidal, learned, or rotary encodings).

Older representations: bag of words and TF-IDF (term frequency times inverse document frequency, which downweights words common across documents) remain strong, cheap baselines for classification and retrieval. word2vec / GloVe gave static word vectors; contextual models give each token a vector that depends on its sentence.

2. Recurrent networks and their limits

An RNN processes tokens one at a time, updating a hidden state: $h_t=\varphi (W_h{h}_{t-1}+W_x{x}_t)$. Problems:

• Vanishing and exploding gradients over long sequences: the same matrix is multiplied at every step.
• Sequential computation: step $t$ needs step $t-1$, so training cannot be parallelised across time.
• Fixed-size memory bottleneck: the whole past must be squeezed into one vector.

LSTM and GRU add gates that control what to keep, write and forget, giving a more direct path for gradients. They handle longer dependencies, but not enough, and they remain sequential. Seq2seq with attention (a decoder looking back over all encoder states) fixed the bottleneck and led directly to the idea that attention alone might be enough.

3. Attention

For each position, attention computes a weighted average of values, where the weights say how relevant every other position is.

Each token produces three vectors via learned projections: a query $q$ (what am I looking for), a key $k$ (what do I contain), and a value $v$ (what do I pass on). With matrices $Q,K,V$:

$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{Q{K}^{\top }}{\sqrt{d_k}}\right)V$

Why divide by $\sqrt{d_k}$? If entries of $q$ and $k$ have unit variance, the dot product has variance $d_k$. Large scores push softmax into regions where it saturates and gradients vanish. Dividing by $\sqrt{d_k}$ restores unit variance.

import numpy as np

def softmax(x, axis=-1):
    x = x - x.max(axis=axis, keepdims=True)
    e = np.exp(x)
    return e / e.sum(axis=axis, keepdims=True)

def attention(Q, K, V, mask=None):
    d = Q.shape[-1]
    scores = Q @ K.swapaxes(-1, -2) / np.sqrt(d)
    if mask is not None:
        scores = np.where(mask, scores, -1e9)            # masked positions get ~0 weight
    w = softmax(scores)
    return w @ V, w

rng = np.random.default_rng(0)
T, d = 5, 8
Q, K, V = (rng.normal(size=(T, d)) for _ in range(3))

out, w = attention(Q, K, V)
assert out.shape == (T, d) and np.allclose(w.sum(-1), 1)          # each row of weights is a distribution

causal = np.tril(np.ones((T, T), dtype=bool))                     # token i may attend only to tokens <= i
out_c, w_c = attention(Q, K, V, mask=causal)
assert np.allclose(np.triu(w_c, 1), 0, atol=1e-8)                 # no weight on future tokens
assert np.allclose(out_c[0], V[0])                                # the first token can only see itself

# the variance argument for the scaling factor
big = rng.normal(size=(100000, 64))
dots = (big * rng.normal(size=(100000, 64))).sum(-1)
assert abs(dots.var() - 64) < 3 and abs((dots / 8).var() - 1) < 0.05

Self-attention versus cross-attention

In self-attention $Q$, $K$ and $V$ all come from the same sequence, so each token gathers context from the others. In cross-attention the queries come from one sequence (such as a decoder) and keys and values from another (the encoder or an image), connecting two modalities or streams.

Multi-head attention

Run $h$ attention operations in parallel on lower-dimensional projections ($d_k=d_{model}/h$), concatenate the results, and project them. Different heads can specialise in different relationships (syntax, coreference, position) at roughly the same total cost as one wide head.

Cost

Self-attention compares every pair of positions: $O(n^{2}d)$ time and $O(n^2)$ memory in sequence length $n$. That quadratic cost is the main constraint on context length and has driven many optimisations: FlashAttention (an exact algorithm that avoids materialising the $n\times n$ matrix, cutting memory traffic), sparse and sliding-window attention, linear-attention variants, and key-value caching at inference.

4. The transformer block

An encoder or decoder layer combines:

1. Multi-head self-attention.
2. Residual connection and layer normalisation.
3. A position-wise feed-forward network (two linear layers with a nonlinearity, hidden size about 4 times the model width). Most of a model's parameters live here.
4. Another residual and normalisation.

Modern models usually apply normalisation before each sublayer (pre-norm), which trains more stably at depth.

Parameter count rule of thumb

A transformer layer with width $d$ has about $12d^2$ parameters (roughly $4d^2$ in attention projections and $8d^2$ in the feed-forward block). Embeddings add $V\cdot d$ for vocabulary size $V$.

def transformer_params(layers, d, vocab):
    per_layer = 4 * d * d + 2 * d * (4 * d)          # attention projections + the 4x feed-forward block (weights only)
    return layers * per_layer + vocab * d

# a GPT-2-small-sized configuration: 12 layers, width 768, vocabulary about 50k
p = transformer_params(12, 768, 50257)
assert 110e6 < p < 130e6                              # about 124M, matching the published scale

5. Positions

Because attention is permutation-invariant, you must tell the model the order.

• Sinusoidal fixed encodings: allow some generalisation to longer lengths.
• Learned absolute embeddings: simple, limited to the trained length.
• Relative and rotary (RoPE): rotate queries and keys by position-dependent angles so attention depends on relative distance. The common choice in current open models, and the basis for context-extension techniques.

6. Training objectives

The next-token loss is cross-entropy over the vocabulary. Perplexity is the exponential of the average per-token cross-entropy: a lower value means the model is less "surprised" by the text.

import numpy as np

# perplexity of a model that assigns probability 0.25 to every true token
probs = np.full(100, 0.25)
ce = -np.mean(np.log(probs))
assert abs(np.exp(ce) - 4.0) < 1e-9          # as uncertain as a fair 4-way choice at every step

uniform_vocab = 50000
assert abs(np.exp(-np.log(1 / uniform_vocab)) - uniform_vocab) < 1e-6   # a uniform guess has perplexity equal to the vocabulary

7. Decoding (generation)

At each step the model outputs a distribution over the next token. Strategies:

• Greedy: take the most likely token. Deterministic, can be repetitive.
• Beam search: keep the $b$ best partial sequences. Good for translation, can be dull for open-ended text.
• Temperature sampling: divide logits by $T$ before softmax. $T<1$ sharpens, $T>1$ flattens.
• Top-k and top-p (nucleus) sampling: sample only from the $k$ most likely tokens, or the smallest set whose cumulative probability exceeds $p$.
• Repetition penalties, constrained decoding (grammars, JSON schemas) for structure.

import numpy as np

def softmax(x):
    x = x - x.max(); e = np.exp(x); return e / e.sum()

logits = np.array([2.0, 1.0, 0.0])
cold, hot = softmax(logits / 0.5), softmax(logits / 2.0)
assert cold[0] > softmax(logits)[0] > hot[0]          # lower temperature sharpens, higher flattens

def top_p_set(p, top_p):
    order = np.argsort(-p); cum = np.cumsum(p[order])
    return set(order[: np.searchsorted(cum, top_p) + 1])

assert top_p_set(np.array([0.5, 0.3, 0.15, 0.05]), 0.8) == {0, 1}

At inference, the KV cache stores each past token's keys and values so that generating a new token costs one new row of attention instead of recomputing the whole sequence. It trades memory (growing with context length and batch size) for compute, and is why long-context serving is memory-bound.

8. Beyond text

Vision transformers split an image into patches and treat them as tokens. Audio models operate on spectrogram frames or codec tokens. Multimodal models align image and text embeddings or feed image tokens into a language model. Time-series transformers exist, but strong baselines (seasonal naive, gradient boosting, simple linear models) often win, so benchmark carefully.

9. Common mistakes

• Forgetting the causal mask in decoder training, so the model sees the answer.
• Missing the scaling factor or applying softmax over the wrong axis.
• Believing attention weights are explanations. They are a mechanism; they do not reliably show why a prediction was made.
• Ignoring the quadratic cost when promising long contexts.
• Evaluating perplexity across different tokenisers as if comparable.
• Misusing encoder-only models for generation, or decoder-only models without prompting or fine-tuning for classification.

10. Practice questions

1. Why did transformers replace RNNs and LSTMs for most sequence tasks?
2. Derive scaled dot-product attention. Why divide by $\sqrt{d_k}$?
3. What does multi-head attention add over a single head?
4. What are the time and memory costs of self-attention, and how are they reduced?
5. How does a causal mask work, and where is it needed?
6. Compare BERT-style, GPT-style and T5-style models.
7. What does the KV cache store, and why does it help?
8. Explain temperature, top-k and top-p sampling.
9. Estimate the parameter count of a transformer with 24 layers and width 1024.