Neural Networks and Backpropagation

A neural network is a composition of simple differentiable functions whose parameters are trained by gradient descent. Interviews ask you to explain the forward pass, derive backpropagation for a small network, choose activations and losses, and reason about vanishing gradients and architecture choices. The most convincing answer is one you can back with a short from-scratch implementation.

1. The building block

A layer computes $h=\varphi (Wx+b)$: a linear map followed by a nonlinearity $\varphi$. Stacking layers gives a network:

$\hat{y}=f_L(\cdots f_2(f_1(x)))$

Without nonlinearities, any stack of linear layers collapses into one linear map ($W_2{W}_{1}x=Wx$), so depth would add nothing. The nonlinearity is what lets the network represent curved decision boundaries.

The universal approximation theorem says a network with one sufficiently wide hidden layer can approximate any continuous function on a compact set. It is an existence result: it says nothing about how to find the weights or how wide is enough. In practice, depth is far more parameter-efficient than width.

2. Activation functions

The output activation and loss pair:

3. Backpropagation

Training needs $\partial L/\partial W$ for every weight. Backpropagation is the chain rule applied efficiently on the computation graph: do a forward pass storing intermediate values, then a backward pass that reuses them, computing every gradient in time proportional to the forward pass.

For a two-layer network with $z_1=W_{1}x+b_1$, $a_1=\varphi (z_1)$, $z_2=W_2{a}_1+b_2$, $\hat{y}=z_2$ (regression, squared error $L=\frac{1}{2}(\hat{y}-y{)}^2$):

${\delta }_2=\hat{y}-y\frac{\partial L}{\partial W_2}={\delta }_2{a}_1^{\top }{\delta }_1=(W_2^{\top }{\delta }_2)\odot {\varphi }^{\prime }(z_1)\frac{\partial L}{\partial W_1}={\delta }_1{x}^{\top }$

Each layer's error signal $\delta$ is the next layer's signal sent backwards through the transposed weights and multiplied by the local derivative. The cost of backward is a small constant multiple of forward.

Here is a complete network that learns XOR, which a single linear layer cannot, with the gradients verified numerically:

import numpy as np

rng = np.random.default_rng(0)
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
y = np.array([[0], [1], [1], [0]], dtype=float)

def init(h=8):
    return {"W1": rng.normal(0, 1, (2, h)), "b1": np.zeros(h),
            "W2": rng.normal(0, 1, (h, 1)), "b2": np.zeros(1)}

def forward(p, X):
    z1 = X @ p["W1"] + p["b1"]; a1 = np.tanh(z1)
    z2 = a1 @ p["W2"] + p["b2"]; out = 1 / (1 + np.exp(-z2))
    return z1, a1, z2, out

def loss_fn(p):
    out = forward(p, X)[3]
    return -np.mean(y * np.log(out) + (1 - y) * np.log(1 - out))

def backward(p):
    z1, a1, z2, out = forward(p, X)
    n = len(X)
    d2 = (out - y) / n                              # sigmoid + cross-entropy: the gradient simplifies to (p - y)
    g = {"W2": a1.T @ d2, "b2": d2.sum(0)}
    d1 = (d2 @ p["W2"].T) * (1 - a1 ** 2)           # tanh'(z) = 1 - tanh(z)^2
    g["W1"] = X.T @ d1; g["b1"] = d1.sum(0)
    return g

# gradient check: compare against finite differences
p = init()
g = backward(p)
eps = 1e-6
for name in ("W1", "b1", "W2", "b2"):
    flat = p[name].reshape(-1)
    for i in range(min(3, flat.size)):
        old = flat[i]
        flat[i] = old + eps; up = loss_fn(p)
        flat[i] = old - eps; down = loss_fn(p)
        flat[i] = old
        num = (up - down) / (2 * eps)
        assert abs(num - g[name].reshape(-1)[i]) < 1e-6, (name, i)

for _ in range(5000):
    g = backward(p)
    for k in p:
        p[k] -= 0.5 * g[k]

pred = forward(p, X)[3]
assert ((pred > 0.5) == y).all()                    # XOR solved
assert loss_fn(p) < 0.05

Two habits worth showing in an interview: the gradient check (numerical differentiation to validate your derivation) and the sigmoid-plus-cross-entropy simplification to $p-y$.

4. Vanishing and exploding gradients

Backprop multiplies many Jacobians together. If their typical size is below 1, the gradient shrinks exponentially with depth (vanishing); above 1, it grows (exploding). Sigmoid and tanh saturate and have derivatives under 1, so deep stacks of them barely learn in the early layers.

Remedies:

• ReLU-family activations, whose derivative is 1 on the active side.
• Careful initialisation (He for ReLU, Xavier for tanh).
• Normalisation layers (batch or layer norm).
• Residual connections: $h=x+F(x)$ gives gradients a direct path, which made very deep networks trainable.
• Gradient clipping against explosions.
• Gated architectures (LSTM, GRU) for recurrent nets.

import numpy as np

# gradient magnitude through 30 sigmoid layers: each local derivative is at most 0.25
max_local = 0.25
assert max_local ** 30 < 1e-17                       # vanishes long before reaching the first layer

# a residual path keeps a gradient of exactly 1 along the skip connection
depth, grad = 30, 1.0
for _ in range(depth):
    grad *= 1.0 + 0.0                                # d(x + F(x))/dx = 1 + F'(x); the 1 is always there
assert grad == 1.0

5. Architectures at a glance

Convolution in two sentences

A convolutional layer slides a small learned filter over the input, producing a feature map. Because the same filter is applied everywhere, parameters are shared and the layer is translation equivariant; pooling or striding then adds approximate invariance and reduces size. The output size for a square input of size $n$, kernel $k$, padding $p$, stride $s$ is $\lfloor (n+2p-k)/s\rfloor +1$.

def conv_out(n, k, p=0, s=1):
    return (n + 2 * p - k) // s + 1

assert conv_out(32, 3, p=1, s=1) == 32          # "same" padding with a 3x3 kernel
assert conv_out(32, 3, p=0, s=1) == 30
assert conv_out(224, 7, p=3, s=2) == 112        # a typical first layer of a ResNet

Parameter count

A dense layer with $n_{in}$ inputs and $n_{out}$ outputs has $n_{in}{n}_{out}+n_{out}$ parameters. A convolution with $C_{in}$ input channels, $C_{out}$ filters and a $k\times k$ kernel has $k^2{C}_{in}{C}_{out}+C_{out}$ parameters, independent of image size, which is the efficiency win.

6. Training recipe

1. Normalise inputs. Shuffle data each epoch.
2. Start small: overfit a tiny batch to catch bugs.
3. Pick Adam or AdamW with a sensible learning rate (around $10^{-3}$ to $10^{-4}$), plus warmup and decay.
4. Monitor training and validation loss, and the metric you care about.
5. Regularise as needed: weight decay, dropout, augmentation, early stopping.
6. Tune learning rate first, then batch size, width and depth.
7. Check the data: label noise and leakage cause more failures than architecture.

7. When not to use a neural network

Small tabular datasets, a need for strict interpretability, tight latency or memory budgets, or little data and no pretrained model to fine-tune. Gradient-boosted trees or linear models are usually faster to build and as accurate.

8. Common mistakes

• Forgetting nonlinearities between layers.
• Wrong loss and output pairing, such as softmax followed by a loss that applies softmax again.
• Not shuffling or leaking validation data through augmentation or preprocessing.
• All-zero initialisation, which keeps every hidden unit identical (symmetry is never broken).
• Evaluating in training mode with dropout or batch norm active.
• Inspecting only the loss, not examples, calibration or per-slice metrics.
• Assuming deeper is always better. Without residuals and normalisation it can train worse.

9. Practice questions

1. Why are nonlinearities needed? What does the universal approximation theorem say and not say?
2. Derive backpropagation for a one-hidden-layer network.
3. Why does softmax with cross-entropy give the gradient $p-y$?
4. Explain vanishing gradients and three ways to mitigate them.
5. Why does initialising all weights to zero fail?
6. Compare ReLU, sigmoid and GELU.
7. How many parameters does a 3x3 convolution with 64 input and 128 output channels have?
8. Your network's training loss is flat from the start. How do you debug it?