Neural networks: backpropagation, and when they are the wrong tool

A neural network is a composition of linear maps and non-linearities, fitted by gradient descent. The mathematics is the chain rule organised carefully; the interesting judgement is whether the problem has the compositional structure that makes the architecture worth its enormous capacity.

Equation 7.1

A feedforward layer

A linear map followed by an element-wise non-linearity. Stack them and you have a network; remove the non-linearity and the whole stack collapses to a single linear map.

a^{(l)} = \phi\!\left(W^{(l)}a^{(l-1)} + b^{(l)}\right)

$\phi$: Activation — ReLU by default, since it does not saturate and its gradient is trivial.
$W^{(l)}$: Weights of layer $l$ , learned by gradient descent.

Derivation 7.2

Backpropagation

Compute every partial derivative in one backward sweep rather than one forward sweep per parameter.

$\delta^{(L)} = \nabla_a L \odot \phi'(z^{(L)})$
Start at the output with the loss gradient.
$\delta^{(l)} = \left(W^{(l+1)\top}\delta^{(l+1)}\right) \odot \phi'(z^{(l)})$
Propagate the error backwards through the transpose of each weight matrix.
$\frac{\partial L}{\partial W^{(l)}} = \delta^{(l)}a^{(l-1)\top}$

\text{all gradients at roughly the cost of one forward pass}

Figure 7.3 · Why nobody differentiates a network by hand. A million-parameter model, one gradient. Backpropagation reuses the forward pass and walks the chain rule backwards once; finite differences re-evaluate the whole network for every weight. The same trick, under the name adjoint mode, is how a bank gets all its Greeks in one sweep.

Why backpropagation is the whole trick. A network may have millions of parameters, and computing each derivative by perturbing it individually would cost a forward pass each — completely infeasible. Backpropagation exploits the fact that the derivatives share almost all of their work: the chain rule from the loss to any weight passes through the same intermediate quantities, so computing them once and reusing them gives every gradient in one sweep. It is dynamic programming on the computation graph, and it is what made deep learning possible at all.

Technique	What it does	Why it helps
Weight decay	$L_2$ penalty on the weights	Same as ridge — shrinks toward zero
Dropout	Randomly zero units during training	Forces redundancy; an implicit ensemble
Early stopping	Halt when validation stops improving	Limits effective capacity
Batch normalisation	Standardise activations per batch	Conditions the optimisation; mildly regularises
Data augmentation	Generate plausible variants	Rarely available for financial data

Table 7.4 · The regularisers. The last row is the quiet reason deep learning transfers badly to finance: images can be rotated and cropped to manufacture more data, and a return series cannot.

Neural networks: backpropagation, and when they are the wrong tool

After this lesson you should be able to

A feedforward layer

Backpropagation

The rest of this lesson is in Premium