Machine learning formulas

The framework: bias, variance, capacity and dimensionality

The error decomposition

\mathbb{E}\big[(y - \hat{f}(x))^2\big] = \underbrace{\mathrm{Bias}^2}_{\text{too rigid}} + \underbrace{\mathrm{Var}}_{\text{too flexible}} + \underbrace{\sigma^2}_{\text{irreducible}}

Three terms, of which only the first two are under your control.

How much worse a model does out of sample

\mathbb{E}[\text{train MSE}] = \sigma^2\left(1 - \frac pn\right), \qquad \mathbb{E}[\text{test MSE}] = \sigma^2\left(1 + \frac{p}{n - p - 1}\right)

For least squares with $p$ predictors, $n$ observations, noise variance $\sigma^2$ and Gaussian regressors, when the true coefficients are all zero. Training error flatters the model by the fraction $p/n$ ; test error is inflated by estimation noise in every coefficient.

Remember

Error is bias squared plus variance plus irreducible noise.

Trees: bagging, random forests and gradient boosting

Out-of-bag error comes free

\Pr(\text{row } i \text{ not in a bootstrap sample}) = \left(1 - \tfrac1n\right)^n \approx e^{-1} \approx 0.368

Each tree in a random forest sees about $63\%$ of the rows, so each row can be predicted by the roughly third of trees that never saw it. Averaging those predictions gives an almost unbiased estimate of test error without a separate validation set — though on time series, where nearby rows are correlated, it is optimistic.

Remember

Trees split axis-aligned and are invariant to monotone feature transformations.

Other supervised methods: kNN, SVMs and the kernel trick

The soft-margin SVM

\min_{w, b}\ \tfrac{1}{2}\lVert w\rVert^2 + C\sum_i \max\big(0,\ 1 - y_i(w^\top x_i + b)\big)

The hinge loss penalises points inside the margin or on the wrong side; $C$ trades margin width against violations. Large $C$ fits the training data hard; small $C$ keeps a wide, stable margin — the SVM’s version of regularisation.

Remember

kNN assumes locality and a meaningful distance; standardise, and beware dimension.

Unsupervised learning: clustering assets and correlation structure

Correlation as a distance

d_{ij} = \sqrt{2(1 - \rho_{ij})}

The standard conversion. It is a proper metric — it satisfies the triangle inequality — which naive alternatives like $1 - \rho$ do not.

Remember

k-means assumes round, equal-sized Euclidean clusters; correlations are not Euclidean.

Why k-fold cross-validation is wrong on financial data

The standard error of an AUC

\operatorname{SE}(A) = \sqrt{\frac{A(1-A) + (n_1 - 1)(Q_1 - A^2) + (n_2 - 1)(Q_2 - A^2)}{n_1n_2}},\ Q_1 = \frac{A}{2 - A},\ Q_2 = \frac{2A^2}{1 + A}

Hanley and McNeil’s approximation, for $n_1$ positives and $n_2$ negatives treated as independent. With overlapping labels the effective counts are smaller and the error larger.

Remember

Financial rows are ordered and overlapping, so k-fold leaks.

Optimisation: gradient descent, momentum and Adam

Stochastic gradient descent

\theta_{t+1} = \theta_t - \eta\,\nabla_\theta L(\theta_t; \mathcal{B}_t)

Step downhill using the gradient computed on a small batch rather than on the whole dataset.

Momentum turns $\kappa$ into $\sqrt\kappa$

\text{GD: } \left(\frac{\kappa - 1}{\kappa + 1}\right)^t, \qquad \text{heavy ball: } \left(\frac{\sqrt\kappa - 1}{\sqrt\kappa + 1}\right)^t

On a quadratic with condition number $\kappa$ , the best fixed step of gradient descent shrinks the error by the first factor per step; optimally tuned momentum by the second. Replacing $\kappa$ by its square root is the whole value of momentum on ill-conditioned problems.

Remember

Stochastic gradients are noisier per step and far more efficient per unit of computation.

Neural networks: backpropagation, and when they are the wrong tool

A feedforward layer

a^{(l)} = \phi\!\left(W^{(l)}a^{(l-1)} + b^{(l)}\right)

A linear map followed by an element-wise non-linearity. Stack them and you have a network; remove the non-linearity and the whole stack collapses to a single linear map.

The gradient of softmax with cross-entropy

\frac{\partial}{\partial z}\Big(-\sum_k y_k\ln\operatorname{softmax}(z)_k\Big) = \operatorname{softmax}(z) - y

Predicted probabilities minus the one-hot target: the cleanest gradient in deep learning, and the reason softmax and cross-entropy are always paired. Each logit is pushed down in proportion to the probability wrongly assigned to it.

Remember

A network without non-linearities collapses to a linear map.