The framework: bias, variance, capacity and dimensionality

Every supervised learning problem is the same trade: a more flexible model fits the training data better and generalises worse. What makes financial data distinctive is where the optimum sits — so far toward the simple end that the usual machine-learning instincts are actively misleading.

Equation 1.1

The error decomposition

Three terms, of which only the first two are under your control.

\mathbb{E}\big[(y - \hat{f}(x))^2\big] = \underbrace{\mathrm{Bias}^2}_{\text{too rigid}} + \underbrace{\mathrm{Var}}_{\text{too flexible}} + \underbrace{\sigma^2}_{\text{irreducible}}

$\sigma^2$: Noise in the target. On daily returns this is around $99\%$ of the total.
$\mathrm{Var}$: How much the fitted model changes if you resample the training data.

When the noise term dominates everything. The usual picture has bias and variance roughly comparable, with a clear interior optimum. On daily returns the irreducible term is overwhelming: even a perfect model would explain about one per cent of the variance. In that regime almost any added capacity buys a negligible reduction in bias and a large increase in variance, so the optimum sits at a model far simpler than intuition — or than experience with images and text — suggests. "Try a bigger model" is good advice in most of machine learning and bad advice here.

Knob	More capacity	Less
Model class	Deep network, boosted trees	Linear model
Tree depth	Deeper	Stumps
Regularisation	Small $\lambda$	Large $\lambda$
Features	More of them	A selected few
Training time	More iterations	Early stopping

Table 1.2 · What controls capacity. These are substitutes, which is why tuning them jointly by grid search is both expensive and an easy way to overfit the validation set.

Proposition 1.3

The curse of dimensionality

In high dimensions, data becomes sparse and every point becomes roughly equidistant from every other. To cover the unit cube at a given density you need a number of points exponential in the dimension — ten per axis is 10 points in one dimension and $10^{20}$ in twenty.

Holds when

Distance-based methods like kNN degrade fastest, since "nearest" stops meaning anything.
Most real high-dimensional data lies near a lower-dimensional manifold, which is why anything works at all.
Adding a useless feature is not free: it adds variance and dilutes every neighbourhood.

Figure 1.4 · The only curve in machine learning. The minimum is where the two curves cross, not where either is small. Everything that calls itself regularisation — fewer trees, a shorter depth, a larger

\lambda

, early stopping — is a way of sliding left along this axis, and the right amount is an empirical question every time.

The framework: bias, variance, capacity and dimensionality

After this lesson you should be able to

The error decomposition

The curse of dimensionality

The rest of this lesson is in Premium