Regularisation: ridge, lasso and choosing lambda

Regularisation deliberately biases the coefficients toward zero in exchange for a large reduction in variance. On financial data, where the signal is faint and the regressors are correlated, that trade is almost always worth making — which is why a penalised regression routinely beats OLS out of sample.

Equation 7.1

The two penalties

The same least-squares fit plus a penalty on the size of the coefficients — squared for ridge, absolute for lasso.

\hat\beta_{\text{ridge}} = \arg\min \|y - X\beta\|^2 + \lambda\|\beta\|_2^2, \qquad \hat\beta_{\text{lasso}} = \arg\min \|y - X\beta\|^2 + \lambda\|\beta\|_1

$\lambda$: The strength of the penalty. At zero this is OLS; as it grows everything shrinks toward zero.
$\|\beta\|_1$: Sum of absolute values, which is what produces exact zeros.

Proposition 7.2

Ridge has a closed form and always works

The solution is $\hat\beta = (X^\top X + \lambda I)^{-1}X^\top y$ . Adding $\lambda$ to the diagonal makes the matrix invertible even when $X^\top X$ is singular, so ridge works when there are more regressors than observations and OLS does not exist at all.

Holds when

Standardise the regressors first, or the penalty charges more for variables measured in small units.
Do not penalise the intercept.
Ridge has a Bayesian reading: it is the posterior mode under a normal prior on $\beta$ centred at zero.

Why lasso selects and ridge does not. Picture the constraint region each penalty defines. Ridge gives a sphere, lasso a diamond with corners on the axes. The solution is where the least-squares contours first touch that region, and a smooth sphere is almost never touched exactly at an axis — so ridge coefficients get small but stay non-zero. A diamond has corners *on* the axes, and contours meet corners readily, so lasso sets coefficients exactly to zero. Selection is a consequence of the geometry, not an extra feature.

Aspect	Ridge	Lasso	Elastic net
Penalty	$\\|\beta\\|_2^2$	$\\|\beta\\|_1$	A mix of both
Exact zeros	No	Yes	Yes
Correlated regressors	Splits weight between them	Picks one arbitrarily	Keeps the group together
Closed form	Yes	No	No
Use when	All regressors plausibly matter	You want a sparse model	Correlated groups of features

Table 7.3 · Ridge, lasso, elastic net. The correlated-regressors row decides it in practice. On a factor library where features come in families, lasso picks one member of each at random and elastic net keeps the family, which is far more stable across refits.

Figure 7.4 · Ridge bends, lasso cuts. A coefficient that starts at

1

. Ridge shrinks it towards zero and never reaches it, so every regressor stays in the model; lasso subtracts a constant and hits zero at

\lambda = 1

, which is the whole of why one selects variables and the other does not.

Regularisation: ridge, lasso and choosing lambda

After this lesson you should be able to

The two penalties

Ridge has a closed form and always works

The rest of this lesson is in Premium