Matrix calculus: the identities behind OLS, ridge and portfolios

LA · Chapter 712 min readAsked at Two Sigma, Citadel, DE Shaw, QuantCo

Assumes Definiteness, Cholesky and generating correlated normals.

After this lesson you should be able to

  • Differentiate the standard linear and quadratic forms.
  • Derive OLS, ridge and the minimum-variance portfolio from scratch.
  • Use a Lagrange multiplier on a constrained quadratic problem.

Four derivative identities generate most of the closed-form results in quantitative finance. Knowing them means you can derive OLS, ridge, the minimum-variance portfolio and the tangency portfolio on a whiteboard rather than recalling them.

ExpressionDerivative with respect to xx
a⊤xa^\top xaa
x⊤Axx^\top A x(A+A⊤)x(A + A^\top)x, which is 2Ax2Ax when AA is symmetric
∥x∥2=x⊤x\|x\|^2 = x^\top x2x2x
∥y−Xb∥2\|y - Xb\|^2 (w.r.t. bb)−2X⊤(y−Xb)-2X^\top(y - Xb)
ln⁡det⁡(A)\ln\det(A) (w.r.t. AA)A−⊤A^{-\top}
Table 7.1 · The identities. The second row is the one that does the work. Covariance matrices are symmetric, so the derivative of a portfolio variance w⊤Σww^\top\Sigma w is simply 2Σw2\Sigma w.

Derivation 7.2

OLS in three lines

Minimise the squared residual.

  1. S(b)=(y−Xb)⊤(y−Xb)S(b) = (y - Xb)^\top(y - Xb)
  2. ∂S∂b=−2X⊤(y−Xb)=0\frac{\partial S}{\partial b} = -2X^\top(y - Xb) = 0

    Apply the fourth identity.

  3. X⊤Xb=X⊤yX^\top X b = X^\top y
b^=(X⊤X)−1X⊤y\hat{b} = (X^\top X)^{-1}X^\top y

Derivation 7.3

Ridge, from the same start

Add the penalty and differentiate again.

  1. S(b)=(y−Xb)⊤(y−Xb)+λb⊤bS(b) = (y - Xb)^\top(y - Xb) + \lambda b^\top b
  2. −2X⊤(y−Xb)+2λb=0-2X^\top(y - Xb) + 2\lambda b = 0

    The penalty contributes 2λb2\lambda b by the third identity.

  3. (X⊤X+λI)b=X⊤y(X^\top X + \lambda I)b = X^\top y
b^ridge=(X⊤X+λI)−1X⊤y\hat{b}_{\text{ridge}} = (X^\top X + \lambda I)^{-1}X^\top y

Derivation 7.4

The minimum-variance portfolio

Minimise variance subject to the weights summing to one.

  1. L=w⊤Σw−γ(1⊤w−1)\mathcal{L} = w^\top\Sigma w - \gamma(\mathbf{1}^\top w - 1)

    Lagrangian with one constraint.

  2. 2Σw−γ1=0⇒w∝Σ−112\Sigma w - \gamma\mathbf{1} = 0 \Rightarrow w \propto \Sigma^{-1}\mathbf{1}

    The direction falls out immediately.

  3. w=Σ−111⊤Σ−11w = \frac{\Sigma^{-1}\mathbf{1}}{\mathbf{1}^\top\Sigma^{-1}\mathbf{1}}
normalise so the weights sum to one\text{normalise so the weights sum to one}

Why every answer has Σ−1\Sigma^{-1} in it. Minimum variance gives Σ−11\Sigma^{-1}\mathbf{1}, the tangency portfolio gives Σ−1μ\Sigma^{-1}\mu, and a hedge ratio gives Σ−1\Sigma^{-1} times a covariance vector. The inverse covariance keeps appearing because it is the natural weighting: it downweights directions with a lot of risk and upweights the quiet ones. That is also why every one of these is so unstable in practice — inverting a noisily estimated Σ\Sigma amplifies exactly its least reliable directions, which is the case for shrinkage restated in one sentence.

-0.20.51.20.030.060.09wᵀΣwWeight in the first assetPortfolio variance
Figure 7.5 · Portfolio variance is a quadratic, so it has one answer. Two assets at 20%20\% and 30%30\% volatility correlated 0.30.3. Setting the derivative to zero is one line of matrix calculus and lands at w≈0.76w \approx 0.76 — and the curve is flat near the bottom, which is why the minimum-variance weights are stable even when the inputs are not.

The rest of this lesson is in Premium

You have read the opening. 10 more sections follow, including 4 worked examples and 3 quick checks.

Start the free 7-day trialSign in

Nothing is charged for 7 days, and you can cancel before then. Or read The law of large numbers and the central limit theorem in full, free.