Linear algebra

The identities to have ready

Transpose and inverse both reverse the order of a product. Forgetting that is the single most common slip in a matrix derivation.

The matrix determinant lemma

A rank-one update changes the determinant by a scalar factor. With $A = I$ it reads $\det(I + uv^\top) = 1 + v^\top u$ — the reason a one-factor covariance matrix has such a simple determinant.

Updating an inverse: Sherman–Morrison

Inverting a matrix after a rank-one change costs $O(n^2)$ instead of $O(n^3)$. It gives equicorrelation and one-factor covariance inverses in closed form, and it is how recursive least squares updates without re-solving.

The expected value of a quadratic form

For a random vector with mean $\mu$ and covariance $\Sigma$, whatever its distribution. It is the trace trick at work, and it gives expected squared tracking errors, expected residual sums of squares and the bias of sample variances in one line.

Rank–nullity

For an $m\times n$ matrix, the null space dimension is $n-r$ when the rank is $r$.

The projection matrix

Projects any vector onto the column space of $A$. Everything about least squares follows from its two defining properties.

The eigenvalue equation

A direction that is merely scaled, and the characteristic polynomial whose roots are the scalings.

The Cholesky factorisation

A "square root" of a symmetric positive-definite matrix. Unique, cheap, and the workhorse of simulation and of solving normal equations.

Singular value decomposition

The columns of $V$ are directions in input space; singular values rank how strongly $A$ stretches those directions.

Two more identities

These, with the linear and quadratic rules, derive the maximum-likelihood estimate of a covariance matrix: the Gaussian log-likelihood is $-\tfrac n2\ln\det\Sigma - \tfrac{1}{2}\operatorname{tr}(\Sigma^{-1}S)$, and setting its derivative to zero gives $\hat\Sigma = S$.