Eigenvalues, diagonalisation and the spectral theorem

LA · Chapter 412 min readAsked at Two Sigma, Citadel, DE Shaw, Jane Street

Assumes Orthogonality, QR and OLS as a projection.

After this lesson you should be able to

  • Say what an eigenvector is and why diagonalisation is useful.
  • State the spectral theorem and what it guarantees for symmetric matrices.
  • Connect the largest eigenvalue to power iteration and to Markov chains.

An eigenvector is a direction the matrix does not rotate — it only stretches. Finding those directions turns a complicated linear map into independent one-dimensional scalings, which is why eigendecomposition is the tool behind PCA, Markov chain limits, and every question about repeated application of a matrix.

Equation 4.1

The eigenvalue equation

A direction that is merely scaled, and the characteristic polynomial whose roots are the scalings.

Av=λv,det⁡(A−λI)=0Av = \lambda v, \qquad \det(A - \lambda I) = 0
λ\lambda
The stretch factor. Zero means the direction is collapsed.
vv
Defined up to scale, so only the direction is determined.

Proposition 4.2

Diagonalisation and why it helps

When A=VΛV−1A = V\Lambda V^{-1}, applying AA repeatedly becomes trivial: Ak=VΛkV−1A^k = V\Lambda^k V^{-1}, and raising a diagonal matrix to a power means raising each entry. Anything involving repeated application — a Markov chain, a difference equation, a matrix exponential — collapses to independent scalar problems in the eigenbasis.

Holds when

  • Not every matrix is diagonalisable; repeated eigenvalues can leave it defective.
  • Every *symmetric* matrix is, which is why the theory is so much cleaner in that case.
  • Similar matrices share eigenvalues, so a change of basis does not change the spectrum.

Definition 4.3

The spectral theorem

Spectral theorem, A=A⊤ ⇒ A=QΛQ⊤, Q⊤Q=IA = A^\top \ \Rightarrow \ A = Q\Lambda Q^\top, \ Q^\top Q = I — A real symmetric matrix has real eigenvalues and an orthonormal basis of eigenvectors. Since covariance matrices, projection matrices and Hessians are all symmetric, this covers nearly everything that turns up in statistics — and it is the theorem that makes PCA work.

Why symmetry buys orthogonality. A symmetric matrix acts the same way in both directions: ⟨Ax,y⟩=⟨x,Ay⟩\langle Ax, y\rangle = \langle x, Ay\rangle. Apply that to two eigenvectors with different eigenvalues and you get λ1⟨v1,v2⟩=λ2⟨v1,v2⟩\lambda_1\langle v_1, v_2\rangle = \lambda_2\langle v_1, v_2\rangle, which forces the inner product to zero. Orthogonality is not an extra assumption but a two-line consequence — and it is what lets PCA produce uncorrelated components rather than merely a change of basis.

Example 4.4

Find the eigenvalues of the 2×22\times 2 correlation matrix with off-diagonal ρ=0.6\rho = 0.6, and say what they mean.

Show the worked solutionHide the worked solution

Worked solution

  1. Formula
    det⁡(1−λρρ1−λ)=0\det\begin{pmatrix}1-\lambda & \rho \\ \rho & 1-\lambda\end{pmatrix} = 0
  2. Substitute
    (1−λ)2−ρ2=0(1-\lambda)^2 - \rho^2 = 0
  3. Solve
    1−λ=±ρ⇒λ=1±ρ1 - \lambda = \pm\rho \Rightarrow \lambda = 1 \pm \rho
  4. λ1=1.6,λ2=0.4\lambda_1 = 1.6, \quad \lambda_2 = 0.4
  5. Answer
    1.6 and 0.41.6 \text{ and } 0.4

Sanity check. They sum to 2, the trace, as they must. The first eigenvector is (1,1)/2(1,1)/\sqrt{2} — the common move — and carries 80%80\% of the variance; the second is (1,−1)/2(1,-1)/\sqrt{2}, the spread between them, with 20%20\%. That is a two-asset PCA done by hand.

The rest of this lesson is in Premium

You have read the opening. 10 more sections follow, including 4 worked examples and 3 quick checks.

Start the free 7-day trialSign in

Nothing is charged for 7 days, and you can cancel before then. Or read The law of large numbers and the central limit theorem in full, free.