PCA, covariance matrices and what an eigenvalue is telling you

LA · Chapter 614 min readAsked at Two Sigma, Citadel, DE Shaw, AQR

After this lesson you should be able to

  • Derive PCA as either maximum variance or minimum reconstruction error.
  • Say why every covariance matrix is positive semi-definite, and test a proposed correlation matrix.
  • Read the principal components of a yield curve.

PCA is the eigendecomposition of a covariance matrix, and almost every question about it is really a question about what eigenvalues and eigenvectors mean. Get that right and the yield-curve interpretation, the risk-model application and the "is this a valid correlation matrix?" question all follow from the same fact.

Derivation 6.1

Why every covariance matrix is positive semi-definite

The one-line argument, and it is the answer to several interview questions at once.

  1. w⊤Σw=Var(w⊤X)w^{\top}\Sigma w = \mathrm{Var}(w^{\top}X)

    A quadratic form in Σ\Sigma is the variance of a portfolio.

  2. Var(⋅)≥0 for every w\mathrm{Var}(\cdot) \ge 0 \ \text{for every } w

    Variances cannot be negative.

  3. ⇒Σ⪰0\Rightarrow \Sigma \succeq 0
all eigenvalues of Σ are≥0\text{all eigenvalues of } \Sigma \text{ are} \ge 0

Proposition 6.2

Testing a proposed correlation matrix

The classic question — "can three assets be pairwise correlated at −0.5-0.5?" — is a positive semi-definiteness check. Build the matrix, find its eigenvalues, and if any is negative the matrix describes a portfolio with negative variance, which cannot exist.

Holds when

  • For the 3×33\times 3 equicorrelation matrix with off-diagonal ρ\rho, the eigenvalues are 1+2ρ1 + 2\rho (once) and 1−ρ1 - \rho (twice).
  • So ρ≥−1/2\rho \ge -1/2 is required, and −0.5-0.5 is exactly the boundary: attainable, but only just.
  • In general, nn assets can be pairwise equicorrelated only down to ρ=−1/(n−1)\rho = -1/(n-1).

Example 6.3

Can four assets all be pairwise correlated at −0.4-0.4?

Show the worked solutionHide the worked solution

Worked solution

  1. Formula
    λmin⁡=1+(n−1)ρ ≥0 ⇒ ρ≥−1n−1\lambda_{\min} = 1 + (n-1)\rho \ \ge 0 \ \Rightarrow\ \rho \ge -\frac{1}{n-1}
  2. Substitute
    n=4⇒ρ≥−13≈−0.333n = 4 \Rightarrow \rho \ge -\tfrac13 \approx -0.333
  3. Solve
    −0.4<−0.333-0.4 < -0.333
  4. Answer
    No — the matrix is not positive semi-definite\text{No — the matrix is not positive semi-definite}

Sanity check. The intuition: if every pair pulls against every other, at some point there is nowhere left for them all to go. With four assets the floor is −1/3-1/3; with many assets it approaches zero.

AspectMaximum varianceMinimum reconstruction error
ObjectiveMaximise w⊤Σww^{\top}\Sigma w subject to ∥w∥=1\|w\| = 1Minimise ∥X−Xk∥F2\|X - X_k\|_F^2 over rank-kk XkX_k
Solved byLagrange multipliers; the multiplier is the eigenvalueEckart–Young: keep the top kk singular values
AnswerThe top eigenvector of Σ\SigmaThe same subspace
Table 6.4 · PCA derived two ways. Two apparently different questions with the same answer, which is the cleanest way to see why PCA is canonical rather than an arbitrary choice.

The rest of this lesson is in Premium

You have read the opening. 10 more sections follow, including 2 worked examples and 3 quick checks.

Start the free 7-day trialSign in

Nothing is charged for 7 days, and you can cancel before then. Or read The law of large numbers and the central limit theorem in full, free.

PCA, covariance matrices and what an eigenvalue is telling you · QuantMax