Orthogonality, QR and OLS as a projection

Projection is the geometric content of least squares, and orthogonalisation is how it is computed. Understanding both turns a list of regression facts — orthogonal residuals, monotone $R^2$ , the meaning of "controlling for" — into consequences of one picture.

Equation 3.1

The projection matrix

Projects any vector onto the column space of $A$ . Everything about least squares follows from its two defining properties.

P = A(A^\top A)^{-1}A^\top

$P^2 = P$: Idempotent — projecting twice is projecting once.
$P^\top = P$: Symmetric, which is what makes it an *orthogonal* projection.

Proposition 3.2

What the picture gives you

The residual $y - Py$ is orthogonal to the column space, because the closest point is reached by dropping a perpendicular. That single fact yields: residuals sum to zero when there is an intercept, residuals are uncorrelated with fitted values, and adding a regressor enlarges the space so the residual can only shrink.

Holds when

Eigenvalues of a projection are 0 and 1, so $\mathrm{tr}(P) = \mathrm{rank}(P)$ counts the dimensions projected onto.
$I - P$ is also a projection — onto the orthogonal complement — which is where the residuals live.
Pythagoras applies: $\|y\|^2 = \|Py\|^2 + \|(I-P)y\|^2$ , which *is* the analysis of variance.

Derivation 3.3

Gram–Schmidt and QR

Turn a set of vectors into an orthonormal basis by removing, from each, its component along the ones already processed.

$u_1 = a_1, \qquad q_1 = u_1/\|u_1\|$
Normalise the first.
$u_k = a_k - \sum_{j<k}\langle a_k, q_j\rangle q_j$
Subtract the part already explained — which is residualising.
$A = QR, \quad Q \text{ orthonormal}, \ R \text{ upper triangular}$

\hat\beta = R^{-1}Q^\top y

Why not just invert $X^\top X$ . Forming $X^\top X$ squares the condition number, so a design matrix that was merely awkward becomes numerically hopeless: you can lose twice as many digits of precision as the problem itself requires. QR works on $X$ directly and never forms the product, so it keeps the original conditioning. On well-conditioned data the two agree; on collinear data, which is the case where you most need the answer, the normal equations can return coefficients that are simply wrong. Every serious least-squares routine uses QR or SVD for this reason.

Example 3.4

Project $y = (3, 4)$ onto the line spanned by $a = (1, 0)$ , and give the residual.

Show the worked solutionHide the worked solution

Worked solution

Formula $Py = \frac{a^\top y}{a^\top a}\,a$
Substitute $a^\top y = 3, \quad a^\top a = 1$
Solve $Py = 3(1, 0) = (3, 0)$
$y - Py = (0, 4)$
Answer $\text{fitted } (3,0), \text{ residual } (0,4)$

Sanity check. The residual is orthogonal to $a$ , as it must be: $(0,4)\cdot(1,0) = 0$ . And Pythagoras holds — $9 + 16 = 25 = \|y\|^2$ — which is the analysis of variance in two dimensions.

Orthogonality, QR and OLS as a projection

After this lesson you should be able to

The projection matrix

What the picture gives you

Gram–Schmidt and QR

The rest of this lesson is in Premium