The four fundamental subspaces, and when Ax = b has a solution

LA · Chapter 211 min readAsked at Two Sigma, Citadel, DE Shaw, Jane Street

Assumes Rank, trace and determinant: what each one measures.

After this lesson you should be able to

  • Name the four subspaces and how they fit together.
  • Say when a linear system has no solution, one, or infinitely many.
  • Connect the null space to multicollinearity.

Every matrix carves its input and output spaces into four pieces, and the whole theory of linear systems is a statement about how those pieces fit. It is worth carrying as a picture rather than a list, because the picture answers existence and uniqueness questions immediately.

SubspaceLives inDimensionMeaning
Column spaceRm\mathbb{R}^mrrEverything AA can produce
Null spaceRn\mathbb{R}^nn−rn - rInputs that map to zero
Row spaceRn\mathbb{R}^nrrOrthogonal complement of the null space
Left null spaceRm\mathbb{R}^mm−rm - rOrthogonal complement of the column space
Table 2.1 · The four subspaces. For an m×nm \times n matrix of rank rr. The row and null spaces split the input; the column and left null spaces split the output — and each pair is orthogonal.

Proposition 2.2

Existence and uniqueness

A solution to Ax=bAx = b exists exactly when bb lies in the column space, and it is unique exactly when the null space is trivial. Those are two independent conditions, which is why a system can have no solution, exactly one, or infinitely many — and never any other count.

Holds when

  • No solution: bb is outside the column space. Least squares projects it in, which is what OLS does.
  • Infinitely many: the null space is non-trivial, so any null vector can be added to a solution.
  • Exactly one: bb is reachable and nothing maps to zero.

Regression is the no-solution case. With more observations than regressors, yy almost never lies in the column space of XX — there is no coefficient vector that reproduces the data exactly, and that is the normal situation rather than a failure. Least squares does the only sensible thing: it finds the point *in* the column space closest to yy, which is the orthogonal projection. The residual is then what is left over, living in the left null space, and it is orthogonal to every column by construction. The four-subspace picture is the geometry of regression.

Proposition 2.3

The null space is multicollinearity

If a linear combination of your regressors is exactly zero, that combination is a null vector of XX — and you can add any multiple of it to the coefficient vector without changing a single fitted value. That is precisely why perfectly collinear regressors are not identified: infinitely many coefficient vectors give the identical fit.

Holds when

  • Near-collinearity means a nearly-null direction, so the coefficients are nearly unidentified and wildly unstable.
  • The dummy variable trap is exactly this: including every category plus an intercept makes the columns sum to the intercept.

The rest of this lesson is in Premium

You have read the opening. 10 more sections follow, including 4 worked examples and 3 quick checks.

Start the free 7-day trialSign in

Nothing is charged for 7 days, and you can cancel before then. Or read The law of large numbers and the central limit theorem in full, free.