OLS from three angles

Least squares is the workhorse, and interviewers probe whether you understand it or merely call it. Three views of the same object — minimise a sum of squares, project onto a column space, take a ratio of covariance to variance — and each answers a different follow-up.

Equation 1.1

The model

Linear in the parameters — not necessarily in the variables, since $x^2$ and $\log x$ are perfectly acceptable columns.

y = X\beta + \varepsilon, \qquad \mathbb{E}[\varepsilon \mid X] = 0

$X$: The $n \times k$ design matrix, including a column of ones for the intercept.
$\varepsilon$: The error, assumed mean-zero conditional on the regressors.

Derivation 1.2

The normal equations

Minimise the sum of squared residuals with respect to $\beta$ .

$S(\beta) = (y - X\beta)^{\top}(y - X\beta)$
$\frac{\partial S}{\partial \beta} = -2X^{\top}(y - X\beta) = 0$
Setting the gradient to zero gives the normal equations.
$X^{\top}X\hat{\beta} = X^{\top}y$

\hat{\beta} = (X^{\top}X)^{-1}X^{\top}y

The projection view. The normal equations say $X^{\top}(y - X\hat{\beta}) = 0$ : the residual is orthogonal to every column of $X$ . So $\hat{y}$ is the point in the column space of $X$ closest to $y$ , and $H = X(X^{\top}X)^{-1}X^{\top}$ is the projection matrix onto that space. Everything mechanical about OLS follows from this picture — residuals sum to zero when there is an intercept, adding a regressor can never increase the residual sum of squares, and fitted values are uncorrelated with residuals.

Equation 1.3

The one-regressor case

Worth memorising in the second form: a beta is a correlation scaled by the ratio of standard deviations.

\hat{\beta} = \frac{\mathrm{Cov}(x, y)}{\mathrm{Var}(x)} = \rho\,\frac{\sigma_y}{\sigma_x}

$\rho$: The correlation between $x$ and $y$ .

Figure 1.4 · Least squares is one parabola. With one regressor the objective is a parabola in the slope, so it has exactly one minimum and a closed form. Everything OLS is — the normal equations, the projection, the uniqueness — is this shape in as many dimensions as you have regressors.

Proposition 1.5

What a coefficient means

In a multiple regression, $\beta_j$ is the expected change in $y$ for a one-unit change in $x_j$ *holding the other regressors fixed*. That clause is the whole content of the estimate, and by the Frisch–Waugh–Lovell theorem it is literally true: $\beta_j$ equals the simple regression coefficient of the residualised $y$ on the residualised $x_j$ .

Holds when

Holding fixed is a statistical operation, not a causal one — it is not the same as intervening.
If $x_j$ is nearly collinear with the others there is little residual variation left, which is why the standard error explodes.

OLS from three angles

After this lesson you should be able to

The model

The normal equations

The one-regressor case

What a coefficient means

The rest of this lesson is in Premium