Diagnostics: reading residuals, leverage and influence

A regression output is four numbers and a great deal of hidden behaviour. The diagnostics exist to surface that behaviour, and each one detects a specific failure — which means knowing what a plot *cannot* reveal is as useful as knowing what it can.

Plot	Looks for	A problem looks like
Residuals against fitted	Non-linearity, heteroskedasticity	Curvature, or a widening fan
Residuals against each regressor	A missing transformation	Structure in one variable only
Q-Q plot of residuals	Non-normality	Heavy tails curving away at the ends
Residuals against time or order	Autocorrelation, regime change	Runs of the same sign
Scale–location	Heteroskedasticity specifically	A trend in the spread
Residuals against leverage	Influential points	A point outside the Cook’s distance contour

Table 4.1 · What each plot shows. None of these can detect endogeneity. Residuals are orthogonal to the regressors by construction, so an omitted variable correlated with a regressor leaves no trace in any plot.

Definition 4.2

Leverage, outlier, influence

Three different things, $h_{ii} = \big[X(X^\top X)^{-1}X^\top\big]_{ii}$ — A high-*leverage* point is unusual in $X$ — far from the centre of the regressors. An *outlier* has a large residual, unusual in $y$ given $x$ . A point is *influential* only when it is both: unusual in $X$ and not on the line the rest of the data describe. The diagonal of the hat matrix measures leverage, and the rule of thumb is that $h_{ii}$ above $2(k+1)/n$ is worth a look.

Why influence needs both. A point in the middle of the data with a large residual barely moves the fit — there is plenty of other data at that $x$ to hold the line in place. A point far out in $X$ but sitting exactly on the trend also changes nothing; it simply confirms it, with a lot of weight. It is the combination that does damage: a far-out point off the line pivots the whole regression toward itself, and can single-handedly create or destroy a coefficient. Cook’s distance is exactly the product of the two ingredients, which is why it is the statistic that matters.

Equation 4.3

Cook’s distance

How much the fitted values move when observation $i$ is deleted — residual size times leverage.

D_i = \frac{e_i^2}{(k+1)s^2}\cdot\frac{h_{ii}}{(1-h_{ii})^2}

$e_i$: The residual: how much of an outlier the point is in $y$ .
$h_{ii}$: The leverage: how unusual it is in $X$ .

Figure 4.4 · One observation pivots the fitted line. The highlighted observation sits far from the others in both factor exposure and response. Including it pulls the fitted slope from about

0.50

0.75

. Cook’s distance detects this combination of leverage and residual size; the honest report shows the fit with and without the point.

Diagnostics: reading residuals, leverage and influence

After this lesson you should be able to

Leverage, outlier, influence

Cook’s distance

The rest of this lesson is in Premium