Goodness of fit: R squared, F tests and information criteria

Fit statistics answer "how much of the variation did I explain?" and are routinely over-read. In return prediction an $R^2$ of one per cent is a strong result, so the absolute number carries almost no information — what matters is comparing models on the same data, honestly penalised for complexity.

Equation 3.1

R squared and its adjustment

The share of variance explained, and the version that charges for parameters.

R^2 = 1 - \frac{\mathrm{RSS}}{\mathrm{TSS}}, \qquad \bar{R}^2 = 1 - \frac{\mathrm{RSS}/(n-k-1)}{\mathrm{TSS}/(n-1)}

$\mathrm{RSS}$: Residual sum of squares — what the model failed to explain.
$k$: Number of regressors. Adjusted $R^2$ can fall, and can even go negative.

Proposition 3.2

Why $R^2$ never falls

Adding a regressor enlarges the space you are projecting onto, and the projection onto a larger space is at least as close to the target. In the worst case the new coefficient comes out zero and nothing changes; in any finite sample it will be slightly non-zero by chance, so $R^2$ rises strictly. That makes $R^2$ useless for comparing models of different sizes.

Holds when

In a simple regression $R^2 = \rho^2$ ; in a multiple regression it is the squared correlation between $y$ and $\hat{y}$ .
It is not comparable across different dependent variables — regressing returns and regressing prices produce incomparable numbers.

Why a one per cent $R^2$ is good news. Returns are mostly unforecastable, so nearly all their variance is noise you were never going to explain. A daily cross-sectional signal with an information coefficient of $0.10$ — very strong — has an $R^2$ of one per cent, and that is enough to build a business on because you apply it across thousands of positions and thousands of days. Reading $R^2$ as a quality score imported from a physics or marketing context is one of the fastest ways to sound unfamiliar with financial data.

Equation 3.3

The F test for nested models

Tests whether the $q$ extra regressors in the unrestricted model add anything jointly.

F = \frac{(\mathrm{RSS}_r - \mathrm{RSS}_u)/q}{\mathrm{RSS}_u/(n - k - 1)}

$q$: Number of restrictions being tested.
$\mathrm{RSS}_r, \mathrm{RSS}_u$: Restricted and unrestricted residual sums of squares.

Figure 3.4 · Why $R^2$ cannot fall. On

100

observations.

R^2

rises with every regressor because adding a column can only shrink the residual; the adjusted version charges rent for each one and turns over, which is the only reason it is worth reporting.

Goodness of fit: R squared, F tests and information criteria

After this lesson you should be able to

R squared and its adjustment

Why $R^2$ never falls

The F test for nested models

The rest of this lesson is in Premium