Regression and econometrics

The model

Linear in the parameters — not necessarily in the variables, since $x^2$ and $\log x$ are perfectly acceptable columns.

The one-regressor case

Worth memorising in the second form: a beta is a correlation scaled by the ratio of standard deviations.

The variance of the estimator

Under homoskedastic, uncorrelated errors. The diagonal gives each coefficient’s squared standard error; the off-diagonals say how their estimation errors move together. Correlated regressors make $X^\top X$ nearly singular and blow up the diagonal of its inverse.

The sandwich estimator

White’s heteroskedasticity-robust covariance: the "bread" is the usual $(X^\top X)^{-1}$ and the "meat" uses each observation’s own squared residual. Newey–West adds weighted cross-products of nearby residuals to handle autocorrelation; clustered versions sum within groups.

R squared and its adjustment

The share of variance explained, and the version that charges for parameters.

The F test for nested models

Tests whether the $q$ extra regressors in the unrestricted model add anything jointly.

What one more regressor adds

A new regressor explains a share of what was left unexplained, equal to its squared partial correlation with $y$ given the existing regressors. It can only raise $R^2$, and it raises it by little when the model already explains most of the variance.

Cook’s distance

How much the fitted values move when observation $i$ is deleted — residual size times leverage.

Studentised residuals

A residual scaled by its own standard error, estimated without observation $i$. High-leverage points have small raw residuals because they pull the line toward themselves; studentising corrects for that, so outliers are compared on a common scale.

The Wald estimator

With a binary instrument, IV is the effect of the instrument on the outcome divided by its effect on the treatment. It scales up the "intention-to-treat" effect by how much the instrument actually moved the treatment, and it estimates the effect for those whose treatment the instrument changed.

Omitted-variable bias

Leaving out $x_2$ biases the coefficient on $x_1$ by the true effect of $x_2$ times the regression of $x_2$ on $x_1$.

The two penalties

The same least-squares fit plus a penalty on the size of the coefficients — squared for ridge, absolute for lasso.

The elastic net

A blend of the two penalties. The $L_1$ part still sets coefficients to zero; the $L_2$ part makes correlated features share weight rather than having the lasso pick one arbitrarily. $\alpha$ between $0$ (ridge) and $1$ (lasso) is chosen by cross-validation alongside $\lambda$.

Logistic regression

Model the log-odds linearly, which maps the whole real line onto $(0,1)$ so the fitted probability is always valid.

The quantile loss

Quantile regression minimises this asymmetric loss. For the median ($\tau = 0.5$) it is half the absolute error; for a low quantile it charges little for over-predicting and a lot for under-predicting, which pushes the fit down to the chosen quantile.