Estimation: bias, variance and maximum likelihood

An estimator is a recipe, and judging one means asking two separate questions: does it point at the right answer on average, and how much does it move about? Almost every practical choice in estimation is a trade between those two, and in finance the trade usually favours accepting bias.

Equation 2.1

The decomposition

Squared error splits cleanly into a systematic part and a noisy part. Minimising the total is what you actually want; unbiasedness constrains only the first term.

\mathrm{MSE}(\hat\theta) = \mathbb{E}\big[(\hat\theta - \theta)^2\big] = \mathrm{Bias}(\hat\theta)^2 + \mathrm{Var}(\hat\theta)

$\mathrm{Bias} = \mathbb{E}[\hat\theta] - \theta$: How far off you are on average.
$\mathrm{Var}$: How much the estimate moves from sample to sample.

Property	Statement	Why it matters
Unbiased	$\mathbb{E}[\hat\theta] = \theta$	Right on average — says nothing about any one sample
Consistent	$\hat\theta \to \theta$ as $n \to \infty$	Enough data eventually gets you there
Efficient	Smallest variance in its class	Least wasteful use of the data you have
Sufficient	Captures all the information about $\theta$	You can throw the rest of the data away

Table 2.2 · What the words mean. Unbiased and consistent are independent properties: an estimator can be unbiased and never converge, or biased at every

n

and still consistent.

Derivation 2.3

Maximum likelihood

Choose the parameter that makes the data you saw most probable. The standard recipe is three steps.

$L(\theta) = \prod_i f(x_i; \theta)$
The likelihood — a function of $\theta$ , with the data fixed.
$\ell(\theta) = \sum_i \ln f(x_i; \theta)$
Take logs, which turns the product into a sum and never moves the maximum.
$\frac{\partial \ell}{\partial \theta} = 0$

\hat\theta_{\text{MLE}}

Model	MLE	Note
Bernoulli $(p)$	$\bar{x}$	The sample proportion
Normal $(\mu, \sigma^2)$	$\bar{x}$ and $\frac{1}{n}\sum(x_i - \bar{x})^2$	The variance MLE is biased low
Exponential $(\lambda)$	$1/\bar{x}$	Reciprocal of the sample mean
Poisson $(\lambda)$	$\bar{x}$	Mean and variance are the same parameter
Uniform $(0, \theta)$	$\max x_i$	Biased low, and not found by differentiating

Table 2.4 · The MLEs worth knowing. The last row is the one interviewers use: the likelihood is maximised at a boundary, so calculus finds nothing and you have to reason about the support instead.

Proposition 2.5

Why the sample variance has $n-1$

The MLE divides by $n$ and is biased low, because it measures deviations from the sample mean rather than the true mean — and the sample mean sits, by construction, in the middle of the data you have. Dividing by $n-1$ corrects exactly for that one degree of freedom spent estimating the mean.

Holds when

The correction matters at small $n$ and is negligible beyond a few hundred observations.
The unbiased *variance* estimator does not give an unbiased *standard deviation*, because the square root is not linear.

Why bias is often the right trade. Unbiasedness has intuitive appeal and no claim on optimality. If a biased estimator has much smaller variance, its mean squared error is lower and it is simply better. That is the entire case for shrinkage — pulling a noisy estimate toward a structured prior — and in finance the case is overwhelming, because the quantities being estimated have tiny signal-to-noise. A shrunk covariance matrix is biased and enormously more useful than the sample one; the same goes for ridge regression and for shrinking a mean return toward zero.

Estimation: bias, variance and maximum likelihood

After this lesson you should be able to

The decomposition

Maximum likelihood

Why the sample variance has $n-1$

The rest of this lesson is in Premium