Bayesian statistics: conjugacy, shrinkage and credible intervals

STAT · Chapter 712 min readAsked at Two Sigma, QuantCo, Citadel, DE Shaw

Assumes Estimation: bias, variance and maximum likelihood.

After this lesson you should be able to

  • Update a beta prior with binomial data and read off the posterior.
  • State the difference between a credible and a confidence interval.
  • Explain why a Bayesian posterior mean is a shrinkage estimator.

Bayesian inference treats the parameter as uncertain and the data as fixed, which is the reverse of the frequentist setup and much closer to how a trader actually reasons. The practical payoff in finance is shrinkage: a prior is a principled way of saying "this estimate is mostly noise, so do not move far from where I started".

Equation 7.1

The update

Posterior is proportional to likelihood times prior. The constant of proportionality is whatever makes it integrate to one, and for conjugate pairs you never need to compute it.

p(θ∣x)∝p(x∣θ) p(θ)p(\theta \mid x) \propto p(x \mid \theta)\,p(\theta)
p(θ)p(\theta)
What you believed before seeing the data.
p(x∣θ)p(x \mid \theta)
The likelihood — the same object as in maximum likelihood.
LikelihoodPriorPosterior
BinomialBeta(α,β)(\alpha, \beta)Beta(α+s,β+f)(\alpha + s, \beta + f)
PoissonGamma(α,β)(\alpha, \beta)Gamma(α+∑x,β+n)(\alpha + \sum x, \beta + n)
Normal, known varianceNormalNormal, with precision-weighted mean
Normal, unknown varianceNormal-inverse-gammaSame family
MultinomialDirichletDirichlet, counts added
Table 7.2 · The conjugate pairs worth knowing. The first row is the one to have at your fingertips: a beta prior means you just add successes to α\alpha and failures to β\beta, which makes the update arithmetic rather than integration.

A prior is a pile of imaginary data. Beta(α,β)(\alpha, \beta) behaves exactly as if you had already seen α\alpha successes and β\beta failures before the experiment began. That reading makes the strength of a prior concrete: Beta(1,1)(1,1) is two observations’ worth and barely moves the answer, while Beta(50,50)(50,50) is a hundred and will dominate a sample of twenty. It also tells you how to elicit one — ask how many observations of evidence your belief is worth, rather than asking someone to draw a density.

Prior mean0.5Observed 7 of 100.7Posterior mean0.643
Figure 7.3 · Where the posterior lands. A Beta(2,2)\mathrm{Beta}(2,2) prior is two heads and two tails of imaginary evidence. Adding seven heads and three real tails gives Beta(9,5)\mathrm{Beta}(9,5), whose mean is 9/149/14 — pulled back from the data towards the prior by exactly the weight of those four pseudo-observations.

Example 7.4

You believe a signal has around a 50%50\% hit rate, with Beta(10,10)(10, 10) worth of conviction. It then hits 7 of 10. What is your posterior?

Show the worked solutionHide the worked solution

Worked solution

  1. Formula
    Beta(α+s,β+f),E=α+sα+β+n\text{Beta}(\alpha + s, \beta + f), \qquad \mathbb{E} = \frac{\alpha + s}{\alpha + \beta + n}
  2. Substitute
    α=10, β=10, s=7, f=3\alpha = 10,\ \beta = 10,\ s = 7,\ f = 3
  3. Solve
    posterior=Beta(17,13)\text{posterior} = \text{Beta}(17, 13)
  4. E=1730\mathbb{E} = \frac{17}{30}
  5. Answer
    ≈56.7%\approx 56.7\%

Sanity check. The sample said 70%70\% and the prior said 50%50\%; the posterior sits at 56.7%56.7\%, much closer to the prior because twenty pseudo-observations outweigh ten real ones. That pull is shrinkage, and it is what stops you over-reacting to a short run.

The rest of this lesson is in Premium

You have read the opening. 9 more sections follow, including 3 worked examples and 3 quick checks.

Start the free 7-day trialSign in

Nothing is charged for 7 days, and you can cancel before then. Or read The law of large numbers and the central limit theorem in full, free.

Bayesian statistics: conjugacy, shrinkage and credible intervals · QuantMax