Conditioning: the tower property

When a quantity is hard to average directly, average it in two stages: first within each scenario, then across scenarios. Choosing what to condition on is the entire skill, and the usual right answer is "the first step", "the hidden parameter", or "which case we are in".

Equation 3.18

Law of total expectation

The inner expectation is a function of $Y$ , so it is itself a random variable. The outer expectation averages it over $Y$ .

\mathbb{E}[X] = \mathbb{E}\big[\,\mathbb{E}[X \mid Y]\,\big]

$X$: The quantity you want the average of.
$Y$: What you condition on — the thing that, once known, makes $X$ easy.

Equation 3.19

The form you will actually write

A weighted average of the conditional answers, weighted by how likely each case is.

\mathbb{E}[X] = \sum_{y} \Pr(Y = y)\,\mathbb{E}[X \mid Y = y]

What conditioning buys you. It converts one hard problem into several easy ones plus some bookkeeping. If you cannot see the distribution of $X$ , ask what single piece of information would make it obvious. Learning the first coin flip, the hidden bias, or which branch you took usually collapses the problem to something you can write down in a line.

Equation 3.20

Law of total variance

Total variability splits into the average spread inside each scenario, plus the spread of the scenario averages.

\operatorname{Var}(X) = \underbrace{\mathbb{E}\big[\operatorname{Var}(X \mid Y)\big]}_{\text{within-group}} + \underbrace{\operatorname{Var}\big(\mathbb{E}[X \mid Y]\big)}_{\text{between-group}}

Proposition 3.21

Reading the two terms

The first term is the noise you would still face even if someone told you $Y$ . The second is the variability that knowing $Y$ would remove. So the second term is exactly the value of the information in $Y$ — which is why this decomposition shows up whenever a question is about how much an observation tells you.

Holds when

Both terms are non-negative, so conditioning can never increase the average conditional variance above the total.
If $Y$ determines $X$ , the first term is zero and all variance is between-group.

Example 3.22

A coin of unknown bias

A coin is drawn at random from a bag: with probability $1/2$ it is fair, and with probability $1/2$ it lands heads with probability $0.8$ . You flip the chosen coin $10$ times. What are the mean and variance of the number of heads?

Show the worked solutionHide the worked solution

Worked solution

Formula $\mathbb{E}[X] = \mathbb{E}\big[\mathbb{E}[X \mid p]\big], \qquad \operatorname{Var}(X) = \mathbb{E}\big[\operatorname{Var}(X \mid p)\big] + \operatorname{Var}\big(\mathbb{E}[X \mid p]\big)$
Substitute $X \mid p \sim \mathrm{Binomial}(10, p), \qquad p \in \{0.5,\ 0.8\} \text{ each with probability } \tfrac{1}{2}$
Solve $\mathbb{E}[X] = \tfrac{1}{2}(10)(0.5) + \tfrac{1}{2}(10)(0.8) = 2.5 + 4 = 6.5$ Conditional means are $5$ and $8$ ; average them.
$\mathbb{E}\big[\operatorname{Var}(X \mid p)\big] = \tfrac{1}{2}(10)(0.5)(0.5) + \tfrac{1}{2}(10)(0.8)(0.2) = 1.25 + 0.8 = 2.05$ Within-group: the binomial noise you face once the coin is known.
$\operatorname{Var}\big(\mathbb{E}[X \mid p]\big) = \operatorname{Var}(\{5, 8\}) = \left(\tfrac{8-5}{2}\right)^{2} = 2.25$ Between-group: a two-point variable takes value $5$ or $8$ with equal probability, so its variance is the square of half the gap.
$\operatorname{Var}(X) = 2.05 + 2.25 = 4.30$
Answer $\mathbb{E}[X] = 6.5, \qquad \operatorname{Var}(X) = 4.30$

Sanity check. A single binomial with $p = 0.65$ would have variance $10(0.65)(0.35) = 2.275$ . The mixture is more variable, as it must be: not knowing which coin you hold adds uncertainty rather than removing it.