Causal cases: treatment effects, confounding and surprising results

A causal question asks what would have happened otherwise, and the otherwise is never observed. Every method in this area is a way of constructing a credible counterfactual, and every failure is a counterfactual that was not credible.

Definition 3.1

The treatment effect

Average treatment effect, $\tau = \mathbb{E}[Y(1) - Y(0)]$ — The difference between the outcome with treatment and without, for the same unit. Only one is ever observed, which is the fundamental problem of causal inference: the data are missing by construction, not by accident.

Proposition 3.2

Selection bias is the default

Whoever received the treatment usually differs from whoever did not, in ways that also affect the outcome. Customers who used the support chat had a problem; patients who took the drug were sicker; funds that report performance are the ones that survived. The naive comparison mixes the treatment effect with those differences and can easily reverse its sign.

Holds when

Ask who *chose* to be treated and why — the selection mechanism is the confounder.
Controlling for observables handles only the confounders you thought of and measured.
When selection is on something unobservable, no regression will fix it and you need a design.

Figure 3.3 · A treatment that wins twice and loses overall. The treatment wins in both subgroups and loses in the pool, because it was given to far more of the severe cases. Nothing here is a statistical artefact to be corrected — the grouping variable is the finding, and reporting the pooled number alone would be the error.

Simpson’s paradox is not a paradox. A treatment can help in every subgroup and appear harmful overall, if the subgroups have different base rates and different treatment shares. Nothing contradictory has happened: the aggregate comparison is between different mixtures of subgroups, so it is not answering the question it appears to answer. The lesson is that the correct level of aggregation is determined by the causal structure — condition on what caused the assignment, not on everything available — and that an aggregate difference with no causal story behind it is not evidence of anything.

Example 3.4

Treatment A succeeds in 72% of 100 cases and B in 78% of 100. Split by severity, A wins in both. How?

Show the worked solutionHide the worked solution

Worked solution

Formula $\text{aggregate} = \text{weighted average over subgroups}$
Substitute $\text{A: } 90/100 \text{ severe}, \quad \text{B: } 10/100 \text{ severe}$
Solve $\text{Severe: A } 63/90 = 70\%, \quad \text{B } 6/10 = 60\%$
$\text{Mild: A } 9/10 = 90\%, \quad \text{B } 72/90 = 80\%$
$\text{A overall } 72/100, \quad \text{B overall } 78/100$
Answer $\text{A is better in each group and worse overall}$

Sanity check. A was given overwhelmingly to severe cases and B to mild ones, so the aggregate compares a hard sample against an easy one. If severity determined which treatment was given, the subgroup comparison is the causal one and the aggregate is meaningless.

Causal cases: treatment effects, confounding and surprising results

After this lesson you should be able to

The treatment effect

Selection bias is the default

The rest of this lesson is in Premium