You roll a fair die and may re-roll once, keeping the second value if you do. What is the game worth?

Work backwards from the second roll. Re-rolling hands you a fresh die worth $3.5$, so the only decision on the first roll is whether the face beats $3.5$: keep $4$, $5$ and $6$, re-roll $1$, $2$ and $3$. That turns the game into the average of $\max(f, 3.5)$ over the six faces, $(3.5 + 3.5 + 3.5 + 4 + 5 + 6)/6 = 25.5/6 = 4.25$. The usual slip is to re-roll a $4$ because it is below the value of the game; the comparison is against the continuation value of $3.5$, not against $4.25$.

Takeaway: The threshold is the continuation value, not the mean of the distribution.