Floating point: IEEE 754, cancellation and why prices are integers

NUM · Chapter 611 min readAsked at Hudson River Trading, Jump, Optiver, IMC

Assumes ODE and PDE solvers: finite differences and stability.

After this lesson you should be able to

  • Say what a double can and cannot represent exactly.
  • Recognise catastrophic cancellation and restructure to avoid it.
  • Explain why exchanges store prices as integers.

Floating point is an approximation that is almost always good enough and occasionally catastrophic. The failures are not random — they happen at predictable places, and knowing those places is the difference between code that is correct and code that has not been tested at the boundary yet.

Proposition 6.1

What a double holds

A 64-bit double has one sign bit, eleven exponent bits and fifty-two mantissa bits, giving about sixteen significant decimal digits. The spacing between representable numbers scales with magnitude, so precision is *relative*: near 1 the gap is about 2×10−162 \times 10^{-16}, and near 10910^9 it is about 10−710^{-7}.

Holds when

  • Integers are exact up to 2532^{53}, which is about 9×10159 \times 10^{15}.
  • Decimal fractions like 0.10.1 have no exact binary representation, in the same way 1/31/3 has no exact decimal one.
  • Never test floats for equality — compare against a tolerance scaled to the magnitude.

Catastrophic cancellation. Subtracting two nearly equal numbers destroys precision in a way that nothing later recovers. Each input carries sixteen good digits; if the first eight agree, they cancel and the result has eight good digits — but it is stored in a double that looks just as precise as before, so the loss is invisible. Everything downstream then inherits an error the code gives no sign of. This is why computing a variance as E[X2]−E[X]2\mathbb{E}[X^2] - \mathbb{E}[X]^2 can return a small negative number: two large nearly equal quantities were subtracted, and the true answer was smaller than the error.

Example 6.2

Compute the variance of a million prices near $10,000\$10{,}000 with a true standard deviation of $0.01\$0.01, using E[X2]−E[X]2\mathbb{E}[X^2] - \mathbb{E}[X]^2. What goes wrong?

Show the worked solutionHide the worked solution

Worked solution

  1. Formula
    Var=E[X2]−(E[X])2\mathrm{Var} = \mathbb{E}[X^2] - \left(\mathbb{E}[X]\right)^2
  2. Substitute
    E[X2]≈108,(E[X])2≈108\mathbb{E}[X^2] \approx 10^{8}, \quad \left(\mathbb{E}[X]\right)^2 \approx 10^{8}
  3. Solve
    true variance=10−4\text{true variance} = 10^{-4}
  4. relative size=10−4/108=10−12\text{relative size} = 10^{-4}/10^{8} = 10^{-12}
  5. double precision≈10−16 relative\text{double precision} \approx 10^{-16} \text{ relative}
  6. Answer
    about four good digits survive, and it can come out negative\text{about four good digits survive, and it can come out negative}

Sanity check. Move the prices to $1,000,000\$1{,}000{,}000 and there is nothing left at all. Welford’s algorithm updates the mean and the sum of squared deviations incrementally and never forms the difference, so it stays accurate regardless of the offset.

Both inputs16Agreeing to 6 digits10Agreeing to 12 digits4Agreeing to 15 digits1
Figure 6.3 · Digits left after subtracting two near-equal numbers. Subtraction does not lose precision — it reveals how little was there. Two doubles agreeing to twelve digits leave four, which is why an implied-volatility solver written around a difference of two nearly equal prices fails long before the maths does.

The rest of this lesson is in Premium

You have read the opening. 11 more sections follow, including 3 worked examples and 3 quick checks.

Start the free 7-day trialSign in

Nothing is charged for 7 days, and you can cancel before then. Or read Complexity: reading it off, and deriving it in full, free.