Idiomatic Python: generators, itertools and the gotchas

Python is the working language of quantitative research, and interviewers use it to check two things: whether you write code that reads well, and whether you know where the language will quietly betray you. Both are a short list.

Proposition 1.1

Generators and laziness

A generator produces values on demand rather than building a list, so it uses constant memory and can represent an unbounded sequence. On a large file or a long market-data stream that is the difference between working and exhausting memory.

Holds when

A generator expression is a comprehension with parentheses: sum(x*x for x in data) never builds the list.
They are single-pass — consuming one exhausts it, and a second loop sees nothing.
yield from delegates to another generator and is the clean way to compose them.

Trap	What happens	Fix
Mutable default argument	The default is created once and shared across calls	Default to `None` and create inside
Late binding in closures	Loop variables are read at call time, not definition	Bind with a default argument
`is` versus `==`	Identity, not equality; small ints are cached and confuse the issue	Use `==` except for `None`
Modifying a list while iterating	Elements are skipped silently	Iterate over a copy, or build a new list
Shallow copy of nested structures	Inner objects are still shared	`copy.deepcopy`, or restructure

Table 1.2 · The gotchas. The first two are the ones asked about most, because both produce code that works in testing and fails in a way that looks like a logic error rather than a language feature.

Why mutable defaults behave that way. A default argument is evaluated once, when the def statement runs, not each time the function is called. So a list written as a default is a single object living on the function, and every call that relies on the default mutates the same one. It is not a bug so much as a consequence of functions being ordinary objects with attributes — and knowing *why* means you will also predict the behaviour of a default built from a computation, which is evaluated once for the same reason.

# Mutable default: the same list is reused across calls.
def bad(x, acc=[]):
    acc.append(x)
    return acc

bad(1)      # [1]
bad(2)      # [1, 2]  -- surprising

def good(x, acc=None):
    acc = [] if acc is None else acc
    acc.append(x)
    return acc

# Late binding: every lambda reads i at call time.
fns = [lambda: i for i in range(3)]
[f() for f in fns]            # [2, 2, 2]

fns = [lambda i=i: i for i in range(3)]
[f() for f in fns]            # [0, 1, 2]

Listing 1.3 · The two classic traps. Both examples run without error and return the wrong thing, which is what makes them worth recognising on sight. Time n/a · Space n/a.

Idiomatic Python: generators, itertools and the gotchas

After this lesson you should be able to

Generators and laziness

The rest of this lesson is in Premium