The problem Link to heading

When you just want to “run” a map/generator pipeline for its side effects (writing each item to a stream, for example), the reflex is a for or a list(...). The list allocates an entire list that gets thrown away right after, wasting memory for nothing. The for solves the memory issue, but the loop runs in pure Python.

The tip Link to heading

collections.deque accepts a maxlen parameter. With maxlen=0, it pulls each element and discards it immediately, never growing: that is the itertools consume recipe. Instead of materializing a list only to throw it away, you drain the pipeline in constant memory.

The pattern is always deque(map(...), maxlen=0). In a very simple case, it looks like this:

from collections import deque

# runs the side effects without keeping anything
deque(map(print, range(3)), maxlen=0)

With for Link to heading

1
2
3
4
5

for line in readlines(reader_stream):
    dump_json(
        process_operations_batch(parse_json_line(line)),
        writer_stream,
    )

With deque Link to heading

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

from collections import deque
from functools import partial

deque(
    map(
        partial(dump_json, output=writer_stream),
        map(process_operations_batch, map(parse_json_line, readlines(reader_stream))),
    ),
    maxlen=0,
)

Read it from the inside out: parse_json_line on each line, process_operations_batch on the result, and dump_json at the end. The deque(..., maxlen=0) is the drain at the top: it accepts everything and keeps nothing.

And one important detail: map is lazy. On its own it does nothing, no side effect runs. There always has to be someone pulling the items, and here that someone is the deque.

When to use each one? Against list(map(...)), the deque always wins: it trades a throwaway list for constant memory. Against a for, the iteration runs in C, so it only pays off when the loop overhead weighs more than the work per item (the map functions still run in Python). For a simple loop body, the for is still more readable, and that is the form that stayed in capital_gains/cli.py, from a refactoring of a technical assignment (slides here).

That’s it, folks!

See you next time!

{}’s