Generators and Iterators in Python

The protocol a for-loop is built on

for x in thing works on anything that implements two methods: __iter__, which returns an iterator, and __next__ on that iterator, which returns the next value or raises StopIteration when there is nothing left. A list is not itself an iterator — it is iterable, meaning iter(my_list) produces a fresh iterator object each time, which is why two separate for loops over the same list both start from the beginning.

numbers = [1, 2, 3]
it = iter(numbers)
next(it)  # 1
next(it)  # 2
next(it)  # 3
next(it)  # raises StopIteration

A for loop is syntax that calls iter() once and next() repeatedly, catching StopIteration for you. Writing the protocol out by hand is rare, but recognising it explains why a generator — which implements exactly this protocol — slots into a for loop, a list comprehension, or sum() without any special-case code anywhere.

What yield actually does

A function with yield anywhere in its body is not a function anymore — calling it returns a generator object immediately, without running a single line of the body:

def count_up_to(n):
    i = 0
    while i < n:
        yield i
        i += 1
        print(f'resumed after yielding {i - 1}')

gen = count_up_to(3)   # nothing has printed yet
next(gen)               # 0
next(gen)               # prints "resumed after yielding 0", then returns 1

Each call to next() runs the body until it hits yield, hands back that value, and then freezes — every local variable, the current line, the state of the while loop — exactly where it stopped. The next call resumes from that frozen point rather than starting over. This is the entire mechanism: a generator is a function whose stack frame the interpreter keeps alive between calls, instead of discarding it on return.

Why that matters for memory

def read_large_file(path):
    with open(path) as f:
        for line in f:
            yield line.strip()

for line in read_large_file('access.log'):
    process(line)

At no point does this hold the whole file in memory — one line exists at a time, processed and discarded before the next is read. The list-building equivalent, [line.strip() for line in open(path)], is fine for a small file and a real problem for one that does not fit in RAM. This is the practical reason generators exist: not elegance, but the ability to process a sequence of unbounded size in bounded memory.

Generator expressions

The same laziness is available without def or yield, using parentheses instead of a comprehension's brackets:

total = sum(n * n for n in range(1_000_000))

No intermediate list of a million squares is ever built — each square is produced, added to the running total, and discarded. Swapping the parentheses for square brackets would build that list first and then sum it, using far more memory for the same result. A generator expression is the right default whenever the values are consumed once and never indexed back into.

A generator is exhausted, not reset

gen = (n for n in range(3))
list(gen)  # [0, 1, 2]
list(gen)  # [] — already exhausted

Unlike a list, a generator has no way to start over: once StopIteration has been raised, every further call to next() raises it again. Code that needs to iterate the same data twice needs either a fresh generator call each time or a plain list, and treating a generator like a reusable collection is a common source of a bug that looks like the data silently disappeared.

Chaining generators into a pipeline

def parse(lines):
    for line in lines:
        yield line.split(',')

def filter_valid(rows):
    for row in rows:
        if len(row) == 3:
            yield row

pipeline = filter_valid(parse(open('data.csv')))

Nothing runs until pipeline is iterated — each stage pulls one item from the stage before it, on demand. This is how libraries process arbitrarily large datasets through multiple transformation steps without ever materialising the data in full at any single stage.