What are and how to use Generators in Python

3 min
Advanced

In Python, generators are a tool that allows generating sequences of data, focused on efficiency as an objective.

To achieve this, unlike lists, generators do not store all elements in memory, but produce them on the fly.

This makes them very useful for working with large datasets or continuous data streams, as elements are generated only when needed (and only if needed).

Their main advantage is that they are much more memory efficient than lists (because they do not store all elements in memory at once)

What are generators

A generator is a function that uses the reserved word yield instead of return (yield could be interpreted as “yield the turn”).

The yield keyword allows a function to produce a value, but at the same time “remember” its state. This allows execution to resume from that point on the next call.

Let’s see it with an example,

def simple_generator():
    yield 1
    yield 2
    yield 3

Copied!

When simple_generator() is called, the code inside the function is not executed immediately. Instead, it returns a generator object that can be iterated to produce the values.

The created generator object is not callable by itself. Instead, the next value is “requested” from it using next(). For example like this,

gen = simple_generator()
print(next(gen))  # Output: 1
print(next(gen))  # Output: 2
print(next(gen))  # Output: 3

Copied!

In this example

We use the generator function simple_generator() to create a generator object gen
next(gen) is used to get the next value produced by the generator
When there are no more values to produce, the generator raises a StopIteration exception

Generators with Loops

Most generators are created using loops to produce a sequence of values.

def count_to(n):
    counter = 1
    while counter <= n:
        yield counter
        counter += 1

Copied!

This generator function

Starts by returning counter with a value of 1
Each time the function is called, it “wakes up” in the while loop. It increments the value, does the loop, and returns the new counter
When counter equals 5, the loop ends, and the sequence terminates.

It is also very common for generators to be used from within a loop. For example, it’s very common to use them with for.

for number in count_to(5):
    print(number)

# output: 1 2 3 4 5

Copied!

Combined Generators

Generators can be nested to handle more complex sequences.

def nested_generator():
    yield from range(3)
    yield from range(10, 13)

for number in nested_generator():
    print(number)

# output: 0 1 2 10 11 12

Copied!

In this example, yield from delegates value production to another generator or iterable.

First, the function returns the values generated by range(3)
When that finishes, it supplies values from range(10, 13)
When both are finished, the sequence ends

Generators with expressions

Python also allows creating generators using a syntax similar to list comprehensions, known as “generator comprehensions”.

gen = (x**2 for x in range(10))

for number in gen:
    print(number)

# output: 0 1 4 9 16 25 36 49 64 81

Copied!