Generators

Advanced ~25 min read

Generators provide an elegant way to create iterators without the boilerplate of __iter__ and __next__ methods. Using the yield keyword, generator functions automatically implement the iterator protocol. They're perfect for memory-efficient iteration, infinite sequences, and lazy evaluation - generating values only when needed!

Generator Functions and yield

A generator function looks like a regular function, but uses yield instead of return. When called, it returns a generator object (an iterator) without executing the function body immediately. Each call to next() resumes execution until the next yield, which produces a value and pauses the function.

# Generator Functions and yield

def simple_generator():
    """A simple generator function using yield."""
    print("Generator started")
    yield 1
    print("After first yield")
    yield 2
    print("After second yield")
    yield 3
    print("Generator finished")

# Calling a generator function returns a generator object
gen = simple_generator()
print("Generator object:", gen)
print("Type:", type(gen))

# Generators are iterators - they have __iter__ and __next__
print("\nIterating through generator:")
for value in gen:
    print(f"Got value: {value}")

# Or use next() manually
print("\nManual iteration:")
gen2 = simple_generator()
print(next(gen2))  # Executes until first yield, returns 1
print(next(gen2))  # Resumes, executes until next yield, returns 2
print(next(gen2))  # Resumes, executes until next yield, returns 3
# next(gen2) would raise StopIteration

def counter(max_value):
    """Generator that counts up to max_value."""
    current = 0
    while current < max_value:
        yield current
        current += 1

print("\nCounter generator:")
for num in counter(5):
    print(num, end=" ")  # 0 1 2 3 4
print()

Output

Click Run to execute your code

How yield Works:
- When a generator function is called, it returns a generator object without executing
- Calling next() executes until the first yield
- The function pauses at yield and returns the yielded value
- Next call to next() resumes after yield and continues until the next yield or function ends
- When the function ends or returns, StopIteration is raised automatically

Memory Efficiency: Generators vs Lists

Generators generate values on-demand, using minimal memory. Compare this to lists that must store all values in memory at once. For large datasets, generators are essential!

# Memory Efficiency: Generators vs Lists

import sys

def squares_list(n):
    """Creates a list of squares - all in memory."""
    result = []
    for i in range(n):
        result.append(i ** 2)
    return result

def squares_generator(n):
    """Generator that yields squares on-demand."""
    for i in range(n):
        yield i ** 2

# Memory comparison
n = 10000

print("Memory comparison for squares of numbers 0 to", n-1)
print("-" * 60)

# List approach
squares_as_list = squares_list(n)
list_size = sys.getsizeof(squares_as_list)
print(f"List size: {list_size:,} bytes")

# Generator approach
squares_as_gen = squares_generator(n)
gen_size = sys.getsizeof(squares_as_gen)
print(f"Generator size: {gen_size:,} bytes")

print(f"\nGenerator uses {list_size // gen_size}x less memory!")

# Practical example: Reading large file
print("\n" + "-" * 60)
print("Practical example: Processing large dataset")

def read_lines_list(filename):
    """Reads all lines into memory (memory intensive)."""
    with open(filename, 'r') as f:
        return f.readlines()  # All lines in memory at once

def read_lines_generator(filename):
    """Yields lines one at a time (memory efficient)."""
    with open(filename, 'r') as f:
        for line in f:
            yield line.strip()  # One line at a time

# For demonstration with a simple range
def process_numbers_list(n):
    """Processes all numbers at once."""
    return [x * 2 for x in range(n)]

def process_numbers_generator(n):
    """Processes numbers one at a time."""
    for x in range(n):
        yield x * 2

print("\nWith generator, you can stop early:")
gen = process_numbers_generator(1000)
count = 0
for value in gen:
    if count >= 5:  # Stop after 5 values
        break
    print(value, end=" ")
    count += 1
print("\n(Processed only what we needed, not all 1000 items)")

Output

Click Run to execute your code

Pro Tip: Use generators when you don't need all values at once or when dealing with large datasets. You can convert a generator to a list with list(generator) if you need random access, but this defeats the memory advantage!

Generator Expressions

Generator expressions are like list comprehensions but create generators instead of lists. They use parentheses () instead of square brackets []. Perfect for simple generator creation!

# Generator Expressions

# List comprehension: creates a list (all values in memory)
squares_list = [x**2 for x in range(5)]
print("List comprehension:", squares_list)
print("Type:", type(squares_list))

# Generator expression: creates a generator (values generated on-demand)
squares_gen = (x**2 for x in range(5))
print("\nGenerator expression:", squares_gen)
print("Type:", type(squares_gen))

# Convert generator to list if needed
print("Values from generator:", list(squares_gen))

# Generator expressions are memory efficient
print("\nMemory efficiency:")
import sys

# List: all values stored
list_size = sys.getsizeof([x**2 for x in range(1000)])
print(f"List size (1000 items): {list_size:,} bytes")

# Generator: minimal memory
gen_size = sys.getsizeof((x**2 for x in range(1000)))
print(f"Generator size (1000 items): {gen_size:,} bytes")

# Using generator expressions with built-in functions
print("\nUsing with sum() (memory efficient):")
total = sum(x**2 for x in range(10))
print(f"Sum of squares 0-9: {total}")

# Using with max()
print("\nUsing with max():")
max_val = max(x**2 for x in range(10))
print(f"Max square 0-9: {max_val}")

# Filtering with generator expressions
print("\nFiltering with generator expression:")
evens = (x for x in range(20) if x % 2 == 0)
print("Even numbers:", list(evens))

# Chaining generator expressions
print("\nChaining generators:")
numbers = range(10)
squared = (x**2 for x in numbers)
filtered = (x for x in squared if x > 10)
print("Squares > 10:", list(filtered))

Output

Click Run to execute your code

Infinite Sequences

Since generators produce values lazily, they can represent infinite sequences that would be impossible to store in memory. This is perfect for mathematical sequences or continuous data streams.

# Infinite Sequences with Generators

def infinite_counter():
    """Generator that counts forever."""
    count = 0
    while True:
        yield count
        count += 1

def fibonacci():
    """Generator that yields Fibonacci numbers indefinitely."""
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

def powers_of_two():
    """Generator that yields powers of 2: 1, 2, 4, 8, 16, ..."""
    power = 1
    while True:
        yield power
        power *= 2

# Using infinite generators with limits
print("First 10 numbers from infinite counter:")
counter = infinite_counter()
for i in range(10):
    print(next(counter), end=" ")
print()

print("\nFirst 10 Fibonacci numbers:")
fib = fibonacci()
for i in range(10):
    print(next(fib), end=" ")
print()

print("\nFirst 10 powers of 2:")
powers = powers_of_two()
for i in range(10):
    print(next(powers), end=" ")
print()

# Using with itertools.islice (cleaner way to limit)
print("\nUsing itertools.islice:")
from itertools import islice

print("Fibonacci numbers 20-29:")
fib2 = fibonacci()
for num in islice(fib2, 20, 30):  # Skip first 20, take next 10
    print(num, end=" ")
print()

# Practical: Reading from infinite stream
def simulate_data_stream():
    """Simulates reading from an infinite data source."""
    import random
    while True:
        yield random.randint(1, 100)

print("\nSimulated data stream (first 5 values):")
stream = simulate_data_stream()
for i, value in enumerate(stream):
    if i >= 5:
        break
    print(f"Data point {i+1}: {value}")

Output

Click Run to execute your code

Caution: Infinite generators can loop forever if not handled carefully. Always use break conditions or limit the number of values consumed. Never try to convert an infinite generator to a list - it will consume all memory!

Common Mistakes

1. Trying to reuse a consumed generator

# Wrong - generator is exhausted after first iteration
def numbers():
    yield 1
    yield 2
    yield 3

gen = numbers()
print(list(gen))  # [1, 2, 3]
print(list(gen))  # [] - empty!

# Correct - create a new generator each time
def numbers():
    yield 1
    yield 2
    yield 3

print(list(numbers()))  # [1, 2, 3]
print(list(numbers()))  # [1, 2, 3] - new generator

2. Converting infinite generators to lists

# Wrong - will consume all memory and crash!
def infinite_numbers():
    i = 0
    while True:
        yield i
        i += 1

all_numbers = list(infinite_numbers())  # Never completes!

# Correct - use generator directly with limits
def infinite_numbers():
    i = 0
    while True:
        yield i
        i += 1

# Use with itertools.islice or for loop with break
for num in infinite_numbers():
    if num > 10:
        break
    print(num)

3. Confusing return and yield

# Wrong - return ends the generator immediately
def bad_generator():
    yield 1
    return 2  # Generator stops here, never yields 3
    yield 3

# Correct - use yield for all values, return is optional for cleanup
def good_generator():
    yield 1
    yield 2
    yield 3
    # Optional: return value (accessed via StopIteration.value)

Exercise: Fibonacci Generator

Task: Create a generator function that produces Fibonacci numbers. The Fibonacci sequence starts with 0, 1, and each subsequent number is the sum of the previous two.

Requirements:

Create a fibonacci() generator function
Yield numbers starting from 0, then 1, then continue the sequence
The function should be able to generate numbers indefinitely
Test it by printing the first 10 Fibonacci numbers

Output

Click Run to execute your code

Show Solution

def fibonacci():
    """Generator that yields Fibonacci numbers indefinitely."""
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b


# Print first 10 Fibonacci numbers
print("First 10 Fibonacci numbers:")
fib = fibonacci()
for i in range(10):
    print(next(fib))

# Or using a for loop with enumerate
print("\nUsing for loop with break:")
for i, num in enumerate(fibonacci()):
    if i >= 10:
        break
    print(f"F({i}) = {num}")

Summary

Generator Functions: Functions that use yield instead of return, automatically creating iterators
yield: Pauses function execution, returns a value, and resumes on next call
Memory Efficient: Generators produce values on-demand, perfect for large datasets
Generator Expressions: Syntax (expr for item in iterable) creates generators like list comprehensions create lists
Lazy Evaluation: Values are generated only when needed, not all at once
Infinite Sequences: Generators can represent infinite sequences impossible to store in memory
One-time Use: Generators are consumed after iteration; create new ones to iterate again
Under the Hood: Generators automatically implement __iter__ and __next__ methods

What's Next?

Generators are powerful tools for iteration! Next, we'll explore decorators, which allow you to modify or extend function behavior without permanently changing the function itself. Decorators are commonly used with generators for timing, logging, and caching!

Previous Iterators

Next Decorators

Generator Functions and yield

Memory Efficiency: Generators vs Lists

Generator Expressions

Infinite Sequences

Common Mistakes

1. Trying to reuse a consumed generator

2. Converting infinite generators to lists

3. Confusing return and yield

Exercise: Fibonacci Generator

Summary

What's Next?

Enjoying these tutorials?