Python Iterators & Generators: The yield Keyword & Memory Optimization

When working with large datasets (such as a 10 GB server log file or an infinite mathematical stream), loading all elements into a Python list will crash your computer with a MemoryError. Python solves this through the Iterator Protocol and Generators, which generate values lazily on demand using the yield keyword.

1. The Iterator Protocol: iter() and next()

Any sequence in Python can be turned into an iterator:

fruits = ["Apple", "Banana", "Cherry"]

# Create an iterator:
fruit_iter = iter(fruits)

# Fetch items one by one:
print(next(fruit_iter))  # 'Apple'
print(next(fruit_iter))  # 'Banana'
print(next(fruit_iter))  # 'Cherry'
# next(fruit_iter)       # Raises StopIteration exception!

2. Creating Generators with the yield Keyword

When a function contains yield, calling it does not execute the code immediately; it returns a generator object:

# Generator for Fibonacci Sequence:
def fibonacci(limit):
    a, b = 0, 1
    while a < limit:
        yield a       # Pauses here and yields value!
        a, b = b, a + b

# Using the generator in a for loop:
for num in fibonacci(50):
    print(num, end=" ")
# Output: 0 1 1 2 3 5 8 13 21 34

3. Memory Comparison: List vs Generator

import sys

# List of 1,000,000 integers:
large_list = [x for x in range(1_000_000)]
print("List Memory Consumption:     ", sys.getsizeof(large_list), "bytes (~8.5 MB)")

# Generator of 1,000,000 integers:
large_gen = (x for x in range(1_000_000))
print("Generator Memory Consumption:", sys.getsizeof(large_gen), "bytes (ONLY 200 bytes!)")

Frequently Asked Questions (FAQ)