Python Iterators

3Python Iterators: explained step by step

In the generators lesson you met iter() and next() in passing. This lesson opens the box: how a for loop really asks for its items, what an iterator is, and how to build your own. Once you can see the protocol, questions like "why does this only work once?" stop being mysteries.

Iterable or iterator?

Two words, two different jobs:

• An iterable is anything you can loop over: a list, a string, a dict, a range, a file. It can give you an iterator when you ask.
• An iterator is the object that does the walking. It remembers its position and hands out the next item each time you call next() on it.

You ask an iterable for its iterator with the built-in iter():

scores = [90, 75, 60]
it = iter(scores)

print(type(scores).__name__, type(it).__name__)
print(next(it))
print(next(it))
print(list(it))

The list never moves. It is just storage. The iterator is a separate small object that says "I am at position 2 of that list". That is why next(it) gives 90 and then 75, and why list(it) afterwards finds only the one item that is left.

What a for loop really does

for item in items: is shorthand for three steps. Python calls iter(items) once, calls next() on the result before every pass, and stops quietly when next() raises StopIteration. Written by hand, a loop looks like this:

cart = ["milk", "bread", "eggs"]

it = iter(cart)
while True:
    try:
        item = next(it)
    except StopIteration:
        break
    print("Packing", item)
print("Done")

StopIteration is not an error here. It is the iterator's way of saying "I have nothing more". The for loop catches it for you, which is why you never see it in everyday code.

Iterators are used up

An iterator only moves forward. After it has handed out its last item it is exhausted and stays empty: it never rewinds. A list can create as many fresh iterators as you like, so looping over a list twice works. Reading one iterator twice does not:

temps = iter([21, 25, 19])

print(list(temps))
print(list(temps))
print(sum(temps))

Everything that reads items uses them up: for, list(), sum(), max(), sorted(), the in operator and unpacking (a, b = it). A loop that ends early with break leaves the remaining items waiting for the next reader:

queue = iter(["Asha", "Ben", "Chen", "Dee"])

for name in queue:
    print("Serving", name)
    if name == "Ben":
        break

print("Still waiting:", list(queue))

next() with a default

Calling next() on an exhausted iterator raises StopIteration. When running out is a normal possibility, pass a second argument. Python returns it instead of raising:

pending = iter(["fix bug", "write tests"])
print(next(pending, "nothing left"))
print(next(pending, "nothing left"))
print(next(pending, "nothing left"))

Building your own iterator

This part uses a class, so look at the classes lesson first if __init__ and self are new to you. An object is an iterator when it has two methods:

• __next__(self) returns the next item, or does raise StopIteration when there are no more.
• __iter__(self) returns the iterator itself (return self), so that for and iter() accept it.

class Evens:
    def __init__(self, limit):
        self.limit = limit
        self.number = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.number > self.limit:
            raise StopIteration
        value = self.number
        self.number += 2
        return value

print(list(Evens(10)))
print(*Evens(4))

The attributes limit and number are the iterator's memory, and that is exactly why it cannot be rewound. Why not just return a list? Because an iterator makes each item only when it is asked for, so it can describe a huge or even endless sequence, or data you cannot hold in memory at once, such as the lines of a big file. A generator function is a shortcut for writing the same thing in fewer lines; the class shows what happens underneath.

Make a class reusable: return a fresh iterator

A class whose __iter__ returns self is a one-shot iterator, like Evens above. Often you want something you can loop over again and again, like a list. Keep the data in the class and let __iter__ hand out a new iterator on every call:

class Playlist:
    def __init__(self, songs):
        self.songs = songs

    def __iter__(self):
        return iter(self.songs)      # a brand-new iterator each time

mix = Playlist(["Intro", "Waves", "Outro"])
print(list(mix))
print(list(mix))
pairs = [(a, b) for a in mix for b in mix]
print(len(pairs))

Playlist is an iterable but not an iterator: it has no __next__. Every loop gets its own private walker, so even two nested loops over the same playlist (3 x 3 = 9 pairs) do not disturb each other.

iter(function, sentinel)

iter() has a second form. iter(function, stop_value) calls function() with no arguments again and again, hands out each result, and stops as soon as the result equals stop_value (the sentinel, which is not handed out). It is handy for "read until an end marker":

import io

log = io.StringIO("start\nrunning\nEND\nignored\n")
for line in iter(log.readline, "END\n"):
    print(line.strip())

io.StringIO behaves like an open file that lives in memory. Notice log.readline has no parentheses: iter() needs the function itself so that it can call it repeatedly.

Which one do I need?

• Just visiting every item? A plain for loop does it all.
• Want one item by hand, such as a header line or the first match? next(it) or next(it, default).
• Need to loop more than once? Keep a list, or build a class whose __iter__ returns a fresh iterator.
• Need a lazy or endless sequence? Write an iterator class or a generator.