Python Nested Data Structures

3Nested Data: explained step by step

Real data is rarely flat. An order has a customer, and the customer has a city. A student has a name and a list of scores. Python copes by letting a list or dictionary hold other lists and dictionaries. That is called nesting, and it stops looking scary once you read it one layer at a time.

Four shapes you will meet again and again

students = [                                        # list of dicts: one record per item
    {"name": "Asha", "score": 82},
    {"name": "Ben", "score": 67},
]
scores = {"Asha": [82, 90], "Ben": [67]}            # dict of lists: several values per key
users = {"u1": {"name": "Asha", "city": "Pune"}}    # dict of dicts: look up a record by id
grid = [[1, 2, 3], [4, 5, 6]]                       # list of lists: rows and columns

print(students[1]["name"])
print(scores["Asha"][-1])
print(users["u1"]["city"])
print(grid[1][2])

• A list of dicts stores records, such as spreadsheet rows or results from a web API. You loop over it.
• A dict of lists groups many values under one label, like all the scores of one student.
• A dict of dicts is a lookup table: jump straight to a record by its id.
• A list of lists is a grid. The first index picks the row, the second the column.

Real programs mix these freely.

Read deep values one layer at a time

Read the path from the outside in. Every [...] peels off one layer: a key if that layer is a dict, an integer index if it is a list.

order = {
    "id": 1042,
    "customer": {"name": "Ben", "city": "Pune"},
    "items": [{"sku": "A1", "qty": 2}, {"sku": "B7", "qty": 1}],
}
items = order["items"]           # a list
second = items[1]                # a dict
print(second["sku"])
print(order["items"][1]["sku"])  # the same path in one expression

When a path gets long, save a middle step in a variable (items above). The name tells the reader what you are holding. If you lose track, print the part you have reached, or its type(), to see which kind of layer comes next.

Read optional data safely

order["coupon"]["code"] raises a KeyError if either key is missing. For fields that are allowed to be missing, use .get() and let the middle steps default to an empty dict so the chain carries on:

profiles = [
    {"name": "Asha", "address": {"city": "Pune"}},
    {"name": "Ben"},                          # no address key at all
    {"name": "Cara", "address": None},        # key exists, but the value is empty
]
for person in profiles:
    city = (person.get("address") or {}).get("city", "unknown")
    print(person["name"], city)

Writing person.get("address", {}) would handle Ben but not Cara: her key is there, so the default is never used and None.get(...) would crash. (person.get("address") or {}) replaces both a missing and an empty value with {}. Use this only for optional data: for required fields a plain [...] is better, because a KeyError shows that the data is broken.

Change nested values

Assigning uses the same path. Every step except the last finds a container, and the last step changes it:

cart = {"owner": "Asha", "items": [], "totals": {"qty": 0}}
cart["items"].append({"name": "pen", "qty": 3})    # grow the inner list
cart["items"][0]["qty"] += 1                       # update a value two layers down
cart["coupon"] = {"code": "SAVE5", "percent": 5}   # add a whole new nested dict
print(cart["items"])
print(cart["coupon"]["code"])

When you build a dict of lists, the inner list may not exist yet. by_city.setdefault(city, []).append(name) returns the list that is already stored, or stores a new empty one first, then appends to it.

Loop through records

Loop over a list of dicts with for record in records: and then read record["field"]. Loop over a dict of lists with for name, marks in scores.items():, which gives each key together with its list. For deeper data, put a loop inside a loop: an outer loop over orders and an inner loop over order["items"].

Filter and sort records

A list comprehension with an if keeps the records you want. sorted() takes a key= function that says which field to sort by. lambda s: s["score"] is a tiny one-line function: it receives one record and returns the value to compare.

students = [
    {"name": "Asha", "score": 82, "city": "Pune"},
    {"name": "Ben", "score": 67, "city": "Delhi"},
    {"name": "Cara", "score": 91, "city": "Pune"},
]
from_pune = [s for s in students if s["city"] == "Pune"]
ranked = sorted(students, key=lambda s: s["score"], reverse=True)
print([s["name"] for s in from_pune])
print([s["name"] for s in ranked])
print(students[0]["name"])

Both lines build new lists and leave students alone. The records inside are still the very same dicts, which the next lesson explains.

Total up by category

"How much per category?" is the counting pattern with amounts instead of ones:

expenses = [
    {"category": "food", "amount": 12.5},
    {"category": "travel", "amount": 40},
    {"category": "food", "amount": 7.5},
]
totals = {}
for expense in expenses:
    key = expense["category"]
    totals[key] = totals.get(key, 0) + expense["amount"]
print(totals)

Grids and transposing

A grid is read as grid[row][column]. Row totals are easy, because each row is a list. Column totals need zip(*grid): the star spreads the rows out as separate arguments, so zip(*sales) means zip(row0, row1), which pairs up the first items, then the second items, and so on. Those pairs are the columns.

sales = [
    [10, 20, 30],      # Monday
    [15, 25, 35],      # Tuesday
]
print([sum(row) for row in sales])
print([sum(col) for col in zip(*sales)])
print([list(col) for col in zip(*sales)])   # a transpose: rows become columns
blank = [[0] * 3 for _ in range(2)]         # build each row separately
print(blank)

Build an empty grid with a comprehension like the last line. [[0] * 3] * 2 would put the same row in the grid twice, so changing one row would change both. The next lesson shows why.

Reading the errors

Using the wrong depth always raises one of a few errors, and each tells you where to look:

• KeyError: 'city': the dict at this layer has no such key. It may live one layer deeper.
• TypeError: list indices must be integers or slices, not str: this layer is a list, but you used a key.
• IndexError: list index out of range: the list is shorter than you thought.
• TypeError: string indices must be integers, not 'str': you reached a string, often because looping over a dict gives its keys.

Print the object just before the failing bracket and the right path is usually obvious.