How to Remove Duplicates from a List in Python
list(dict.fromkeys(items)) to remove duplicates and keep the original order. Use list(set(items)) when order does not matter, because a set drops duplicates but does not promise any order.Duplicates sneak into lists all the time: repeated sign-ups, the same product added twice, log lines that appear more than once. Python gives you several ways to clean them up, and the right one depends on two questions: does the order of the items matter, and are the items hashable (strings, numbers and tuples are, lists and dictionaries are not)?
This page shows the three approaches you will actually use, explains why each works, and warns you about the traps, such as a set shuffling your data.
| Keep order | list(dict.fromkeys(items)) |
|---|---|
| Order does not matter | list(set(items)) |
| Items must be hashable | Yes for both one-liners (strings, numbers, tuples) |
| Changes the original list | No, it builds a new list |
| Time complexity | O(n) on average |
Remove duplicates and keep the original order with dict.fromkeys()
Since Python 3.7 a dictionary remembers the order in which keys were first inserted, and a dictionary can hold each key only once. dict.fromkeys(items) therefore creates a dictionary whose keys are your items without repeats, in order of first appearance. Wrapping it in list() turns the keys back into a list.
orders = ["pen", "book", "pen", "ink", "book"]
unique = list(dict.fromkeys(orders))
print(unique)The first copy of each item is the one that stays.
Remove duplicates with set() when order is not important
A set stores each value once and is very fast at checking whether it has seen a value. Converting a list to a set and back is the shortest way to deduplicate, but the result can come out in a different order than the input, so sort it if you need a predictable result.
ids = [3, 1, 3, 2, 1]
print(sorted(set(ids)))For sets of strings the order can even change between runs of the same program, so never rely on it.
Remove duplicates from a list of lists or dictionaries
Lists and dictionaries are not hashable, so set() and dict.fromkeys() raise TypeError. You have two choices: convert each inner list to a tuple first, or fall back to a simple loop that checks not in on the result. The loop is O(n squared) but works for any item type and keeps order.
pairs = [[1, 2], [3], [1, 2]]
# option 1: tuples are hashable
unique = [list(t) for t in dict.fromkeys(tuple(p) for p in pairs)]
# option 2: plain loop, works for any items
unique = []
for p in pairs:
if p not in unique:
unique.append(p)Which method should you choose?
| Method | Keeps order? | Works with unhashable items? | Best for |
|---|---|---|---|
list(dict.fromkeys(items)) | Yes | No | Most everyday cases |
list(set(items)) | No | No | Large lists where order is irrelevant |
Loop with not in | Yes | Yes | Small lists of lists or dicts |
Loop with a seen set | Yes | No | When you need extra logic per item |
Examples you can run
Press Run, then edit the code and run it again.
Keep the first occurrence of each item
dict.fromkeys() keeps the order in which items first appeared.
orders = ["pen", "book", "pen", "ink", "book"]
print(list(dict.fromkeys(orders)))Expected output['pen', 'book', 'ink']
Use a set and sort the result
When only the unique values matter, a set is enough. Sorting makes the output predictable.
ids = [3, 1, 3, 2, 1]
print(sorted(set(ids)))
print(len(ids) - len(set(ids)), "duplicates removed")Expected output[1, 2, 3]
2 duplicates removed
Deduplicate a list of lists
Lists cannot go in a set, so check the result list with not in.
pairs = [[1, 2], [3], [1, 2]]
unique = []
for p in pairs:
if p not in unique:
unique.append(p)
print(unique)Expected output[[1, 2], [3]]
Deduplicate case-insensitively
Build the dictionary from lower-cased keys so "Asha" and "asha" count as the same name.
names = ["Asha", "ben", "asha", "Ben", "Chen"]
unique = {n.lower(): n for n in reversed(names)}
print(list(unique.values())[::-1])Expected output['Asha', 'ben', 'Chen']
Common mistakes
Expecting set() to keep the order
✗ Wrong
tags = ["c", "a", "c", "b"]
print(list(set(tags)))✓ Right
tags = ["c", "a", "c", "b"]
print(list(dict.fromkeys(tags)))A set has no guaranteed order. Use dict.fromkeys() or sort the result.
Putting unhashable items in a set
✗ Wrong
rows = [[1, 2], [1, 2]]
print(set(rows))✓ Right
rows = [[1, 2], [1, 2]]
print(set(tuple(r) for r in rows))Convert each row to a tuple, or use a loop with not in.
Expecting the original list to change
✗ Wrong
items = [1, 1, 2]
list(set(items))
print(items)✓ Right
items = [1, 1, 2]
items = list(dict.fromkeys(items))
print(items)set() and dict.fromkeys() create new objects. Assign the result, or use slice assignment items[:] = ... to update in place.
How to Remove Duplicates from a List in Python: questions people ask
How do I remove duplicates from a list in Python without changing the order?
Use list(dict.fromkeys(items)). Dictionaries keep keys in insertion order (guaranteed since Python 3.7), so the first copy of each item stays in its original position.
Is converting a list to a set the fastest way to remove duplicates?
Yes, set(items) is usually the fastest because membership checks are O(1), but it loses the order. dict.fromkeys() is almost as fast and keeps the order.
How do I remove duplicates from a list of dictionaries?
Dictionaries are unhashable, so loop through the list and append an item only if item not in unique, or build a key such as tuple(sorted(d.items())) for each dictionary and track the keys you have seen.
Keep learning
Step-by-step lessons:
Python ListsPython SetsPython Dictionaries
Related:
How to Count Occurrences in a List in PythonPython List vs Set: Differences and When to Use EachPython sort() vs sorted(): What Is the Difference?Python List count() Method
More in lists, dicts and sets:
How to Sort a Dictionary by Value in PythonHow to Merge Two Dictionaries in PythonHow to Reverse a List in PythonHow to Flatten a List of Lists in PythonHow to Check if a List Is Empty in PythonHow to Get the Last Element of a List in PythonHow to Loop Through a Dictionary in PythonHow to Count Occurrences in a List in Python See all →