Deduplicate

uniq [options] [input [output]]

sort access.log | uniq -c | sort -rn | head -20

    847 GET /api/users
    623 GET /api/health
    412 POST /api/login

Note uniq only removes adjacent duplicates, so you almost always need to sort first. -c prefixes each line with its count, -d shows only duplicates, -i ignores case.

{expression for item in iterable if condition}

emails = ["[email protected]", "[email protected]", "[email protected]", "[email protected]"]
unique_lower = {e.lower() for e in emails}
print(unique_lower)

{'[email protected]', '[email protected]', '[email protected]'}

Note Set comprehensions automatically deduplicate results. Useful for extracting unique transformed values from a collection.

SELECT DISTINCT column FROM table;

SELECT DISTINCT city
FROM users
ORDER BY city;

-- Returns each city only once, no duplicates

Note DISTINCT applies to the entire row when used with multiple columns. SELECT DISTINCT city, state treats (city, state) pairs as the unit of uniqueness.