How do you remove duplicates?

This task is covered in 4 stacks on this page: Bash & Linux, Interview Prep, Python, SQL. The "Filter Duplicate Lines" snippet in Bash & Linux uses `uniq [options] [input [output]]`.

Which command does the Bash & Linux example use?

The "Filter Duplicate Lines" snippet uses `uniq [options] [input [output]]`, from the File Content section of the Bash & Linux cheat sheet.

Which stacks cover "remove duplicates" on this page?

Bash & Linux, Interview Prep, Python, SQL. Together they hold 4 copy-ready snippets for this task.

Remove duplicates

4 snippets across 4 stacks - Bash & Linux, Interview Prep, Python, SQL

SHBash & Linux

Filter Duplicate Lines

SH · File Content

Syntax

uniq [options] [input [output]]

Example

sort access.log | uniq -c | sort -rn | head -20

Output

    847 GET /api/users
    623 GET /api/health
    412 POST /api/login

Note uniq only removes adjacent duplicates, so you almost always need to sort first. -c prefixes each line with its count, -d shows only duplicates, -i ignores case.

DSAInterview Prep

Pattern: Two Pointers

DSA · Common Patterns Summary

Syntax

When to use:
- Sorted array pair problems
- Removing duplicates in place
- Container with most water
- Palindrome checking

Variants:
- Opposite ends (converging)
- Same direction (fast/slow or lagging)

Example

// JavaScript - Remove duplicates from sorted array
function removeDuplicates(arr) {
  if (arr.length === 0) return 0;
  let writeIdx = 1;
  for (let i = 1; i < arr.length; i++) {
    if (arr[i] !== arr[i - 1]) {
      arr[writeIdx] = arr[i];
      writeIdx++;
    }
  }
  return writeIdx;
}

# Python
def remove_duplicates(arr):
    if not arr:
        return 0
    write_idx = 1
    for i in range(1, len(arr)):
        if arr[i] != arr[i - 1]:
            arr[write_idx] = arr[i]
            write_idx += 1
    return write_idx

Output

remove_duplicates([1,1,2,2,3]) → 3, arr becomes [1,2,3,...]

Note O(n) time, O(1) space. The 'read pointer / write pointer' variant is essential for in-place array problems. Always ask: is the input sorted? If yes, two pointers likely applies. If unsorted, consider hash set instead.

PYPython

Creating Sets

PY · Tuples & Sets

Syntax

s = {val1, val2, ...}
s = set(iterable)

Example

tags = {"python", "tutorial", "beginner"}
from_list = set([1, 2, 2, 3, 3, 3])
print(from_list)

empty_set = set()
print(type(empty_set))

Output

{1, 2, 3}
<class 'set'>

Note Use set() for an empty set, NOT {}. Empty braces {} create an empty dictionary, not a set.

SQLSQL

DISTINCT

SQL · Basic Queries

Syntax

SELECT DISTINCT column FROM table;

Example

SELECT DISTINCT city
FROM users
ORDER BY city;

Output

-- Returns each city only once, no duplicates

Note DISTINCT applies to the entire row when used with multiple columns. SELECT DISTINCT city, state treats (city, state) pairs as the unit of uniqueness.

Frequently asked questions

How do you remove duplicates?: This task is covered in 4 stacks on this page: Bash & Linux, Interview Prep, Python, SQL. The "Filter Duplicate Lines" snippet in Bash & Linux uses `uniq [options] [input [output]]`.
Which command does the Bash & Linux example use?: The "Filter Duplicate Lines" snippet uses `uniq [options] [input [output]]`, from the File Content section of the Bash & Linux cheat sheet.
Which stacks cover "remove duplicates" on this page?: Bash & Linux, Interview Prep, Python, SQL. Together they hold 4 copy-ready snippets for this task.
Is there anything to watch out for?: Yes. For "Filter Duplicate Lines": uniq only removes adjacent duplicates, so you almost always need to sort first. -c prefixes each line with its count, -d shows only duplicates, -i ignores case.