Remove duplicates

4 snippets across 4 stacks — Bash & Linux, Interview Prep, Python, SQL

SHBash & Linux

Filter Duplicate Lines

syntax

uniq [options] [input [output]]

example

sort access.log | uniq -c | sort -rn | head -20

output

    847 GET /api/users
    623 GET /api/health
    412 POST /api/login

Note uniq only removes adjacent duplicates, so you almost always need to sort first. -c prefixes each line with its count, -d shows only duplicates, -i ignores case.

DSAInterview Prep

Pattern: Two Pointers

DSA · Common Patterns Summary

syntax

When to use:
- Sorted array pair problems
- Removing duplicates in place
- Container with most water
- Palindrome checking

Variants:
- Opposite ends (converging)
- Same direction (fast/slow or lagging)

example

// JavaScript — Remove duplicates from sorted array
function removeDuplicates(arr) {
  if (arr.length === 0) return 0;
  let writeIdx = 1;
  for (let i = 1; i < arr.length; i++) {
    if (arr[i] !== arr[i - 1]) {
      arr[writeIdx] = arr[i];
      writeIdx++;
    }
  }
  return writeIdx;
}

# Python
def remove_duplicates(arr):
    if not arr:
        return 0
    write_idx = 1
    for i in range(1, len(arr)):
        if arr[i] != arr[i - 1]:
            arr[write_idx] = arr[i]
            write_idx += 1
    return write_idx

output

remove_duplicates([1,1,2,2,3]) → 3, arr becomes [1,2,3,...]

Note O(n) time, O(1) space. The 'read pointer / write pointer' variant is essential for in-place array problems. Always ask: is the input sorted? If yes, two pointers likely applies. If unsorted, consider hash set instead.

PYPython

Creating Sets

PY · Tuples & Sets

syntax

s = {val1, val2, ...}
s = set(iterable)

example

tags = {"python", "tutorial", "beginner"}
from_list = set([1, 2, 2, 3, 3, 3])
print(from_list)

empty_set = set()
print(type(empty_set))

output

{1, 2, 3}
<class 'set'>

Note Use set() for an empty set, NOT {}. Empty braces {} create an empty dictionary, not a set.

SQLSQL

DISTINCT

SQL · Basic Queries

syntax

SELECT DISTINCT column FROM table;

example

SELECT DISTINCT city
FROM users
ORDER BY city;

output

-- Returns each city only once, no duplicates

Note DISTINCT applies to the entire row when used with multiple columns. SELECT DISTINCT city, state treats (city, state) pairs as the unit of uniqueness.