About Remove Duplicates
Remove Duplicates strips repeated lines from a list, leaving only one of each. Paste a column of emails, IDs, URLs, log lines — anything line-separated — and get back the unique set, optionally sorted or reversed. Comparison can be case-sensitive or case-insensitive, and you can choose to ignore surrounding whitespace so 'foo' and ' foo ' count as the same. Handles lists of hundreds of thousands of lines without breaking a sweat, which makes it useful for one-off dedupe jobs you'd otherwise reach for `sort -u` to handle.
- No uploads
- Browser-only
- Works offline
- 100% free
How it works
- 1
Paste your list
Drop your line-separated text into the input box. There's no row cap beyond available memory — hundreds of thousands of lines work fine.
- 2
Pick comparison rules
Toggle case-insensitive matching and 'trim whitespace' if you want 'Foo', 'foo' and ' foo ' treated as the same line. Optionally sort the output A→Z, Z→A, or by length.
- 3
Copy the deduped output
The result panel shows the unique-line count alongside the cleaned list. Copy to clipboard or download as a .txt.
Removing duplicate lines without losing the ones you need
Deduplication looks trivial — drop repeated lines — but the result depends on three decisions you have to make up front: whether comparison is case-sensitive, whether surrounding whitespace counts, and whether the surviving lines keep their original order or come out sorted. Get those wrong and you either keep duplicates you wanted gone or delete lines that were only superficially identical. The right settings are different for an email list than for a log file, so it pays to think about the data before you click.
The core rule is simple: a line is a duplicate only if it matches an earlier line under the comparison rules you chose. "Match" is where all the nuance lives.
The three settings that decide the outcome
| Setting | Off (strict) | On (loose) | Use "on" when |
|---|---|---|---|
| Case sensitivity | Alice@x.com and alice@x.com both survive | Treated as the same line | Emails, usernames, tags |
| Trim whitespace | "foo" and "foo " both survive | Leading/trailing space ignored | Hand-pasted lists, CSV columns, copied tables |
| Sort vs preserve | Lines stay in first-seen order | Output is sorted A-Z | Building a canonical set; keep off for logs/transcripts |
Email addresses: the domain is case-insensitive, and in practice every major provider treats the whole address case-insensitively, so case-insensitive dedupe is safe for real-world lists.
Two jobs, two completely different setups
- Cleaning an email or contact list: turn case sensitivity OFF and trimming ON, so bob@x.com, Bob@x.com and " bob@x.com" collapse to one. Sorting is usually fine here — you want a canonical, de-duplicated set, and order rarely matters.
- Condensing a log or chat transcript: keep case sensitivity ON, trimming ON, and sorting OFF. You want to squash exact repeated lines (the same stack trace printed 400 times) while preserving chronological order — sorting a log destroys the timeline that makes it readable.
- Building a wordlist or unique-tokens set: case-insensitive, trimmed, sorted gives you a clean alphabetized vocabulary with each term once.
- De-duping a CSV by a single column is not this tool's job — line dedupe compares whole lines, so two rows that differ only in a trailing field both survive. For column-level work, split the field out first.
Where line-dedupe quietly fails
The most common surprise is invisible difference. Two lines that look identical can differ by a trailing space, a non-breaking space, or a stray carriage return from mixed line endings — and a strict, untrimmed comparison will keep both. If "identical" lines refuse to merge, the data is dirty, not the tool: run it through Text Clean first to normalize whitespace and line endings, then dedupe.
The second is the order-versus-set trade-off. Sorting makes the unique set easy to scan, but it permanently discards the original sequence. If that order carries meaning — message timestamps, build steps, a ranked list — sort a copy, never the source. And remember dedupe only removes whole-line repeats; near-duplicates (same email, different display name on the line) are distinct lines and survive untouched. To see exactly what two near-identical files differ by, compare them in Text Diff.
Frequently asked questions about Remove Duplicates
Which duplicate is kept — the first or the last?
The first occurrence is kept; subsequent duplicates are dropped. Relative order of the remaining (unique) lines is preserved, so a deduped list of timestamps, log lines, or any naturally-ordered input keeps its original sequence. Switch on a sort option if you want the output reordered.
Can it dedupe by a specific column instead of the whole line?
Not yet — comparison is always against the full trimmed line. For column-based dedupe (e.g. CSV by email column), import the file into the Spreadsheet Editor and use the Remove Duplicates filter there, which lets you pick which column to compare.
What's the difference between this and the Text Cleaner?
Dedupe removes whole duplicate lines. Text Cleaner normalises whitespace, line endings, smart quotes and invisible characters but keeps every line. Run Text Cleaner first if your duplicates only differ by weird whitespace — that way Dedupe will actually catch them.
Privacy, offline use, browser support, and pricing questions are answered on the site-wide FAQ.