How to Remove Duplicate Lines: Online, Excel, Notepad++, Linux

To remove duplicate lines online, paste your list into our remove duplicate lines tool. Every repeated line is removed and the first occurrence of each is kept, in its original order. Options let you ignore case, trim stray spaces and drop empty lines. It runs in your browser, so lists of emails, phone numbers or customer names are never uploaded.

If you work in Excel, Notepad++, VS Code or a Linux terminal, the built-in methods are below.

Remove duplicate lines online

1. Paste your list, one item per line.
2. Choose the options:
   • Ignore case: treats "Mumbai" and "MUMBAI" as the same line.
   • Trim spaces: treats "abc" and "abc " (with a trailing space) as the same. On by default, because invisible spaces are the most common reason duplicates "survive".
   • Remove empty lines: drops blank lines. On by default.
3. Copy the result or download it as a .txt file.

The output keeps the order in which lines first appeared. If you also want the list alphabetised, run it through sort lines afterwards; our guide to sorting lines alphabetically covers the options.

If your list is in one long line separated by commas, put each item on its own line first (in Word, replace , with ^p). If items are broken across lines, remove line breaks can help tidy it up.

Remove duplicate lines in Excel

Remove Duplicates button

1. Select the column (or the whole table).
2. Go to Data > Remove Duplicates.
3. Tick the columns that should be compared. If you tick several, a row is only a duplicate when all ticked columns match.
4. Click OK. Excel tells you how many duplicates it removed.

This deletes rows in place, so work on a copy if you might need the original. It is not case-sensitive, and it keeps the first occurrence.

UNIQUE function (Excel 365 and 2021)

=UNIQUE(A2:A500) returns the unique values in a new range without touching the original. Google Sheets has the same UNIQUE function.

Highlight duplicates instead of deleting

Home > Conditional Formatting > Highlight Cells Rules > Duplicate Values colours every repeated value, so you can review before deleting.

Remove duplicate lines in Notepad++

• Edit > Line Operations > Remove Duplicate Lines removes every repeat across the file and keeps the first one.
• Edit > Line Operations > Remove Consecutive Duplicate Lines removes only repeats that are next to each other. Combine it with Sort Lines Lexicographically Ascending first if your version lacks the other option.

Remove duplicate lines in VS Code

1. Select the lines (Ctrl+A for the whole file).
2. Open the Command Palette with Ctrl+Shift+P (Cmd+Shift+P on Mac).
3. Type Delete Duplicate Lines and press Enter.

The command acts on the selection, so if nothing seems to happen, make sure text is selected. Older versions of VS Code may not have it; update, or use one of the other methods.

Remove duplicate lines in Linux and Mac

sort -u

sort -u file.txt > clean.txt

Sorts the file and removes duplicates in one step. Add -f (sort -uf) to ignore case.

sort | uniq

sort file.txt | uniq > clean.txt

uniq only removes adjacent duplicates, which is why it is almost always used after sort. Useful variations:

• sort file.txt | uniq -c counts how many times each line appears.
• sort file.txt | uniq -d shows only the lines that are duplicated.
• sort file.txt | uniq -u shows only the lines that appear exactly once.

Remove duplicates without sorting (awk)

awk '!seen[$0]++' file.txt > clean.txt

This prints each line the first time it is seen and skips later copies, so the original order is kept. It is the command-line equivalent of our online tool.

Vim

:sort u sorts the buffer and removes duplicate lines.

Windows PowerShell

Get-Content file.txt | Sort-Object -Unique | Set-Content clean.txt

Sort-Object -Unique is not case-sensitive by default.

Which method to use

Why duplicates sometimes survive

• Trailing spaces or tabs. "rahul@example.com" and "rahul@example.com " look identical but are not. Turn on Trim spaces.
• Different case. "Delhi" vs "delhi". Turn on Ignore case if they should be treated as one.
• Invisible characters pasted from web pages or PDFs, such as non-breaking spaces or zero-width characters.
• Different line endings when combining files from Windows and Linux. Opening and re-saving in one editor usually fixes this.
• Near-duplicates like "Mr. Sharma" and "Mr Sharma". No line-based tool catches these; review by sorting the list so similar lines sit together.

Common mistakes

• Using uniq without sort and thinking duplicates were removed.
• Running Excel's Remove Duplicates on one column of a table and breaking the link between rows. Select the whole table and choose the columns to compare.
• Deduplicating email lists case-sensitively. Email addresses are effectively case-insensitive in practice, so turn on Ignore case.
• Not keeping a copy of the original list before deleting rows in place.

Questions