Remove Duplicate Lines

Dedup takes a list of lines and gives you back the same list with the repeats removed. It runs as you type, keeps your original order by default, and never sends your text anywhere.

Who this is for

Anyone holding a list that has been pasted together from more than one place. Common cases: an email list merged from two exports, a column of part numbers copied out of Excel where the same item was picked twice, a log file where you want the distinct error messages rather than every occurrence, or a set of URLs collected over several sessions. If your list lives in a database or spreadsheet you may be better served by that tool's own DISTINCT or Remove Duplicates feature — this page is for the times when you just have text on the clipboard and want it cleaned in a few seconds.

How to use it

1. Paste your list into the left box, one item per line.
2. The result appears on the right immediately. Nothing needs to be pressed.
3. Adjust the matching options if lines that look identical are not being treated as duplicates.
4. Press Copy, or Download .txt to save the result as a file.

What the options do

• Ignore case — treats Smith and smith as the same line. Off by default, because for things like case-sensitive IDs and passwords that would be wrong.
• Trim spaces — ignores leading and trailing whitespace when comparing. On by default, because invisible trailing spaces are the single most common reason two identical-looking lines are not matched. The line you get back keeps its original spacing.
• Drop blank lines — removes empty lines entirely rather than keeping one of them.
• Keep last, not first — when a line repeats, keep the later copy instead of the earlier one. This matters when your list is in chronological order and the newest entry is the one you want.
• Show ×count — prefixes each output line with how many times it appeared, like 3× widget-a. Useful for finding out which entry is causing the trouble.

The three modes

Unique lines is the normal one: every distinct line, once each. Only repeated lines shows just the lines that appeared more than once — use it to see what was duplicated rather than to clean the list. Only lines used once is the opposite: it shows the lines that appeared exactly once and drops anything that repeated at all. That last one is worth knowing about, because it answers a different question — not "what is my clean list" but "what only turned up a single time".

Worked example

Say you paste eight lines: apple, Banana, apple, cherry, banana, apple (with a trailing space), a blank line, and cherry.

• With the defaults (trim on, case sensitive), you get apple, Banana, cherry, banana — four lines. The trailing-space apple was matched thanks to trimming, but Banana and banana are still separate.
• Tick Ignore case and you get three: apple, Banana, cherry. Note that the surviving line is Banana with the capital, because it appeared first and the tool returns the original text, not a lowercased version.
• Switch to Only lines used once with ignore case on and you get nothing at all, because every fruit appeared at least twice.

Common mistakes

• Expecting words to be deduplicated. This works on whole lines. If your items are separated by commas on a single line, they are one line to this tool. Replace the commas with line breaks first.
• Assuming case is ignored. It is not, unless you tick the box. This trips people up on name and email lists.
• Invisible characters. Text pasted from a PDF or a web page often carries non-breaking spaces, which look exactly like normal spaces but are a different character. Trimming will not help there, because a non-breaking space is not whitespace as far as the comparison is concerned.
• Losing the original. The result box is a copy, but the tool saves nothing. If you press Use as input the original paste is replaced, so copy it somewhere first if you still need it.

Accessibility

Every control is a real button, checkbox or textarea, so the whole page works from the keyboard with Tab and Space. The mode and sort groups are marked up as button groups with aria-pressed so a screen reader announces which one is active, and the counts are announced when they change. Focus outlines are left visible rather than styled away.

Limits and honest caveats

Everything runs in one browser tab on your own device, so the amount you can process is bounded by that tab's memory rather than by any limit set here. Very large pastes — in the order of hundreds of thousands of lines — will work but may freeze the tab for a moment while they are processed, because the work happens on the main thread rather than in a background worker. There is no undo: the input box is yours to edit, and the result is recalculated from scratch each time.

Comparison is done on the exact characters in the line. Two strings that look the same but use different Unicode representations — for example an accented character written as one code point versus a letter plus a combining accent — will not be treated as duplicates. That is rare in practice but it does happen with text copied out of some systems.

Privacy

Your text stays in the tab. There is no upload endpoint on this page and nothing is written to a server, which also means nothing is stored — close the tab and the text is gone. See the privacy policy for how the site as a whole behaves.

Related tools

The Notepad has a Tools menu with sort and dedupe if you would rather work in a full editor, the Word Counter gives you counts and keyword frequency for the same text, and the CheckOff tool is the one to use when you want to tick items off a list rather than clean it.

Questions

More help: Dedup FAQ · All tools · Contact · About