%22%2F%3E%3Ccircle%20cx%3D%2232%22%20cy%3D%2232%22%20r%3D%2223%22%20fill%3D%22%23fff%22%2F%3E%3Cg%20fill%3D%22none%22%20stroke%3D%22url(%23gu)%22%20stroke-width%3D%225%22%20stroke-linecap%3D%22round%22%3E%3Cpath%20d%3D%22M21.5%2024h21%22%2F%3E%3Cpath%20d%3D%22M21.5%2032h21%22%2F%3E%3Cpath%20d%3D%22M21.5%2040h11%22%2F%3E%3C%2Fg%3E%3C%2Fsvg%3E)
TextLabFree duplicate line remover
Remove duplicate lines and see which ones repeated
Paste a list and get it back with the repeats gone, in the order you pasted it, plus a table naming every entry that appeared more than once and how many times. It is free, asks for nothing and handles lists of a hundred thousand lines without complaint. Matching can ignore capitals, space at either end or punctuation; the surviving copy can be the first appearance or the last; and a third mode hands you only the repeats, which is how you find what two merged lists had in common.
- 100% free
- No signup
- Names every repeat
- Keep first or last copy
- Order preserved
The survivor stays where that copy sat, so keeping the last one reorders your list.
When are two lines the same line
Which entries were repeated
Paste a list and every entry that shows up more than once will be named here, with a count.
How to remove duplicate lines from a list
Three steps, and the count of distinct entries appears the moment you paste.
Paste the list, one entry per line
Two merged exports, a column lifted out of a spreadsheet, a log, a bibliography — anything where a line is a record. Paste straight onto the page — the box does not need the cursor in it first. The header immediately reports how many distinct entries hide inside how many lines, which is usually the number you actually came for.
Decide what you want handed back
One of each is the ordinary answer. Only the repeats gives you the entries that appeared more than once, which is how you find the overlap between two lists you merged. Only the one-offs throws away every repeated entry completely, leaving the records that appeared exactly once — useful when a second appearance means the row was already processed.
Loosen the matching, then check the table
Capitals, space at either end and punctuation can each be declared irrelevant, which is how Sales@example.com, sales@example.com and "sales@example.com " become one entry rather than three. The table at the bottom names every repeated entry with its number of appearances and how many copies were dropped, so you can confirm the merge did what you expected before you copy the list out.
Technical specifications
| Matching rules | Exact by default; capitals, space at either end and punctuation can each be declared irrelevant, in any combination |
|---|---|
| Which copy survives | The first appearance, which preserves the original order, or the last, which moves each entry to the position of its final appearance |
| Output modes | One of each entry, only the entries that appeared more than once, or only the entries that appeared exactly once |
| Blank lines | Excluded from matching by default, so section gaps survive; include them and every blank line becomes a repeat of every other |
| What the report names | Every repeated entry with its number of appearances and copies dropped, sorted with the most repeated first, showing the first twelve |
| Order | Preserved. Nothing is sorted, so a list with a meaningful sequence keeps it |
| Speed | One indexing pass: a 100,000-line list is de-duplicated in roughly 60 ms, and the report is rebuilt on every option change |
| Price and processing | Free, no signup, and the list never leaves this browser tab |
Frequently asked questions
Two lines look identical but the tool kept both. Why?
One of them has a space on the end. A trailing space is invisible in every editor and makes two otherwise identical lines different strings, which is the single most common reason a de-duplication appears to fail. Switching on the space-at-either-end option makes the comparison ignore it, and the surviving copy is still returned with whatever spacing it originally had. If that does not resolve it, the difference is an invisible character rather than a space, and it needs identifying rather than ignoring.
Should I match case-insensitively for email addresses?
Yes, in practice. The domain half of an address is officially case-insensitive, and while the part before the @ is technically allowed to be case-sensitive, no mail provider in general use treats it that way — Sales@example.com and sales@example.com reach the same inbox. Leave capitals mattering when your list is of names, product codes or anything where AAA and aaa are genuinely different things.
What is the difference between keeping the first copy and keeping the last?
Which copy survives, and where in the list it ends up. Keeping the first preserves the original order of the list, so the entry stays where it first appeared. Keeping the last moves each surviving entry down to the position of its final appearance, which reorders the list — that is the point when the later copy is the more recent one, as in a log or an updated export, and you want the newest version of each record. The text kept is that copy's own text, so it will differ in case or spacing if the copies did.
How do I find what two lists have in common?
Paste one list, then the other underneath it, and choose only the repeats. Every entry that appears in both lists appeared twice, so the output is the intersection of the two. It works for three or more lists too, though the count column becomes the interesting part: an entry with a count of three was in all three lists. Choosing only the one-offs instead gives you everything unique to a single list.
Does it sort the list at the same time?
No — the order you pasted is preserved exactly, minus the removed copies. This is deliberate: sorting and de-duplicating are separate decisions, and many lists have a meaningful order that alphabetising would destroy. If you do want it alphabetised, de-duplicate first and sort afterwards, since sorting a list that still contains repeats simply parks them next to each other.
What happens to the blank lines between my sections?
They are left exactly where they are as long as the last tick box is on, which it is by default. Every blank line is identical to every other blank line, so a de-duplicating pass that treats them as ordinary entries would collapse all of them into a single gap at the top of your list and destroy its structure. Untick the box and blanks are treated as a repeated entry like any other.
Can it cope with a list of a hundred thousand lines?
Yes, and the size makes very little difference. Matching is done by building an index of the entries in one pass, so the work grows in step with the length of the list rather than with its square — a 100,000-line file is de-duplicated in around 60 ms. Nothing is uploaded, so the practical limit is your own machine's memory rather than any quota.
Identical to the eye, different to the machine
De-duplication looks like the simplest operation on a list and turns into an argument about what “the same” means. Two lines are the same string only if every character matches, and lists assembled by human beings are full of pairs that fail that test while being obviously the same record: one has a trailing space, one was typed with a capital, one ends with a full stop because it came out of a sentence. A strict comparison keeps all three, and the list you get back is barely shorter than the one you pasted. That is why the matching rules above are separate switches rather than a single strictness dial: for a list of email addresses you want capitals and outer space ignored and punctuation respected, because the dots in an address are load-bearing; for a list of book titles you want the opposite.
Which copy survives sounds like a technicality until the copies differ. Keeping the first preserves the order you pasted, which is usually what a merged list needs. Keeping the last is what a log or a re-exported dataset needs, because a later line is a more recent version of the same record — and it necessarily reorders the list, since each surviving entry moves down to where its final appearance sat. The text kept is that copy's own text too, so if the earlier line read sales@example.com and the later one read Sales@example.com, your choice decides which capitalisation you end up with. The report underneath the tool exists so this is never a surprise: it names the entries that repeated rather than only counting them, which is how you notice that the duplicate you were worried about was not in the file at all.
The third output mode inverts the operation and turns it into a comparison tool. Paste two lists one after the other, ask for only the repeats, and what comes back is everything that appeared in both — the overlap between last month's subscribers and this month's, or between the invited and the attended. Ask for only the one-offs instead and you get everything unique to one side. Neither needs a formula or a spreadsheet. Two habits make all of this cleaner: close the gaps first, since blank lines are all identical to each other, and leave sorting until afterwards — the sort lines tool works better on a list with no repeats in it than the other way round. When the same list also needs its bullets, tabs and stray spaces removed, the text cleaner does all of that and this in one go.
Your list is never uploaded
Every comparison happens inside this browser tab, and that matters more here than almost anywhere else on this site: the lists people de-duplicate are usually customer emails, subscriber exports or internal records. None of it is transmitted, stored or logged, and the table of repeated entries is rebuilt from your text each time rather than saved anywhere. Close the tab and nothing remains.