Fuzzy Duplicate Finder

In short A plain deduplicator only removes lines that are byte-identical, which means "Acme Ltd." and "Acme Ltd" both survive and your list still has the same company twice. This compares lines by similarity instead, so near-matches surface: differing punctuation, a trailing space, a swapped word order, a typo, or one entry carrying a title the other does not. That is the difference between a list that looks clean and one that is clean. The threshold is the control that matters. Set it high and you catch only obvious pairs; set it too low and genuinely different entries start matching each other — two people who share a surname, or two branches of the same company in different cities, are legitimately distinct and will merge if you push similarity far enough. Start strict, review what it finds, and loosen only if real duplicates are still slipping through. Review is not optional at any threshold: fuzzy matching produces candidates rather than decisions, and the cost of merging two records that should have stayed separate is usually higher than the cost of a duplicate. The line limit exists because the comparison is quadratic — every line against every other one — and grows quickly.