Remove Duplicate Lines

In short Removing duplicate lines compares each line against the ones before it and keeps only the first occurrence, so a list of 500 addresses with twenty repeats comes back as 480. The comparison is an exact string match after the optional trim and case-folding you choose, and both of those options change the answer: "Alice" and "alice " are different lines by default and the same line with trimming and case-folding on. Which behaviour you want depends on the data — email addresses are case-insensitive in the domain part and conventionally treated as such throughout, while passwords or codes are not. Order is preserved by default: the first occurrence of each line keeps its original position, so the output is the input with later repeats removed rather than a sorted set. That matters when the order carries meaning, such as a ranked list or a log. The option to keep only the duplicates instead is the more useful direction surprisingly often, because it answers the opposite question — not "what is my clean list" but "what exactly was repeated".