What's the fastest method of removing duplicates from an array or text file?

The method I'm using right now is reading the file then adding what isn't found in a string into that string using InstrB.

With very few duplicates (like.. 10), it's very fast, in a 3.5 million list, it takes less than 3 seconds on my computer.

With a lot of duplicates, it just dies, freezes.
While a have a program written in C or something that demolishes it in ~10 seconds..

Having InstrB function read through large amounts of data in a loop, looking for every word doesn't work out well with a lot of dupes.


There's also sorting the list first then comparing each one with the next, removing dupes in 1 pass, but how would you rearrange the sorted list back into the previous order?