How to handle very huge text/log files

[ntg]

I am looking create an application which can find a line in the text file using regex, removing duplicate entries.

Just so it's clear to me, you need to search the files in order to remove duplicate entries or do you just need to search in the files?