Hi,

I have a very huge server log files like files are more than 500 to 600 mb and even some files are over 2 gb data. the files are maintained for few years. it will be maintained as is. each log file has at least 1 million lines to 20 million lines.

I am looking create an application which can find a line in the text file using regex, removing duplicate entries.

please let me know how these huge file can be handled in a way it works very quickly.

thanks in advance.