Handle large text file superfast

Hi all,

I have a large text file like 50000 lines (minimum). My program reads this file and do some modification in each line and write it to another text file.

That's what I do?

Normally if the text file has only 100 lines then it takes only 2 seconds to process it. But if the file is large then it takes even 10 seconds to process each 100 lines.

How to fix this issues? I need to keep the process time as 2 seconds for each 100 lines for any number of lines.

show me the best way to do this.

Thanks in advance.

How are you currently reading the file? Please post some code.

Did you try reading the file at once? something like..

VB Code:

Dim FileContent  As String
Open "C:\file.txt" For Input As #1
      FileContent = Input(LOF(1), 1)
Close #1
 
'or..
 
Dim tmp As String
Open "C:\file.txt" For Binary As #1
      tmp = Space(LOF(1))
      Get #1, , tmp
Close #1

then you could split using vbNewLine, select the line to modify (an index of the array), modify it, and then write back to file.

the faster way to process text files is via the line input function.
your number's seem a little slow, how much work are you doing on each line?

i have an app that scans 250,000 line txt files, and on a match, pulls various info into 5 vars of different types.
it returns 5000 matches from 250,000 lines in 1.3 seconds.
post your code, and we can probably help you a lot more.

Would this work?

VB Code:

Dim FF_Read As Integer
Dim FF_Write As Integer
Dim StrTxt As String
 
FF_Read = 1
FF_Write = 2
 
OriginalFile = "D:\File.txt"
NewFile = "D:\NewFile.txt"
 
Open OriginalFile For Input As #FF_Read
Open NewFile For Output As #FF_Write
    While Not EOF(FF_Read)
        Line Input #FF_Read, StrTxt
        StrTxt = StrReverse(StrTxt)   'Change the text
        Print #FF_Write, StrTxt
    Wend
Close #FF_Read
Close #FF_Write