Handle large text file superfast

[cssriraman]

Hi all,

I have a large text file like 50000 lines (minimum). My program reads this file and do some modification in each line and write it to another text file.

That's what I do?

Normally if the text file has only 100 lines then it takes only 2 seconds to process it. But if the file is large then it takes even 10 seconds to process each 100 lines.

How to fix this issues? I need to keep the process time as 2 seconds for each 100 lines for any number of lines.

show me the best way to do this.

Thanks in advance.

[Logophobic]

How are you currently reading the file? Please post some code.

[jcis]

Did you try reading the file at once? something like..

Dim FileContent  As String
Open "C:\file.txt" For Input As #1
      FileContent = Input(LOF(1), 1)
Close #1
 
'or..
 
Dim tmp As String
Open "C:\file.txt" For Binary As #1
      tmp = Space(LOF(1))
      Get #1, , tmp
Close #1

then you could split using vbNewLine, select the line to modify (an index of the array), modify it, and then write back to file.

[rnd me]

the faster way to process text files is via the line input function.
your number's seem a little slow, how much work are you doing on each line?

i have an app that scans 250,000 line txt files, and on a match, pulls various info into 5 vars of different types.
it returns 5000 matches from 250,000 lines in 1.3 seconds.
post your code, and we can probably help you a lot more.

[Andrew G]

Would this work?

Dim FF_Read As Integer
Dim FF_Write As Integer
Dim StrTxt As String
 
FF_Read = 1
FF_Write = 2
 
OriginalFile = "D:\File.txt"
NewFile = "D:\NewFile.txt"
 
Open OriginalFile For Input As #FF_Read
Open NewFile For Output As #FF_Write
    While Not EOF(FF_Read)
        Line Input #FF_Read, StrTxt
        StrTxt = StrReverse(StrTxt)   'Change the text
        Print #FF_Write, StrTxt
    Wend
Close #FF_Read
Close #FF_Write