[RESOLVED] How to parse big files ?

[Daniel Duta]

No I think you misunderstood. This is just a general text file reader.

A replacement for the following:

' Classic VB File IO
Open FileName For Input As #1
Do While Not EOF(1)
  Line Input #1, sLine
Loop 
Close #1

Dim cFile As clsFile
Set cFile = New clsFile

' File IO using cFile object
If cFile.OpenTextStream(FileName) Then
  Do While Not cFile.EndOfFile
    sLine = cFile.ReadLine
  Loop
  cFile.CloseTextStream
End If

Is it correct that the file is a normal text file?
The chunks you want to process are separated by "============================"
Then just process the file line by line until you encounter the chunk separator

Can you explain how you handle the blocks of data?
Because your original blocks are still strings with newline separators

Yes, it is a normal text file with row separators LF. My ''chunks'' are separated by "============================" (...I call them blocks because chunks are associated with space of a string and usually they have a predefined size). Now, I took your advice and I run the Do While - Loop above in my context just to check any possible Out of memory error. The result says that 21965012 lines were read (without other conditions) in 94 seconds. But considering that these blocks of data require themselves a secondary parsing and the results have be saved further as a csv format I expect to exceed 10 minutes... You are right, the schema of my parsing process has that logic. Thank you.