[RESOLVED] [2005] Error streamreading number of lines in text file

[penguin5000]

Hello,
A utility I have written is causing an OutOfMemory exception error on the following line:

System.Text.RegularExpressions.Regex.Split(myStreamChecker.ReadToEnd(), Environment.NewLine).Length

The utility works in the majority of cases, but the text file I am reading on this occasion is huge in comparison to the others that the utility is generally used for - 692,684KB, whereas it is normally asked to look at files below 145,000KB.

All the line is doing is assigning a "maximum" value to a variable so that it may use it in a progress bar as it reads each line.

Is there a better way of doing what I want to do, or does anybody have an advice regarding the type of variable I should be assigning the value to? I've tried the full range of "Int"'s, to no avail. At the time of writing, the variable is an Int64.

The full error text is:
Exception of type 'System.OutOfMemoryException' was thrown.
at System.String.GetStringForStringBuilder(String value, Int32 startIndex, Int32 length, Int32 capacity)
at System.Text.StringBuilder.GetNewString(String currentString, Int32 requiredLength)
at System.Text.StringBuilder.Append(Char[] value, Int32 startIndex, Int32 charCount)
at System.IO.StreamReader.ReadToEnd()

[jmcilhinney]

Firstly, there's no need to use that convoluted method in VB 2005. If you want all the lines in a text file just use IO.File.ReadAllLines. That said, if all you want is the number of lines then you shouldn't be creating a 700MB string, or even a 145MB string, to do it. You should be reading one line at a time an incrementing a counter, so you never tie up more memory at a time than is required for one line from the file.

[penguin5000]

Thanks for your reply.
I thought about reading one line at a time, and adding 1 to a counter, but I'd obviously need to read each line - just to determine how many lines there are so that I can display a progress bar.
I actually disabled all progress bar functionality during run time when I encountered the exception a while ago, but I've been asked to put it back in again.
I wonder if there is any useful correlation between file size and the number of lines in the text file?

[FourBlades]

Set the ProgressBar Style to Marquee.
Start it just before you begin
Stop it when you are through

There will be times when you just don't know the length of time a process will take...so there's the Marquee style to handle those cases.

[penguin5000]

I thought about that too - but it doesn't really give the user any indication about how long they've got to wait.

What I'm thinking of doing now is getting the 'source' file size, and then set this as the progress bar max.

Then, whilst I was analysing the text file (the purpose of the utility), create a new file, and append each StreamRead line to it. Its the only real way that I can think of updating the progress bar.

[jmcilhinney]

You say this:

I thought about reading one line at a time, and adding 1 to a counter, but I'd obviously need to read each line - just to determine how many lines there are so that I can display a progress bar.

like it is somehow worse than what you're doing now. This:

myStreamChecker.ReadToEnd()

reads the entire file anyway. Basically, there's no way you can determine how many lines are in a text file without reading the whole thing. One way or another you will have to do it. For small files reading the whole contents in one go is no big deal but for files of that size it's ridiculous.

What you should do is get the file size first and set your ProgressBar's Maximum to that value. You can then increment the ProgressBar's Value property after each line by adding its Length, thus providing an accurate progress indicator for the user.

[penguin5000]

Thanks again for your reply.
Is there a way of determining the actual number of bytes that the read line equates to?

[jmcilhinney]

If the file is ASCII then each character is one byte. If it's Unicode each character is two bytes.

[stanav]

You can get an approximate line count based on the file size if you know the average length of each line by deviding the file size by the line's length.

[penguin5000]

Thanks for your replies.
I used jmcilhinney's advice: 1 byte = 1 character, and its works perfectly.
The idea that I try to work out how many lines based on the file size didn't quite work as a few 'test' text files indicated that, with one file at least, even though there were more lines in it, the file size was actually smaller than the other test files. I suspect its something to do with the software used to create the text files in the first place (they changed the way it wrote the text files at some point).