[RESOLVED] Working with massive strings - memory errors

[InvisibleDuncan]

I'm working on some VB.Net 2008 code that extracts a massive chunk of data from a database and then formats it and sticks it into a text file.

Currently, it builds everything by spooling through a data table and putting the results into an IO.StringWriter, using the .WriteLine method for each record, and then at the end dumps it all out to a text file using the FileSystem.WriteAllText method. Unfortunately, some of the extracts are massive, and it's giving System.OutOfMemoryException errors part-way through the table.

Does anyone know of a way to get around this, short of writing each line out to the text file independently?

Any help would be gratefully received...

[Evil_Giraffe]

Does anyone know of a way to get around this,

Yes, you need to write each line out to the text file independently.

short of writing each line out to the text file independently?

Oh.

Seriously, though, why would you try not to do that? The problem is with having objects that are too large (specifically, larger than 85,000 bytes). These objects go on the Large Object Heap and that's something you really want to avoid if you can.
So if the problem is dealing with objects that are too large, why wouldn't you try the obvious thing and work with smaller objects?

[InvisibleDuncan]

Seriously, though, why would you try not to do that?

Simply because the writing of each line is a long way down the function chain from where the file name is determined (as it happens, the data is all collated before the file name is determined). I was hoping to avoid passing the file name as a parameter through each function call, and also avoid expanding the scope of the file name variable.

I also (probably wrongly) assumed that it would operate faster if it was handled in memory rather than writing to the hard disk after every record.

Thanks for the impressively fast reply!

[Evil_Giraffe]

Instead of writing to a StringWriter, then, perhaps add each line as its own string to a List(Of String), and then loop through that when you are writing the file. You needn't change the architecture, just don't mush all the strings into one!

[InvisibleDuncan]

Hmmmm - that sounds like a very promising plan! I'll give that a whirl and see what happens.

Incidentally, I'm just back from a couple of weeks in Kenya, and all the giraffes I encountered were quite friendly. Their coding wasn't up to much, though, so maybe they were just too nice.

[Evil_Giraffe]

Don't know whether coding comes into it, but I eat more leaves than I should, then other giraffes may die.

[InvisibleDuncan]

That's quite an explanation!

Your suggestion of a list successfully avoided the memory error, but as I feared it was hideously slow to save the file line by line. I've therefore come up with a compromise: when it comes to saving the file, I collate the list into text blocks of 3,000 records each, and save them like that.

All in all, a successful result. Thanks for the advice.