Replace function runs out of memory

I don't use VB.Net very much, still mostly do VB6, but I am doing this project in .Net. I have a huge string of data, 168 Mb that I stream in from a file to a string variable.
Then I have to perform a number of replacements on it before using the data to compile statistics.
Because of the nature of the data, I can't break this up into smaller strings.

The problem is that I get a system out of memory error after a few replacements.

I've used both Replace(x,y,z) and X.Replace(y,z).
The second one gets a little farther in the code before it crashes, but they both crash.


When I run it in the IDE, the IDE is using about 130 Mb of RAM, the Exe of the app is running about 350 Mb, and one svchost.exe runs about 300 Mb

Other than that there are no major RAM uses going on.
The PC has an i7 980x cpu with 12 GB of RAM.

Any suggestions?

Every one of those calls is creating a whole new string. Strings are immutable, so whenever you think you are changing one, you are actually abandoning it and creating a new one with the changes. Abandoning the one string doesn't remove it, and in your case, it almost certainly means that not only has it not been deleted, but it CAN'T be deleted, so the old string and the new string remain in memory. That is probably what is killing the memory. You are probably creating at least two, and possibly more, strings of the full size with each call to Replace.

As for what to do...on that I have no good idea as long as you can't come up with a way to work on parts of the string rather than the whole thing.

Just to be clear, is it a System Out Of Memory error, or a System.OutOfMemoryException exception? The first comes form the underlying OS and the second is from .NET.

I highly suspect the latter, and the problem is that all those strings you are creating are going to be put on the Large Object Heap. The problem with this is that the Garbage Collector really doesn't like to touch things in here, as they are very expensive to collect or move about, so once something is in here, it generally stays, and worse, once it is collected, the rest of the objects in the heap won't get compacted down, so the memory in there gets fragmented. If your new strings can't fit in a 'hole' left by a reclaimed string, then you'll get an OOM before long.

There is little you can do to avoid this in .NET other than making sure those strings don't get on to the LOH. You'll need to keep them under 85k to do that, which basically means, as SH has said, you'll need to figure out a way of handling chunks of the string, not all t once.