Working with large amounts of data without getting Out Of Memory

[chris128]

Hi Guys,

I recently wrote a program that collects information from the network can displays it in a DGV for the user to see. I tested this on networks where I would end up with several thousand items brought back and all worked fine, very little memory increase and once the search had finished the memory usage went back to pretty much what it was before the search, so I assume I have no major memory leaks that I should be concerned about (I'm careful to always dispose of things and clear large variables where possible).

However, a new company has now started using my application and they have several hundred thousand items that my program will find. They reported that they leave the program running its search for a few hours and most of the time it crashes with an Out Of Memory exception. Now I did have my app do some caching of certain information to avoid querying the network more than was necessary so I thought well maybe this is what is causing it but I added an option to turn that off and they say it still happens. So the only thing I can see it being is simply the amount of data that the program is finding and having to store in memory... once my application has gathered information about the items it stores them in a List(Of MyItemClass) and then when the search is complete the items are added to the DGV. So I'm thinking maybe its just the size of this List(Of MyItemClass) that is simply getting too large. Bear in mind that each instance of MyItemClass in this List can have quite a lot of information in it as it has several properties that are List(Of String) that may contain a few thousand strings in some scenarios.

So, with the background story out of the way, I'll get to the actual question:
How do you deal with such large amounts of data without running out of memory?
I'm assuming the only option I have got is to 'page' some of the data to a temporary file on disk once I get over a certain number of items in the List(Of MyItemClass) but interested to hear how other people would do it and if paging to disk is the best option does anyone have any advice on exactly how to do this?

Thanks!
Chris

[jmcilhinney]

You can always raise the maximum amount of memory that can be assigned to your app. If the amount of physical memory in the system is the issue though, then your only option may be to do as you suggest. You might be able to use a GZipStream to store some data in memory in a compressed state, which would likely make access quicker than saving it to disk, but that would not be an option in all circumstances. Note also that, while it's generally to be avoided, you may also find that an explicit call to GC.Collect may help in situations where large amounts of memory are assigned and then are no longer required en masse.

[chris128]

Thanks, and yeah I did consider GC.Collect but I don't think it would really help here because the issue is not really that I'm using lots of memory then suddenly clearing it and then using a load more memory, the issue seems to be more that I'm slowly growing memory usage naturally because of more and more items being added to my List.
If I do decide to page data to disk, I'm wondering what the best way of doing this is... I can't seem to find any recommendations or standard practices on the web on how to actually handle it and whilst I could come up with my own solution I'm thinking I will still hit problems when I come to actually load it all back into the application when the search has finished. One thing I'm considering is having an option to just write the results straight to CSV file instead of showing them in the DGV first, that way there never needs to be much in memory at all as I could just use a Queue(Of MyItemClass) and every 100 items I write anything in the queue to the file or something like that, until the search is complete. Only problem with that is that the items would not be sorted at all, they would just be written in whatever order they were found in, but I guess its easy enough for the user to sort it in the CSV file in Excel.