how much memory can streamreader.ReadToEnd() hold

[benmartin101]

I have 2GB ram, and I checked that the taskmanager and I have more than enough memory left. I tried to ReadToEnd() a file that has 194 mb. Yet as soon as it passes the code, the returned string is a null. The file exists, and I tried other files(which are smaller) and it works. The code is as follows:

System.IO.StreamReader sr = new System.IO.StreamReader(@"c:\file1.csv");
string str = sr.ReadToEnd();

Why is it giving me out of memory exception?

[stanav]

Since StreamReader.ReadToEnd() returns a String and the Length property of a String object is an int, theoretically you can have a string with its length = int.MaxValue, which is roughly 2GB. Having said that, I really don't know how the ReadToEnd function works internally... Could it be creating multiple strings when it reads before returning the final string as a whole? As a test, can you use something like ReadLine() function and append the line to a stringbuilder (of course you will loop until there is no more lines to read).

[axion_sa]

From the StreamReader.ReadToEnd docs:

OutOfMemoryException: There is insufficient memory to allocate a buffer for the returned string.

If the current method throws an OutOfMemoryException, the reader's position in the underlying Stream object is advanced by the number of characters the method was able to read, but the characters already read into the internal ReadLine buffer are discarded. Since the position of the reader in the stream cannot be changed, the characters already read are unrecoverable, and can be accessed only by reinitializing the StreamReader object. If the initial position within the stream is unknown or the stream does not support seeking, the underlying Stream object also needs to be reinitialized.

To avoid such a situation and produce robust code you should use the Read method and store the read characters in a pre-allocated buffer.

Simply allocate a smaller buffer, 1MB or the likes, and loop/process away. Should skip past the OutOfMemoryException.

using (StreamReader reader = new StreamReader(path))
{
	char[] buffer = new char[1024 * 1024];
	int position = 0;
	int charsRead = 0;

	while ((charsRead = reader.Read(buffer, 0, buffer.Length)) > 0)
	{
		string contentToProcess = new string(buffer, 0, charsRead);
		// ... Do some work.
		position += charsRead;
	}
}

Of course, an alternative to this course is to bulk insert the data into a SQL database [MSDE, possibly?] and use the SQL engine to process the data.

// Edit: fixed sample.

[ComputerJy]

by the way the ReadToEnd method reads the whole file into a StringBuilder and returns the StringBuilder.ToString() value