Out of Memory writing text files

Hey guys,

I wrote a vb .net Windows application, and it rewrites IIS log files on a schedule.

I am containing the re-written log file in a string and then it writes to disk and loops. The log files can be upwards of 500MB.

When running it, i see the CPU go to 100%, and the memory is about 500mb. The thing is, the server this runs on has 6gb of memory.

When the log files reach a certain size, i get out of memory exceptions. I think im missing something here, like clearing the string and dumping to the GC.. Can someone show me where in my code is causing this?

Dim _24HoursAgo As Date = Date.Now.AddHours(-168)
For Each file1 As IO.FileInfo In (New IO.DirectoryInfo("D:\Logs\Nissan\")).GetFiles("*.*")
If file1.LastWriteTime >= _24HoursAgo Then

Dim strfile As String = file1.Name
Dim reader As StreamReader = File.OpenText("D:\Logs\N2\" & strfile)
Dim contents As String = reader.ReadToEnd()
Dim ds As New DataSet
ds.ReadXml("d:\Logs\n.xml")

Dim sb As New StringBuilder(contents)
For Each ro As DataRow In ds.Tables("row").Rows
If ro("find").ToString().Length = ro("count") Then
sb.Replace(ro("find").ToString(), ro("repl").ToString())
End If
Next ro


Dim output As String
output = sb.ToString

Dim objwriter As New StreamWriter("D:\Logs\N\" & strfile)
'Const OPEN_FILE_FOR_APPENDING = 8

' generate a filename base on the script name
Dim strOutputFile2 = "D:\Logs\N\" & strfile

Dim objFileSystem2 = CreateObject("Scripting.fileSystemObject")
Dim objOutputFile2 = objFileSystem2.OpenTextFile(strOutputFile2)
'objwriter.Flush()

objwriter.WriteLine(sb.ToString)
objOutputFile2.Close()

objFileSystem2 = Nothing
objOutputFile2 = Nothing
sb = Nothing

reader.Close()
objwriter.Close()

End If

Next file1

Thanks so much!

Try moving as many of the Dim keywords as you can to outside the loop.
That should help some.

Take a closer look at your code. You load a 500 MB file to a string. You the create a stringbuilder object from the 500 MB string, that again takes another 500 MB. You then call stringbuilder.ToString, which creates another 500 MB string. That's not counting the dataset read from your xml file, and then you open another file (objFileSystem2.OpenTextFile(strOutputFile2)
)... At the minimum, I'm seeing that you're using more than 1.5 GB of memory.

I have never used these read/write everything io statements. When I first saw them I thought they were useful, but I bet the assumption was that the user would know they were designed for smaller??? files.

stanav is correct in his assessment of your memory usage, and I think stanav is saying he sees 1.5 GB as the minimum.

Quote:

---

Originally Posted by dbasnett

I have never used these read/write everything io statements. When I first saw them I thought they were useful, but I bet the assumption was that the user would know they were designed for smaller??? files.

stanav is correct in his assessment of your memory usage, and I think stanav is saying he sees 1.5 GB as the minimum.

---

Even 1.5gb is fine.. the server is beefed up enough to handle it.. it has 6gb installed. But it's still maxing out. Sometimes i get out of memory after writing 20 30mb files and then it hits a 100mb file. That should only be a 700MB operation.

Dont you think somewhere in the code this isnt dumping the memory properly?

6GB free, total, ??? Is this the only thing running when you have the problem? Does the problem happen in the IDE? If so what statement causes the problem?

Without looking at your code in detail I think you have more huge strings than you might suspect, as well as the large files, and datasets. The fact that the problem is not deterministic should tell you something.

100% CPU is typically hard to attain when an application is working mostly with files, unless there are some very long and tight loops with no IO.

Most probably it's not your application running out of memory but some component inside it is not releasing its memory and there is some framework internal limit on that. First (if this is 2005) try the new Using block for your streamreader and streamwriter and make sure you include 'as New'.

Code:

---

Dim contents As String

Using reader As New StreamReader ("D:\Logs\N2\" & strfile)
contents = reader.ReadToEnd()
End Using

Dim ds As New DataSet
ds.ReadXml("d:\Logs\n.xml")
'the rest of your code here...

'then
Dim strOutputFile2 = "D:\Logs\N\" & strfile

Dim objFileSystem2 = CreateObject("Scripting.fileSystemObject")
Dim objOutputFile2 = objFileSystem2.OpenTextFile(strOutputFile2)

Using objwriter As New StreamWriter("D:\Logs\N\" & strfile)
objwriter.WriteLine(sb.ToString)
End Using

---