WinForms Memory Leak

[sdfax_work]

I'm trying to get to grips with how Windows Forms applications manage memory allocation. I'll give you an illustration of the problem. Take this simple winforms app which is a main form with two buttons: one that opens a form containing some random data, and another button that closes all open forms (except the main form)

Public Class MainForm

    Private WithEvents _timer As System.Timers.Timer = New System.Timers.Timer()

    Private Sub UltraButton1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles UltraButton1.Click
        Try
            Dim frm As Form = New GridForm()
            frm.Show()

        Catch ex As Exception
            MessageBox.Show(ex.Message)
        End Try
        
    End Sub

    Private Sub _timer_Elapsed(ByVal sender As Object, ByVal e As System.Timers.ElapsedEventArgs) Handles _timer.Elapsed
        Dim bytesInUse As Long = GC.GetTotalMemory(False)
        Dim mbInUse As Double = Math.Round(bytesInUse / 1024 / 1024, 1)

        Dim msg As String = String.Format("Form Count: {0}        {1}       {2}MB       ({3} bytes)", (My.Application.OpenForms.Count - 1).ToString("000"), System.DateTime.Now.ToString("HH:mm:ss fff"), mbInUse.ToString("000"), bytesInUse.ToString("0000000000"))

        Debug.Print(msg)
    End Sub

    Private Sub MainForm_Load(ByVal sender As Object, ByVal e As System.EventArgs) Handles Me.Load
        _timer.Interval = 5000
        _timer.Enabled = True
    End Sub

    Private Sub UltraButton2_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles UltraButton2.Click
        Try
            Dim closeForms As List(Of Form) = New List(Of Form)()

            For Each frm As Form In My.Application.OpenForms
                If frm.Name = "GridForm" Then
                    closeForms.Add(frm)
                End If
            Next

            For Each frm As Form In closeForms
                frm.Close()
                frm.Dispose()
            Next


        Catch ex As Exception
            MessageBox.Show(ex.Message)
        End Try
    End Sub
End Class





Public Class GridForm

    Private _randomData As List(Of RandomData) = Nothing
    
    Private Sub GridForm_Load(ByVal sender As Object, ByVal e As System.EventArgs) Handles Me.Load
        Try
            _randomData = RandomDataHelper.GetRandomData()

            Me.DataGridView1.DataSource = _randomData

        Catch ex As Exception

        End Try
    End Sub
End Class




Friend Class RandomData

    Private _guid As String
    Private _id As Integer

    Public Property MyGuid() As String
        Get
            Return _guid
        End Get
        Set(ByVal value As String)
            _guid = value
        End Set
    End Property

    Public Property Id() As Integer
        Get
            Return _id
        End Get
        Set(ByVal value As Integer)
            _id = value
        End Set
    End Property
End Class






Public Class RandomDataHelper

    Private Shared _randomData As List(Of RandomData) = Nothing

    Friend Shared Function GetRandomData() As List(Of RandomData)
        If _randomData Is Nothing Then
            _randomData = New List(Of RandomData)()
            For i As Integer = 1 To 100000
                _randomData.Add(New RandomData() With {.Id = i, .MyGuid = System.Guid.NewGuid().ToString()})
            Next
        End If

        Dim newList As List(Of RandomData) = New List(Of RandomData)()

        For Each rd In _randomData
            newList.Add(New RandomData() With {.Id = rd.Id, .MyGuid = rd.MyGuid})
        Next

        Return newList
    End Function

End Class

If you run this and click away opening lots of forms, 50 at a time, 100 at a time, and then closing them all, the memory goes up and down over time but gradually gets higher and higher, never returning to anything like its initial state. Now I assume part of this is the .NET runtime being clever and assuming that as the app needed a lot of memory previously, it will need more again and better to keep some allocated (I am running this on a machine with 4GB RAM).

This is the output

Form Count: 000        08:44:39 755       001MB       (0000813092 bytes)
Form Count: 001        08:44:44 728       017MB       (0017485136 bytes)
Form Count: 016        08:44:49 716       062MB       (0064453476 bytes)
Form Count: 016        08:44:54 705       062MB       (0064887652 bytes)
Form Count: 016        08:44:59 694       062MB       (0064887652 bytes)
Form Count: 016        08:45:04 682       062MB       (0064887652 bytes)
Form Count: 016        08:45:09 671       062MB       (0064887652 bytes)
Form Count: 016        08:45:14 660       062MB       (0064904036 bytes)
Form Count: 016        08:45:19 648       062MB       (0064904036 bytes)
Form Count: 016        08:45:24 637       062MB       (0065084260 bytes)
Form Count: 026        08:45:29 626       103MB       (0107992376 bytes)
Form Count: 049        08:45:34 614       178MB       (0186475920 bytes)
Form Count: 000        08:45:39 603       178MB       (0187032976 bytes)
Form Count: 006        08:45:44 594       202MB       (0212140692 bytes)
Form Count: 028        08:45:49 699       242MB       (0253800456 bytes)
Form Count: 063        08:45:54 585       389MB       (0407684876 bytes)
Form Count: 001        08:45:59 581       401MB       (0420033088 bytes)
Form Count: 019        08:46:04 576       268MB       (0280643940 bytes) -
Form Count: 029        08:46:09 572       313MB       (0328246272 bytes)
Form Count: 060        08:46:14 567       202MB       (0211634588 bytes) -
Form Count: 000        08:46:19 563       228MB       (0238579268 bytes)
Form Count: 021        08:46:24 559       303MB       (0317532092 bytes)
Form Count: 050        08:46:29 554       181MB       (0189643032 bytes) -
Form Count: 050        08:46:34 550       181MB       (0189643032 bytes)
Form Count: 007        08:46:39 592       205MB       (0214563092 bytes)
Form Count: 046        08:46:44 541       318MB       (0333827408 bytes)
Form Count: 000        08:46:49 537       335MB       (0350860716 bytes)
Form Count: 006        08:46:54 532       360MB       (0377380588 bytes)
Form Count: 041        08:46:59 528       293MB       (0306788368 bytes) -
Form Count: 049        08:47:04 524       329MB       (0344529752 bytes)
Form Count: 049        08:47:09 519       329MB       (0344529752 bytes)
Form Count: 049        08:47:14 515       329MB       (0344529752 bytes)
Form Count: 000        08:47:19 510       329MB       (0345054040 bytes)
Form Count: 000        08:47:24 506       329MB       (0345070424 bytes)
Form Count: 000        08:47:29 502       329MB       (0345095000 bytes)
Form Count: 000        08:47:34 497       329MB       (0345095000 bytes)
Form Count: 019        08:47:39 524       404MB       (0423631984 bytes)
Form Count: 049        08:47:44 489       166MB       (0173639576 bytes) -
Form Count: 049        08:47:49 484       166MB       (0173655960 bytes)
Form Count: 000        08:47:54 480       160MB       (0168018476 bytes)
Form Count: 000        08:47:59 475       160MB       (0168018476 bytes)
Form Count: 013        08:48:04 471       215MB       (0225774536 bytes)
Form Count: 050        08:48:09 623       306MB       (0320518980 bytes)
Form Count: 090        08:48:14 525       474MB       (0496950512 bytes)
Form Count: 099        08:48:19 458       514MB       (0538941444 bytes)
Form Count: 099        08:48:24 453       514MB       (0538941444 bytes)
Form Count: 099        08:48:29 449       514MB       (0538941444 bytes)
Form Count: 099        08:48:34 445       514MB       (0538949636 bytes)
Form Count: 099        08:48:39 440       514MB       (0538949636 bytes)
Form Count: 099        08:48:44 436       514MB       (0539039748 bytes)
Form Count: 000        08:48:49 432       515MB       (0540022788 bytes)
Form Count: 012        08:48:54 427       565MB       (0592271776 bytes)
Form Count: 049        08:48:59 423       473MB       (0496036112 bytes) -
Form Count: 088        08:49:04 418       646MB       (0677261468 bytes)
Form Count: 099        08:49:09 414       691MB       (0725020668 bytes)
Form Count: 099        08:49:14 410       691MB       (0725028860 bytes)
Form Count: 099        08:49:19 405       691MB       (0725028860 bytes)
Form Count: 099        08:49:24 401       691MB       (0725028860 bytes)
Form Count: 099        08:49:29 396       691MB       (0725028860 bytes)
Form Count: 033        08:49:34 392       692MB       (0725710488 bytes)
Form Count: 000        08:49:39 388       692MB       (0726046360 bytes)
Form Count: 005        08:49:44 383       316MB       (0331798056 bytes) -
Form Count: 005        08:49:49 379       318MB       (0333050676 bytes)
Form Count: 005        08:49:54 375       318MB       (0333050676 bytes)
Form Count: 030        08:49:59 370       101MB       (0106193620 bytes) --
Form Count: 063        08:50:04 506       200MB       (0209653496 bytes)
Form Count: 065        08:50:09 361       213MB       (0223073476 bytes)
Form Count: 065        08:50:14 357       213MB       (0223081668 bytes)
Form Count: 000        08:50:19 353       212MB       (0222580224 bytes)
Form Count: 004        08:50:24 348       230MB       (0241493684 bytes)
Form Count: 005        08:50:29 344       250MB       (0262433028 bytes)
Form Count: 005        08:50:34 339       250MB       (0262433028 bytes)
Form Count: 005        08:50:39 335       250MB       (0262433028 bytes)
Form Count: 005        08:50:44 332       250MB       (0262441220 bytes)
Form Count: 005        08:50:49 330       250MB       (0262441220 bytes)
Form Count: 005        08:50:54 328       250MB       (0262449412 bytes)
Form Count: 035        08:50:59 326       375MB       (0393604792 bytes)
Form Count: 070        08:51:04 324       443MB       (0463951896 bytes)
Form Count: 098        08:51:09 338       558MB       (0585160608 bytes)
Form Count: 000        08:51:14 336       564MB       (0591128768 bytes)
Form Count: 003        08:51:19 334       576MB       (0604198604 bytes)
Form Count: 003        08:51:24 333       588MB       (0616855512 bytes)
Form Count: 000        08:51:29 331       601MB       (0629999648 bytes)
Form Count: 020        08:51:34 329       683MB       (0716641424 bytes)
Form Count: 005        08:51:39 327       704MB       (0738295324 bytes)
Form Count: 013        08:51:44 372       766MB       (0802846820 bytes)
Form Count: 048        08:51:49 323       552MB       (0578560104 bytes) -
Form Count: 001        08:51:54 322       565MB       (0592206272 bytes)
Form Count: 005        08:51:59 320       581MB       (0609027444 bytes)
Form Count: 009        08:52:04 318       598MB       (0627505656 bytes)
Form Count: 004        08:52:09 316       617MB       (0646774500 bytes)
Form Count: 004        08:52:14 314       617MB       (0646774500 bytes)
Form Count: 025        08:52:19 313       253MB       (0265532620 bytes) - 
Form Count: 000        08:52:24 311       254MB       (0265835724 bytes)
Form Count: 000        08:52:29 309       254MB       (0265835724 bytes)
Form Count: 000        08:52:34 307       254MB       (0265876684 bytes)
Form Count: 000        08:52:39 305       254MB       (0265876684 bytes)

You can see the memory dropping at intervals (I've marked some of them above) which is good but it still slowly has a minimum that creeps up and up.

This is a very basic illustration of a much bigger production issue with a huge application with 100,000s lines of code and users that keep the app open all day. Over time they open and close a lot of forms and the memory creeps up.

Is there any way to force the allocation down? I know I can force GC but this won't do anything as it's the memory post-GC that is remaining high as far as I can see.

This isn't a problem with references as far as I can see because I have spent some time experimenting with just one form in the main application opening and closing it and there are no references after it is closed and disposed and yet the memory still creeps up in a similar way to that logged above. Also if it was a problem with references, I'm not sure how that would explain my test app above.

Any help or advice on what to use to troubleshoot greatly received.

[boops boops]

You RandomDataHelper class has a Shared List of RandomData objects. A shared object like that list can't go out of scope while the program is running, because it belongs to the class not to an instance. So every time you instantiate a GridForm, the call to RandomDataHelper adds another 10,000 RandomData items to the list. I wonder if that is the cause of your memory leak?

Maybe you could fix that by using List.Clear before adding the new items, although a better design could be to get rid of the Shared keywords in RandomDataHelper. Then you could instantiate and dispose of it every time you need it, for example with a Using block in GridForm.Load:

Using RDH As New RandomDataHelper
       _randomData = RDH.GetRandomData()
End Using

BB

[Shaggy Hiker]

What is the ultimate problem? Does the program crash, or are you simply disturbed by the memory usage?

After all, you made a key point early on when you suggested that part of this was .NET cleverly recognizing that the program used lots of memory before and might do so again. That's not necessarily .NET, but it is effectively correct. If the system has lots of RAM, any application can keep asking for more, and the OS will provide...up to a point. The OS can also recover RAM, but it isn't going to do that without sound reason. If there is no other demand on the RAM, why bother recovering it? What would even be the trigger for such a thing?

This kind of question used to come up more often, and the basic answer is: Unless it is causing trouble, ignore it. Memory is managed for efficiency, not for aesthetics. This means that there will be times when it will appear to be using more and more memory simply because it is easier to give the app more memory than to re-manage the memory the app already has.

[sdfax_work]

You RandomDataHelper class has a Shared List of RandomData objects. A shared object like that list can't go out of scope while the program is running, because it belongs to the class not to an instance. So every time you instantiate a GridForm, the call to RandomDataHelper adds another 10,000 RandomData items to the list. I wonder if that is the cause of your memory leak?

Maybe you could fix that by using List.Clear before adding the new items, although a better design could be to get rid of the Shared keywords in RandomDataHelper. Then you could instantiate and dispose of it every time you need it, for example with a Using block in GridForm.Load:

Using RDH As New RandomDataHelper
       _randomData = RDH.GetRandomData()
End Using

BB

Thanks for the reply. I don't think this is the cause because of the massive amount of memory that is sometimes cleared. The static (shared) class was to keep the GUIDs in memory and generate lists from them because generating GUIDs every time took too long. I should really do it properly and read from a file or a database to eliminate this but I wanted to exclude unmanaged resources. I will make some changes and see what effect they have

[sdfax_work]

What is the ultimate problem? Does the program crash, or are you simply disturbed by the memory usage?

After all, you made a key point early on when you suggested that part of this was .NET cleverly recognizing that the program used lots of memory before and might do so again. That's not necessarily .NET, but it is effectively correct. If the system has lots of RAM, any application can keep asking for more, and the OS will provide...up to a point. The OS can also recover RAM, but it isn't going to do that without sound reason. If there is no other demand on the RAM, why bother recovering it? What would even be the trigger for such a thing?

This kind of question used to come up more often, and the basic answer is: Unless it is causing trouble, ignore it. Memory is managed for efficiency, not for aesthetics. This means that there will be times when it will appear to be using more and more memory simply because it is easier to give the app more memory than to re-manage the memory the app already has.

Thanks for this. I will reply to this in some more detail a little later - rushed for time now

[Shaggy Hiker]

I forgot to mention that I remember from earlier discussions that minimizing then restoring the app caused the memory consumption to reset.

[sdfax_work]

What is the ultimate problem? Does the program crash, or are you simply disturbed by the memory usage?

After all, you made a key point early on when you suggested that part of this was .NET cleverly recognizing that the program used lots of memory before and might do so again. That's not necessarily .NET, but it is effectively correct. If the system has lots of RAM, any application can keep asking for more, and the OS will provide...up to a point. The OS can also recover RAM, but it isn't going to do that without sound reason. If there is no other demand on the RAM, why bother recovering it? What would even be the trigger for such a thing?

This kind of question used to come up more often, and the basic answer is: Unless it is causing trouble, ignore it. Memory is managed for efficiency, not for aesthetics. This means that there will be times when it will appear to be using more and more memory simply because it is easier to give the app more memory than to re-manage the memory the app already has.

OK, thanks again for the reply and the information. The code posted above is a bit contrived but in the production environment we get some users who have the win forms app open all day and in constant use and it involves a lot of opening and closing of forms with large grids and a lot of data.

It is a problem for them and .NET slows down and usually throws an IO out of memory exception. We can see that their memory creeps over time by monitoring using perf mon and also logging using GC.GetTotalMemory.

I stripped down one form in the app and then opened it programmatically multiple times to push the memory up, and then closed all the open forms in a similar way to that seen above. The memory does get freed up but despite this, over time, it slowly has a minimum level that creeps up and up.

I'm pretty sure if I ran that test app above for long enough I would see a similar thing but I can't find any evidence of references that are being held onto - and if there were references you would expect the memory never to come down at all until the whole app was closed. What we see is up-down-up-down trend where the "down" gets higher and higher over time and we end up with memory pressure. These users have high spec new machines, 4GB/6GB RAM and the app starts out about 20MB and creeps up throughout the day (with some "downs") to about 1.2GB and this is usually when they start to get issues (obviously depending on what else they are doing)

I hoped to write a test app to prove that there was something wrong with the production app and that just opening and closing data-rich forms in a winforms application shouldn't push up the memory over time but I seem to be showing that this happens with even the most basic applications if you do enough opening and closing of forms.

The minimize window thing is only temporary and as soon as the app has the focus again the memory jumps back up to where it was before you minimized.

Ideally, what I would like is to tell the runtime never to use more than 500MB and if it ever reaches this level, to release the allocation (or just the 500MB or already has reserved because there should be no way the app ever needs this much at any point in time). I can't seem to do anything within the application to prevent it creeping up over time but I'm open to suggestions.

[Shaggy Hiker]

Where do you feel the problem lies? Do you feel that there is a flaw in your program that isn't releasing the memory, or do you feel that there is something inherently wrong with .NET that causes a slow, steady, memory leak? I would say that this is more of a gut feeling kind of thing rather than a logical conclusion, and that's all I'm asking for. The choice you make seems likely to influence the direction you take with this. You appear to be looking for an external, .NET method that will fix what sounds likely to be a design problem. If the real problem comes from resources not being freed, then placing a cap of any sort on the maximum memory usage would only serve to crash the program sooner rather than later. The only way that seems like a viable solution is if you feel that .NET is just being selfish and that if you had the right tool to limit it, it would mend its ways.

As far as I can see, the problem, while it may be VERY tricky, is one of something not being released properly such that some block of technically unused memory is still theoretically accessible, and therefore the GC will never release it. Where that would be in a large app I can't say. Fortunately, you have a simple test app that shows the same problem (or at least a similar manifestation), so you can tinker around with that.

[sdfax_work]

Where do you feel the problem lies? Do you feel that there is a flaw in your program that isn't releasing the memory, or do you feel that there is something inherently wrong with .NET that causes a slow, steady, memory leak? I would say that this is more of a gut feeling kind of thing rather than a logical conclusion, and that's all I'm asking for. The choice you make seems likely to influence the direction you take with this. You appear to be looking for an external, .NET method that will fix what sounds likely to be a design problem. If the real problem comes from resources not being freed, then placing a cap of any sort on the maximum memory usage would only serve to crash the program sooner rather than later. The only way that seems like a viable solution is if you feel that .NET is just being selfish and that if you had the right tool to limit it, it would mend its ways.

As far as I can see, the problem, while it may be VERY tricky, is one of something not being released properly such that some block of technically unused memory is still theoretically accessible, and therefore the GC will never release it. Where that would be in a large app I can't say. Fortunately, you have a simple test app that shows the same problem (or at least a similar manifestation), so you can tinker around with that.

I assume it's with our app somewhere 100%, not generally in winforms. I was just surprised and confused by the memory graph in a such a small basic test app. Fair enough that the timing can seem random but great that memory is reclaimed and I know memory allocation is a dark art. What makes no sense to me is the lower bound trend upwards over time. I will tweak the test app and remove the static class to rule this out. I've used a .NET Memory Profiler in the past so may revisit this but it's excruciating because of the amount of data you have to sift through

[Shaggy Hiker]

The lower bound trending upwards is certainly disturbing, but, as you say, memory allocation is a dark art, so it isn't entirely implausible. I would suggest getting a good understanding of how the Garbage Collector works in .NET. It's kind of an interesting topic, though I know of no single source to suggest. The key, in my mind, is understanding how the GC recognizes that some object is unreachable. It seems to me that the most likely problem here is that some resource is unreachable in your mind (you certainly believe it to be discarded), yet the GC object map suggests that the item really isn't unreachable, so it never feels safe in recovering that memory. Of course, that can be quite the subject in itself.

[Grimfort]

I've used a .NET Memory Profiler in the past so may revisit this but it's excruciating because of the amount of data you have to sift through

Actually it is really easy to find leaks using a profiler. The thing to look at is delta, you can see the difference between created and removed, the ones that are not removed have a really high delta, and those are the ones that are causing your leak. Open your base form, take a snapshot to get a baseline, open loads and loads and loads of forms, leaving them open take another snapshot, close them all back to the base form, take another snapshop. Diff the first and last, they should be similar. A decent profiler will show you the link to the object and code that is keeping it referenced.

[Evil_Giraffe]

It is a problem for them and .NET slows down and usually throws an IO out of memory exception. We can see that their memory creeps over time by monitoring using perf mon and also logging using GC.GetTotalMemory.

If it's throwing an OOM exception, that means that GC is unable to collect "unused" memory because a reference to it exists still. (That's why unused is in quotes).

If there was truly unused memory, the GC would kick in before allocation and make it available for use.

I hoped to write a test app to prove that there was something wrong with the production app and that just opening and closing data-rich forms in a winforms application shouldn't push up the memory over time but I seem to be showing that this happens with even the most basic applications if you do enough opening and closing of forms.

If you are allocating memory, your memory usage will go up. It will only go back down again when GC kicks in. If you don't allocate enough memory that you trigger a GC, then that memory will never be reclaimed until you exit the application. This is not a problem. However, your app is hitting those thresholds, which is why you see the dips in your memory usage.

As to the "minimum memory" creeping up: this I would hazard a guess is due to some objects surviving a garbage collection and being promoted to Gen1 objects. The .NET GC is a generational GC algorithm. It has three generations (Gen0, GEn1 and Gen2). A Gen1 collection (collects from Gen0 and Gen1) is more costly than a Gen0 collection (only collects from Gen0), so you would expect it to happen every tenth run of a Gen0 collection (approx, ish). A Gen2 (Full) collection is even more costly still, so would run every ten runs of a Gen1 collection (again: ish).

So, if you have some transient objects that are still alive when you do a Gen0 collection (that you will stop referencing very soon), they will be promoted to the next generation and a) won't be eligible for collection until a Gen1 collection occurs b) won't count towards the Gen0 threshold of memory usage and c) will still show up in the total memory count.

This is what I suspect is happening in your test app. I hope you can see how it explains the memory usage you are seeing.

[Evil_Giraffe]

    Private Sub UltraButton2_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles UltraButton2.Click
        Try
            Dim closeForms As List(Of Form) = New List(Of Form)()

            For Each frm As Form In My.Application.OpenForms
                If frm.Name = "GridForm" Then
                    closeForms.Add(frm)
                End If
            Next

            For Each frm As Form In closeForms
                frm.Close()
                frm.Dispose()
            Next


        Catch ex As Exception
            MessageBox.Show(ex.Message)
        End Try
    End Sub

This is just an example that sticks out at me. Suppose a Gen0 collection happens whilst in this Sub (I think this is unlikely, as I can't really see what might trigger it by inspection, but it is possible and serves a handy illustration of some of the problems you might face).

There is a List(Of Form) that is being kept alive by the closeForms variable, so that list survives the collection and is promoted to the next generation, and not collected until the next Gen1 collection. Okay, not too bad on its own, it only contains as many references as the number of your "GridForm"s open at the time. BUT, by referencing those forms, those are also alive and promoted to the next generation. You call Dispose on the forms, true, but that does not force them to be Garbage Collected! You don't override Form's implementation of Dispose to remove the reference to all the random data, so by virtue of the Form objects being alive, the random data on each of those forms are still alive as well, and get promoted to the next generation also!

So, that's just one example of how objects can stay alive longer than you'd think.

[Evil_Giraffe]

You RandomDataHelper class has a Shared List of RandomData objects. A shared object like that list can't go out of scope while the program is running, because it belongs to the class not to an instance. So every time you instantiate a GridForm, the call to RandomDataHelper adds another 10,000 RandomData items to the list. I wonder if that is the cause of your memory leak?

Not sure if anyone has responded to this yet (can't see anything but I may have missed it).

I don't think that is an issue. When the RandomDataHelper.GetRandomData method is called, it only allocates the list object and populates it with data the first time through. Subsequent calls to the method create a new List with new Data objects that get returned, but are not stored in the Shared member.

This will allocate a bunch of memory that is never released only on the first time the method is called. Subsequent to that, the memory allocated will be collectible before the end of the program.

[Evil_Giraffe]

For your test program only: I think this is one of the few times where you (the programmer) are smarter than the GC. You know that when you close all those forms, you are done with those objects. You know that the user may very well tolerate some fractional delay in the application because they are "done" with an immediate task. Therefore, this is one of the very few times where you could probably get away with forcing a collection. I would try adding a GC.Collect() in the UltraButton2_Click handler outside the try/catch block (such that the closeForms variable has gone out of scope! (see #13)).

Compare the memory usage you see with and without that call.

NOTE: I only suggest this approach for this particular scenario and if you do it you must test the effect (compare to without doing so) to ensure that you are actually getting something for your money here. Forcing a collection isn't cheap, and worse, the GC adapts to your app's memory profile as it runs so it can perform better - forcing a Collection wipes out those internal counters and resets the GC to dumb I-don't-know-what-this-app-looks-like mode.


As to your production app: you should find the memory leak that I strongly suspect is there before you try anything like this.

[sdfax_work]

Actually it is really easy to find leaks using a profiler. The thing to look at is delta, you can see the difference between created and removed, the ones that are not removed have a really high delta, and those are the ones that are causing your leak. Open your base form, take a snapshot to get a baseline, open loads and loads and loads of forms, leaving them open take another snapshot, close them all back to the base form, take another snapshop. Diff the first and last, they should be similar. A decent profiler will show you the link to the object and code that is keeping it referenced.

I will revisit the memory profiling options. We have some RedGate tools so I'll look to get their profiler. The size of the app and the number of Infragistics controls we use makes it much more difficult than it would be for most apps but to be fair I didn't do much complex delta analysis other than looking for a class that only got referenced in one of the forms and identifying the number of object instances of that class still in memory

[sdfax_work]

The lower bound trending upwards is certainly disturbing, but, as you say, memory allocation is a dark art, so it isn't entirely implausible. I would suggest getting a good understanding of how the Garbage Collector works in .NET. It's kind of an interesting topic, though I know of no single source to suggest. The key, in my mind, is understanding how the GC recognizes that some object is unreachable. It seems to me that the most likely problem here is that some resource is unreachable in your mind (you certainly believe it to be discarded), yet the GC object map suggests that the item really isn't unreachable, so it never feels safe in recovering that memory. Of course, that can be quite the subject in itself.

I had a good look at garbage collection in the past (and yes I agree that it is interesting). We also profiled the gen0,1,2 object collection counts but couldn't see anything untoward in the ratios between the generations (following some MS guidelines).

I also agree on the most likely problem being items identified as reachable. Each time I explain this issue or read up about it I get an idea about what may be the cause and you've just given me another idea. We have the equivalent of an Alt-Tab application switched internally in the app so that you can see all of the open forms visually and switch between them. When the form dies I'll check what is happening to this list collection

[sdfax_work]

For your test program only: I think this is one of the few times where you (the programmer) are smarter than the GC. You know that when you close all those forms, you are done with those objects. You know that the user may very well tolerate some fractional delay in the application because they are "done" with an immediate task. Therefore, this is one of the very few times where you could probably get away with forcing a collection. I would try adding a GC.Collect() in the UltraButton2_Click handler outside the try/catch block (such that the closeForms variable has gone out of scope! (see #13)).

Compare the memory usage you see with and without that call.

If do a GC.Collect() for 0,1,2 after closing all of the forms, it has no immediate effect, but if you then open just one form and repeat the close all forms code (including the the GC.Collect()) the memory seemed to be consistently compacted back to about 15MB. I need some more time to test but this is a huge improvement in the test app - I realise this is not a good idea in production

[MastAvalons]

I'm trying to get to grips with how Windows Forms applications manage memory allocation. I'll give you an illustration of the problem. Take this simple winforms app which is a main form with two buttons: one that opens a form containing some random data, and another button that closes all open forms (except the main form)

Public Class MainForm

    Private WithEvents _timer As System.Timers.Timer = New System.Timers.Timer()

    Private Sub UltraButton1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles UltraButton1.Click
        Try
            Dim frm As Form = New GridForm()
            frm.Show()

        Catch ex As Exception
            MessageBox.Show(ex.Message)
        End Try
        
    End Sub

    Private Sub _timer_Elapsed(ByVal sender As Object, ByVal e As System.Timers.ElapsedEventArgs) Handles _timer.Elapsed
        Dim bytesInUse As Long = GC.GetTotalMemory(False)
        Dim mbInUse As Double = Math.Round(bytesInUse / 1024 / 1024, 1)

        Dim msg As String = String.Format("Form Count: {0}        {1}       {2}MB       ({3} bytes)", (My.Application.OpenForms.Count - 1).ToString("000"), System.DateTime.Now.ToString("HH:mm:ss fff"), mbInUse.ToString("000"), bytesInUse.ToString("0000000000"))

        Debug.Print(msg)
    End Sub

    Private Sub MainForm_Load(ByVal sender As Object, ByVal e As System.EventArgs) Handles Me.Load
        _timer.Interval = 5000
        _timer.Enabled = True
    End Sub

    Private Sub UltraButton2_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles UltraButton2.Click
        Try
            Dim closeForms As List(Of Form) = New List(Of Form)()

            For Each frm As Form In My.Application.OpenForms
                If frm.Name = "GridForm" Then
                    closeForms.Add(frm)
                End If
            Next

            For Each frm As Form In closeForms
                frm.Close()
                frm.Dispose()
            Next


        Catch ex As Exception
            MessageBox.Show(ex.Message)
        End Try
    End Sub
End Class





Public Class GridForm

    Private _randomData As List(Of RandomData) = Nothing
    
    Private Sub GridForm_Load(ByVal sender As Object, ByVal e As System.EventArgs) Handles Me.Load
        Try
            _randomData = RandomDataHelper.GetRandomData()

            Me.DataGridView1.DataSource = _randomData

        Catch ex As Exception

        End Try
    End Sub
End Class




Friend Class RandomData

    Private _guid As String
    Private _id As Integer

    Public Property MyGuid() As String
        Get
            Return _guid
        End Get
        Set(ByVal value As String)
            _guid = value
        End Set
    End Property

    Public Property Id() As Integer
        Get
            Return _id
        End Get
        Set(ByVal value As Integer)
            _id = value
        End Set
    End Property
End Class






Public Class RandomDataHelper

    Private Shared _randomData As List(Of RandomData) = Nothing

    Friend Shared Function GetRandomData() As List(Of RandomData)
        If _randomData Is Nothing Then
            _randomData = New List(Of RandomData)()
            For i As Integer = 1 To 100000
                _randomData.Add(New RandomData() With {.Id = i, .MyGuid = System.Guid.NewGuid().ToString()})
            Next
        End If

        Dim newList As List(Of RandomData) = New List(Of RandomData)()

        For Each rd In _randomData
            newList.Add(New RandomData() With {.Id = rd.Id, .MyGuid = rd.MyGuid})
        Next

        Return newList
    End Function

End Class

If you run this and click away opening lots of forms, 50 at a time, 100 at a time, and then closing them all, the memory goes up and down over time but gradually gets higher and higher, never returning to anything like its initial state. Now I assume part of this is the .NET runtime being clever and assuming that as the app needed a lot of memory previously, it will need more again and better to keep some allocated (I am running this on a machine with 4GB RAM).

This is the output

Form Count: 000        08:44:39 755       001MB       (0000813092 bytes)
Form Count: 001        08:44:44 728       017MB       (0017485136 bytes)
Form Count: 016        08:44:49 716       062MB       (0064453476 bytes)
Form Count: 016        08:44:54 705       062MB       (0064887652 bytes)
Form Count: 016        08:44:59 694       062MB       (0064887652 bytes)
Form Count: 016        08:45:04 682       062MB       (0064887652 bytes)
Form Count: 016        08:45:09 671       062MB       (0064887652 bytes)
Form Count: 016        08:45:14 660       062MB       (0064904036 bytes)
Form Count: 016        08:45:19 648       062MB       (0064904036 bytes)
Form Count: 016        08:45:24 637       062MB       (0065084260 bytes)
Form Count: 026        08:45:29 626       103MB       (0107992376 bytes)
Form Count: 049        08:45:34 614       178MB       (0186475920 bytes)
Form Count: 000        08:45:39 603       178MB       (0187032976 bytes)
Form Count: 006        08:45:44 594       202MB       (0212140692 bytes)
Form Count: 028        08:45:49 699       242MB       (0253800456 bytes)
Form Count: 063        08:45:54 585       389MB       (0407684876 bytes)
Form Count: 001        08:45:59 581       401MB       (0420033088 bytes)
Form Count: 019        08:46:04 576       268MB       (0280643940 bytes) -
Form Count: 029        08:46:09 572       313MB       (0328246272 bytes)
Form Count: 060        08:46:14 567       202MB       (0211634588 bytes) -
Form Count: 000        08:46:19 563       228MB       (0238579268 bytes)
Form Count: 021        08:46:24 559       303MB       (0317532092 bytes)
Form Count: 050        08:46:29 554       181MB       (0189643032 bytes) -
Form Count: 050        08:46:34 550       181MB       (0189643032 bytes)
Form Count: 007        08:46:39 592       205MB       (0214563092 bytes)
Form Count: 046        08:46:44 541       318MB       (0333827408 bytes)
Form Count: 000        08:46:49 537       335MB       (0350860716 bytes)
Form Count: 006        08:46:54 532       360MB       (0377380588 bytes)
Form Count: 041        08:46:59 528       293MB       (0306788368 bytes) -
Form Count: 049        08:47:04 524       329MB       (0344529752 bytes)
Form Count: 049        08:47:09 519       329MB       (0344529752 bytes)
Form Count: 049        08:47:14 515       329MB       (0344529752 bytes)
Form Count: 000        08:47:19 510       329MB       (0345054040 bytes)
Form Count: 000        08:47:24 506       329MB       (0345070424 bytes)
Form Count: 000        08:47:29 502       329MB       (0345095000 bytes)
Form Count: 000        08:47:34 497       329MB       (0345095000 bytes)
Form Count: 019        08:47:39 524       404MB       (0423631984 bytes)
Form Count: 049        08:47:44 489       166MB       (0173639576 bytes) -
Form Count: 049        08:47:49 484       166MB       (0173655960 bytes)
Form Count: 000        08:47:54 480       160MB       (0168018476 bytes)
Form Count: 000        08:47:59 475       160MB       (0168018476 bytes)
Form Count: 013        08:48:04 471       215MB       (0225774536 bytes)
Form Count: 050        08:48:09 623       306MB       (0320518980 bytes)
Form Count: 090        08:48:14 525       474MB       (0496950512 bytes)
Form Count: 099        08:48:19 458       514MB       (0538941444 bytes)
Form Count: 099        08:48:24 453       514MB       (0538941444 bytes)
Form Count: 099        08:48:29 449       514MB       (0538941444 bytes)
Form Count: 099        08:48:34 445       514MB       (0538949636 bytes)
Form Count: 099        08:48:39 440       514MB       (0538949636 bytes)
Form Count: 099        08:48:44 436       514MB       (0539039748 bytes)
Form Count: 000        08:48:49 432       515MB       (0540022788 bytes)
Form Count: 012        08:48:54 427       565MB       (0592271776 bytes)
Form Count: 049        08:48:59 423       473MB       (0496036112 bytes) -
Form Count: 088        08:49:04 418       646MB       (0677261468 bytes)
Form Count: 099        08:49:09 414       691MB       (0725020668 bytes)
Form Count: 099        08:49:14 410       691MB       (0725028860 bytes)
Form Count: 099        08:49:19 405       691MB       (0725028860 bytes)
Form Count: 099        08:49:24 401       691MB       (0725028860 bytes)
Form Count: 099        08:49:29 396       691MB       (0725028860 bytes)
Form Count: 033        08:49:34 392       692MB       (0725710488 bytes)
Form Count: 000        08:49:39 388       692MB       (0726046360 bytes)
Form Count: 005        08:49:44 383       316MB       (0331798056 bytes) -
Form Count: 005        08:49:49 379       318MB       (0333050676 bytes)
Form Count: 005        08:49:54 375       318MB       (0333050676 bytes)
Form Count: 030        08:49:59 370       101MB       (0106193620 bytes) --
Form Count: 063        08:50:04 506       200MB       (0209653496 bytes)
Form Count: 065        08:50:09 361       213MB       (0223073476 bytes)
Form Count: 065        08:50:14 357       213MB       (0223081668 bytes)
Form Count: 000        08:50:19 353       212MB       (0222580224 bytes)
Form Count: 004        08:50:24 348       230MB       (0241493684 bytes)
Form Count: 005        08:50:29 344       250MB       (0262433028 bytes)
Form Count: 005        08:50:34 339       250MB       (0262433028 bytes)
Form Count: 005        08:50:39 335       250MB       (0262433028 bytes)
Form Count: 005        08:50:44 332       250MB       (0262441220 bytes)
Form Count: 005        08:50:49 330       250MB       (0262441220 bytes)
Form Count: 005        08:50:54 328       250MB       (0262449412 bytes)
Form Count: 035        08:50:59 326       375MB       (0393604792 bytes)
Form Count: 070        08:51:04 324       443MB       (0463951896 bytes)
Form Count: 098        08:51:09 338       558MB       (0585160608 bytes)
Form Count: 000        08:51:14 336       564MB       (0591128768 bytes)
Form Count: 003        08:51:19 334       576MB       (0604198604 bytes)
Form Count: 003        08:51:24 333       588MB       (0616855512 bytes)
Form Count: 000        08:51:29 331       601MB       (0629999648 bytes)
Form Count: 020        08:51:34 329       683MB       (0716641424 bytes)
Form Count: 005        08:51:39 327       704MB       (0738295324 bytes)
Form Count: 013        08:51:44 372       766MB       (0802846820 bytes)
Form Count: 048        08:51:49 323       552MB       (0578560104 bytes) -
Form Count: 001        08:51:54 322       565MB       (0592206272 bytes)
Form Count: 005        08:51:59 320       581MB       (0609027444 bytes)
Form Count: 009        08:52:04 318       598MB       (0627505656 bytes)
Form Count: 004        08:52:09 316       617MB       (0646774500 bytes)
Form Count: 004        08:52:14 314       617MB       (0646774500 bytes)
Form Count: 025        08:52:19 313       253MB       (0265532620 bytes) - 
Form Count: 000        08:52:24 311       254MB       (0265835724 bytes)
Form Count: 000        08:52:29 309       254MB       (0265835724 bytes)
Form Count: 000        08:52:34 307       254MB       (0265876684 bytes)
Form Count: 000        08:52:39 305       254MB       (0265876684 bytes)

You can see the memory dropping at intervals (I've marked some of them above) which is good but it still slowly has a minimum that creeps up and up.

This is a very basic illustration of a much bigger production issue with a huge application with 100,000s lines of code and users that keep the app open all day. Over time they open and close a lot of forms and the memory creeps up.

Is there any way to force the allocation down? I know I can force GC but this won't do anything as it's the memory post-GC that is remaining high as far as I can see.

This isn't a problem with references as far as I can see because I have spent some time experimenting with just one form in the main application opening and closing it and there are no references after it is closed and disposed and yet the memory still creeps up in a similar way to that logged above. Also if it was a problem with references, I'm not sure how that would explain my test app above.

Any help or advice on what to use to troubleshoot greatly received.

you can use debuggers - Valgrind (for Linux) or Deleaker (for Windows). With Deleaker you can detect and localize resource leaks such as memory, gdi and user objects.

[Jokeman]

If you have a problem with memory leaks, you need to be more careful. If you allocate memory, you have the duty to liberate. Do not forget this important aspect.
The debugger only helps to look for leaks, but it does not work for you.

[MastAvalons]

memory leaks - it is always evil

[Evil_Giraffe]

If you allocate memory, you have the duty to liberate. Do not forget this important aspect.

Um, not really the case in a managed memory language like VB. And when you do need to take care, there are different considerations to unmanaged languages.

Essentially, if there is no pressure for the memory to be released, why on earth should any time be spent on releasing the memory? This can make things look much worse than they are if you just look at 'memory usage' of a .NET process (the same with Java).

[MastAvalons]

Java rules OK!

[sdfax_work]

Thank you to everyone who contributed to this thread. The recent posts have served as a reminder that am I remiss in not updating what happened in this particular case.

The memory leak was traced to the way a third party component, Infragistics UltraGrid and UltraComboBox held onto a reference. In summary:

1. Instantiate and show a new form containing a grid where one column is bound to a combobox.
2. Close the form when you're done
3. A reference survives related to the combobox bound to the column (this was detected using Redgate's ANTS memory profiler)
4. Because of the surviving reference the form and its memory are never garbage collected

It was fixed by introducing something like this is the class that serves as a base to all the data bound forms

Private Sub frmBoundForm_FormClosed(ByVal sender As Object, ByVal e As System.Windows.Forms.FormClosedEventArgs) Handles Me.FormClosed
    Try
        ClearDownGridCombos(Me)

    Catch ex As Exception
        HandleException(ex)
    End Try
End Sub

Private Sub ClearDownGridCombos(ByVal ctl As Control)
    If TypeOf ctl Is UltraGrid Then
        ClearDownGridCombos(CType(ctl, UltraGrid))
    End If

    If ctl.HasChildren Then
        For Each child As Control In ctl.Controls
            ClearDownGridCombos(child)
        Next
    End If
End Sub

Private Sub ClearDownGridCombos(ByVal ug As UltraGrid)
    For Each band As UltraGridBand In ug.DisplayLayout.Bands
        For Each col In band.Columns.Cast(Of UltraGridColumn)().Where(Function(c) c.EditorComponent IsNot Nothing)
            col.EditorComponent.Dispose()
            col.EditorComponent = Nothing
        Next
    Next
End Sub

There is still underlying strange behaviour and a general increase in memory over time in a very basic windows forms application that just opens and closes 100s of forms, but some memory does get released and it's not as big an issue as the problem above in the production app.

As has been stated, the best way to trace this kind of thing is to hunt for the root of surviving references using a tool like Redgate's ANTS memory profiler. This proved very useful in my case

[Big Pig]

a very interesting theme. recently, he was faced with memory leaks, however, appealed for help to http://stackoverflow.com. Now I know that live here, and powerful MINDS

[Grimfort]

That must be an awful babelfish translation or yoda is back in business!

[Big Pig]

Next time I'll be more attentive.
I read many useful things in the forum. I was impressed!
yoda always alive !!!!

[MastAvalons]

Next time I'll be more attentive.
I read many useful things in the forum. I was impressed!
yoda always alive !!!!

[Big Pig]

it's time to take a towel and travel the galaxy.
hitchhiking...