webbrowser control - memory leak - HTTP web request instead of a webbrowser

[carspoo200]

Ok so i have created an app to load an access database into a datagridview, which contains web urls. When button is clicked it webbrowser1 navigates to each url and each webpages document.inertext is put into textbox. This all work fine but after a while the webbrowser navigation becomes increasingly slower. How could i fix this.

Any help much appreciated.

For Each RW As DataGridViewRow In Me.DataGridView1.SelectedRows
                '''''''''''#######cell values into strings ########''''''''''''''
                If RW.Selected = True Then

                    Dim domain As String
                    Dim keyword As String
                    Dim location As String
                    Dim space As String
                    space = " "
                    location = RW.Cells(10).Value.ToString
                    keyword = RW.Cells(9).Value.ToString
                    domain = RW.Cells(3).Value.ToString

                    Application.DoEvents()
                    If Not WebBrowser1.ReadyState = WebBrowserReadyState.Loading Then
                        If RadioButton1.Checked = True Then
                            WebBrowser1.Navigate("http://www.google.com.au/#q=" & keyword & space & location & "&start=0&hl=en&safe=off&biw=1280&bih=666&prmd=ivnsu&source=lnt&tbs=ctr:countryAU&cr=countryAU&sa=X&ei=WkRNTY-FLIfQcbr2vbkJ&ved=0CAcQpwUoAQ&fp=84eb9b8bc3203d0f")

                           

                        ElseIf RadioButton2.Checked = True Then
                            WebBrowser1.Navigate("http://www.google.com.au/#q=" & keyword & space & location & "&start=0&hl=en&safe=on&biw=1280&bih=666&prmd=ivnsu&source=lnt&tbs=ctr:countryAU&cr=countryAU&sa=X&ei=WkRNTY-FLIfQcbr2vbkJ&ved=0CAcQpwUoAQ&fp=84eb9b8bc3203d0f")
                            

                        ElseIf RadioButton3.Checked = True Then
                            WebBrowser1.Navigate("http://www.google.com.au/#q=" & keyword & space & location & "&start=0&hl=en&safe=moderate&biw=1280&bih=666&prmd=ivnsu&source=lnt&tbs=ctr:countryAU&cr=countryAU&sa=X&ei=WkRNTY-FLIfQcbr2vbkJ&ved=0CAcQpwUoAQ&fp=84eb9b8bc3203d0f")

                        End If
                    Else
                        MsgBox(" Did not send url! ")
                    End If
                    Application.DoEvents()

                    Delay(NumericUpDown1.Value)
                   
                    TextBox11.Text = WebBrowser1.Document.Body.InnerText.ToString()
                    Delay(NumericUpDown2.Value)

                    If Not TextBox11.Text = "" Then
                        If RW.Selected = True Then
                            location = RW.Cells(10).Value.ToString
                            keyword = RW.Cells(9).Value.ToString
                            domain = RW.Cells(3).Value.ToString

                            If TextBox11.Text.ToLower.Contains("www." & domain) Then
                                RW.Cells(7).Value = "1"

                            ElseIf TextBox11.Text.ToLower.Contains(domain) Then
                                RW.Cells(7).Value = "1-MC"

                            ElseIf Not TextBox11.Text.ToLower.Contains(domain) Then
                                RW.Cells(7).Value = "0"

                            End If
                            ProgressBar2.Value = 95
                            TextBox11.Text = ""
                        End If
                    Else
                        RW.Cells(7).Value = "Recheck"
                    End If
                End If
            Next
        End If

[carspoo200]

Any help/suggestions would be great.

[bergerkiller]

Any help...well I used webbrowsers in threads once and they were accessed and created using delegated subs. (Thread asked for the Main form to invoke some subroutine). It also got increasingly slower after a while, and the RAM memory kept on increasing.

Then I started to discover that any created webbrowser instance wasn't disposed from memory when it got out of focus. So, I had to dispose every control after I used it. (another invoke).

You might have to create a new webbrowser control, make it process the navigation event (AddHandler mytempbrowser.navigating, AddressOf <subname>) and finally dispose this webbrowser.

I do not believe you are creating multiple instances, but maybe the old data doesn't get disposed when navigating? I'd say...try to dispose some.

Besides the webbrowser, I see you are calling the "CreateObject("WScript.Shell")" quite often. Be careful with "in-code variable creation", they usually remain after the loop ended.

Add all above the loop this line:

Dim shell As Object = CreateObject("WScript.Shell")

And call it like this:

shell.sendkeys("{ENTER}")
shell.Popup("Hello", 3, "hello!!")
etc.

Is there no way to use an HTTP web request instead of a webbrowser? A webbrowser stores all page images and objects, pure HTML would take up very little compared to that.

[carspoo200]

Is there no way to use an HTTP web request instead of a webbrowser? A webbrowser stores all page images and objects, pure HTML would take up very little compared to that.

I could use httpweb request but i have know idea how to use that function
how would you do it with httpweb requests?

p.s looking into some of the things you mentioned thanks. Oh and the wscript was commented out

[bergerkiller]

Ah ok, didn't notice the '.

In your code, all you want is to load the HTML from the url you pass into "Webbrowser.navigate". So, first step is getting a String variable with the URL to load:

Dim URL As String = ""
If RadioButton1.Checked = True Then
     URL = "http://www.google.com.au/#q=" & keyword & space & location & "&start=0&hl=en&safe=off&biw=1280&bih=666&prmd=ivnsu&source=lnt&tbs=ctr:countryAU&cr=countryAU&sa=X&ei=WkRNTY-FLIfQcbr2vbkJ&ved=0CAcQpwUoAQ&fp=84eb9b8bc3203d0f"

ElseIf RadioButton2.Checked = True Then
     URL = "http://www.google.com.au/#q=" & keyword & space & location & "&start=0&hl=en&safe=on&biw=1280&bih=666&prmd=ivnsu&source=lnt&tbs=ctr:countryAU&cr=countryAU&sa=X&ei=WkRNTY-FLIfQcbr2vbkJ&ved=0CAcQpwUoAQ&fp=84eb9b8bc3203d0f"
                            
ElseIf RadioButton3.Checked = True Then                            
     URL = "http://www.google.com.au/#q=" & keyword & space & location & "&start=0&hl=en&safe=moderate&biw=1280&bih=666&prmd=ivnsu&source=lnt&tbs=ctr:countryAU&cr=countryAU&sa=X&ei=WkRNTY-FLIfQcbr2vbkJ&ved=0CAcQpwUoAQ&fp=84eb9b8bc3203d0f"

End If

Then you process this URL:

If URL <> "" Then
            Dim request As Net.HttpWebRequest = Net.HttpWebRequest.Create(URL)
            Dim response As Net.HttpWebResponse = request.GetResponse
            Dim stream As New IO.StreamReader(response.GetResponseStream)
            Dim HTML As String = stream.ReadToEnd.ToLower 'all following commands were tolower anyways
            stream.Close()
            response.Close()
            'process HTML
            If HTML.Contains("www." & domain) Then

            ElseIf HTML.Contains(domain) Then
                'etc
            End If
End If

You may have to add some more request information in between the Dim request and Dim response bit.

[carspoo200]

You may have to add some more request information in between the Dim request and Dim response bit.

Like what??? can seem to get it to work

[carspoo200]

Ah ok, didn't notice the '.

In your code, all you want is to load the HTML from the url you pass into "Webbrowser.navigate". So, first step is getting a String variable with the URL to load:

Dim URL As String = ""
If RadioButton1.Checked = True Then
     URL = "http://www.google.com.au/#q=" & keyword & space & location & "&start=0&hl=en&safe=off&biw=1280&bih=666&prmd=ivnsu&source=lnt&tbs=ctr:countryAU&cr=countryAU&sa=X&ei=WkRNTY-FLIfQcbr2vbkJ&ved=0CAcQpwUoAQ&fp=84eb9b8bc3203d0f"

ElseIf RadioButton2.Checked = True Then
     URL = "http://www.google.com.au/#q=" & keyword & space & location & "&start=0&hl=en&safe=on&biw=1280&bih=666&prmd=ivnsu&source=lnt&tbs=ctr:countryAU&cr=countryAU&sa=X&ei=WkRNTY-FLIfQcbr2vbkJ&ved=0CAcQpwUoAQ&fp=84eb9b8bc3203d0f"
                            
ElseIf RadioButton3.Checked = True Then                            
     URL = "http://www.google.com.au/#q=" & keyword & space & location & "&start=0&hl=en&safe=moderate&biw=1280&bih=666&prmd=ivnsu&source=lnt&tbs=ctr:countryAU&cr=countryAU&sa=X&ei=WkRNTY-FLIfQcbr2vbkJ&ved=0CAcQpwUoAQ&fp=84eb9b8bc3203d0f"

End If

Then you process this URL:

If URL <> "" Then
            Dim request As Net.HttpWebRequest = Net.HttpWebRequest.Create(URL)
            Dim response As Net.HttpWebResponse = request.GetResponse
            Dim stream As New IO.StreamReader(response.GetResponseStream)
            Dim HTML As String = stream.ReadToEnd.ToLower 'all following commands were tolower anyways
            stream.Close()
            response.Close()
            'process HTML
            If HTML.Contains("www." & domain) Then

            ElseIf HTML.Contains(domain) Then
                'etc
            End If
End If

You may have to add some more request information in between the Dim request and Dim response bit.

the html does not contain the webpage text and that is what i want to search. How i did that with webbrowser control was: WebBrowser1.Document.Body.InnerText.ToString
how do i do that with the httpwebrequest?

[bergerkiller]

Mmh well the HTTP request only returns a single piece of HTML. The webbrowser document should contain the same HTML, but only ordered into elements.

It could be you have to read a second .HTM link in the HTML and do another request for that, since the webbrowser loads the .htm links in the document as a new "sub document". For example, this is another document loaded by the webbrowser:

<frame src="frames/info.htm" scrolling=auto target="main">

A problem with using HTML parsing is that you can't access scripts.

Any ways. If the request does not return the HTML you need, the site probably has either a "user agent check" or contains multiple embedded frames.

You asked how you could limit the memory usage in a webbrowser. Well, you can't, so you will have to do it differently.

What you do:
- when you need the HTML, you make a new webbrowser.
- You navigate
- You wait till it is navigated
- You store the HTML
- You dispose EVERYTHING of the webbrowser, you close the stream, clear any outside data and basically just make it break from the inside.

    Private WB As WebBrowser
    Public Function GetHTML(ByVal URL As String, Optional ByVal innerHTML As Boolean = True) As String
        WB = New WebBrowser
        WB.Navigate(URL)
        Do While WB.ReadyState <> WebBrowserReadyState.Complete
            Application.DoEvents()
            Threading.Thread.Sleep(100)
        Loop
        If innerHTML Then
            GetHTML = WB.Document.Body.InnerText.ToString()
        Else
            GetHTML = WB.DocumentText
        End If
        'get rid of the control and all related streams and data
        WB.DocumentText = Nothing
        WB.DocumentStream.Close()
        WB.DocumentStream = Nothing
        For Each c As Control In WB.Controls
            c.Dispose()
            c = Nothing
        Next
        WB.Site = Nothing
        WB.Dispose()
        WB = Nothing

        GC.Collect()
    End Function

Even this left 2 mb of a memory leak, but it is a lot less than without the disposing; 10 mb.

I tried to access everything inside and destroy it all, but it even then some things are left in the background.

If anyone knows of a webbrowser part I haven't disposed, please reply.

EDIT

Alternatively you could make a separate application for getting the HTML. Then a memory leak is not possible at all, and you can run multiple "Get HTML" commands at a time.