[RESOLVED] WebRequest gives Time out second time

[Santana23]

Hi All,

I have a Windows program (sort of a crawler) that queries an internet website to collect data from. I use WebRequest and Webresponse. The first time I run the program everything works fine. However the second time I get a Time out exception on the WebRequest.GetRequestStream method.
Here is what I basically do:
1. Do a WebRequest with a GET method (to get Cookie info)
2. From the response I determin the number of query result pages I get.
3. I loop through the result pages and collect the rest of the data
4. Display result in a listview

This works fine. However when I change my search parameters and search again I immediately get a timeout on the WebRequest.GetRequestStream method. Now some other strange thing happens.. when I have Fiddler open to monitor the traffic, my program works fine all the time! Somehow Fiddler has some influence on it. I also tried to do some browsing to see if that had any impact but it didn't.

Here is the code:

Imports System.Net
Imports System.Text
Imports System.IO
Imports System.Web.HttpUtility
Imports System.Xml


Public Class MainForm
    Private WebRequest As Net.HttpWebRequest
    Private WebResponse As Net.HttpWebResponse
    Private cookies As Net.CookieContainer
    Private boolOnePage As Boolean = True

    Private Sub ButtonGo_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles ButtonGo.Click
        cookies = New Net.CookieContainer

        ListViewPeople.Items.Clear()
        ListViewDetails.Items.Clear()
        Call stamboom()
        WebRequest = Nothing
        WebResponse = Nothing
    End Sub
    Private Sub stamboom()
        ' Create a request using a URL that can receive a post. 
        'Doe eerst een GET om de cookie info op te halen

        Dim myUrl As String = "http://www.markiezenhof.nl/index.php?option=com_genealogie_zoeken&Itemid=36&sub=resultaat"

        WebRequest = CType(Net.HttpWebRequest.Create(myUrl), HttpWebRequest)
        WebRequest.ContentType = "application/x-www-form-urlencoded"
        WebRequest.Method = "GET"
        WebRequest.Referer = myUrl
        WebRequest.CookieContainer = New Net.CookieContainer()
        WebRequest.CookieContainer = cookies

        WebResponse = CType(WebRequest.GetResponse(), HttpWebResponse)
        cookies.Add(WebResponse.Cookies)

        WebRequest = CType(Net.HttpWebRequest.Create(myUrl), HttpWebRequest)
        WebRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30)"
        WebRequest.ContentType = "application/x-www-form-urlencoded"
        WebRequest.Method = "POST"
        WebRequest.Referer = myUrl
        WebRequest.CookieContainer = New Net.CookieContainer()
        WebRequest.CookieContainer = cookies

        Dim postData As String
        postData = "achternaam=" + UrlEncode(TextBoxAchternaam.Text) + "&zoekmethode_achternaam=" + UrlEncode("exact") + "&voornaam=" + TextBoxVoornaam.Text + "&zoekmethode_voornaam=" + UrlEncode("exact")
        postData += "&plaats=all&bron_a=1&bron_d=1&bron_g=1&bron_t=1&bron_h=1&bron_b=&bron_o=1&bron_br=1"
        postData += "&jaar_van=&jaar_tot&rol=all"

        WriteStream(postData)
        WebResponse = CType(WebRequest.GetResponse(), HttpWebResponse)

        cookies.Add(WebResponse.Cookies)
        Dim strHtml As String
        strHtml = ReadStream(WebResponse.GetResponseStream())
        If strHtml.IndexOf("geen resultaten gevonden") > -1 Then
            Exit Sub
        End If
        'Determin number of pages
        Dim intAantalPaginas As Integer = 1
        If strHtml.IndexOf("(pagina ") > -1 Then
            'Several pages
            Dim intStart As Integer = strHtml.IndexOf("(pagina ")
            Dim intEind As Integer = strHtml.IndexOf(")", intStart)
            Dim arrSplit() As String = strHtml.Substring(intStart, intEind - intStart).Split
            intAantalPaginas = CInt(arrSplit(3))
        End If
        pbMarkizaat.Maximum = intAantalPaginas
        pbMarkizaat.Minimum = 1
        boolOnePage = intAantalPaginas = 1
        For t = 1 To intAantalPaginas
            pbMarkizaat.Value = t
            'if t=1 first page already present
            If t > 1 Then
                'Eerst de volgende pagina ophalen
                Dim newUrl As String = "http://www.markiezenhof.nl/index.php?option=com_genealogie_zoeken&Itemid=36&sub=resultaat&cp=" + t.ToString + "&pp=8&of=an_waarde&od=a&ft=&ff="
                WebRequest = CType(Net.HttpWebRequest.Create(newUrl), HttpWebRequest)
                WebRequest.ContentType = "application/x-www-form-urlencoded"
                WebRequest.Method = "POST"
                WebRequest.Referer = newUrl
                WebRequest.CookieContainer = New Net.CookieContainer()
                WebRequest.CookieContainer = cookies

                WriteStream(postData)
                WebResponse = CType(WebRequest.GetResponse(), HttpWebResponse)
                strHtml = ReadStream(WebResponse.GetResponseStream())
            End If
            fillListView(strHtml)
        Next
        'WebResponse.Close()
        'WebResponse = Nothing
        'WebRequest = Nothing
    End Sub
    Private Sub fillListView(ByVal strHtml As String)
        Dim intStart As Integer = strHtml.IndexOf("<table class=" + Chr(34) + "dt_data_table")
        Dim intEind As Integer = strHtml.IndexOf("</table>", intStart)
        Dim strHtmlPart As String = strHtml.Substring(intStart, intEind - intStart + 8)
        strHtmlPart = strHtmlPart.Replace("&Itemid", "&amp;Itemid")
        strHtmlPart = strHtmlPart.Replace("&sub", "&amp;sub")
        strHtmlPart = strHtmlPart.Replace("&id", "&amp;id")
        strHtmlPart = strHtmlPart.Replace("&euml;", "ë")
        Dim xmlDoc As New XmlDocument
        xmlDoc.LoadXml(strHtmlPart)
        If boolOnePage Then
            pbMarkizaat.Minimum = 0 : pbMarkizaat.Maximum = xmlDoc.ChildNodes(0).ChildNodes.Count
        End If
        Dim intRow As Integer
        For Each xnl As XmlElement In xmlDoc.ChildNodes(0)
            If Not intRow = 0 Then
                Dim lvi As ListViewItem
                lvi = New ListViewItem(getID(xnl.InnerXml))
                Dim lviSub As New ListViewItem.ListViewSubItem

                For intTeller = 1 To 5
                    lvi.SubItems.Add(xnl.ChildNodes.Item(intTeller).InnerText)
                Next
                ListViewPeople.Items.Add(lvi)
            End If
            intRow += 1
            If boolOnePage Then
                pbMarkizaat.Value = intRow
            End If
        Next
    End Sub
    Private Function getID(ByVal strXML As String) As String
        Dim intFindStartId As Integer = strXML.IndexOf("&amp;id=")
        Dim intFindEindId As Integer = strXML.IndexOf(Chr(34), intFindStartId)
        Return strXML.Substring(intFindStartId + 8, intFindEindId - (intFindStartId + 8))
    End Function
    Private Function getHtmlPart(ByVal strHtml As String, ByVal strBegin As String, ByVal strFindStart As String, ByVal strFindEind As String) As String
        Dim intFindBegin As Integer = strHtml.IndexOf(strBegin)
        Dim intFindStart As Integer
        Dim intFindEind As Integer
        If intFindBegin > -1 Then
            intFindStart = strHtml.Substring(intFindBegin + 1).IndexOf(strFindStart)
            If intFindStart > 0 Then
                intFindEind = strHtml.Substring(intFindBegin + intFindStart).IndexOf(strFindEind)
                If intFindEind > 0 Then
                    Return strHtml.Substring(intFindBegin + 1 + intFindStart, (intFindEind + intFindBegin + intFindStart) - (intFindBegin + intFindStart) - 1)
                Else
                    Return "Error: De Website voor de verenigingen heeft niet meer de gebruikelijke layout.(Eind punt)"
                End If
            Else
                Return "Error: De Website voor de verenigingen heeft niet meer de gebruikelijke layout.(Start punt)"
            End If
        Else
            Return "Error: De Website voor de verenigingen heeft niet meer de gebruikelijke layout.(Begin punt)"
        End If
        Return strHtml
    End Function

    Private Sub WriteStream(ByVal PostData As String)
        WebRequest.ContentLength = PostData.Length
        WebRequest.Timeout = 10000
        WebRequest.KeepAlive = False
        Using RequestWriter As New IO.StreamWriter(WebRequest.GetRequestStream)
            RequestWriter.Write(PostData)
            RequestWriter.Flush()
            RequestWriter.Close()
        End Using
    End Sub
    Private Function ReadStream(ByVal Stream As IO.Stream) As String
        Dim ResponseReader As IO.StreamReader
        Dim ResponseData As String
        ResponseReader = New IO.StreamReader(Stream)
        ResponseData = ResponseReader.ReadToEnd()
        ResponseReader.Close()
        Return ResponseData
    End Function

    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        End
    End Sub
End Class

I tried a numerous things but now I'm running out of ideas .. any body any thoughts?

[cicatrix]

Try disposing of the old webrequest and webresponse before creating new ones.

[Santana23]

Hello Cicatrix,

Thanx for responding I have set the WebRequest en WebResponse both to nothing but to no effect.

[cicatrix]

No, disposing means invoking .Dispose member, setting them to nothing does not release the resources unlike it was in vb6.

[Santana23]

Yes I understand. But as there is no dispose method I thought I might use the set to nothing. I do close the WebResponse object but that's as far as I can come. I could rewrite my code and use the Using construct.

[Edit]Have to correct myself.. I did find a way to dispose the WebResponse by:

CType(WebResponse, IDisposable).Dispose()

But that didn't help either. So as this didn't help and disposing the WebRequest by: CType(WebRequest, IDisposable).Dispose() gives an Unable to cast object of type 'System.Net.HttpWebRequest' to type 'System.IDisposable'. I wonder if rewriting my code with a Using construct is going to help..

[Santana23]

Hi All,

To everyone who likes to know .. got the problem solved. Don't ask me how.. but I'll tell you anyway . Just added a simple GC.Collect() at the end of the Query part. Did the trick. As I understand you cannot dispose the WebRequest 'cause there is nothing to dispose. You should however close your streams and readers and if necessary .. do a Garbage collect.

Have fun.