[RESOLVED] WebRequest gives Time out second time

Hi All,

I have a Windows program (sort of a crawler) that queries an internet website to collect data from. I use WebRequest and Webresponse. The first time I run the program everything works fine. However the second time I get a Time out exception on the WebRequest.GetRequestStream method.
Here is what I basically do:
1. Do a WebRequest with a GET method (to get Cookie info)
2. From the response I determin the number of query result pages I get.
3. I loop through the result pages and collect the rest of the data
4. Display result in a listview

This works fine. However when I change my search parameters and search again I immediately get a timeout on the WebRequest.GetRequestStream method. Now some other strange thing happens.. when I have Fiddler open to monitor the traffic, my program works fine all the time! Somehow Fiddler has some influence on it. I also tried to do some browsing to see if that had any impact but it didn't.

Here is the code:

Code:

---

Imports System.Net
Imports System.Text
Imports System.IO
Imports System.Web.HttpUtility
Imports System.Xml


Public Class MainForm
    Private WebRequest As Net.HttpWebRequest
    Private WebResponse As Net.HttpWebResponse
    Private cookies As Net.CookieContainer
    Private boolOnePage As Boolean = True

    Private Sub ButtonGo_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles ButtonGo.Click
        cookies = New Net.CookieContainer

        ListViewPeople.Items.Clear()
        ListViewDetails.Items.Clear()
        Call stamboom()
        WebRequest = Nothing
        WebResponse = Nothing
    End Sub
    Private Sub stamboom()
        ' Create a request using a URL that can receive a post.
        'Doe eerst een GET om de cookie info op te halen

        Dim myUrl As String = "http://www.markiezenhof.nl/index.php?option=com_genealogie_zoeken&Itemid=36&sub=resultaat"

        WebRequest = CType(Net.HttpWebRequest.Create(myUrl), HttpWebRequest)
        WebRequest.ContentType = "application/x-www-form-urlencoded"
        WebRequest.Method = "GET"
        WebRequest.Referer = myUrl
        WebRequest.CookieContainer = New Net.CookieContainer()
        WebRequest.CookieContainer = cookies

        WebResponse = CType(WebRequest.GetResponse(), HttpWebResponse)
        cookies.Add(WebResponse.Cookies)

        WebRequest = CType(Net.HttpWebRequest.Create(myUrl), HttpWebRequest)
        WebRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30)"
        WebRequest.ContentType = "application/x-www-form-urlencoded"
        WebRequest.Method = "POST"
        WebRequest.Referer = myUrl
        WebRequest.CookieContainer = New Net.CookieContainer()
        WebRequest.CookieContainer = cookies

        Dim postData As String
        postData = "achternaam=" + UrlEncode(TextBoxAchternaam.Text) + "&zoekmethode_achternaam=" + UrlEncode("exact") + "&voornaam=" + TextBoxVoornaam.Text + "&zoekmethode_voornaam=" + UrlEncode("exact")
        postData += "&plaats=all&bron_a=1&bron_d=1&bron_g=1&bron_t=1&bron_h=1&bron_b=&bron_o=1&bron_br=1"
        postData += "&jaar_van=&jaar_tot&rol=all"

        WriteStream(postData)
        WebResponse = CType(WebRequest.GetResponse(), HttpWebResponse)

        cookies.Add(WebResponse.Cookies)
        Dim strHtml As String
        strHtml = ReadStream(WebResponse.GetResponseStream())
        If strHtml.IndexOf("geen resultaten gevonden") > -1 Then
            Exit Sub
        End If
        'Determin number of pages
        Dim intAantalPaginas As Integer = 1
        If strHtml.IndexOf("(pagina ") > -1 Then
            'Several pages
            Dim intStart As Integer = strHtml.IndexOf("(pagina ")
            Dim intEind As Integer = strHtml.IndexOf(")", intStart)
            Dim arrSplit() As String = strHtml.Substring(intStart, intEind - intStart).Split
            intAantalPaginas = CInt(arrSplit(3))
        End If
        pbMarkizaat.Maximum = intAantalPaginas
        pbMarkizaat.Minimum = 1
        boolOnePage = intAantalPaginas = 1
        For t = 1 To intAantalPaginas
            pbMarkizaat.Value = t
            'if t=1 first page already present
            If t > 1 Then
                'Eerst de volgende pagina ophalen
                Dim newUrl As String = "http://www.markiezenhof.nl/index.php?option=com_genealogie_zoeken&Itemid=36&sub=resultaat&cp=" + t.ToString + "&pp=8&of=an_waarde&od=a&ft=&ff="
                WebRequest = CType(Net.HttpWebRequest.Create(newUrl), HttpWebRequest)
                WebRequest.ContentType = "application/x-www-form-urlencoded"
                WebRequest.Method = "POST"
                WebRequest.Referer = newUrl
                WebRequest.CookieContainer = New Net.CookieContainer()
                WebRequest.CookieContainer = cookies

                WriteStream(postData)
                WebResponse = CType(WebRequest.GetResponse(), HttpWebResponse)
                strHtml = ReadStream(WebResponse.GetResponseStream())
            End If
            fillListView(strHtml)
        Next
        'WebResponse.Close()
        'WebResponse = Nothing
        'WebRequest = Nothing
    End Sub
    Private Sub fillListView(ByVal strHtml As String)
        Dim intStart As Integer = strHtml.IndexOf("<table class=" + Chr(34) + "dt_data_table")
        Dim intEind As Integer = strHtml.IndexOf("</table>", intStart)
        Dim strHtmlPart As String = strHtml.Substring(intStart, intEind - intStart + 8)
        strHtmlPart = strHtmlPart.Replace("&Itemid", "&amp;Itemid")
        strHtmlPart = strHtmlPart.Replace("&sub", "&amp;sub")
        strHtmlPart = strHtmlPart.Replace("&id", "&amp;id")
        strHtmlPart = strHtmlPart.Replace("&euml;", "ë")
        Dim xmlDoc As New XmlDocument
        xmlDoc.LoadXml(strHtmlPart)
        If boolOnePage Then
            pbMarkizaat.Minimum = 0 : pbMarkizaat.Maximum = xmlDoc.ChildNodes(0).ChildNodes.Count
        End If
        Dim intRow As Integer
        For Each xnl As XmlElement In xmlDoc.ChildNodes(0)
            If Not intRow = 0 Then
                Dim lvi As ListViewItem
                lvi = New ListViewItem(getID(xnl.InnerXml))
                Dim lviSub As New ListViewItem.ListViewSubItem

                For intTeller = 1 To 5
                    lvi.SubItems.Add(xnl.ChildNodes.Item(intTeller).InnerText)
                Next
                ListViewPeople.Items.Add(lvi)
            End If
            intRow += 1
            If boolOnePage Then
                pbMarkizaat.Value = intRow
            End If
        Next
    End Sub
    Private Function getID(ByVal strXML As String) As String
        Dim intFindStartId As Integer = strXML.IndexOf("&amp;id=")
        Dim intFindEindId As Integer = strXML.IndexOf(Chr(34), intFindStartId)
        Return strXML.Substring(intFindStartId + 8, intFindEindId - (intFindStartId + 8))
    End Function
    Private Function getHtmlPart(ByVal strHtml As String, ByVal strBegin As String, ByVal strFindStart As String, ByVal strFindEind As String) As String
        Dim intFindBegin As Integer = strHtml.IndexOf(strBegin)
        Dim intFindStart As Integer
        Dim intFindEind As Integer
        If intFindBegin > -1 Then
            intFindStart = strHtml.Substring(intFindBegin + 1).IndexOf(strFindStart)
            If intFindStart > 0 Then
                intFindEind = strHtml.Substring(intFindBegin + intFindStart).IndexOf(strFindEind)
                If intFindEind > 0 Then
                    Return strHtml.Substring(intFindBegin + 1 + intFindStart, (intFindEind + intFindBegin + intFindStart) - (intFindBegin + intFindStart) - 1)
                Else
                    Return "Error: De Website voor de verenigingen heeft niet meer de gebruikelijke layout.(Eind punt)"
                End If
            Else
                Return "Error: De Website voor de verenigingen heeft niet meer de gebruikelijke layout.(Start punt)"
            End If
        Else
            Return "Error: De Website voor de verenigingen heeft niet meer de gebruikelijke layout.(Begin punt)"
        End If
        Return strHtml
    End Function

    Private Sub WriteStream(ByVal PostData As String)
        WebRequest.ContentLength = PostData.Length
        WebRequest.Timeout = 10000
        WebRequest.KeepAlive = False
        Using RequestWriter As New IO.StreamWriter(WebRequest.GetRequestStream)
            RequestWriter.Write(PostData)
            RequestWriter.Flush()
            RequestWriter.Close()
        End Using
    End Sub
    Private Function ReadStream(ByVal Stream As IO.Stream) As String
        Dim ResponseReader As IO.StreamReader
        Dim ResponseData As String
        ResponseReader = New IO.StreamReader(Stream)
        ResponseData = ResponseReader.ReadToEnd()
        ResponseReader.Close()
        Return ResponseData
    End Function

    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        End
    End Sub
End Class

---

I tried a numerous things but now I'm running out of ideas .. any body any thoughts?

Try disposing of the old webrequest and webresponse before creating new ones.

Hello Cicatrix,

Thanx for responding I have set the WebRequest en WebResponse both to nothing but to no effect.

No, disposing means invoking .Dispose member, setting them to nothing does not release the resources unlike it was in vb6.

Yes I understand. But as there is no dispose method I thought I might use the set to nothing. I do close the WebResponse object but that's as far as I can come. I could rewrite my code and use the Using construct.

[Edit]Have to correct myself.. I did find a way to dispose the WebResponse by:

CType(WebResponse, IDisposable).Dispose()

But that didn't help either. So as this didn't help and disposing the WebRequest by: CType(WebRequest, IDisposable).Dispose() gives an Unable to cast object of type 'System.Net.HttpWebRequest' to type 'System.IDisposable'. I wonder if rewriting my code with a Using construct is going to help..

Hi All,

To everyone who likes to know .. got the problem solved. Don't ask me how.. but I'll tell you anyway :p. Just added a simple GC.Collect() at the end of the Query part. Did the trick. As I understand you cannot dispose the WebRequest 'cause there is nothing to dispose. You should however close your streams and readers and if necessary .. do a Garbage collect.

Have fun.