dcsimg
Results 1 to 6 of 6

Thread: [RESOLVED] WebRequest gives Time out second time

  1. #1

    Thread Starter
    New Member
    Join Date
    Apr 2010
    Posts
    6

    Resolved [RESOLVED] WebRequest gives Time out second time

    Hi All,

    I have a Windows program (sort of a crawler) that queries an internet website to collect data from. I use WebRequest and Webresponse. The first time I run the program everything works fine. However the second time I get a Time out exception on the WebRequest.GetRequestStream method.
    Here is what I basically do:
    1. Do a WebRequest with a GET method (to get Cookie info)
    2. From the response I determin the number of query result pages I get.
    3. I loop through the result pages and collect the rest of the data
    4. Display result in a listview

    This works fine. However when I change my search parameters and search again I immediately get a timeout on the WebRequest.GetRequestStream method. Now some other strange thing happens.. when I have Fiddler open to monitor the traffic, my program works fine all the time! Somehow Fiddler has some influence on it. I also tried to do some browsing to see if that had any impact but it didn't.

    Here is the code:

    Code:
    Imports System.Net
    Imports System.Text
    Imports System.IO
    Imports System.Web.HttpUtility
    Imports System.Xml
    
    
    Public Class MainForm
        Private WebRequest As Net.HttpWebRequest
        Private WebResponse As Net.HttpWebResponse
        Private cookies As Net.CookieContainer
        Private boolOnePage As Boolean = True
    
        Private Sub ButtonGo_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles ButtonGo.Click
            cookies = New Net.CookieContainer
    
            ListViewPeople.Items.Clear()
            ListViewDetails.Items.Clear()
            Call stamboom()
            WebRequest = Nothing
            WebResponse = Nothing
        End Sub
        Private Sub stamboom()
            ' Create a request using a URL that can receive a post. 
            'Doe eerst een GET om de cookie info op te halen
    
            Dim myUrl As String = "http://www.markiezenhof.nl/index.php?option=com_genealogie_zoeken&Itemid=36&sub=resultaat"
    
            WebRequest = CType(Net.HttpWebRequest.Create(myUrl), HttpWebRequest)
            WebRequest.ContentType = "application/x-www-form-urlencoded"
            WebRequest.Method = "GET"
            WebRequest.Referer = myUrl
            WebRequest.CookieContainer = New Net.CookieContainer()
            WebRequest.CookieContainer = cookies
    
            WebResponse = CType(WebRequest.GetResponse(), HttpWebResponse)
            cookies.Add(WebResponse.Cookies)
    
            WebRequest = CType(Net.HttpWebRequest.Create(myUrl), HttpWebRequest)
            WebRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30)"
            WebRequest.ContentType = "application/x-www-form-urlencoded"
            WebRequest.Method = "POST"
            WebRequest.Referer = myUrl
            WebRequest.CookieContainer = New Net.CookieContainer()
            WebRequest.CookieContainer = cookies
    
            Dim postData As String
            postData = "achternaam=" + UrlEncode(TextBoxAchternaam.Text) + "&zoekmethode_achternaam=" + UrlEncode("exact") + "&voornaam=" + TextBoxVoornaam.Text + "&zoekmethode_voornaam=" + UrlEncode("exact")
            postData += "&plaats=all&bron_a=1&bron_d=1&bron_g=1&bron_t=1&bron_h=1&bron_b=&bron_o=1&bron_br=1"
            postData += "&jaar_van=&jaar_tot&rol=all"
    
            WriteStream(postData)
            WebResponse = CType(WebRequest.GetResponse(), HttpWebResponse)
    
            cookies.Add(WebResponse.Cookies)
            Dim strHtml As String
            strHtml = ReadStream(WebResponse.GetResponseStream())
            If strHtml.IndexOf("geen resultaten gevonden") > -1 Then
                Exit Sub
            End If
            'Determin number of pages
            Dim intAantalPaginas As Integer = 1
            If strHtml.IndexOf("(pagina ") > -1 Then
                'Several pages
                Dim intStart As Integer = strHtml.IndexOf("(pagina ")
                Dim intEind As Integer = strHtml.IndexOf(")", intStart)
                Dim arrSplit() As String = strHtml.Substring(intStart, intEind - intStart).Split
                intAantalPaginas = CInt(arrSplit(3))
            End If
            pbMarkizaat.Maximum = intAantalPaginas
            pbMarkizaat.Minimum = 1
            boolOnePage = intAantalPaginas = 1
            For t = 1 To intAantalPaginas
                pbMarkizaat.Value = t
                'if t=1 first page already present
                If t > 1 Then
                    'Eerst de volgende pagina ophalen
                    Dim newUrl As String = "http://www.markiezenhof.nl/index.php?option=com_genealogie_zoeken&Itemid=36&sub=resultaat&cp=" + t.ToString + "&pp=8&of=an_waarde&od=a&ft=&ff="
                    WebRequest = CType(Net.HttpWebRequest.Create(newUrl), HttpWebRequest)
                    WebRequest.ContentType = "application/x-www-form-urlencoded"
                    WebRequest.Method = "POST"
                    WebRequest.Referer = newUrl
                    WebRequest.CookieContainer = New Net.CookieContainer()
                    WebRequest.CookieContainer = cookies
    
                    WriteStream(postData)
                    WebResponse = CType(WebRequest.GetResponse(), HttpWebResponse)
                    strHtml = ReadStream(WebResponse.GetResponseStream())
                End If
                fillListView(strHtml)
            Next
            'WebResponse.Close()
            'WebResponse = Nothing
            'WebRequest = Nothing
        End Sub
        Private Sub fillListView(ByVal strHtml As String)
            Dim intStart As Integer = strHtml.IndexOf("<table class=" + Chr(34) + "dt_data_table")
            Dim intEind As Integer = strHtml.IndexOf("</table>", intStart)
            Dim strHtmlPart As String = strHtml.Substring(intStart, intEind - intStart + 8)
            strHtmlPart = strHtmlPart.Replace("&Itemid", "&amp;Itemid")
            strHtmlPart = strHtmlPart.Replace("&sub", "&amp;sub")
            strHtmlPart = strHtmlPart.Replace("&id", "&amp;id")
            strHtmlPart = strHtmlPart.Replace("&euml;", "")
            Dim xmlDoc As New XmlDocument
            xmlDoc.LoadXml(strHtmlPart)
            If boolOnePage Then
                pbMarkizaat.Minimum = 0 : pbMarkizaat.Maximum = xmlDoc.ChildNodes(0).ChildNodes.Count
            End If
            Dim intRow As Integer
            For Each xnl As XmlElement In xmlDoc.ChildNodes(0)
                If Not intRow = 0 Then
                    Dim lvi As ListViewItem
                    lvi = New ListViewItem(getID(xnl.InnerXml))
                    Dim lviSub As New ListViewItem.ListViewSubItem
    
                    For intTeller = 1 To 5
                        lvi.SubItems.Add(xnl.ChildNodes.Item(intTeller).InnerText)
                    Next
                    ListViewPeople.Items.Add(lvi)
                End If
                intRow += 1
                If boolOnePage Then
                    pbMarkizaat.Value = intRow
                End If
            Next
        End Sub
        Private Function getID(ByVal strXML As String) As String
            Dim intFindStartId As Integer = strXML.IndexOf("&amp;id=")
            Dim intFindEindId As Integer = strXML.IndexOf(Chr(34), intFindStartId)
            Return strXML.Substring(intFindStartId + 8, intFindEindId - (intFindStartId + 8))
        End Function
        Private Function getHtmlPart(ByVal strHtml As String, ByVal strBegin As String, ByVal strFindStart As String, ByVal strFindEind As String) As String
            Dim intFindBegin As Integer = strHtml.IndexOf(strBegin)
            Dim intFindStart As Integer
            Dim intFindEind As Integer
            If intFindBegin > -1 Then
                intFindStart = strHtml.Substring(intFindBegin + 1).IndexOf(strFindStart)
                If intFindStart > 0 Then
                    intFindEind = strHtml.Substring(intFindBegin + intFindStart).IndexOf(strFindEind)
                    If intFindEind > 0 Then
                        Return strHtml.Substring(intFindBegin + 1 + intFindStart, (intFindEind + intFindBegin + intFindStart) - (intFindBegin + intFindStart) - 1)
                    Else
                        Return "Error: De Website voor de verenigingen heeft niet meer de gebruikelijke layout.(Eind punt)"
                    End If
                Else
                    Return "Error: De Website voor de verenigingen heeft niet meer de gebruikelijke layout.(Start punt)"
                End If
            Else
                Return "Error: De Website voor de verenigingen heeft niet meer de gebruikelijke layout.(Begin punt)"
            End If
            Return strHtml
        End Function
    
        Private Sub WriteStream(ByVal PostData As String)
            WebRequest.ContentLength = PostData.Length
            WebRequest.Timeout = 10000
            WebRequest.KeepAlive = False
            Using RequestWriter As New IO.StreamWriter(WebRequest.GetRequestStream)
                RequestWriter.Write(PostData)
                RequestWriter.Flush()
                RequestWriter.Close()
            End Using
        End Sub
        Private Function ReadStream(ByVal Stream As IO.Stream) As String
            Dim ResponseReader As IO.StreamReader
            Dim ResponseData As String
            ResponseReader = New IO.StreamReader(Stream)
            ResponseData = ResponseReader.ReadToEnd()
            ResponseReader.Close()
            Return ResponseData
        End Function
    
        Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
            End
        End Sub
    End Class
    I tried a numerous things but now I'm running out of ideas .. any body any thoughts?

  2. #2

  3. #3

    Thread Starter
    New Member
    Join Date
    Apr 2010
    Posts
    6

    Re: WebRequest gives Time out second time

    Hello Cicatrix,

    Thanx for responding I have set the WebRequest en WebResponse both to nothing but to no effect.

  4. #4
    PowerPoster cicatrix's Avatar
    Join Date
    Dec 2009
    Location
    Moscow, Russia
    Posts
    3,654

    Re: WebRequest gives Time out second time

    No, disposing means invoking .Dispose member, setting them to nothing does not release the resources unlike it was in vb6.

  5. #5

    Thread Starter
    New Member
    Join Date
    Apr 2010
    Posts
    6

    Re: WebRequest gives Time out second time

    Yes I understand. But as there is no dispose method I thought I might use the set to nothing. I do close the WebResponse object but that's as far as I can come. I could rewrite my code and use the Using construct.

    [Edit]Have to correct myself.. I did find a way to dispose the WebResponse by:

    CType(WebResponse, IDisposable).Dispose()

    But that didn't help either. So as this didn't help and disposing the WebRequest by: CType(WebRequest, IDisposable).Dispose() gives an Unable to cast object of type 'System.Net.HttpWebRequest' to type 'System.IDisposable'. I wonder if rewriting my code with a Using construct is going to help..
    Last edited by Santana23; Apr 6th, 2010 at 02:29 AM. Reason: new insight ;-)

  6. #6

    Thread Starter
    New Member
    Join Date
    Apr 2010
    Posts
    6

    Re: WebRequest gives Time out second time

    Hi All,

    To everyone who likes to know .. got the problem solved. Don't ask me how.. but I'll tell you anyway . Just added a simple GC.Collect() at the end of the Query part. Did the trick. As I understand you cannot dispose the WebRequest 'cause there is nothing to dispose. You should however close your streams and readers and if necessary .. do a Garbage collect.

    Have fun.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Featured


Click Here to Expand Forum to Full Width