Results 1 to 12 of 12

Thread: HttpWebRequest and GZip Http Responses

  1. #1

    Thread Starter
    ASP.NET Moderator gep13's Avatar
    Join Date
    Nov 2004
    Location
    The Granite City
    Posts
    21,963

    HttpWebRequest and GZip Http Responses

    Hey,

    In a recent forum post, the question was asked how to make an HttpWebRequest when the site in question uses Compression? For instance, digg.com.

    If you attempt to use a normal HttpWebRequest, you will get a TimeOut on the request. Even if you extend the standard length of the time out, you still won't get a response.

    A solution to this problem is provided here:

    http://www.west-wind.com/WebLog/posts/102969.aspx

    But I thought I would create a VB.Net sample of the same operation.

    Code:
        Public Function GetUrl(ByVal Url As String, ByVal PostData As String, ByVal GZip As Boolean) As String
            Dim Http As HttpWebRequest = WebRequest.Create(Url)
    
            If GZip = True Then
                Http.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate")
            End If
    
            If Not String.IsNullOrEmpty(PostData) Then
                Http.Method = "POST"
                Dim lbPostBuffer As Byte() = Encoding.Default.GetBytes(PostData)
    
                Http.ContentLength = lbPostBuffer.Length
    
                Using PostStream As Stream = Http.GetRequestStream()
                    PostStream.Write(lbPostBuffer, 0, lbPostBuffer.Length)
                End Using
            End If
    
            Using WebResponse As HttpWebResponse = Http.GetResponse()
    
                Dim responseStream As Stream = WebResponse.GetResponseStream()
    
                If (WebResponse.ContentEncoding.ToLower().Contains("gzip")) Then
                    responseStream = New GZipStream(responseStream, CompressionMode.Decompress)
                ElseIf (WebResponse.ContentEncoding.ToLower().Contains("deflate")) Then
                    responseStream = New DeflateStream(responseStream, CompressionMode.Decompress)
                End If
    
                Dim reader As StreamReader = New StreamReader(responseStream, Encoding.Default)
    
                Dim html As String = reader.ReadToEnd()
    
                responseStream.Close()
    
                Return html
            End Using
    
        End Function
    Attached is a working sample of the above.

    To use it, simply type in the URL that you would like to request, click the checkbox to indicate if you want to compress or not, and then hit go.

    For example, type http://digg.com in the textbox, check the checkbox and then hit go. The response is almost immediate. If however, you type http://digg.com, don't check the checkbox and then hit go, a exception will be raised (which of course isn't handled, as this is just a simple example )

    Hope this helps!!

    Gary
    Attached Files Attached Files

  2. #2
    Frenzied Member
    Join Date
    Aug 2000
    Posts
    1,091

    Re: HttpWebRequest and GZip Http Responses

    Nice!

    Visual Studio 2010

  3. #3

    Thread Starter
    ASP.NET Moderator gep13's Avatar
    Join Date
    Nov 2004
    Location
    The Granite City
    Posts
    21,963

    Re: HttpWebRequest and GZip Http Responses

    Thanks!

    Hope it helped!!

  4. #4
    Fanatic Member coolcurrent4u's Avatar
    Join Date
    Apr 2008
    Location
    *****
    Posts
    993

    Re: HttpWebRequest and GZip Http Responses

    hello gep

    this line
    If (WebResponse.ContentEncoding.ToLower().Contains("gzip"))
    would give an error according to this

    In GZipEncodePage, your test for gzip is to examine the Accept-Encoding header and see if it contains "gzip".

    According to RFIC 2616 http://www.w3.org/Protocols/rfc2616/...4.html#sec14.3

    1. Accept-Encoding header could contain "gzip;q=0" which would mean that gzip is NOT accepted.

    2. It could contain "*;q=1.0", which would mean gzip IS accepted.

    3. There may be no header, in which case the standard is to accept any encoding.

    Your test would produce incorrect results in each of these three cases.
    Programming is all about good logic. Spend more time here


    (Generate pronounceable password) (Generate random number c#) (Filter array with another array)

  5. #5

    Thread Starter
    ASP.NET Moderator gep13's Avatar
    Join Date
    Nov 2004
    Location
    The Granite City
    Posts
    21,963

    Re: HttpWebRequest and GZip Http Responses

    Hello,

    The posting that I made here is simply a VB.Net Port of the original post which you can find here:

    http://www.west-wind.com/weblog/post...Http-Responses

    I didn't spend any time looking into the validity of the check that was being made.

    Perhaps I should spend some time on it, I will see what I can do.

    Gary

  6. #6
    New Member
    Join Date
    May 2012
    Posts
    5

    Re: HttpWebRequest and GZip Http Responses

    Hi gep 13
    Its not working in my case, giving error " The remote server returned an error: (500) Internal Server Error.
    Please help me. My code is

    Public Function PostXml(ByVal Url As String, ByVal PostData As String) As String
    Try
    Dim Http As HttpWebRequest = WebRequest.Create(Url)
    Http.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate")
    If Not String.IsNullOrEmpty(PostData) Then
    Http.Method = "POST"
    Dim lbPostBuffer As Byte() = Encoding.Default.GetBytes(PostData)
    Http.ContentLength = lbPostBuffer.Length
    Using PostStream As Stream = Http.GetRequestStream()
    PostStream.Write(lbPostBuffer, 0, lbPostBuffer.Length)
    End Using
    End If
    Using WebResponse As HttpWebResponse = Http.GetResponse()
    Dim responseStream As Stream = WebResponse.GetResponseStream()
    If (WebResponse.ContentEncoding.ToLower().Contains("gzip")) Then
    responseStream = New GZipStream(responseStream, CompressionMode.Decompress)
    ElseIf (WebResponse.ContentEncoding.ToLower().Contains("deflate")) Then
    responseStream = New DeflateStream(responseStream, CompressionMode.Decompress)
    End If
    Dim reader As StreamReader = New StreamReader(responseStream, Encoding.Default)
    Dim html As String = reader.ReadToEnd()
    responseStream.Close()
    Return html
    End Using

    Catch ex As Exception

    End Try
    End Function

  7. #7
    New Member
    Join Date
    May 2012
    Posts
    5

    Re: HttpWebRequest and GZip Http Responses

    I copy and past your code
    But it is throwing Error The remote server returned an error: (500) Internal Server Error.

    Please Help me. In xml http request response it is getting 150 to 200 Second.

  8. #8
    Frenzied Member MattP's Avatar
    Join Date
    Dec 2008
    Location
    WY
    Posts
    1,227

    Re: HttpWebRequest and GZip Http Responses

    Ran into this previously when asked about www.yellowpages.com.au. I posted 2 solutions to the problem here: http://www.vbforums.com/showpost.php...52&postcount=3
    This pattern in common to all great programmers I know: they're not experts in something as much as experts in becoming experts in something.

    The best programming advice I ever got was to spend my entire career becoming educable. And I suggest you do the same.

  9. #9

    Thread Starter
    ASP.NET Moderator gep13's Avatar
    Join Date
    Nov 2004
    Location
    The Granite City
    Posts
    21,963

    Re: HttpWebRequest and GZip Http Responses

    Hello vvsingh,

    Did the above work for you, or are you still having problems?

    Gary

  10. #10
    New Member
    Join Date
    May 2012
    Posts
    5

    Re: HttpWebRequest and GZip Http Responses

    sorry for delay reply.

    My problem not solved.

  11. #11
    New Member
    Join Date
    May 2012
    Posts
    5

    Re: HttpWebRequest and GZip Http Responses

    without gzip method below code is working but it taking 150 to 200 second to read response data.

    Protected Function PostXml(ByVal url As String, ByVal xml As String) As String
    Dim bytes As Byte() = Encoding.UTF8.GetBytes(xml)
    Dim strResult As String = String.Empty
    Dim request As HttpWebRequest = DirectCast(WebRequest.Create(url), HttpWebRequest)
    request.Method = "POST"
    request.ContentLength = bytes.Length
    request.ContentType = "text/xml" '; charset=utf-8

    Using requestStream As Stream = request.GetRequestStream()
    requestStream.Write(bytes, 0, bytes.Length)
    End Using
    Using response As HttpWebResponse = DirectCast(request.GetResponse(), HttpWebResponse)
    If response.StatusCode <> HttpStatusCode.OK Then
    Dim message As String = [String].Format("POST failed. Received HTTP {0}", response.StatusCode)
    Throw New ApplicationException(message)
    Else
    Dim reader As StreamReader = Nothing
    Dim responseStream As Stream = response.GetResponseStream()
    reader = New StreamReader(responseStream, Encoding.Default)
    strResult = reader.ReadToEnd() ' Here taking time
    response.Close()
    responseStream.Close()
    reader.Close()
    End If
    End Using
    End Function

  12. #12
    New Member
    Join Date
    May 2012
    Posts
    5

    Re: HttpWebRequest and GZip Http Responses

    Thank you very much
    My problem solved
    now i am getting result within 30 Second

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width