HttpWebRequest and GZip Http Responses

[gep13]

Hey,

In a recent forum post, the question was asked how to make an HttpWebRequest when the site in question uses Compression? For instance, digg.com.

If you attempt to use a normal HttpWebRequest, you will get a TimeOut on the request. Even if you extend the standard length of the time out, you still won't get a response.

A solution to this problem is provided here:

http://www.west-wind.com/WebLog/posts/102969.aspx

But I thought I would create a VB.Net sample of the same operation.

    Public Function GetUrl(ByVal Url As String, ByVal PostData As String, ByVal GZip As Boolean) As String
        Dim Http As HttpWebRequest = WebRequest.Create(Url)

        If GZip = True Then
            Http.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate")
        End If

        If Not String.IsNullOrEmpty(PostData) Then
            Http.Method = "POST"
            Dim lbPostBuffer As Byte() = Encoding.Default.GetBytes(PostData)

            Http.ContentLength = lbPostBuffer.Length

            Using PostStream As Stream = Http.GetRequestStream()
                PostStream.Write(lbPostBuffer, 0, lbPostBuffer.Length)
            End Using
        End If

        Using WebResponse As HttpWebResponse = Http.GetResponse()

            Dim responseStream As Stream = WebResponse.GetResponseStream()

            If (WebResponse.ContentEncoding.ToLower().Contains("gzip")) Then
                responseStream = New GZipStream(responseStream, CompressionMode.Decompress)
            ElseIf (WebResponse.ContentEncoding.ToLower().Contains("deflate")) Then
                responseStream = New DeflateStream(responseStream, CompressionMode.Decompress)
            End If

            Dim reader As StreamReader = New StreamReader(responseStream, Encoding.Default)

            Dim html As String = reader.ReadToEnd()

            responseStream.Close()

            Return html
        End Using

    End Function

Attached is a working sample of the above.

To use it, simply type in the URL that you would like to request, click the checkbox to indicate if you want to compress or not, and then hit go.

For example, type http://digg.com in the textbox, check the checkbox and then hit go. The response is almost immediate. If however, you type http://digg.com, don't check the checkbox and then hit go, a exception will be raised (which of course isn't handled, as this is just a simple example )

Hope this helps!!

Gary