1 Attachment(s)
HttpWebRequest and GZip Http Responses
Hey,
In a recent forum post, the question was asked how to make an HttpWebRequest when the site in question uses Compression? For instance, digg.com.
If you attempt to use a normal HttpWebRequest, you will get a TimeOut on the request. Even if you extend the standard length of the time out, you still won't get a response.
A solution to this problem is provided here:
http://www.west-wind.com/WebLog/posts/102969.aspx
But I thought I would create a VB.Net sample of the same operation.
Code:
Public Function GetUrl(ByVal Url As String, ByVal PostData As String, ByVal GZip As Boolean) As String
Dim Http As HttpWebRequest = WebRequest.Create(Url)
If GZip = True Then
Http.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate")
End If
If Not String.IsNullOrEmpty(PostData) Then
Http.Method = "POST"
Dim lbPostBuffer As Byte() = Encoding.Default.GetBytes(PostData)
Http.ContentLength = lbPostBuffer.Length
Using PostStream As Stream = Http.GetRequestStream()
PostStream.Write(lbPostBuffer, 0, lbPostBuffer.Length)
End Using
End If
Using WebResponse As HttpWebResponse = Http.GetResponse()
Dim responseStream As Stream = WebResponse.GetResponseStream()
If (WebResponse.ContentEncoding.ToLower().Contains("gzip")) Then
responseStream = New GZipStream(responseStream, CompressionMode.Decompress)
ElseIf (WebResponse.ContentEncoding.ToLower().Contains("deflate")) Then
responseStream = New DeflateStream(responseStream, CompressionMode.Decompress)
End If
Dim reader As StreamReader = New StreamReader(responseStream, Encoding.Default)
Dim html As String = reader.ReadToEnd()
responseStream.Close()
Return html
End Using
End Function
Attached is a working sample of the above.
To use it, simply type in the URL that you would like to request, click the checkbox to indicate if you want to compress or not, and then hit go.
For example, type http://digg.com in the textbox, check the checkbox and then hit go. The response is almost immediate. If however, you type http://digg.com, don't check the checkbox and then hit go, a exception will be raised (which of course isn't handled, as this is just a simple example :))
Hope this helps!!
Gary
Re: HttpWebRequest and GZip Http Responses
Re: HttpWebRequest and GZip Http Responses
Thanks! :)
Hope it helped!!
Re: HttpWebRequest and GZip Http Responses
hello gep
this line
Quote:
If (WebResponse.ContentEncoding.ToLower().Contains("gzip"))
would give an error according to this
In GZipEncodePage, your test for gzip is to examine the Accept-Encoding header and see if it contains "gzip".
Quote:
According to RFIC 2616
http://www.w3.org/Protocols/rfc2616/...4.html#sec14.3
1. Accept-Encoding header could contain "gzip;q=0" which would mean that gzip is NOT accepted.
2. It could contain "*;q=1.0", which would mean gzip IS accepted.
3. There may be no header, in which case the standard is to accept any encoding.
Your test would produce incorrect results in each of these three cases.
Re: HttpWebRequest and GZip Http Responses
Hello,
The posting that I made here is simply a VB.Net Port of the original post which you can find here:
http://www.west-wind.com/weblog/post...Http-Responses
I didn't spend any time looking into the validity of the check that was being made.
Perhaps I should spend some time on it, I will see what I can do.
Gary
Re: HttpWebRequest and GZip Http Responses
Hi gep 13
Its not working in my case, giving error " The remote server returned an error: (500) Internal Server Error.
Please help me. My code is
Public Function PostXml(ByVal Url As String, ByVal PostData As String) As String
Try
Dim Http As HttpWebRequest = WebRequest.Create(Url)
Http.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate")
If Not String.IsNullOrEmpty(PostData) Then
Http.Method = "POST"
Dim lbPostBuffer As Byte() = Encoding.Default.GetBytes(PostData)
Http.ContentLength = lbPostBuffer.Length
Using PostStream As Stream = Http.GetRequestStream()
PostStream.Write(lbPostBuffer, 0, lbPostBuffer.Length)
End Using
End If
Using WebResponse As HttpWebResponse = Http.GetResponse()
Dim responseStream As Stream = WebResponse.GetResponseStream()
If (WebResponse.ContentEncoding.ToLower().Contains("gzip")) Then
responseStream = New GZipStream(responseStream, CompressionMode.Decompress)
ElseIf (WebResponse.ContentEncoding.ToLower().Contains("deflate")) Then
responseStream = New DeflateStream(responseStream, CompressionMode.Decompress)
End If
Dim reader As StreamReader = New StreamReader(responseStream, Encoding.Default)
Dim html As String = reader.ReadToEnd()
responseStream.Close()
Return html
End Using
Catch ex As Exception
End Try
End Function
Re: HttpWebRequest and GZip Http Responses
I copy and past your code
But it is throwing Error The remote server returned an error: (500) Internal Server Error.
Please Help me. In xml http request response it is getting 150 to 200 Second.
Re: HttpWebRequest and GZip Http Responses
Ran into this previously when asked about www.yellowpages.com.au. I posted 2 solutions to the problem here: http://www.vbforums.com/showpost.php...52&postcount=3
Re: HttpWebRequest and GZip Http Responses
Hello vvsingh,
Did the above work for you, or are you still having problems?
Gary
Re: HttpWebRequest and GZip Http Responses
sorry for delay reply.
My problem not solved.
Re: HttpWebRequest and GZip Http Responses
without gzip method below code is working but it taking 150 to 200 second to read response data.
Protected Function PostXml(ByVal url As String, ByVal xml As String) As String
Dim bytes As Byte() = Encoding.UTF8.GetBytes(xml)
Dim strResult As String = String.Empty
Dim request As HttpWebRequest = DirectCast(WebRequest.Create(url), HttpWebRequest)
request.Method = "POST"
request.ContentLength = bytes.Length
request.ContentType = "text/xml" '; charset=utf-8
Using requestStream As Stream = request.GetRequestStream()
requestStream.Write(bytes, 0, bytes.Length)
End Using
Using response As HttpWebResponse = DirectCast(request.GetResponse(), HttpWebResponse)
If response.StatusCode <> HttpStatusCode.OK Then
Dim message As String = [String].Format("POST failed. Received HTTP {0}", response.StatusCode)
Throw New ApplicationException(message)
Else
Dim reader As StreamReader = Nothing
Dim responseStream As Stream = response.GetResponseStream()
reader = New StreamReader(responseStream, Encoding.Default)
strResult = reader.ReadToEnd() ' Here taking time
response.Close()
responseStream.Close()
reader.Close()
End If
End Using
End Function
Re: HttpWebRequest and GZip Http Responses
Thank you very much
My problem solved
now i am getting result within 30 Second