[RESOLVED] WebClient.DownloadString problem.

[Pradeep1210]

This doesn't work. Sometimes I get an empty string while at other times I get an error saying connection was closed by remote client.
The url opens OK in all web browsers.
Anyone knows why?

Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
    Dim wc As New System.Net.WebClient
    Dim s As String = wc.DownloadString("http://www.yellowpages.com.au")
    TextBox1.Text = s
End Sub

[jmcilhinney]

I just tried and got the same result. I don't know the answer but I'm going to take a guess and say that the web server is checking the user agent and only responding to "genuine" browsers. You might have to add a header to your WebClient that it recognises as a legitimate user agent.

[MattP]

Looks like for some reason only the .au site sends gzip encoded data.

This worked for me.

Imports System.Net

Public Class Form1

    Private Sub Button1_Click(sender As System.Object, e As System.EventArgs) Handles Button1.Click
        Dim wc As New WebClientEx()
        TextBox1.Text = wc.DownloadString("http://www.yellowpages.com.au")
    End Sub

End Class

Public Class WebClientEx
    Inherits WebClient

    Protected Overrides Function GetWebRequest(address As System.Uri) As System.Net.WebRequest
        Dim req = CType(MyBase.GetWebRequest(address), HttpWebRequest)
        req.AutomaticDecompression = DecompressionMethods.GZip Or DecompressionMethods.Deflate
        Return req
    End Function

End Class

*Edit*

If you'd rather not create your own class to extend the WebClient here's how you'd go about it.

        Dim request = CType(WebRequest.Create("http://www.yellowpages.com.au"), HttpWebRequest)
        request.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705;)"
        request.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate")
        request.AutomaticDecompression = DecompressionMethods.GZip Or DecompressionMethods.Deflate
        Using response = request.GetResponse()
            Using responseStream = response.GetResponseStream()
                Using sr = New IO.StreamReader(responseStream)
                    TextBox1.Text = sr.ReadToEnd()
                End Using
            End Using
        End Using