[02/03] Retrieving files using HTTP, then converting file to .txt format

[Eddd88]

Hi everyone, I'd like to know if its possible in VB to visit a webpage, then save the webpage from a .asp to a .txt format.

And if it's possible, can it be done hidden too? So the user doesnt see the visiting of the webpage and all, the user just sees the end result of the .txt version of the webpage in the folder on the deskstop or something..

[Jumpercables]

The following code will save the page source of a given webpage to a string.

Dim request As WebRequest = WebRequest.Create("http://www.vbforums.com/")
Dim response As WebResponse = request.GetResponse
Dim stream As System.IO.Stream = request.GetResponse.GetResponseStream
Dim streamReader As System.IO.StreamReader = New System.IO.StreamReader(stream)
String pageSource = streamReader.ReadToEnd

[Eddd88]

Alright, so it saves the webpage down, can it save the webpage in a different file format too? like if i'm not wrong most webpages are saved in .asp (or at least in my case it is), and i need it in a .txt format

[Eddd88]

And er, is the code you gave in VS 2003? 'cause I've just tried it and got the wiggles under the WebRequest, WebResponse and String pageSource..

For the WebRequest and WebResponse, it says they're not declared, so do I declare them as Strings too or something?

For the String, it says 'String' is a class type and so is not a valid expression.

And lastly for the pageSource, it says '.' is expected

[jmcilhinney]

Use the WebClient.DownloadString method. You specify the URL and it returns the output as a String, e.g.

Using wc As New WebClient
    IO.File.WriteAllText("file path here", wc.DownloadString("URL here"))
End Using

[Eddd88]

I created a button on a winform and pasted the code you posted jmcilhinney in the button and it gave wiggles for Using wc As, IO.File.WriteAllText and End

For Using and wc it says they aren't declared

For As it says Comma, ')', or a valid expression continuation is needed

For IO.File.WriteAllText it says WriteAllText is not a member of 'System.IO.File'..so i thought it's because i didn't import System.IO, but after i did, the wiggle still remains

And the 'End', it says 'End' statement not valid

[jmcilhinney]

Ah, bummer. I missed that this is .NET 1.x.

Dim wc As New WebClient
Dim sr As New StreamWriter("file path here")
 
sr.Write(wc.DownloadString("URL here")
 
sr.Dispose()
wc.Dispose()

[Eddd88]

Thats strange, VS 2003 refuses to give me WebClient

As in when i type Dim wc As New then i press the spacebar and that list of stuff i can use pops up, it doesnt list WebClient as one of them, and i forcefully type it in and it gives the wiggles saying Type 'WebClient' not defined =/

Is this only usable in VS 2005 sadly?

[jmcilhinney]

The WebClient class has existed since .NET 1.0. Just like any other class, you have to either qualify it with its namespace or else import its namespace. If you read the documentation for any class in the MSDN library it will tell you which namespace and which assembly a type is declared in.

[Eddd88]

Okays, so I had to import System.Net

and then the wiggles came off from the WebClient in

Dim wc As New WebClient

but the wiggle remained for wc.DownloadString in

sr.Write(wc.DownloadString("URL here")

the wiggle says that 'DownloadString' is not a member of 'System.Net.WebClient'

so when i added a WebClient to the System.Net that i already imported, it takes away the wiggle from wc.DownloadString but then it gives a wiggle for WebClient in

Dim wc As New WebClient

saying that Type 'WebClient' is not defined

so i added another import line to import System.Net back and when i did that, now WebClient has no wiggles but wc.DownloadString has a wiggle again, and now i have both System.Net and System.Net.WebClient imported

[jmcilhinney]

Man, I really botched this one didn't I. DownloadString wasn't added until .NET 2.0. The DownloadFile method has existed from the start though. If it works it would actually be better because there's no need for a StreamWriter. I'm just not sure whether the URL you supply has to refer to an existing file or it can refer to variable output. Hopefully the latter, in which case this will do:

Dim wc As New WebClient
 
wc.DownloadFile("URL here", "file path here")
wc.Dispose()

If that doesn't work then go back to what JC suggested and forget I was ever here. Better yet, upgrade to VS 2005 and make my life easier.

[Eddd88]

Well now i got something like this

Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
		Dim wc As New WebClient
		wc.DownloadFile("http://www.yahoo.com", "C:\Documents and Settings\jy\My Documents\Visual Studio Projects\WindowsApplication8")
		wc.Dispose()
	End Sub

Unfortunately, an error keeps coming up saying

Additional information: An exception occured during a WebClient request

I guess thats because what you said was right that the URL i had to provide would either refer to an existing file or it can refer to a variable output, and i think that it seems to want to refer to an existing file

[jmcilhinney]

Look at what I posted:

wc.DownloadFile("URL here", "file path here")

Now look at what you posted:

wc.DownloadFile("http://www.yahoo.com", "C:\Documents and Settings\jy\My Documents\Visual Studio Projects\WindowsApplication8")

Is that a file path?

[Eddd88]

i thought file path means the place where i want the file (url) to get stored?

[jmcilhinney]

The file path means the path of the file. You've supplied the path of the folder you want to save the file under.