[2008] Grab html source after logging in

[rhijaen]

Using the code below I can get a page that doesn't require you to be logged in. But I need to be logged in to get the page I want..

How would I modify it to be able to use cookies to log in and retrieve the page?

What I eventually want to do is have a user and password input box on the form. Once the user clicks the button, it will grab the source and then manipulate it to how I want it.

        Dim wc As New System.Net.WebClient
        Dim html As String = wc.DownloadString("https://www.webpage.com")
        RichTextBox1.Text = html

[stanav]

Use a WebBrowser control... It uses the same IE's cookies.

[jmcilhinney]

Does setting the Credentials property of the WebClient work?

[rhijaen]

I've tried using the code below but it gives me the same result as if I used no credentials. Could it be because the login is in a form and not a popup box?

        Dim objWebClient As New WebClient()
        Dim Cred As New System.Net.NetworkCredential
        Cred.UserName = TextBox1.Text
        Cred.Password = TextBox2.Text
        objWebClient.Credentials = Cred

        Dim html As String = objWebClient.DownloadString("https://www.webpage.com")
        RichTextBox1.Text = html

[Kasracer]

Credentials will only work if it's a pop-up asking for your information (typically NTLM or whatever).

There is no specific way to do what you want. Some sites might be able to use a cookie, others may use only sessions or other ways.

What you should do is contact the webmaster and verify that you can do this. they will be able to tell you how. Also, if you plan on screen scraping once logged in, then don't. It's also a bad design idea and you should be contacting the webmaster to see if they have an API available.

[rhijaen]

Using cookies I can do it fine with wget, but I'd like to remove that dependancy and just have to rely on vb.

Shell("C:\gnucore\bin\wget.exe --load-cookies c:\cookies.txt -O c:\mansion.html http://www.the4thcoming.net/players/House.asp?Function=SelectCharacter&PropPage=1&CharPage=1&UserID=" & a, AppWinStyle.Hide, True)
            Dim strLines As String() = IO.File.ReadAllLines("C:\mansion.html")

[rhijaen]

Bump..can anyone help me out here?