[RESOLVED] DownloadString Login site

[Banjo]

Hello I'm having a small problem, I have a listbox with a websites in that need checking. To check the sites I open the follwing url and it will return either 0, 1 or -1:

http://www.mysite.com/check.php?id=item

Where the text saying "item" is, that is the item in the listbox. My code needs to loop through every item checking them. So far I'v tried this:

    Public Sub RemoveSites(ByVal Lst As ListBox, ByVal ListName As String)
        Dim Client As New System.Net.WebClient

        For Each Item In Lst.Items
            'Downloads the source
            Dim Html As String = Client.DownloadString("http://username:[email protected]/check.php?id=" & GetSiteName(Item))

            'Checks
            If Html = "1" Then
                'Need to remove
                Lst.Items.Remove(Item)
            ElseIf Html = "-1" Then
                'Need to remove
                Lst.Items.Remove(Item)
            End If
        Next
    End Sub

The problem with this is, even though it has the user and password in for the .htaccess protection, it still doesn't let me download the string and says unortharised access.

I know you can login like:
http://username:[email protected]/check.php?id=
So thats why I tried above.


Another way I tried was to use a web browser, this seems to work as far as the site opening with out the unortharised access. But it doesn't download the content so cant' check.

    Public Sub RemoveSuspendedFast(ByVal Lst As ListBox, ByVal ListName As String)
        'For each item check if suspended
        For Each Item In Lst.Items
            'Open the page
            wbSuspend.Navigate(New Uri("http://username:[email protected]/check.php?id=" & GetSiteName(Item)))

            'Read the page
            If wbSuspend.DocumentText = "1" Then
                'Has been suspended - remove.
                Lst.Items.Add(Item)
            ElseIf wbSuspend.DocumentText = "-1" Then
                'Doens't exist - remove.
                Lst.Items.Add(Item)
            Else
                MsgBox("Nothing :(")
            End If
        Next
     End Sub

The above code doesn't work because they all get the message box saying "Nothing :(", some of them should return 1 or -1 because I'v checked.

Is there anyway I can open a site that is password protected by .htaccess and download the content of it for checking.

[kleinma]

Instead of trying to include your username and password in the URL, try setting them via a networkcredentials object, and assign that object to the webclient prior to trying to download the string.

Dim Client As New System.Net.WebClient
Client.Credentials = New Net.NetworkCredential("UserName", "Password")
'''
'''
Dim Html As String = Client.DownloadString("http://www.mysite.com/check.php?id=" & GetSiteName(Item))

[Banjo]

Ooh didn't know that, thankyou soo much works a treat

[Banjo]

Hmm this is the code I'm using now, but it gets 403 errors :S

 Public Sub RemoveSites(ByVal Lst As ListBox, ByVal ListName As String)
        On Error Resume Next

        'Creates client
        Dim Client As New System.Net.WebClient

        'Creates a variable for the counter
        Dim Counter As Integer

        'Sets credentials
        Client.Credentials = New Net.NetworkCredential("user", "pass")

        'Does the following for all items
        For Each Item In Lst.Items
            'Downloads the source
            Dim Html As String = Client.DownloadString("http://www.mysite.com/check.php?id=" & GetSiteName(Item))

            'Checks if to remove or not
            If Html = "1" Then
                'Removes - already suspended
                Lst.Items.Remove(Item)
                Counter += 1
            ElseIf Html = "-1" Then
                'Removes - site doens't exist anymore.
                Lst.Items.Remove(Item)
                Counter += 1
            End If
        Next

        'Tells me its done :D
        MsgBox("Sites that have been suspended have been removed from the list (" & ListName & ") - " & Counter & " sites.", MsgBoxStyle.Information, "Sites removed")

    End Sub

How can I resolve this?

[kleinma]

before you said it was working. Did you change something? or did you realize it was never working?

[Banjo]

Well when I tested it i used this code:

        'Creates client
        Dim Client As New System.Net.WebClient
        'Does the following for all items
        For Each Item In ListBox1.Items
            'Downloads the source

            'Sets credentials
            Client.Credentials = New Net.NetworkCredential("user", "password")

            Dim Html As String = Client.DownloadString(http://www.mysite.com/check.php?id=" & Item)

            MsgBox(Html)
        Next

It's weird because If I put that code on a button then it works, but after I press the button a few times then it gets the error, however in the code in my previous post it just gets the error.

Is there a way I could check if the webclient got that error and then retry?

[kleinma]

have you set a break point and stepped through on the code that works to make sure the URL looks correct on the attempt that fails versus the one that works?

[Banjo]

Yeh the URL's are correct, hmm Ill try putting some wait functions in it see if that works because If I go through the list slowly (putt a MessageBox before each item saying something) it works fine.

[kleinma]

you could also try not setting the credentials everytime. You should only need to specify them once, outside the loop, although I don't know why that would directly relate to your issue.

[Banjo]

Yeh In my previous code I set them outside the loop still the same prob though. Ill have a fiddle around with the code, unless there is another way to download strnigs apart from the WebClient?

[kleinma]

There are, but webclient is the easiest, and even if you were try tro the HTTPWebRequest/HTTPWebResponse route, I think the same issue will crop up.

[Banjo]

Ahh ok, well thanks for the help much appreciated. Iv put in a wait and used uri for the url seems to be ok. Rather odd though.

[kleinma]

maybe the site has something that blocks from too many requests coming in from the same IP in a given interval?

Have you tried with any other sites? Or are you just doing this with one specific site?

When you are trying to do something with a website, that was not its original intention, you really never know what is going to happen.

[Banjo]

Hmm yeh that's possible ill find out and no I haven't tried with other sites, can't really.

[Banjo]

Just to let you know it was an ip block thing, it's a secure site thats why. Thanks for the help.

[kleinma]

glad you got it all sorted