Scraping Links from Web??? help please?

[khuntballa]

I am using webbrowser control to login to a webpage and auto submit login, email and password with this code:

Webbrowser1.Document.All("usrname").Focus()
        Webbrowser1.Document.ActiveElement.InnerText = txt_user.Text
        Webbrowser1.Document.All("uemail").Focus()
        Webbrowser1.Document.ActiveElement.InnerText = txt_email.Text
        Webbrowser1.Document.All("peeword").Focus()
        Webbrowser1.Document.ActiveElement.InnerText = txt_pass.Text
        Webbrowser1.Document.Forms(0).InvokeMember("Submit")

Yes, I'm a noob but it works so after a long time finding out vb.net 2005 didn't support formname.submit() I was very happy to get this working

Anyhow, I am now trying to use webbrowser control to scrape links with stats.php?id=6digit#here in it's link. I've tried everything I can possibly think of over the past few days and am lost. ANY help will be much appreciated. Thanks


Kevin

[khuntballa]

^ Sorry to bump but I really am completely lost here if I could get any help I'd love it. Thanks

[penagate]

Here is an example that should get you started:

Dim scrapedLinks As New List(Of HtmlAnchorElement)
Dim allLinks As HtmlElementCollection = Webbrowser1.GetElementsByTagName("a")
For Each link as HtmlAnchorElement In allLinks
  If (link.GetAttribute("href").IndexOf("stats.php?id=6digit#here") > 0) Then
    scrapedLinks.Add(link)
  End If
Next

Untested, hope the class names are right.

[khuntballa]

Thanks for replying, I tried

Webbrowser1.Navigate("http://www.kingsofchaos.com/battlefield.php?" & "start=" & txt_low.Text)
        Dim scrapedLinks As New List(Of HTMLAnchorElement)
        Dim allLinks As HTMLElementCollection = Webbrowser1.Document.GetElementsByTagName("a")
        For Each link As HTMLAnchorElement In allLinks
            If (link.getAttribute("href").IndexOf("stats.php?id=") > 0) Then
                scrapedLinks.Add(link)
                   End If
        Next

without adding an

Imports System.Collections.Generic

I got an error list not defined.

Now the error I am getting is unable to cast object of type system.windows.forms.htmlcollection to type mshtml.htmlelementcollection. I have added the imports statement for mshtml.... any help is apreciated

[penagate]

Oh... change HTMLElementCollection to HTMLCollection then.

[khuntballa]

htmlcollection is not defined



Again thanks for helping me penagate

[penagate]

I was right its HtmlElementCollection.

Dim allLinks As HtmlElementCollection = Webbrowser1.Document.GetElementsByTagName("a")

that should work

http://msdn2.microsoft.com/en-us/lib...bytagname.aspx

[khuntballa]

It is still getting
unable to cast object of type system.windows.forms.htmlcollection to type mshtml.htmlelementcollection

Any ideas what could be causing that?

[penagate]

Well then the documentation is incorrect because that's what they've used

I'll give it a try

[penagate]

It worked for me although I needed to change HtmlAnchorElement to HtmlElement.

[khuntballa]

Hrmm,
I am using 2005 could that be a reason?

[penagate]

So am I and so is the documentation which explicitly says it is a new method in the Framework 2.0

[khuntballa]

This is what I am trying w/ a completely new application now

WebBrowser2.Navigate("www.kingsofchaos.com/battlefield.php")
        Dim scrapedLinks As New List(Of HtmlElement)
        Dim allLinks As HTMLElementCollection = webBrowser2.Document.GetElementsByTagName("a")
        For Each link As HtmlElement In allLinks
            If (link.GetAttribute("href").IndexOf("stats.php?id=") > 0) Then
                scrapedLinks.Add(link)
            End If
        Next

And the Error I am getting is Unable to cast object of type system.Windows.forms.htmlelementcollection to type mshtml.htmlelementcollection

Did you add any additional references? I added mshtml and that was it



A first chance exception of type 'System.NullReferenceException' occurred in DarkFuryToolz.exe
A first chance exception of type 'System.NullReferenceException' occurred in DarkFuryToolz.exe
In order to evaluate an indexed property, the property must be qualified and the arguments must be explicitly supplied by the user.

I get this for:

Dim allLinks As HTMLElementCollection = webBrowser2.Document.GetElementsByTagName("a")

[khuntballa]

bump ^^

[penagate]

I added mshtml and that was it

Daah, that's the problem. Take it out

It's an ambigous reference.

[khuntballa]

Still getting
Object reference not set to an instance of an object.
and it is highlighting

Dim allLinks As HTMLElementCollection = webBrowser2.Document.GetElementsByTagName("a")

[ZaNi]

You probably don't have a document loaded in webBrowser2. Try adding a check that webBrowser2 has a document.

[khuntballa]

I'll probably sound stupid for asking this, but how do I actually view that list to check it's exact output?

I tried doing listbox2.items.add(link) in the loop and it came up as htmlelement 50 times

[ZaNi]

I don't know for sure, but I'd guess a member of link would have what you want... probably value or text or something similar

[penagate]

It needs to be done in the DocumentComplete event

[khuntballa]

So the last thing before I close this thread, is how do I export that list to something readable by me in thne correct format? like when I tried adding it to a listbox all I got was htmlelement instead of the link

[penagate]

Make sure you have "Option Strict On" at the top of your code file. This should prevent you from doing things which don't make sense, like listbox2.items.add(link).

Next, type a dot "." after "link" and you'll see everything you need.

[VaRz]

can u make it in normal VB 6.0??

[penagate]

rentacoder.com

[stanav]

rentacoder.com

Nice recommendation