Scraping Links from Web??? help please?

I am using webbrowser control to login to a webpage and auto submit login, email and password with this code:

VB Code:

Webbrowser1.Document.All("usrname").Focus()
        Webbrowser1.Document.ActiveElement.InnerText = txt_user.Text
        Webbrowser1.Document.All("uemail").Focus()
        Webbrowser1.Document.ActiveElement.InnerText = txt_email.Text
        Webbrowser1.Document.All("peeword").Focus()
        Webbrowser1.Document.ActiveElement.InnerText = txt_pass.Text
        Webbrowser1.Document.Forms(0).InvokeMember("Submit")

Yes, I'm a noob but it works so after a long time finding out vb.net 2005 didn't support formname.submit() I was very happy to get this working

Anyhow, I am now trying to use webbrowser control to scrape links with stats.php?id=6digit#here in it's link. I've tried everything I can possibly think of over the past few days and am lost. ANY help will be much appreciated. Thanks


Kevin

^ Sorry to bump but I really am completely lost here if I could get any help I'd love it. Thanks :)

Here is an example that should get you started:

VB Code:

Dim scrapedLinks As New List(Of HtmlAnchorElement)
Dim allLinks As HtmlElementCollection = Webbrowser1.GetElementsByTagName("a")
For Each link as HtmlAnchorElement In allLinks
  If (link.GetAttribute("href").IndexOf("stats.php?id=6digit#here") > 0) Then
    scrapedLinks.Add(link)
  End If
Next

Untested, hope the class names are right.

Thanks for replying, I tried

VB Code:

Webbrowser1.Navigate("http://www.kingsofchaos.com/battlefield.php?" & "start=" & txt_low.Text)
        Dim scrapedLinks As New List(Of HTMLAnchorElement)
        Dim allLinks As HTMLElementCollection = Webbrowser1.Document.GetElementsByTagName("a")
        For Each link As HTMLAnchorElement In allLinks
            If (link.getAttribute("href").IndexOf("stats.php?id=") > 0) Then
                scrapedLinks.Add(link)
                   End If
        Next

without adding an

VB Code:

Imports System.Collections.Generic

I got an error list not defined.

Now the error I am getting is unable to cast object of type system.windows.forms.htmlcollection to type mshtml.htmlelementcollection. I have added the imports statement for mshtml.... any help is apreciated ;)

Oh... change HTMLElementCollection to HTMLCollection then.

htmlcollection is not defined

:(

Again thanks for helping me penagate

I was right its HtmlElementCollection.

VB Code:

Dim allLinks As HtmlElementCollection = Webbrowser1.Document.GetElementsByTagName("a")

that should work :confused:

http://msdn2.microsoft.com/en-us/lib...bytagname.aspx

It is still getting
unable to cast object of type system.windows.forms.htmlcollection to type mshtml.htmlelementcollection

Any ideas what could be causing that?

Well then the documentation is incorrect because that's what they've used :confused:

I'll give it a try

It worked for me although I needed to change HtmlAnchorElement to HtmlElement.

Hrmm,
I am using 2005 could that be a reason?

So am I and so is the documentation which explicitly says it is a new method in the Framework 2.0

This is what I am trying w/ a completely new application now

VB Code:

WebBrowser2.Navigate("www.kingsofchaos.com/battlefield.php")
        Dim scrapedLinks As New List(Of HtmlElement)
        Dim allLinks As HTMLElementCollection = webBrowser2.Document.GetElementsByTagName("a")
        For Each link As HtmlElement In allLinks
            If (link.GetAttribute("href").IndexOf("stats.php?id=") > 0) Then
                scrapedLinks.Add(link)
            End If
        Next

And the Error I am getting is Unable to cast object of type system.Windows.forms.htmlelementcollection to type mshtml.htmlelementcollection

Did you add any additional references? I added mshtml and that was it



A first chance exception of type 'System.NullReferenceException' occurred in DarkFuryToolz.exe
A first chance exception of type 'System.NullReferenceException' occurred in DarkFuryToolz.exe
In order to evaluate an indexed property, the property must be qualified and the arguments must be explicitly supplied by the user.

I get this for:

Dim allLinks As HTMLElementCollection = webBrowser2.Document.GetElementsByTagName("a")

bump ^^

Quote:

---

Originally Posted by khuntballa

I added mshtml and that was it

---

Daah, that's the problem. Take it out :D

It's an ambigous reference.

Still getting
Object reference not set to an instance of an object.
and it is highlighting

VB Code:

Dim allLinks As HTMLElementCollection = webBrowser2.Document.GetElementsByTagName("a")

You probably don't have a document loaded in webBrowser2. Try adding a check that webBrowser2 has a document.

I'll probably sound stupid for asking this, but how do I actually view that list to check it's exact output?

I tried doing listbox2.items.add(link) in the loop and it came up as htmlelement 50 times

I don't know for sure, but I'd guess a member of link would have what you want... probably value or text or something similar

It needs to be done in the DocumentComplete event

So the last thing before I close this thread, is how do I export that list to something readable by me in thne correct format? like when I tried adding it to a listbox all I got was htmlelement instead of the link

Make sure you have "Option Strict On" at the top of your code file. This should prevent you from doing things which don't make sense, like listbox2.items.add(link).

Next, type a dot "." after "link" and you'll see everything you need.

can u make it in normal VB 6.0??

rentacoder.com

Quote:

---

Originally Posted by penagate

rentacoder.com

---

Nice recommendation :D