Need Assistance with Html Agility Pack and Google
Hey guys I'm trying to create a windows form on my application that will scrape the results of a Google search for the search result links (which are the green links).
I have this code that I found online however it doesn't return any results.
Any ideas on who I can fix this code to scrape the search results?
Thanks
Code:
Dim web As New HtmlWeb()
Dim htmlDoc As HtmlAgilityPack.HtmlDocument = web.Load("http://www.google.com/search?num=10&q=Hello+World")
' get search result URLs
Dim items = htmlDoc.DocumentNode.SelectNodes("//div[@id='ires']/ol[@id='rso']/li/div[@class='vsc']/h3/a/@href")
For Each node As HtmlNode In items
Console.WriteLine(node.Attributes)
Next
In my app I am also trying to return the results in a ListView rather than using console.
Thanks for any support.
Re: Need Assistance with Html Agility Pack and Google
No amount of fixing will solve the problem, which is that these links are generated by script and simply do not appear in any form in the HTML. If you want to interact with Google you have to do it via their API.
Re: Need Assistance with Html Agility Pack and Google
Roger that.. Thanks dunfiddlin. I decided to go the WebRequest with html agility pack and a little Reg Ex to do the trick.