[RESOLVED] Get a website scapper to click on a button on a website.

[Justin M]

Hi there and good day to you all. I am working on a scapper that will allow me to collect contact information from various sites without me doing through all of them one by one. The website has a drop down list for each state and for each state has its own set of contact info. I am looking to automate this by having the program move the mouse to the drop down list on the website, or at least mimicking the action of clicking, then clicking the first item on the list, then press ok.

Would this be a matter of just programming what part of the screen to move the mouse to, then clicking on the button?

Thanks!

[Max187Boucher]

You do not have to click with mouse you can do with the website's document just click with code or select which item from combo box

[Justin M]

Ok, that is good news. Would I have to use a web browser control on the program form, get the webpage on the form and then "click" the button? I guess what I am asking is how could I click with code a button on a website? Thanks!

[jmsrickland]

If you are referring to using IE and not the WebBrowser control that would be a lot easier than trying to find the handle of the button which is the normal way but for I.E. it's a much more difficult process than it is for a window that owns the button. However, if you wanted to go to the trouble of finding the handle then it would be a more precise way and you wouldn't have to fiddle around with the screen location of the button which may be in a different place from one site to another and even with the same site for that matter.

If you are going to use the WebBrowser control then max has the right approach

The problem I see here is if you are going to visit various web sites then each site would have it's own button name and HTML source which you will need to look at so you can get the name of the buttons.

[Justin M]

Ok, it is no problem to use I.E. Ok, so I got the program to open I.E., go to the website I want to get info, but I am not sure what you mean by handle, is that the code for the button?

Can Vb 6 find the handle? The button says "Submit", but that is prob not what I need to know right lol.

[Justin M]

Ok I had the chance to play around. I have all the urls I will need in a listbox and the program will navigate to them one at a time. I found a snippet of code that uses the web browser control and will even allow me to view the page source...

Private Sub Command2_Click()
Dim pageSource As String
pageSource = WebBrowser1.Document.body.parentElement.innerHTML
End Sub

However, I am not sure how to make use of this. I can get a webpage to show in the browser control, but where would the source be displayed? Would I add a textbox to have it put it. I realize this is sort of going away from the topic at hand, and can make a new thread if wanted.

[Max187Boucher]

Ie or webbrowser control they can both use document which would be the best
For Internet Explorer

And for its Document

Need more help feel free to ask but as jmsrickland said you will need to control object so you will need to find its name first thru the source code

[Justin M]

Nvm I just tried a multi line textbox by adding

text1.text = pageSource and it does show up which is good

Now I need to figure out how to search for specific text for contact info. I see it in the source, but it would be soooo cool if I get the prog to look for it and copy it to a listview.

Hmmm I'll play around and see what happens.


This chunk of source from the webpage has the contact info.

contentString += '<h2 class="com_locator_title">FLAMAN SALES</h2><br class="none"/><span class="line_item address">3405 16TH AVENUE NORTH</span><br/><span class="line_item address2"></span><span class="line_item city">LETHBRIDGE</span>, <span class="line_item state">AB</span> <span class="line_item postalcode">T1H 5E8</span><br/>T: <span class="line_item phone"> (888) 883-8081</span> <br/><span class="line_item tollfree"></span>F: <span class="line_item fax"></span> <br/><span class="line_item email"><a href="mailto:"></a></span><span class="line_item link"><a href="http://www.flaman.com" target="_blank">http://www.flaman.com</a></span><br/><br/><strong/><div class="com_locator_taglist">Aluma</div></strong/><br/>';

I guess where I am stuck Max is I know this is where the text I need to extract info from. There are prob 10 different contacts on here. I need to figure out how I can extra things from this such as the Company Name, Address, Postal Code, PH Number, Website, etc without extracting the html code with it.

Thank you guys for your help so far on this. I wouldn't have got very far without you two!

[Max187Boucher]

Since you seem like a decent person and you are trying to learn not just ask how to do and give me code and you answer back... I will tell you one of my tricks to find document objects have a look at this program it is really good and 99% of time the document objects are right ex IE.Document.GetElemenById("ButtonOnSite") and then i gave you the msdn of all the object properties as .innerText or .Value or .InnerHTML etc... With that program and msdn you'll be unstoppable if your not sure about something go on google and write for example

Document.getelementbyid msdn

Or

Document.body msdn

Or

Internet Explorer msdn

These example will help you get the information you need

[Max187Boucher]

Once you have the program you can select the object on the website with one clixk and it will give you the document name for the object very easy to use

[Max187Boucher]

If resolved please mark as resolved if not ask your questions and if my post were helpful to you please rate me

[Justin M]

Max, thank you for that program. I opened the web link in IE and I can see all the objects on the page. It is SWEET. Thanks!

[jmsrickland]

... have a look at this program it is really good and 99% of time the document objects are right ex IE.Document.GetElemenById("ButtonOnSite")

So, after the 15 days free trial what did you do? Buy it?

[Max187Boucher]

Well first 30 days i did not buy it since i got 15 then laptop got messed up then bought a new one used 15 days ... I the. Bought it since i still need to use it but in 15 days you should be able to get all information you need dor you program to work? Unless if you need it longer install on other computer for 15 days more or simply buy it it depends how much you will be uaing it dis you get everythin working?