Manipulate/Change/Form Fill data in webpages using the Webbrowser control

[kleinma]

I started a 2008 version, like my other things, I just haven't finished it yet.

I had a bunch of new features lined up for it, but I suppose getting a mirror image of this current sample, but done in the managed browser control would be just as benefitial for the time being.

In any event, that error is just because you need that file. Try searching your hard drive for it, if you don't have it, let me know, and I can upload it here. I sort of thought I had done that in one of these previous replies already, but I don't have time to read back through 9 pages

[Xancholy]

No need to read 9! pages. I managed to find the missing file. That's great work. Thanks.

On a side note, you've shown us how to manipulate textboxes, checkboxes, etc..

What about autoclicking an image that does not use ahref ? eg:

<div class="iconbody">
<input id="ctl00_cp1_btnDownload" type="image" style="border-width: 0px;" onclick="openWindow('download','3pwBhzhigrk5PL4HGdAq2bDHDBlmHO0qkJvITXr5sRc1'); return false;" src="App_Themes/NA_Default/Images/Icon_Download.gif" name="ctl00$cp1$btnDownload"/>

How do I click image named ctl00$cp1$btnDownload ?

[kleinma]

you can either run its javascript function directly, or you can invoke a click event on it.

[gmchun83]

Hai,we would like thanks to kleinma that provide sample code to us for make our development more easy.But,we are having some issue about button.We have found most button at html pages that available at internet nowadays.Their don't provide id or name for button.At sample code,it only work for button when have id or name.Please refer following for the html code(submit button).

<form method="GET" action="http://www.myweb.com/addurl.html">
<tr><td align=left><font face="helvetica,arial,sans-serif">
Url to add:<br>
<input type="text" name="url" size=30 value="http://"><br><br>
Your email address:<br>
<input type="text" name="email" size=30 value=""><br>
<input type="checkbox" name="optin" checked value="1"> Send me the newsletter<br><br>

<input type="submit" value=" Submit URL ">
</td></tr>
</form>

Please advice.
Thanks a lot.

[razohad]

Hai,we would like thanks to kleinma that provide sample code to us for make our development more easy.
But,we are having some issue about button.We have found most button at html pages that available at internet nowadays.
Their don't provide id or name for button.
At sample code,it only work for button when have id or name.Please refer following for the html code(submit button).

if i understand you correct then you can try this

Dim theWElementCollection2 As HtmlElementCollection = WebBrowser1.Document.GetElementsByTagName("input")
        For Each curElement As HtmlElement In theWElementCollection
            Dim controlName As String = curElement.GetAttribute("type").ToString
            If controlName = "submit" Then
                curElement.InvokeMember("click")
            End If
        Next

[tronni]

Thanks for this great example. It’s working excellent.

I'm able to logon to my website and navigate to right page, but how can I download a file? (I want to download the file to a defined location ek: c:\tmp\file.txt)

The download dialog pops up after I click a button:

HTML: <input type="submit" name="pdirsall" value="Recursive" class="fatbuttom"/>

I've no idea where I should look. Can you help me please?

Thanks

Tronni

[kleinma]

if you can get the URL of the file to download, you can just download it directly using VB code. You don't need to automate anything on the part of the browser. Look at my sig for another example code on how to download files from the web with a progress bar. You could work that into your browser app.

[tronni]

Sorry, I forget to mention that, I don’t have a direct URL, if I had I could use your app or something simple like:

My.Computer.Network.DownloadFile("webURL", "downloadDestination")

Guess I must accept and handle the file stream from the webserver? Anyways I have no idea how I should do this (I have some experience with file streams).

Do you know a way to do this or where I can find some information?

Thanks for answering

[kleinma]

the code that has the

<input type="submit" name="pdirsall" value="Recursive" class="fatbuttom"/>

that input is obviously to submit a form. There isn't a field in the form, or the action value of the form tag, that indicates what file will be downloaded?

How does it know which file to download for you?

[Mark107]

kleinma, I want to fill the name first and the random numbers at the end (like from the start mark0 to the end mark1000) on the first textbox of the webpage. I want to do the same text what i did to the first textbox and the email address i want to input at the end like this [email protected] so do the same to fill on the email address require box. I also want to click the tickbox, the program type the security code itself, click the webpage button and click my form button to go back and do the next random number like what I did first mark0 and do the second mark1.....It is bit like counting up so i want to keep going like loops until it stop 1000.



my register site:
http://mark107.awardspace.com/register.php



I'd appreciate it if you get what i mean from my description on above. lol



Hope you can post the code.


Thanks,
Mark

[stlaural]

Hi guys,

I have a web form which is already open in an IE Window. What I want to do is to fill two of its fields, respectively id = "df_1_0" and id = "df_2_1", with my VB.NET application.

So, First, I need the application to find the correct IE Window, access the two fields and fill them.

And, Second, The web page is using framesets and frames, the form is contained into one of the frames so I am not sure how I could access the fields by the VB.Net application. I don't think that I can use the WebBrowser Control because to get to the page that contains the form, I need to authenticate myself, and then on the page that I am redirected to I can open the form, which will open in an other IE Window (kinda popup).

So the real question is this : can I control an already opened IE window and access its components ?

thanks for helping !

[kleinma]

the only reliable way to do that would be writing a plugin for IE. I am not sure if you can write IE plugins in .NET code, or if you have to use native C++ or something like that.

The other method would be using a cheap hack like sendkeys, which is totally unreliable.

[Mark Douglas]

Hi kleinma,

I've been successfully using your example here to pull a listing of hyperlinks off one of my Intranet pages for a basic listing of links.

Well i have like 100 web pages i would like to run the program on. However because .net cycles through the pages so fast, i don't think the pages are fully loading, and it just repeats the hyperlinks from the first web page 100 times.

Do you know if there is a way to execute some code once the page is fully re-freshed? So cycle through each ".htm" file, and once its finished totally loading, then run the hyperlink code. The code then pastes it into an excel sheet. Which i have working. Just not the wait for page load type thing?

Any ideas? Thank you for any help!!

So im using

  Strpath = "\\server\Intranet\Intranet\CAD Dept"
     

        For Each strdir In IO.Directory.GetFiles(Strpath, "*.htm", IO.SearchOption.TopDirectoryOnly)

            wb.Navigate(strdir)
      
            For Each MyLink As mshtml.HTMLAnchorElement In GetCurrentWebDoc.links
             
                Temp &= MyLink.href
                Wksht.Cells(MyrowNum, 8) = Temp
                i += 1
                MyrowNum = MyrowNum + 1
                Temp = ""
            Next
        Next

[kleinma]

use the documentcompleted event. That event fires when the page is fully loaded. In that event you can check the current URL to take specific action based on what page has just loaded.


Just also note it will fire more than once if the page is actually a frames page.

[Mark Douglas]

Seems like this is only fireing at the very end of the cycle of htm files. Only the 100th htm is accessing the documentcompleted code?

        Strpath = "\\server\Intranet\Intranet\CAD Dept"

        For Each strdir In IO.Directory.GetFiles(Strpath, "*htm", IO.SearchOption.TopDirectoryOnly)

            wb.Navigate(strdir)


        Next

If just keeps cycling through each htm page and once its done, then the documentcompleted aspect fires?

[kleinma]

how are you accessing each page? My guess would be probably in some sort of loop, but that will not fit in to the asynchronous way the browser control works.

You need to initially fire off a navigation to the first htm page. Then documentcomplete will fire, then you do whatever you need to do with that page, and then navigate to the next url in your list, until its all done.

[Mark Douglas]

Ok here is how im accessing the files, and the looping. Basically looping the webpages, and hoping for the document complted code to fire after i navigate to each page. But isnt working like that.

Im running VS2005 forgot to mention that if that helps?

Thanks for the help!!

   
    Dim Temp As String = String.Empty
    Dim myExcel As Excel.Application
    Dim MyrowNum As Integer = 1
    Dim i As Integer = 1
    Dim Strpath As String
    Dim strfile, strdir, strdir2 As String
    Dim Wksht As Excel.Worksheet

 Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click

        myExcel = CType(CreateObject("Excel.Application"), Excel.Application)
        Dim Owb As Excel.Workbook = myExcel.Workbooks.Add()
        myExcel.Visible = True
        Wksht = DirectCast(myExcel.Sheets("Sheet1"), Excel.Worksheet)
        Strpath = "\\server\Intranet\Intranet\CAD Dept"
        For Each strdir In IO.Directory.GetFiles(Strpath, "*htm", IO.SearchOption.TopDirectoryOnly)
            wb.Navigate(strdir)
        Next
            End Sub

    Private Sub wb_DocumentComplete(ByVal sender As Object, ByVal e As AxSHDocVw.DWebBrowserEvents2_DocumentCompleteEvent) Handles wb.DocumentComplete
        Wksht = DirectCast(myExcel.Sheets("Sheet1"), Excel.Worksheet)
        For Each MyLink As mshtml.HTMLAnchorElement In GetCurrentWebDoc.links
            Temp &= i.ToString & ")" & Environment.NewLine
            Temp &= MyLink.href
            Dim MyFileArray As Object
            MyFileArray = Split(Temp, "/")
            Dim MyArrayCount As Integer
            MyArrayCount = UBound(MyFileArray)
            Dim MyLinkFile As String = Replace(MyFileArray(MyArrayCount), ".dwf", "")
            Wksht.Cells(MyrowNum, 8) = MyLinkFile
            i += 1
            MyrowNum = MyrowNum + 1
            Temp = ""
        Next
    End Sub

[kleinma]

again, you can't call navigate in a loop like that. You can't hope 1 routine will fire before the other. Well I guess you can hope, but it just won't happen.

Rethink your logic a bit, and read what I wrote above as far as how you need to do this.

Create a listing of the URLs (in an array of list of string, whatever) that you want to nav too. Then kick it all off by navigating to the first URL, and ONLY the first URL. When DocumentComplete fires, do the code you are doing now, but when you are done, call the navigate method again (in documentcomplete) and navigate to the next URL, and so on... until you are done.

[stlaural]

Thanks for the quick answer kleinma, that confirms what I thought. I will have to use the SendKeys and try to make it as reliable as possible. Should be able to work a solution at 95% reliability. at least I hope so.

I'll make sure that the keyboard and mouse are locked during the process, which should only last 4 to 5 seconds.

thanks again

[kleinma]

I have heard (but not confirmed) issues with using SendKeys and Vista due to its more secure user account structure. You may want to check that out.

[Mark Douglas]

Hi kleinma, your solution on how to approach it worked, putting the page change in the document completed portion. Thanks!!

P.S. On the send keys in Vista, yes I'm working on another program right now that runs inside AutoCAD, and Vista wont let me sendkeys to it or any other program. Bummer...

Thanks for the help!!

Mark

[kleinma]

It is security to avoid a rouge program from being able to have access to another program. Like you enter your password in 1 program, and another program is monitoring it to steal it.

You may have success with SendKeys in Vista if both processes are running with admin rights, but I am not sure of that.

[Mark107]

@kleinma: Is it possible to get pass of the automatic captacha image??




Thanks,
Mark

[kleinma]

Nope.

[swallowbob]

Hi Kleinma,

How do emulate this button ? It has no id. DirectCast(GetCurrentWebForm.item("submit.add-to-cart", 0), mshtml.HTMLButtonElement).click() can not get anything.

Thanks.

html:

<input type="image" src="images/G/01/x-locale/common/transparent-pixel._V42752373_.gif" width="160" alt="Add to Shopping Cart" name="submit.add-to-cart" class="bbSprite btn-atc_s" height="27" border="0" />

[kleinma]

it doesn't look like a button to me.. it looks like an image..

[swallowbob]

It is a button. Can you help me take a look "Add to shoping cart button" on amazon? It should be a button. I suspect that I have to specify the form id since they have a lot of forms there.
thanks for your help.

[kleinma]

sorry, I was not fully clear.

It IS very much a button.

an HTML input elements type is determined by the type attribute. You posted

<input type="image" src="images/G/01/x-locale/common.....>

So it is an input element, of type image. It may feel like a button because you can click it and it does something, but that doesn't make it a button, so you can't cast it to a button element type from the MSHTML library, it simply isn't that type.

You should be able to cast it to an image element or a generic input element type, and then invoke a click on it to get the desired behavior. If you are unable to get a reference to it because it has no ID and you can't access it via the name property, then really the only solution is to loop all <input> elements in the DOM and evaluate their name property until you find the one named submit.add-to-cart, and then you would have your reference to the correct element.

[swallowbob]

Thanks, Kleinma

I got it work using the following method:
Dim thisObj As Object
Dim butInput As mshtml.HTMLInputElement
thisObj = webBrowser.Document.getElementsByTagName("input")
For Each butInput In thisObj
If butInput.name = "submit.add-to-cart" Then
butInput.click()
End If
Next

Another question:
Can I hide the browser and still get everthing work? If I set visible to false, it can not get the same result as visible.

Thanks you give us very good example and help.

[kleinma]

Try setting the size of the browser very small, and then position it OFF the form by giving it negative x/y location settings (or x/y that are larger than the forms size). That should still allow it to technically be visible, even though you don't see it on screen.

[swallowbob]

Very good! It works.

One last question, is there a good way to parse the price.
<td class="priceBlockLabelPrice">Price:</td>
<td><b class="priceLarge">$26.05</b>

Right now I loop block element and find class name priceLarge. It looks not very safe or stable.
Appreciate your help.

[swallowbob]

by the way, I have several operations, each click/submit wil take some time , the second operation is based the first operation result. In this case, how to avoid the second operation is too fast and doesn't run it if not get the first page result return. I tried sleep a while but looks like it will sleep the first operation too. multi-thread? thanks.

[kleinma]

no unfortunately, when you automate someone elses site, you are doing something they didnt intend or design for, so you get lots of cases where data you want doesn't have any easy way to access it other than some raw parsing. The downside is that when the page changes at all, it could break your parsing logic. However the same is true if they change something like an elements id value, which is generally considered the "easy way" to get data from a page.

As far as your question about timing, you need to use the documentcompleted event of the browser to do subsequent navigations and operations. This is the event that fires when a page fully loads into the browser and has been rendered. If you try to do ANYTHING to a webpage before this event fires, you will likely end up with an error sooner or later, because the page has not fully loaded.

So then people always ask the same question "well documentcompleted is only 1 event, and I need to do something different based on what page is actually loading", and the answer to that is you need to evaluate the browsers current URL when this event occurs, and that tells you which page just loaded, and you can take appropriate action from there.

Again, the process of manipulating webpages is a tricky one, and some sites are way easier than others. However I have done a few consulting projects and personal projects for web automation, and so far there was not a site I couldn't automate. Some projects require more maintenance than others, but that is just how it goes.

[swallowbob]

Thanks, Kleinma

You looks always online. What a great help!

I can imagine what we can not automate is the website you have to download a client(java program) on your computer.

[kleinma]

well yes, I was strictly speaking from an HTML/Javascript manipulation standpoint. Things like java applets and activex controls may have some automation capabilities, but it would be done totally different than the HTML manipulation.

[swallowbob]

Hi Kleinma,

Nice weekend. I met another problem.

When I submit a button, it works at most time but it will prompt a error dialog sometime which make it can not proceed. I have to click to close the dialog. What was I missing? Thanks a lot.

The error dialog is
Title is Internet Explorer Script Error
An error has occurred in the script on this page.
Line: 72
Char: 1
Error: Invalid argument
Code: 0
URL: xxxx
Do you want to continue running scripts on this page?

Here is my code to submit:
webBrowser.Document.forms.item(, 0).elements("twotabsearchtextbox").value = txtField1.Text
'webBrowser.Document.forms.item(, 0).submit()


The following is the form html:
<form style="width:100%; margin:0 0 0 0; border:0; padding:0 0 0 0;" method="get" action="/s/ref=nb_ss_vg" name="site-search">

<table width="100%" border="0" cellpadding="0" cellspacing="0">
<tr>
<td class="navSearchLeft"><img src="transparent-pixel._V42752373_.gif" width="58px" alt="Search" height="38px" border="0" /></td>
<td width="100" class="navSearchBar"><select name="url" class="searchSelect" title="Search in">
<option value="search-alias=aps">xxx.com</option>
<option value="search-alias=videogames" selected="selected">Video Games</option>
<option value="search-alias=videogames-used">Used Video Games</option>
</select></td>
<td width="3px" class="navSearchBar">&nbsp;</td>
<td class="navSearchBar" align='left'><input type="text" id="twotabsearchtextbox" class="searchSelect" name="field-keywords" value="" size="50" title="Search for" style="width:100%; background-color: #FFF;" /></td>
<td align="right" width="3px" class="navSearchBar">&nbsp;</td>
<td width="25" align="left">
<div id="navGoButton">
<input type="image" src=transparent-pixel._V42752373_.gif alt="Go" id="navGoButtonPanel"
onmouseover="shiftImage('navGoButtonPanel',0,-38,'navGoButton',2,1,1);"
onmouseout="shiftImage('navGoButtonPanel',0,0,'navGoButton',2,0,1);"
onfocus="shiftImage('navGoButtonPanel',0,-38,'navGoButton',2,1,1);"
onblur="shiftImage('navGoButtonPanel',0,0,'navGoButton',2,0,1);" />
</div>
</td>
</tr>
</table>
</form></td>

[tanjiunnyann]

kleinma, I want to do an auto login using webbrowser, I already done the auto-fill username and password in the form, but the web site contain the verify code. How to get the verify code from the showing picture? Any idea?

The web site is : http://www.17mso.com

[kleinma]

that picture looks to be the equivelant of a captcha verification, so no, there is no way to get around a captcha, except to write some code that can exame the image and determine the characters from the image.

The whole point of sites putting captcha verification on them, is to prevent automation of their site. To force an actual person to be there filling in the form.

[swallowbob]

Hi Kleinma,

Can you help me take a look above #366 problem? I suspect I need setup some properties/add some parameters manually for the embedded browser since IE works ok.
thanks.

[WHATAGOAL1]

kleinma, many thanks for your superb tutorial, everything i know about html has been learnt from this thread

i am trying to automate tradeXfair.com*WITHOUT THE X, can login and progress to the main page, but then it becomes proper difficult, my first question

is it true that if it is on the webpage it can be automated?

the parsing of the final webpage returns very little, i have managed to glean the info that there are 47 elements, cannot find id, name or any thing else about them, although theire are children, same again no info can be gleaned

anyway i tried invoke a click on the child, (not knowing which), and the website returned "HTTP 403 Forbidden, website declined to show page


while navigating to the page, i have noticed info regarding cursor position
could this be the reason for the decline?. is it possible the website is checking the position of my cursor to stop automation?