Manipulate/Change/Form Fill data in webpages using the Webbrowser control

[Sjors]

anyone that can help me out saving the dynamic picture?

[kleinma]

I have nothing to say about captcha images. Period. Please do not ask for info here about bypassing, downloading, saving, or interpreting captcha images. They exist for a reason, and that reason is the sites they exist on DO NOT want to be automated, and want a human there doing the data entry.

[abcat]

is there a way to disable popup windows using your codes?

[kleinma]

if you are using the activex browser control (the one this article was written for), then you can handle the NewWindow3 event, which exposes the URL that is about to be popuped up, and exposes a cancel event arg you can set to true to cancel the popup.

If I remember correctly, the NewWindow3 event only fires on Windows XP SP2 and higher (which by no coincidence was when MS added the popup blocker to IE)

[piklio]

Hi dear kleinma
tnx for ur hlpful project !
but i have 2 question

first :

DirectCast(GetCurrentWebForm.item(ID or Name), mshtml.HTMLInputElement).click()

in this syntax we must use id on name of submt button . but sometimes for submit buttons there is no ID/Name . how can i click this button ?


second :
how u find the name or ID of button in web page ? i save the page and then browse page in frontpage or i read html codes and find name . is there any speedy way or any extension for browser that help us for finding button name faster than now ?

thanks in advance sir

[kleinma]

If an element has no id or name attribute, then the best you can do is grab all controls of a specific element type (like grabbing all input elements) and then looping them to see if one of the input elements has a type=submit attribute.

You can further verify with checking additional known attributes, like the value attribute which would have the submit buttons text.

As far as "how do you find elements names" and all that, if you use IE8, you can hit F12 when you are on a webpage, and it brings up a tool to go through the webpage DOM easily and find things like element names. There is even an option to allow you to simply click on the element you want on the page, and the utility tells you the info about it.

If you don't have IE8, you can get a similar tool via download (ie developer toolbar) for IE6 and IE7. I believe firefox also has similar tools, however I don't recommend using firefox when working with web automation, because some sites deliver different content (and have different behavior) based on browser version, and if you use firefox, you may find different results when using the IE based browser control in a windows app.

[piklio]

so great !
tnx
i solved my problem and i can login

with this code :

with or loop i checked all value of items and then grab index of item that value equal to "log in" for example .
then please ignore my last private message

for grabbing id and name in page tnx 2 . great idea .
//
one orher small question :

Private Sub wb_NavigateComplete2(ByVal sender As Object, ByVal e As AxSHDocVw.DWebBrowserEvents2_NavigateComplete2Event) Handles wb.NavigateComplete2
        txt_page_source.Text = GetCurrentWebDoc.documentElement.outerHTML
    End Sub

i use this code for grabbing page source
but when i call wb.navigate 1 time , then txt_page_source changed more times !
it means with one navigation , wb_NavigateComplete2 rase more times !
for wat ?
i need only last event and when everythings is complete !

tnx again sir

[kleinma]

naviatecomplete events fire when the initial navigation is done, not when the page has fully loaded into the browser. Use the DocumentCompleted event for that.

Either one of these events can fire multiple times if the page has frames or something similar where it is actually loading its content from different places.

[piklio]

Either one of these events can fire multiple times if the page has frames or something similar where it is actually loading its content from different places.

tnx
but how can i know when page is loaded complete ?

[blazen2009]

kleinma Thanks for the project! I'm a noob to VB.NET and Visual Studios 2008, but I have been interested in apps like this project and it will help speed up my learning.

[piklio]

hi

how can make cockies dedicated from internet explorer ?

[kleinma]

I have no idea what that statement means.

[piklio]

I have no idea what that statement means.

tnx mate
but it means i need login to gmail for example via m
and i dont want if anybody open IE and type Gmail.com can use my session on gmail

[kleinma]

I still have no idea what you are asking for. I'm sorry. You really need to state your intent clearly and use full words.

"for example via m" doesn't mean anything to me...

[piklio]

I still have no idea what you are asking for. I'm sorry. You really need to state your intent clearly and use full words.

"for example via m" doesn't mean anything to me...

"via m " in fact was :via my application" . it was my fault . sorry
i said ur application used IE coockies . and coockies in ur application and IE is same .
how can we segregate these coockies ?
and ur programms dont use IE coockies and IE dont use ur Program Cookies

is it cleary ?

[kleinma]

is it cleary? well not really, but if you are asking about why COOKIES (not cockies, and not coockies) are the same in IE and the same in an app you make when using the webbrowser control, that is because they are one in the same. IE and the browser control use many common things, including settings. This is because the browser control is nothing but a part of IE that Microsoft exposes for you to use in your applications.

[Skatebone]

Hey kleinma,
First of all tnx alot for your project because it was very helpful
I managed to fill in textboxes but i have some problems regarding submit button.

First of all here is the site i am working with:
https://weblogin.runescape.com/login...mentoptions.ws

1) I could not find Mshtml as a reference in the .net framework. (you need to download?)
2) Submit button didnt work for me
3) How can i do all this with web request and using a proxy?

Tnx alot

[stateofidleness]

wow hope you're still reading this kleinma! this thread is priceless!

I have a DVR (digital video recorder) that I am able to log in to via browser to view the cameras. I have successfully logged in once using your provided code which takes me to a second page.

I'm having a problem where, if I click the Submit button manually, it works fine. If I let my code do it (which worked the first time I ran the program), it gives me this "error": "Session ID removed from recorder & browser. " which is specific to the DVR, not IE.

Is it cookies or something that needs to be cleared before loading the page? I'm not sure why it worked the first time, then when I relaunched the program, it is basically telling me my login "timed out"

Thoughts?

[Mailoo]

Thanks for this thread, kleinma! It's been very helpful and it's great thing that you share your knowledge.

Now, to my topic. I need to retrieve anchor text from all links on some website, which contain specified string like "?seed=". The example HTML code:

 <div class="tablename">
                    <h2>blast</h2><table summary="Keywords"><thead><tr><th colspan="2" style="background-color: #505D71">3,886 searches (top 100 only)</th></tr><tr><th width="5&#37;">Searches</th><th class="keyword" width="94%">Keyword</th></tr></thead><tfoot><tr><td>3,886</td><td>total searches</td></tr></tfoot><tbody><tr><td>380</td><td><a href="?seed=scrabble+blast&amp;adult_filter=remove_offensive">scrabble blast</a></td></tr><tr><td>360</td><td><a href="?seed=blast&amp;adult_filter=remove_offensive">blast</a>

Now, as you see, links don't have any id's, so it needs to be done by searching the specified string within all links on the website, check if they contain string and then get only the text, like from :

<a href="?seed=scrabble+blast&amp;adult_filter=remove_offensive">scrabble blast</a>

I need to get only the words "scrabble blast" and do the same to all links on the website.

Now I have the code below, which doesn't work, I use standard WebBrowser from .NET 2.0. When I remove "TextBox2.Text &" from "TextBox2.Text & (element.OuterText) & vbCrLf" it gets text only from the last link on that website. When I run this one here, the app hangs my PC.

 Dim i As Integer

        For i = 0 To WebBrowser1.Document.Links.Count - 1


            For Each element As HtmlElement In WebBrowser1.Document.Links


                If element.GetAttribute("href").Contains("?seed=") Then
                    TextBox2.Text = TextBox2.Text & (element.OuterText) & vbCrLf


                End If

            Next
        Next

And also, I have a question - will it be working better if I use Regex? First, download the HTML source and then use Regex to retrieve what I want?

[abcat]

when i submit a unicode/ascii thingy eg. ★ string, it will display garbage character in the form response. The client program can display the program very well but not the server form.

Do i need to do some conversion, i tried something like
DirectCast(GetCurrentWebForm.item("varName"), mshtml.HTMLInputElement).value = System.Text.Encoding.ASCII.GetString(System.Text.Encoding.unicode.GetBytes("★"))

but it came out other type of character instead.

[VisualBasic6]

How can we supress msgbox button on a webpage?
For example, there is a delete button on a webpage to delete something. However when you click this button it reconfirms before deleting the thing. It pops up a msgbox stating "Are you sure you want to delete the thing?" With Ok or Cancel button.
I do not want to disclose this button to the user and want to press ok button through my program coding. Or lets say when we click on logout on this website it display a msgbox to confirm it first. Is there any way to do supress this box and press ok through program?

[kleinma]

I had done some example code about supressing an alert box, you may be able to use it to do the same for a confirm box as well.

Another approach that SOMETIMES works (it all depends on the site) is to look at exactly what the button does when you click it. For example if the page is setup in such a way that when the button is clicked, it calls a JS routine to confirm with the messagebox, and as long as the JS routine returns true, it submits the form, then you could just directly submit the form via code. Again that all depends on how the specific page is setup and works.

Just note anything I do is in .NET, I notice your user name looks like you use VB6.

[Mailoo]

It's good to know that this thread is still alive

Is there any way to check if certain JS script has finished executing, sort of like "DocumentCompleted" event for page loading? Because DocumentCompleted doesn't work with javascript - there's no way to check if script has finished executing and the data are loaded. I'm using VB.NET 2005. Any help will be appreciated, thanks.

[kleinma]

There is nothing I am currently aware of that would allow you to know when javascript has finished executing, other than if there was something on the page you could keep polling and querying that could tell you.

[Mailoo]

Thanks for the reply. Too bad...right now I'm doing the way you suggested (checking for specific text on page), but it's not really reliable and elegant method.

EDIT: Actually, I've just "discovered" WAY simpler method for checking if JS has finished. It's way better than actually looking through text with MSHTML TextRange (for my needs). InnerHtml method converts the current doc into plain html, thanks to that we can check if specific html element or text has already been loaded by Javascript. You can put the method in Timer or some loop. I'm pasting it here in case anyone needed it:

        If WebBrowser1.Document.Body.InnerHtml.Contains("html element name or text ") = True Then
            Timer1.Enabled = False

           ' code for saving elements from webpage - whatever you want to do with it :)



        End If

Sorry for a little chaotic post, but I'm really excited right now LOL I've been fighting with this **** for like 2 days to solve it :P I guess I've tried to complicate things which are plain simple. And THANKS KLEINMA again for showing me the beautiful world of WebBrowser in VB.NET LOL

[abcat]

is there a way to open a new instance or browser or change the default browser to firefox?

[sonicsx]

Hi, I have a little big trouble with C#, here it goes>
*************************htm code
<FORM onsubmit=centerMap() action=javascript:void(0) >
<INPUT id=txtAddress >

<INPUT type=submit>
</FORM>
**************************************C# code to click on for submit..
webBrowser1.Document.Forms[0].InvokeMember("onsubmit");// it really works!!!

BUTT!!!!!!!! if I had this htm code in the same file.htm*********************
<TABLE>
<TBODY>
<TR>
<TD align=right>Latitud:</TD>
<TD><INPUT class=inputField id=txtLatitude></TD></TR>
<TR>
<TD align=right>Longitud:</TD>
<TD><INPUT class=inputField id=txtLongitude></TD></TR>
<TR>
<TD align=right>Nivel m&#237;nimo de acercamiento:</TD>
<TD>
<INPUT id=txtLevel size=3 value=3> <INPUT onclick=addPoint() type=button>
</TD>
</TR>
<TR>
<TD align=right colSpan=2></TD>
</TR>
</TBODY>
</TABLE>
*****************************I can't simulate the click event, I tried with...
webBrowser1.Document.Forms[0].InvokeMember("onclick");//fail!! I think the problem is there is no form on this part of the code, so I can't perform that...what I know that works is "webBrowser1.Document.???????????.InvokeMember("onclick");" I made some test and this is what I got. Somebody I need some help!!!!!!

[kleinma]

It looks to me like addPoint() is just a javascript function defined somewhere else in the HTML. Why not just make a direct call to addPoint() ? My example code download shows how to execute javascript directly.

[Mailoo]

@sonicsx: Check this out...maybe it'll work in your case too...I use it javascript based form

For Each element2 As HtmlElement In wb.Document.GetElementsByTagName("input")
            If element2.GetAttribute("onclick") = "addPoint()" Then element2.InvokeMember("click")
 
        Next

[tjcj]

kleinma I a noob to vb.net and visual studio, I have a couple of questions I hope you can answer.

Can your code for filling a textarea be used for the regular webbrowser?
Also how do we add the browser from your project to a new project?

Also in a new project I created when I use "mshtml.HTMLTextAreaElement" in your code, visual studios say it is not defined, but this not a problem when I load your project. What could be the reason for this?

[kleinma]

1) regular webbrowser? what is a "regular" webbrowser (other than one that is not google chrome ) Do you mean the standalone IE browser, and not the webbrowser control that you would put on your own form? The answer is no, you can't. You can, but it requires you to write a plugin to IE, not just write code in a VB.NET app that uses a browser control.

2) Right click on your toolbox and select "choose items", the go get coffee while that dialog box loads, and the click the COM Components tab, and find "Microsoft Web Browser" and check it off. This should add the COM browser to your toolbox. Note it does not replace the managed .NET browser.

3) do you have a reference to mshtml set? Do you even have mshtml.dll on your computer? You may need to search for it. I am pretty certain somewhere above in the last 430 posts there is a link to download it if needed

[tjcj]

Thank you for responding kleinma.

So then would this work on the webbroswer control that go on the form?
As for mshtml.dll I have the file, but I didn't have a reference set to mshtml.

1) regular webbrowser? what is a "regular" webbrowser (other than one that is not google chrome ) Do you mean the standalone IE browser, and not the webbrowser control that you would put on your own form? The answer is no, you can't. You can, but it requires you to write a plugin to IE, not just write code in a VB.NET app that uses a browser control.

2) Right click on your toolbox and select "choose items", the go get coffee while that dialog box loads, and the click the COM Components tab, and find "Microsoft Web Browser" and check it off. This should add the COM browser to your toolbox. Note it does not replace the managed .NET browser.

3) do you have a reference to mshtml set? Do you even have mshtml.dll on your computer? You may need to search for it. I am pretty certain somewhere above in the last 430 posts there is a link to download it if needed

[Shane Lowe]

kleinma --

Are there any ways to perform browser manipulation on Compact Framework 3.5? I'm looking to auto-login to a website on a Mobile 6 phone. Your .net demo app was great and is exactly what I was looking for a example but I'm working with the compact framework..Any help would be GREATLY Appreciated. I have the skeleton of my app built but can't get pass this problem.

Thanks!

[sunnydsouza]

First of all,let me THANK you for providing WebpageManupilation code.I am a beginner in VB.net and I found the mshtml method extremely easy and understandable.

I am creating an autosurfing application(where the application automatically surfs webpages and fills in information) to make my work simple.
Now my application using your WebpageManupilation methods which works perfectly on phpBB boards.

However on vBullettin boards it is giving me some problems

Here is my code for entering some info in a form for submitting new thread

Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
 
        DirectCast(GetCurrentWebForm.item("subject"), mshtml.HTMLInputElement).value = "Sunny"    'this TOTALLY DOESNT WORK GIVES ERROR
        wb.Document.All("subject").setAttribute("value", "sdsdsdsdsd")           ' this WORKS!!!
        wb.Document.All("message").setAttribute("value", "sdsdsdsdsd")           ' this DOESNT WORK but no ERROR    
        TextBox1.Text = wb.Document.All("preview").getAttribute("name")
        'DirectCast(GetCurrentWebForm.item("message"), mshtml.HTMLTextAreaElement).value = "Sunny"
        'DirectCast(GetCurrentWebForm.item("cmdSubmitMe", 0), mshtml.HTMLButtonElement).click()
        'GetCurrentWebForm.submit()
    End Sub

The error I get is "Object reference not set to an instance of an object."

the code

wb.Document.All("subject").setAttribute("value", "sdsdsdsdsd")           ' this WORKS!!!
        wb.Document.All("message").setAttribute("value", "sdsdsdsdsd")

somewhat works but still the "message " field in the form doent fill up

The source code of of that form is exactly the same as the one use use to create a new topic in this forum(since its a vBullettin board) except the textarea field is hidden and there is an addtional iframe under it(dunno what it does???)


Subject field

<input class="bginput" type="text" tabindex="1" maxlength="90" size="40" value="" name="subject"/>

MEssage Field

<table width="100&#37;" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr valign="top">
<td class="controlbar">
<textarea id="vB_Editor_001_textarea" dir="ltr" tabindex="1" style="display: none; width: 540px; height: 250px;" cols="60" rows="10" name="message"/>
<iframe id="vB_Editor_001_iframe" tabindex="1" style="border: 2px inset ; width: 540px; height: 250px;">
<html>
<head>
<style type="text/css">
1.wysiwyg { background: rgb(244, 244, 244) none repeat scroll 0% 0% !important; -moz-background-clip: border !important; -moz-background-origin: padding !important; -moz-background-inline-policy: continuous !important; color: rgb(0, 0, 0) !important; font-family: verdana,geneva,lucida,'lucida grande',arial,helvetica,sans-serif !important; font-style: normal !important; font-variant: normal !important; font-weight: normal !important; font-size: 10pt !important; line-height: normal !important; font-size-adjust: none !important; font-stretch: normal !important; -x-system-font: none !important; }
2 p { margin: 0px; } .inlineimg { vertical-align: middle; }
</style>
</head>
<body class="wysiwyg" spellcheck="true" linkifying="true" linkifycurrent="0" linkifymax="0">
<br/>
</body>
</html>
</iframe>
</td>

Somewhat I am having problems clicking "Submit post" and "Preview" buttons.
I dont understand why mshtml doesnt work??
Am I doing something wrong?? PLEASE HELP!!!

[kleinma]

sounds like a bot, so I hope you don't plan to use this on vbforums vbulletin to make posts...

GetCurrentWebForm if I remember correctly is sort of assuming that there is only 1 form on the page. If the page has more than 1 form, then you need to make sure you are grabbing the correct form that actually contains those fields you wish to fill in. wb.document.all() is going to search the entire DOM for the given field. So check that first and post back.

As far as the textarea issue, textareas have an innertext property to set I believe versus a value attribute.

Standard HTML is different for textareas..

<textarea>this is the text in the textarea box</textarea>
<input value="this is text in a textbox">

see the difference?

Let me know how you make out.

[sunnydsouza]

Yeah Thanks for replying....I checked out and indeed there were some hidden forms on the page so I had to change the index value to 3. Now things are working out smoothly except that the TEXTAREA problem still exists.

i tried innertext,innerHTML etc but nothing seems to work.The textarea simply doesnt fill in
I am really getting frustrated with this problem.Please could u have a look on my previous post #434 and see if u could find a solution

I guess the problem is the textarea field is hidden...Does that affect the program??

<td>
<textarea> '''' Its HIDDEN
</textarea>
<iframe>
<html>

</html>
</iframe>

</td>

GOD Knows whose value to set to get the message field filled up correctly.Thats the last of my problems bugging me now


P.S EDIT: Maybe I got the Problem but no solution till now.Maybe this will make things easier for u to find a solution

I found that the hidden textarea field value is only changed after submitting the post.However what really changes when I type my post,is the the value within the body tag which is inside the IFRAME.Maybe I need to set the Ifram's innertext but for some reason am not able to get to the iframe since it doesnt have a name only Id.I tried putting the ID but not Working...throws an exception

Here is an image I took with firebug working in action


Hope this makes it clear of my problem

[EWiggins]

Senerio: I am selecting values on the web page and clicking submit button via code and it returns a Excel sheet. I would like to be able via code get that file and place it in a designated folder.

For Each cdHTMLS In GetCurrentWebDoc.getElementsByTagName("input")
If cdHTMLS.type = "radio" And cdHTMLS.name = "ogrp" And cdHTMLS.value = "GW" Then
cdHTMLS.click()
Exit For
End If
Next

For Each cdHTMLS In GetCurrentWebDoc.getElementsByTagName("input")
If cdHTMLS.type = "radio" And cdHTMLS.name = "output" And cdHTMLS.value = "xls" Then
cdHTMLS.click()
Exit For
End If
Next

For Each cdHTMLS In GetCurrentWebDoc.getElementsByTagName("input")
If cdHTMLS.type = "submit" And cdHTMLS.value = "Submit" Then
cdHTMLS.click()
Exit For
End If
Next

[Abantu]

I am reading text from a webpage using the webrowser control. How do i read the value of the text in the "class" snippet below:

<div id="baseDiv" class="date-20090918 expired-info ">


I want my app to just get 20090918

My code snippet below does not seem to return the value

Dim theElementCollection As HtmlElementCollection = AxWebBrowser1.Document.GetElementsByTagName("div")
        Dim controlName As String
        For Each curElement As HtmlElement In theElementCollection

            controlName = curElement.GetAttribute("id").ToString

            If controlName = "baseDiv" Then
                'this brings empty msg box
                MsgBox(curElement.GetAttribute("class").ToString())
                Return True
                Exit Function
            End If
        Next

Any help is appreciated

[kleinma]

Use "classname" as the string instead of just "class". I know it doesn't make a ton of sense when the actual HTML just has a class attribute, but it works.

[psmith65]

sunnydsouza,

I have bee struggling with the same problem with the textarea's in the iframe. I only just worked it out today after countless frustrations on it. Here is the soltuion that seems to work:

WebBrowser1.Document.Window.Frames("vB_Editor_001_iframe").Document.Body.InnerHtml = "abc"

Allelulia !!!! I can't believe how hard this was to work out.