Manipulate/Change/Form Fill data in webpages using the Webbrowser control

Quote:

---

Originally Posted by beldoy

Hi all im new to this one...

I am trying to make an app in vb.net (part of visual studio 2005) that will do the following:

goto website login page and attempt login
if login fail goto website signup page and attempt signup
if login or signup success then
goto upload page and auto fill upload page and submit.

I would like to do this for say 10 sites.

I have tried 2-3 and managed ok but found that that doing more is very difficult.

Does anyone have any ideas as to how i can keep track within the code of say different upload page form items from the different sites.

Basically I want to make an auto submitter for my video to different video sites and the test for different site form elements and their names are making my code huge...

Just wondering if anyone knows an easier way.

Thanks

---

If you want to do it on 10 sites, I would recommend you manually make sure you have accounts created at each one, then define credentials/path to upload form/form elements based on current url. That's my opinion based on the way I would code a site with a massive user base while trying to discourage bots. I wouldn't use standard form element names to make it easily automated, and even when not trying to make things difficult you will still most likely end up with 5 or more different form field name variations.

Hello, I am trying to autofill a form that has no name or id. There are two forms on the page; the first is hidden while the second is the only visible form.

The visible form is a captcha and I have my COM webbrowser setup so that when it finds a captcha it asks for user input using inputbox()

I have tried countless different ways but I CANNOT find out how to fill the captcha form. Form some reason it refuses to fill.

Please help, oh and if needed I can supply my code and the html I am trying to use it on

hello. im using vb 2008 and trying to make a login program for sfront.ijji.com

currently i have this:

vb Code:

Public Class Form1
 
 
 
 
    Private Sub CmdLogin_Click()
        WebBrowser1.Navigate("http://sfront.ijji.com")
        lblstatus.Text = "Connecting to IJJI Soldier Front...."
    End Sub
    Private Sub WebBrowser1_DocumentComplete(ByVal pDisp As Object, ByVal URL As Object)
        If URL = "http://sfront.ijji.com" Then
            WebBrowser1.Document.All("UserID").InnerHtml = user.Text
            WebBrowser1.Document.All("Password").InnerHtml = pass.Text
            WebBrowser1.Document.All("Sign In").Click()
 
        End If
 
 
 
    End Sub
 
End Class

and the only error i get is this:
Error 1 'Public Event Click(sender As Object, e As System.Windows.Forms.HtmlElementEventArgs)' is an event, and cannot be called directly. Use a 'RaiseEvent' statement to raise an event. C:\Users\Filip\AppData\Local\Temporary Projects\WindowsApplication1\Form1.vb 14 13 WindowsApplication1

please help

I believe you need to use InvokeMember() to call click on a button like you are trying to do:

Something like:

Code:

---

webbrowser1.Document.All("Sign In").InvokeMember("click")

---

If that doesn't work try, "Click" or "click()", but I am pretty sure it is the one in the above code block.

Quote:

---

Originally Posted by bl34ch127

Hello, I am trying to autofill a form that has no name or id. There are two forms on the page; the first is hidden while the second is the only visible form.

The visible form is a captcha and I have my COM webbrowser setup so that when it finds a captcha it asks for user input using inputbox()

I have tried countless different ways but I CANNOT find out how to fill the captcha form. Form some reason it refuses to fill.

Please help, oh and if needed I can supply my code and the html I am trying to use it on

---

I can't help you with bypassing a captcha.. The point of the code I provided is for automating websites that permit it. captcha is an obvious sign of a page that should NOT be automated.

hai.., how can i fill textbox inside frame? i had arround searching for this..,but still no result
thx before.

you can try looking through the past pages of this thread. I am pretty sure frame access was discussed at some point.

Quote:

---

Originally Posted by kleinma

I can't help you with bypassing a captcha.. The point of the code I provided is for automating websites that permit it. captcha is an obvious sign of a page that should NOT be automated.

---

The user will still be watched the page and inputting the captcha manually. I just want to fill the textbox!

nvm ^^:wave:

Found out how to check the first radio button:

Code:

---

WebBrowser1.Document.GetElementsByTagName("Input").Item(0).SetAttribute("checked", "checked")

---

anyone that can help me out saving the dynamic picture?

I have nothing to say about captcha images. Period. Please do not ask for info here about bypassing, downloading, saving, or interpreting captcha images. They exist for a reason, and that reason is the sites they exist on DO NOT want to be automated, and want a human there doing the data entry.

is there a way to disable popup windows using your codes?

if you are using the activex browser control (the one this article was written for), then you can handle the NewWindow3 event, which exposes the URL that is about to be popuped up, and exposes a cancel event arg you can set to true to cancel the popup.

If I remember correctly, the NewWindow3 event only fires on Windows XP SP2 and higher (which by no coincidence was when MS added the popup blocker to IE)

Hi dear kleinma
tnx for ur hlpful project !
but i have 2 question

first :

Code:

---

DirectCast(GetCurrentWebForm.item(ID or Name), mshtml.HTMLInputElement).click()

---

in this syntax we must use id on name of submt button . but sometimes for submit buttons there is no ID/Name . how can i click this button ?


second :
how u find the name or ID of button in web page ? i save the page and then browse page in frontpage or i read html codes and find name . is there any speedy way or any extension for browser that help us for finding button name faster than now ?

thanks in advance sir

If an element has no id or name attribute, then the best you can do is grab all controls of a specific element type (like grabbing all input elements) and then looping them to see if one of the input elements has a type=submit attribute.

You can further verify with checking additional known attributes, like the value attribute which would have the submit buttons text.

As far as "how do you find elements names" and all that, if you use IE8, you can hit F12 when you are on a webpage, and it brings up a tool to go through the webpage DOM easily and find things like element names. There is even an option to allow you to simply click on the element you want on the page, and the utility tells you the info about it.

If you don't have IE8, you can get a similar tool via download (ie developer toolbar) for IE6 and IE7. I believe firefox also has similar tools, however I don't recommend using firefox when working with web automation, because some sites deliver different content (and have different behavior) based on browser version, and if you use firefox, you may find different results when using the IE based browser control in a windows app.

so great !
tnx
i solved my problem and i can login :)

with this code :

with or loop i checked all value of items and then grab index of item that value equal to "log in" for example .
then please ignore my last private message :)

for grabbing id and name in page tnx 2 . great idea .
//
one orher small question :

Code:

---

Private Sub wb_NavigateComplete2(ByVal sender As Object, ByVal e As AxSHDocVw.DWebBrowserEvents2_NavigateComplete2Event) Handles wb.NavigateComplete2
        txt_page_source.Text = GetCurrentWebDoc.documentElement.outerHTML
    End Sub

---

i use this code for grabbing page source
but when i call wb.navigate 1 time , then txt_page_source changed more times !
it means with one navigation , wb_NavigateComplete2 rase more times !
for wat ?
i need only last event and when everythings is complete !

tnx again sir

naviatecomplete events fire when the initial navigation is done, not when the page has fully loaded into the browser. Use the DocumentCompleted event for that.

Either one of these events can fire multiple times if the page has frames or something similar where it is actually loading its content from different places.

Quote:

---

Originally Posted by kleinma

Either one of these events can fire multiple times if the page has frames or something similar where it is actually loading its content from different places.

---

tnx
but how can i know when page is loaded complete ?
:(

kleinma Thanks for the project! I'm a noob to VB.NET and Visual Studios 2008, but I have been interested in apps like this project and it will help speed up my learning. :)

hi

how can make cockies dedicated from internet explorer ?

I have no idea what that statement means.

Quote:

---

Originally Posted by kleinma

I have no idea what that statement means.

---

tnx mate
but it means i need login to gmail for example via m
and i dont want if anybody open IE and type Gmail.com can use my session on gmail :)

I still have no idea what you are asking for. I'm sorry. You really need to state your intent clearly and use full words.

"for example via m" doesn't mean anything to me...

Quote:

---

Originally Posted by kleinma

I still have no idea what you are asking for. I'm sorry. You really need to state your intent clearly and use full words.

"for example via m" doesn't mean anything to me...

---

:(

"via m " in fact was :via my application" . it was my fault . sorry
i said ur application used IE coockies . and coockies in ur application and IE is same .
how can we segregate these coockies ?
and ur programms dont use IE coockies and IE dont use ur Program Cookies

is it cleary ?

is it cleary? well not really, but if you are asking about why COOKIES (not cockies, and not coockies) are the same in IE and the same in an app you make when using the webbrowser control, that is because they are one in the same. IE and the browser control use many common things, including settings. This is because the browser control is nothing but a part of IE that Microsoft exposes for you to use in your applications.

Hey kleinma,
First of all tnx alot for your project because it was very helpful :)
I managed to fill in textboxes but i have some problems regarding submit button.

First of all here is the site i am working with:
https://weblogin.runescape.com/login...mentoptions.ws

1) I could not find Mshtml as a reference in the .net framework. (you need to download?)
2) Submit button didnt work for me
3) How can i do all this with web request and using a proxy?

Tnx alot :)

wow hope you're still reading this kleinma! this thread is priceless!

I have a DVR (digital video recorder) that I am able to log in to via browser to view the cameras. I have successfully logged in once using your provided code which takes me to a second page.

I'm having a problem where, if I click the Submit button manually, it works fine. If I let my code do it (which worked the first time I ran the program), it gives me this "error": "Session ID removed from recorder & browser. " which is specific to the DVR, not IE.

Is it cookies or something that needs to be cleared before loading the page? I'm not sure why it worked the first time, then when I relaunched the program, it is basically telling me my login "timed out"

Thoughts?

Thanks for this thread, kleinma! It's been very helpful and it's great thing that you share your knowledge.

Now, to my topic. I need to retrieve anchor text from all links on some website, which contain specified string like "?seed=". The example HTML code:

Code:

---

<div class="tablename">
                    <h2>blast</h2><table summary="Keywords"><thead><tr><th colspan="2" style="background-color: #505D71">3,886 searches (top 100 only)</th></tr><tr><th width="5&#37;">Searches</th><th class="keyword" width="94%">Keyword</th></tr></thead><tfoot><tr><td>3,886</td><td>total searches</td></tr></tfoot><tbody><tr><td>380</td><td><a href="?seed=scrabble+blast&amp;adult_filter=remove_offensive">scrabble blast</a></td></tr><tr><td>360</td><td><a href="?seed=blast&amp;adult_filter=remove_offensive">blast</a>

---

Now, as you see, links don't have any id's, so it needs to be done by searching the specified string within all links on the website, check if they contain string and then get only the text, like from :

Code:

---

<a href="?seed=scrabble+blast&amp;adult_filter=remove_offensive">scrabble blast</a>

---

I need to get only the words "scrabble blast" and do the same to all links on the website.

Now I have the code below, which doesn't work, I use standard WebBrowser from .NET 2.0. When I remove "TextBox2.Text &" from "TextBox2.Text & (element.OuterText) & vbCrLf" it gets text only from the last link on that website. When I run this one here, the app hangs my PC.

Code:

---

Dim i As Integer

        For i = 0 To WebBrowser1.Document.Links.Count - 1


            For Each element As HtmlElement In WebBrowser1.Document.Links


                If element.GetAttribute("href").Contains("?seed=") Then
                    TextBox2.Text = TextBox2.Text & (element.OuterText) & vbCrLf


                End If

            Next
        Next

---

And also, I have a question - will it be working better if I use Regex? First, download the HTML source and then use Regex to retrieve what I want?

when i submit a unicode/ascii thingy eg. ★ string, it will display garbage character in the form response. The client program can display the program very well but not the server form.

Do i need to do some conversion, i tried something like
DirectCast(GetCurrentWebForm.item("varName"), mshtml.HTMLInputElement).value = System.Text.Encoding.ASCII.GetString(System.Text.Encoding.unicode.GetBytes("★"))

but it came out other type of character instead.

How can we supress msgbox button on a webpage?
For example, there is a delete button on a webpage to delete something. However when you click this button it reconfirms before deleting the thing. It pops up a msgbox stating "Are you sure you want to delete the thing?" With Ok or Cancel button.
I do not want to disclose this button to the user and want to press ok button through my program coding. Or lets say when we click on logout on this website it display a msgbox to confirm it first. Is there any way to do supress this box and press ok through program?

I had done some example code about supressing an alert box, you may be able to use it to do the same for a confirm box as well.

Another approach that SOMETIMES works (it all depends on the site) is to look at exactly what the button does when you click it. For example if the page is setup in such a way that when the button is clicked, it calls a JS routine to confirm with the messagebox, and as long as the JS routine returns true, it submits the form, then you could just directly submit the form via code. Again that all depends on how the specific page is setup and works.

Just note anything I do is in .NET, I notice your user name looks like you use VB6.

It's good to know that this thread is still alive :)

Is there any way to check if certain JS script has finished executing, sort of like "DocumentCompleted" event for page loading? Because DocumentCompleted doesn't work with javascript - there's no way to check if script has finished executing and the data are loaded. I'm using VB.NET 2005. Any help will be appreciated, thanks.

There is nothing I am currently aware of that would allow you to know when javascript has finished executing, other than if there was something on the page you could keep polling and querying that could tell you.

Thanks for the reply. Too bad...right now I'm doing the way you suggested (checking for specific text on page), but it's not really reliable and elegant method.

EDIT: Actually, I've just "discovered" WAY simpler method for checking if JS has finished. It's way better than actually looking through text with MSHTML TextRange (for my needs). InnerHtml method converts the current doc into plain html, thanks to that we can check if specific html element or text has already been loaded by Javascript. You can put the method in Timer or some loop. I'm pasting it here in case anyone needed it:

Code:

---

        If WebBrowser1.Document.Body.InnerHtml.Contains("html element name or text ") = True Then
            Timer1.Enabled = False

          ' code for saving elements from webpage - whatever you want to do with it :)



        End If

---

Sorry for a little chaotic post, but I'm really excited right now LOL :D I've been fighting with this **** for like 2 days to solve it :P I guess I've tried to complicate things which are plain simple. And THANKS KLEINMA again for showing me the beautiful world of WebBrowser in VB.NET :D LOL

is there a way to open a new instance or browser or change the default browser to firefox?

Hi, I have a little big trouble with C#, here it goes>
*************************htm code
<FORM onsubmit=centerMap() action=javascript:void(0) >
<INPUT id=txtAddress >

<INPUT type=submit>
</FORM>
**************************************C# code to click on for submit..
webBrowser1.Document.Forms[0].InvokeMember("onsubmit");// it really works!!!

BUTT!!!!!!!! if I had this htm code in the same file.htm*********************
<TABLE>
<TBODY>
<TR>
<TD align=right>Latitud:</TD>
<TD><INPUT class=inputField id=txtLatitude></TD></TR>
<TR>
<TD align=right>Longitud:</TD>
<TD><INPUT class=inputField id=txtLongitude></TD></TR>
<TR>
<TD align=right>Nivel m&#237;nimo de acercamiento:</TD>
<TD>
<INPUT id=txtLevel size=3 value=3> <INPUT onclick=addPoint() type=button>
</TD>
</TR>
<TR>
<TD align=right colSpan=2></TD>
</TR>
</TBODY>
</TABLE>
*****************************I can't simulate the click event, I tried with...
webBrowser1.Document.Forms[0].InvokeMember("onclick");//fail!! I think the problem is there is no form on this part of the code, so I can't perform that...what I know that works is "webBrowser1.Document.???????????.InvokeMember("onclick");" I made some test and this is what I got. Somebody I need some help!!!!!!

It looks to me like addPoint() is just a javascript function defined somewhere else in the HTML. Why not just make a direct call to addPoint() ? My example code download shows how to execute javascript directly.

@sonicsx: Check this out...maybe it'll work in your case too...I use it javascript based form

vb Code:

For Each element2 As HtmlElement In wb.Document.GetElementsByTagName("input")
            If element2.GetAttribute("onclick") = "addPoint()" Then element2.InvokeMember("click")
 
        Next

kleinma I a noob to vb.net and visual studio, I have a couple of questions I hope you can answer.

Can your code for filling a textarea be used for the regular webbrowser?
Also how do we add the browser from your project to a new project?

Also in a new project I created when I use "mshtml.HTMLTextAreaElement" in your code, visual studios say it is not defined, but this not a problem when I load your project. What could be the reason for this?