Manipulate/Change/Form Fill data in webpages using the Webbrowser control

[Massino]

Hi Kleinma!

Believe it or not I have the same problem...
But I couldn't resolve mine by this.
I'd like to autofill the two login fields(username,password) and submit it automatically,
but the above codes doesn't work somehow.
Would you try it with this url for me as well? : http://www.lemuria.hu
Thanks in advance, Massi.

[kleinma]

What code did you try?

[Massino]

Well I already figured out the first part,
I could succesfully fill the username and the password
elements with this code :
WebBrowser1.Document.GetElementsByTagName("Input").Item(0).SetAttribute("Value", "Username")
WebBrowser1.Document.GetElementsByTagName("Input").Item(1).SetAttribute("Value", "Password")
Now the next thing should be to figure it out how to "click" the submit button to submit the form with the new attributes. Cause VB says : i cannot raise the click event directly, only by the raiseevent handler...
And I forgot to tell ya, I'm using Visual Studio 2005

[kleinma]

My example code shows how to click a button via code.

[Massino]

It says : Error 1 'submit' is not a member of 'System.Windows.Forms.HtmlElement'. D:\_Temporary_\Visual Basic Programs\Lemuria\Lemuria\Lemuria\Form1.vb 11 9 Lemuria
I tried the click and the submit as well.
Sometimes it says : Value does not fall within the expected range.
Now I'm stucked.

[Massino]

Thank God I've managed it finally.
Here's the full code what works for me :
Public Class Form1
Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
WebBrowser1.Document.GetElementsByTagName("Input").Item(0).SetAttribute("Value", "Username")
WebBrowser1.Document.GetElementsByTagName("Input").Item(1).SetAttribute("Value", "Password")
WebBrowser1.Document.GetElementsByTagName("Input").Item(2).InvokeMember("Click")
End Sub
End Class

[kleinma]

You probably aren't using my example code, because that uses the MSHTML library and you can directly access members and click buttons and such using that. It looks like you are using the standard 2.0 Webbrowser control, which my example code doesn't use.

[Massino]

If you mean I didn't add the reference to mshtml, I did.

[kleinma]

No I didn't mean you didn't add the reference, I just mean it doesn't look (from the code you posted) that you actually used MSHTML at all.

[Winla]

Hi kleinma
I download your code and it works great! thx
Now I have two related questions for a web site like
www.whitepages.com:

1)for this submit click code

DirectCast(GetCurrentWebForm.item("cmdSubmitMe", 0), 
mshtml.HTMLButtonElement).click()

to work , you have to find the button id, but for above web site, the submit
buttons has no id to refer to. How to go around this?

2)I want to extract all the names and address, phone numbers from
searching page into a datagridview control. What is the best way to do it?
I am thinking of regex pattern match, but for this web site, the returned
source code is too complicated to handle.

Thanks

[kleinma]

You could just grab the forms themselves, and call the submit method of the form, instead of using the click method of the button.

Regex or just a complex parsing routine may be your best bet... you could probably trim down the returned HTML source and just find the table or whatever that all the data you want to parse is located in, and just parse that part of the HTML.

[Winla]

thx for reply. Could you be more specific about how to parse and trim down the returned HTML source to the table I need. A basic sample code would be very helpful. I know you can get all links by using mshtml as you suggested in other posts but it is more complicated here for my purpose. btw, I am using VS 2005.

[ImaVBNoob]

How would this work with VB 6.0?

[FlyingBear]

Please let me know if you have any questions/comments/suggestions.

I can add more functionality examples if someone can think of one....

Is it possible to manipulate a file uploader? the html is:
==========================
<div>
<input type="file" name="uploadfile">
</div>
==========================

just like the one for 'manage attachment'.

Can I set the file name directly from program instead of click the 'file' button? I tried to change its value, outerhtml, etc... all seem don't work.

thanks


bear

[kleinma]

I would imagine you would simply just set the "value" attribute of the input tag to whatever file name you wanted.

Follow the example code for auto filling a textbox, but apply it to an input file box instead... it should work the same exact way.

[FlyingBear]

no, I tried it. If I just set the value in program, it will reset the uploader, which means the file textbox becomes blank. I use .net web browser, did not try the activex browser yet. If I input the file name, then in debug, I can see the value of the uploader is changed to the file name.


thanks


bear

[Passgad]

Hi kleinma,

I found great info in this post but you didn't answer how to access a frame with the webbrowser.

I have a page with 2 frames and I want the value of a textbox (txtVersion) in the second frame.

Thanks,
Pass

[kleinma]

do you have a URL I can test with?

[Passgad]

Sorry, it's not online and it uses the windows authentication...

But basically, I have an application like yours with the webbrowser control who refers the a basic html page with 2 frames in it. Each frame refers to a simple aspx page. The second page contains the txtVersion that I need the value to set a textbox in my application.

Thanks,
Pass

[Passgad]

I tried this :

Dim document As mshtml.HTMLDocument = DirectCast(WebBrowser1.Document, mshtml.HTMLDocument)
Dim form As mshtml.HTMLFormElement = DirectCast(document.forms.item(0), mshtml.HTMLFormElement)
Dim Version As String = DirectCast(form.item("txtVersion"), mshtml.HTMLInputElement).value
 
MsgBox("Version : " & Version)

I get this error :

System.NullReferenceException: Object reference not set to an instance of an object.

For now, the page doesn't contain a frameset. I wanted to try to access the textbox in the form first...

Also, will all this will work in VB.Net 2003 ?

[kleinma]

Yes all code should work in 2003, as the example uses the COM webbrowser, not the windows forms webbrowser which was introduced in 2005.

The windows forms webbrowser is basically a wrapper around the COM interface, run all in managed code, however it has many short comings and missing features (I assume for full managed code compatibility). For example (just one of many) the 2.0 webbrowser doesn't have a navigate_error event, but the COM one does.

As far as your error, I am not sure why you are getting it, as it could be related to the specific HTML page you are parsing the data out of. If you run the example code, you will see that it does infact work to get text from a textbox in a webpage.

[Passgad]

Ok, I found a way to get the value a the txtVersion when the page is not in a frameset :

Private Sub AxWebBrowser1_DownloadComplete(ByVal sender As Object, ByVal e As System.EventArgs) Handles AxWebBrowser1.DownloadComplete
    Dim document As mshtml.HTMLDocument = DirectCast(AxWebBrowser1.Document, mshtml.HTMLDocument)
    Dim form As mshtml.HTMLFormElement = DirectCast(document.forms.item(0), mshtml.HTMLFormElement)
    Dim Version As String = DirectCast(form.item("txtVersion"), mshtml.HTMLInputElement).value
    MsgBox("Version : " & Version)
End Sub

Do you have an idea when the page is in the second frame of a frameset ?

This doesn't work :

Private Sub AxWebBrowser1_DownloadComplete(ByVal sender As Object, ByVal e As System.EventArgs) Handles AxWebBrowser1.DownloadComplete
    Dim document As mshtml.HTMLDocument = DirectCast(AxWebBrowser1.Document, mshtml.HTMLDocument)
    Dim frame1 As mshtml.HTMLFrameElement = DirectCast(document.frames.item(0), mshtml.HTMLFrameElement)
    Dim frame2 As mshtml.HTMLFrameElement = DirectCast(document.frames.item(1), mshtml.HTMLFrameElement)
    Dim form As mshtml.HTMLFormElement = DirectCast(frame2.forms.item(0), mshtml.HTMLFormElement)
    Dim Version As String = DirectCast(form.item("txtVersion"), mshtml.HTMLInputElement).value
    MsgBox("Version : " & Version)
End Sub

[kleinma]

If you want to make me a little dinky 2 frame HTML page, I will be more than happy to use it here to write some code to grad the data out of it.

The issue MAY be in the fact that inside each frame might be considered a document, so you would have to grad the document of the frame, then the form from the document. (and then the textbox element)

If you want to upload me a sample page to use for analysis, I can confirm that.

[Passgad]

I will try to PM the source code tonight or tomorrow, but please, do not share... Just take what you need.

I also found something interresting; you cannot grab value of a textbox with property visible = false...

Thanks,
Pass

[kleinma]

do you mean an ASP.NET textbox with visible set to false? If so, that is because ASP.NET simply doesn't even render HTML for it when its not visible.

If you are talking about a regular <input type=text> tag, then visible isn't a valid HTML property of the tag anyway.

Also, I don't need your whole source, I just need a sample HTML page with frames in it, so I can test grabbing data in the frames. If you provide one, I will take a look at it and try to provide an answer.

[mailtosubbu7]

Hello sir,

I am new member to this forum.
I have one doubt in vb.net 2005.
How to set the file upload element value using web browser control?.
Thank you for advance.

Regards,
Subu

[mailtosubbu7]

I am tried using setattribute and other command in .net 2005 but i can set the value of file upload element but i am trying send key to set the value but this is very slow.
There is anyother solution?.

[nnolte]

I have a newbie question that hopefully the experts here can answer quickly without me being too much of a nuisance. In the past, I have created simple apps to fill in username and password fields using the webbrowser control in VB.

However, now I am trying to get past a page that implements frames, one of which contains the username and password fields in question. For some reason the method I have used in the past (with varying field names, obviously), namely:

With wWeb.Document
.All("username").Value = "loginid"
.All("password").Value = "pword"
.getElementById("Submit").Click
End With

doesn't work with frames. The web page in question will not let me navigate just to the frame that I need (probably for security reasons).

My questions are:

1. Can anyone point me to where I can get some documentation for what properties and methods can be used under the Document property (i.e. some documentation on WebBrowser.Document.GetElementById or WebBrowser.Document.GetElementByName or WebBrowser.Document.Frames(x) or other properties and methods). I can't help but think if I could get to that information I could figure it out.

2. Can anyone figure out how I should modify the above code to satisfy this web page --> https://sportal.uspto.gov/authentica...rLocalEPF.html

I would prefer to use VB6 but also have access to VB.NET 2005.

Thanks for any help you can render.

[kleinma]

I am not aware of any easy way to auto fill content within frames.

I also didn't find anything that looked promising in my searches.

If I do figure out a way to manipulate content in frames, I will update this example code accordingly

[coheed]

Ok... this should be easy, but I'm having a terrible time...

kleinma, I've looked over your sample and still can't get this to work:

All I'm trying to do is fill in the info on the paypal website and log in. The email field is named "login_email", the password field is "login_password", and the submit button is named "submit.x"

I've scrapped all of my repeated attempts, because I haven't been able to get anything to work.

I've ended up with this, even though I know it's not exactly how you coded your sample, I ended up straying away because I couldn't get yours to work.

        If IsBrowserBusy(WebBrowser1) = False Then
            Dim HTML As HtmlDocument
            Dim HTMLS As mshtml.HTMLInputButtonElement
            HTML = WebBrowser1.Document
            HTML.All.Item("login_email").InnerText = "***"
            HTML.All.Item("login_password").InnerText = "*****"
            HTML.All.Item("submit").Click()
            For Each HTMLS In WebBrowser1.Document.GetElementsByTagName("input")
                If HTMLS.type = "submit" And HTMLS.value = "Log In" Then
                    HTMLS.click()
                    Exit For
                End If
            Next


        End If

IsBrowserBusy is just a function I made to return whether or not the browser is busy. With this code, the textboxes were filled in, but I can't get it to submit, before even compiling i get this error:

"Error 1 'Public Event Click(sender As Object, e As System.Windows.Forms.HtmlElementEventArgs)' is an event, and cannot be called directly. Use a 'RaiseEvent' statement to raise an event."

Ugh, this is frustrating! Hopefully you have some time to lend a hand-

[kleinma]

Just use the 2005 webbrowser, and use the InvokeMember() method

I have attached example code.

I actually plan on rewriting the original example in this thread using the 2005 webbrowser (even though it still does have some limitations that the COM browser doesn't)

however it also does have some advantages.

So anyway take a look at the attached zip file, you will see its only a few lines of code to do a paypal auto login.

[coheed]

Perfect, you're the man!

Any clue where I can find a good writeup on the 2005 web browser?

[kleinma]

well its a control that is actually a built in part of the .NET 2.o framework, so a good starting place would be to go through the MSDN help documentation on it. It will list all the properties, methods, events, etc...

[ggodwin]

Sorry for the mistake.

[kleinma]

Sorry but you are using VB6 and IE automation, and all this example code is done with VB.NET, and ActiveX/.NET webbrowser controls.

[GP_Agrestic]

Hi fellows,
i am trying to do a similar stuff and i am very much successful as of yet. I am building a desktop application in C# which will submit form by HttpWebRequest and HttpWebResponse in addition to other functionalities. i came across a problem when i was trying to submit form on a ASP.Net site as it wont yield the page after login. One reason a friend told me was that i was not incorporating ViewState value in the data being sent. Now as i am building a desktop application and i need UrlEncode function of HttpUtility class for converting viewstate value into base64 equivalent(reason being that this representation caters for escape sequences). I cant access this class in a desktop application.
Can anyone tell me that am i going on right track and if yes what else i need to do.
ur assistance will be of great help to me.

[ArmchairAthlete]

I'm using VS2005 and VB.NET, .NET framework 2.0.

I've read most of this thread... in hopes to be able to "click" on HtmlElements with a WebBrowser, without luck. The issue is I don't want to do this in a windows form app.

I've tried both the InvokeMember("click") method and the reflections/DomElement, which work fine if the browser is on a Windows Form but just hang and don't do anything in my ASP.NET application (where I instantiate the browser myself).

I can get the browser to navigate to pages however. I have to be running in a separate thread with ApartmentState STA.

The AxSHDocVw.AxWebBrowser() ends up throwing this out trying to navigate:
An unhandled exception of type 'System.Windows.Forms.AxHost.InvalidActiveXStateException' occurred in AxInterop.SHDocVw.DLL

[kleinma]

wait, so what exactly are you trying to do? Invoke your own clicks in an ASP.NET application?

[ArmchairAthlete]

Yea that's what I need to do. I wouldn't need the WebBrowser object except for javascript use in some pages I'll need to crawl. The dependence on a web browser object doesn't lend itself well to the parallel nature of crawling, but usually httpwebrequest/response is what I'll use.

We have an old VB6 winform app that is able to do this fine, but in a winform. I'd like to have a sort of independent library that I can choose to use wherever, (asp.net, cmd line, etc). Wanted to use the web browser despite not having one actually visible as it normally would be, and still be able to simulate clicks.

[kleinma]

why not just use a winform and put the webbrowser off screen?