Manipulate/Change/Form Fill data in webpages using the Webbrowser control

[kleinma]

All the time I see people asking how to autofill in forms on webpages or how to click a button in a webpage that is hosted in the webbrowser control, via VB code.

It is really not all that difficult, once you get around the casting of types to the correct kind when using the COM reference.

This example project highlights the following manipulations (as well as opening the door to manipulate just about anything on a webpage that is possible)

* Getting a value of an HTML input element
* Setting a value of a HTML input element
* Getting a value of an HTML text area
* Setting a value of a HTML text area
* Set HTML radio buttons selected or not
* Click an HTML button
* Submit an HTML Form
* Get the sources for all images in HTML document
* Get and display all the links in the HTML document
* Alter non form elements (like changing the color of a DIV tag)
* Getting values from an HTML Select element (value and selected index)
* Display page HTML source
* Run a javascript that is in the HTML Page
* click a checkbox in an HTML form
* Print the current page (with printer selection dialog)
* Added 8/24/2006 - Highlight webpage text via code

This specific example code is done using

Visual Studio 2005 (VB.NET)
However I use the COM WB Control, not the built in .NET 2.0 WB Control (because its functionality is rather limited)
I also use a reference to the MSHTML scripting engine, which is what allows you to parse a webpage into all its elements so you can manipulate them.


Please let me know if you have any questions/comments/suggestions.

I can add more functionality examples if someone can think of one....

[Mc_F]

I need something like that for VS 2003. As much as I like VS, I hate the way MS uses upward compatibility. Basically all I want to do is learn how to programmatically fill in fields on a webpage from within my application.

Here I am attempting a login attempt on a website. (username text box = edit_username)
So far:

Dim CookieContainer As New CookieContainer
Dim strPost As String = "edit_username=my_username&edit_password=my_password" 
Dim myWriter As StreamWriter
Dim objRequest As HttpWebRequest = CType(WebRequest.Create(uiURL.Text), HttpWebRequest)
Dim cookies As CookieContainer = New CookieContainer
  'objRequest.AllowWriteStreamBuffering = True
   objRequest.Method = "POST"
   objRequest.ContentType = "application/x-www-form-urlencoded"
   objRequest.CookieContainer = cookies
 
   myWriter = New StreamWriter(objRequest.GetRequestStream())
   myWriter.Write(strPost)
   myWriter.Close()
 
Dim objResponse As HttpWebResponse = CType(objRequest.GetResponse(), HttpWebResponse)
Dim str As StreamReader
   str = New StreamReader(objResponse.GetResponseStream)
   uiInfo.Text = str.ReadToEnd
   myWriter.Close()
 
   objRequest.GetResponse().Close()

Thanks.

[kleinma]

well my example is using the webbrowser control, which displays the HTML page rendered the same was the IE browser does.

What it looks like you are doing, is a more "behind the scenes" approach, by posting form data to a URL, and getting the resulting response.

Is this what you want? or would you rather use the webbrowser control and auto fill the fields and submit the form?

[Mc_F]

Lets say I have a text box on a form that I put the HTML source into after issuing an HTTPWebRequest and receiving a response.

Below that I have a WebBrowser control that shows the page.

The textbox with the HTML is not the same as the WebBrowser.

In other words, the HTML is a login page whereas the WebBrowser shows what the HTML should be after log in.

Is this what you want?

YES

[kleinma]

why not just automate the login itself by autofilling the fields right in the browser control, and then submitting the form. that way you don't have to use webrequests at all.

Check out this example I had done for someone a while ago in this post
http://www.vbforums.com/showthread.p...ght=webbrowser
it is done in .NET 2003, so you can open it and check it out.

[Mc_F]

WebBrowser.Document.forms.item(, 0).elements("field1").value = "myvalue"

Why doesn't anything after the Document object get acknowledged?
It won't work for me this way. Could you explain?

[kleinma]

do you mean why doesn't intellisense come up? Its because it uses late binding since the .NET framework sees everything past the webbrowser itself as a plain object (including the document)

to get around this, and get intellisense for all the elements of the HTML form, you would have to add a reference to MSHTML which is the .NET wrapper DLL around microsofts HTML parsing engine. Then you can cast the webbrowser.document to a MSHTML.HTMLDocument and gain access to its stuff through there. This is what I did in the code bank example above, but its for 2005 and not 2003 (although it would work just the same in 2003)

[Mc_F]

'Cast the web browser document
         With DirectCast(wb.Document, mshtml.HTMLDocument)
            .forms.item(, 0) 'as far is it goes with intellisense.
         End With

the name of my field I want to access is "edit_username" and I want to put the value "guest" in it.

Thanks.

[kleinma]

ahh... well all you SHOULD need to do is this:

even though intellisense doesn't work for it.. the code will still run

WebBrowser.Document.forms.item(,0).elements("edit_username").value = "guest"

This code basically translates as

In the WebBrowser's Document, I want the First Form (which is index 0, hence the forms.item(,0) part) and I want the element in this form called "edit_username", and I want to make its value "guest"

If you want to give me the URL you are trying to autofill, I can modify the example code to reflect that site.

[Mc_F]

I see how that works, but this is broken down like this, so you may help me with a solution:

go here: http://bright-shadows.net/challenges...ted/tryout.php
there is the login form

once logged in, there will be text on the screen and if you 'view source' there are no tags or anything, just the text that you see on the screen. This text is what I want to put in my text box.

So if at all possible, I would like to automate the process by passing login info to the page and getting the text in my text box. Is there a way to determine if the login page is showing or just the text after the automation process?

Thanks.

[kleinma]

well I would need a login/password.

I totally understand if you don't want me to use yours, I can only tell you that I wouldn't use it for anything (I don't even know what bright shadows is)

If you want to PM me with your login info, I will see if I can make you a sample that works. You can then change your password for security reasons after that.

If you don't want to give me the login info, I totally understand, however I am not going to make myself an account just to make sample code for you (no offense or anything.. its just a bit much )

[kleinma]

I got your PM... I get something like

The "text" is: 'some random text'

when I log in.. is that what should be displayed?

[Mc_F]

Exactly. I just want to auto login by code and retrieve that text.

[kleinma]

ok... ill be back

[Mc_F]

Depending on what you come up with I would really like to use something along the lines of:
using the httpwebrequest / httpwebresponse along with using the credentials property.
Below sends the user/password.

' Create an empty instance of the NetworkCredential class.
         Dim myCredentials As New NetworkCredential("", "", "")
         'myCredentials.Domain = domain
         myCredentials.UserName = "my_username"
         myCredentials.Password = "my_password"
 
         ' Create a WebRequest with the specified URL. 
         Dim myWebRequest As WebRequest = WebRequest.Create(uiURL.Text)
         myWebRequest.Credentials = myCredentials
         uiInfo.Text = ("User Credentials: " & vbcrlf & "Username=" & myCredentials.UserName & "password=" & myCredentials.Password)
 
         ' Send the request and wait for a response.
         Dim myWebResponse As WebResponse = myWebRequest.GetResponse()
 
         '=================================================
         Dim sr As StreamReader
         sr = New StreamReader(myWebResponse.GetResponseStream)
         uiInfo.Text += str.ReadToEnd 'not getting the desired HTML
         ' Release the resources of the response object.
         myWebResponse.Close()

thanks for your help.

[khuntballa]

Hi I am a VB noob, but I am trying to learn and I have been beating my head over the past few days trying to come up with a solution for my problem.

I am making a web browser utility for a text based game that does a few things.

1. It logs into a a site and submits- Finally figured that out when I realized 2005 didn't have form.submit function I had to use invokememeber("submit")

2. It goes to a ranking page submitted by an upper and lower bounds in increments of 50 and retrieves the links of players there. The players links are in the form of stats.php?id=sixdigitnumbergoeshere.

3. It will go one by one and send recruiting messages to the existing players and will have to submit a form again. In order to submit that form one will have to type in a 6 letter captcha that should be displayed on the page.


All that I am worrying about for now is the 2nd part. I have tried many things , even tried using your GetCurrentwebdoc function and my latest attempt at grabbing links was this:

Dim pageSource As String
        pageSource = Webbrowser1.Document.All("stats.php?id=").InnerHtml

again the idea from these forums because I am just desperate at this point.

If any help could be provided it would be much appreciated. Thank you.


Kevin

[andrewteg]

Just wanted to say thanks for this. I have a website that needs certain parts on a CD for distribution and this has come in very valuable! I appreciate the effort and was happily surprised I could get it working on my own just based on your examples even though I haven't used VB regularly in 6+ years!

[kleinma]

Just wanted to say thanks for this. I have a website that needs certain parts on a CD for distribution and this has come in very valuable! I appreciate the effort and was happily surprised I could get it working on my own just based on your examples even though I haven't used VB regularly in 6+ years!

Glad to hear it helped you out.

[andrewteg]

kleinma (or others),

Has anyone had any success in going the other way? By this I mean clicking on something in a Webrowser control (like a link or button) and getting that to run VB code?

This stuff is great for clicking a VB object and manipulating a browser but I was just curious as this would open up a whole new realm of fun and usefulness I think

Thanks,
Andrew

[kleinma]

Yeah you can attach events to specific elements... I know I saw a MSKB article about hooking up the client side events of the DOM to fire in a VB app. I will see if I can find it

[mduclon]

This is a great example. I have a question, I have a form using the Com WebBrowser like this. I use it to fill in and submit a form but on submit I go to a page and get a popup javascript messagebox. I need to close that so I can continue on.
Does anyone know how I can make it close or click the ok button, in code?

MD

[kleinma]

This is a great example. I have a question, I have a form using the Com WebBrowser like this. I use it to fill in and submit a form but on submit I go to a page and get a popup javascript messagebox. I need to close that so I can continue on.
Does anyone know how I can make it close or click the ok button, in code?

MD

Not that I can think of at the moment, but if I come up with something I will post it.

[matosguide]

It is possible to get links that are inside a frame ?
How can I achieve that ?

Thanks in advance!!

[kleinma]

It is possible to get links that are inside a frame ?
How can I achieve that ?

Thanks in advance!!

Yes I am sure you can easily do this. You would just need to access the frame itself, and then you can loop its links. Do you have a specific URL you are trying to work with?

[matosguide]

I can't post the page I'm trying to open.... sorry.
The page have two frames (head & main) and the source are two cgi scripts.
I could go directely to the one I need, but the problem is that it updates the other frame...

I tried to use framecollection but with no sucess... Can you help me with this ?

Thanks in advance

Best regards

[edwardm]

Hi,

This was realy good reading, but how can I fill a site with more than one form (one of the forms that is). How can I find out the syntax in the Document.Forms.Item(, 0).elements, I'm refering to the "0" which I guess is the value that trows my code off(?).

Edward.

[ldadams]

Do you know of a way to auto download a file using your browser?

Cheers
Luke

[kleinma]

If you look at my signature, you will see an example code for downloading a file from the web with progress...

All you do is pass the URL to it and it can download any file.

[ldadams]

Sorry if this is a repeat post.

I have already tried something similar to that. The problem I am having is, I have to script my way past a logon screen to get to the page where I want to download from. So… I thought I could just create a VBScript that opens IE logs me on and then runs my exe. Well that doesn’t quite work. The problem is that logon is per session. And since that utility opens a new response it wants me to logon again. Is there anyway to get the response straight from the WebBrowser Control? Or do you know of a way to fire the Click event of the Save/Open Dialog box? Thanks for you help!!

Cheers,
Luke

[kleinma]

once you get past the logon page, is there a link you click to download the file?

[ldadams]

No not currently. Its a perl function that I pass a date to in the URL. So the URL I actually download the file from is something like: https://ecf.flnb.uscourts.gov/cgi-bi....pl?09/26/2006

If the date exisits in the URL it gives me back a zip. If not page requires user input. Even if there was a link how would I download without creating a new Response. Is there a way to access the response of the webbrowser control?Hope this makes sense.

Cheers,
Luke

[kleinma]

it kind of makes sense to me, however it sounds like the type of thing I would really need to mess around with to get working. If you want to provide me with the actual URL and give me enough info to get in and test with it, I can see if I can do something.

[ldadams]

I really wish I could but I cant give out that information, its out of my hands. I will play with it and see what I can come up with. If you think of anything please share

Cheers,
Luke

[kleinma]

Sure.. if you make any progress, but get stuck somewhere let me know too, I will try to help out where possible

[Nouvelian]

Kleinma, you seem incredibly helpful. Many thanks for the efforts... I've downloaded your .zip file(s) on filling forms on .html documents, but I'm using VB 6... you wouldn't have a similar .zip for VB 6 users would you?

[kleinma]

Sorry.

I have written code in VB6 using the WB control before and done some page manipulations, however that was all commercial code, and not really in any type of code sample that I could provide.

I don't code in VB6 anymore unless I have to for a project, so all my code is now either .NET 1.1, or .NET 2.0

[Nouvelian]

Many thanks anyway.

[slice]

btw this same code can be used with visual basic 6.0 ?

[kleinma]

Many of the concepts can be used in VB6, however the specific .NET code will not be a simple copy/paste into VB6.

If you know .NET and VB6 you could probably easily port this code over to VB6

If I ever have some free time, maybe I will try, since you are the second person to ask in just a few days, but for now its really just .NET 2003/2005

[slice]

thanks .. actually i have not vs.net installed so its problem for me not for you to convert it into vb6.
i know its not big change code.