Manipulating Java Applet Fields Within Webbrowser

[mm040e]

Hello,

I'm currently designing a program to automate the submission of data to a Federal regulatory agency. It will pull data from excel sheets and populate the corresponding fields online. I had no issue with general webpages, I just use the following command for each field:
WebBrowser1.Document.GetElementById("ElementID").SetAttribute("value", "ExcelField".Value2)

Now this works great with typical html pages with forms to fill so I was planning on just expanding upon these basics to fill out an form with multiple fields for population. However, after further examination it looks like the Agency uses a Java Applet interface for data input.

This has made everything much more difficult and I'm not completely sure how I can automate the population of this form. Previously I was just opening the pages in Firefox and using the "Inspect Element" tool to examine each field, get the element ID, and possible values. I did this to avoid having to scan through a bunch of page source code. But with the Java Applet this tool no longer is an option. I can still open up the page's source code however.

If it would be helpful I could post the source code as well as my VB code. But has anyone ever had any experience with manipulating fields in an online Java Applet?

Thanks,
Michael

[dunfiddlin]

Hmm. Figures that a Federal regulatory agency would use the world's least secure language (allegedly, in case my lawyer is reading this!)

As for changing a value in a java field, like a psychotherapist's lightbulb it kinda depends on whether it wants to change! It may come down to a SendKeys sequence but even that's going to require the ability to get focus to the appropriate spot.

I've been messing about a bit with the Awesomium browser control which has a whole bunch of methods designed to deal with scripting etc. but I haven't yet got as far as seeing if that makes it better for trying to use a Java applet (apart from anything else interactive applets aren't all that common these days). If I get some time over the weekend I'll give it a try and let you know what's occurring.

[mm040e]

haha yes that's what I was thinking as well... Please let me know if you find anything. The first thing I really need is just the ID's of the fields in the applet, I haven't even figured that out so far. From there I could give it a shot and see if I can successfully manipulate it.

[mm040e]

Hmm. Figures that a Federal regulatory agency would use the world's least secure language (allegedly, in case my lawyer is reading this!)

As for changing a value in a java field, like a psychotherapist's lightbulb it kinda depends on whether it wants to change! It may come down to a SendKeys sequence but even that's going to require the ability to get focus to the appropriate spot.

I've been messing about a bit with the Awesomium browser control which has a whole bunch of methods designed to deal with scripting etc. but I haven't yet got as far as seeing if that makes it better for trying to use a Java applet (apart from anything else interactive applets aren't all that common these days). If I get some time over the weekend I'll give it a try and let you know what's occurring.

Have you reached any new findings on interacting with a java applet? I posed this same question to a Java forum to see if they might provide any perspective but I feel like they were over-thinking the problem. The suggestion I received was a mouse pointer and keyboard control robot haha

[dunfiddlin]

Ah. Right. Well, thing is, I'm getting to the point of thinking that may be the only solution myself. Awkward!

[mm040e]

Ah. Right. Well, thing is, I'm getting to the point of thinking that may be the only solution myself. Awkward!

How tough do you think this would be?

[dunfiddlin]

It's not that difficult provided you can be reasonably sure that the areas you need to 'click' will stay put give or take a few pixels. Here's a few pointers (pun unintended!)

[mm040e]

It's not that difficult provided you can be reasonably sure that the areas you need to 'click' will stay put give or take a few pixels. Here's a few pointers (pun unintended!)

Eh that's what I was thinking... The problem is that this is being developed for multiple users potentially with different size monitors I'm assuming (which I would imagine would affect this). And the winform is adjustable in size, defaulting to a fullscreen view. So is there some way around this issue? I mean sure, it would be cake if this was just a program for myself and I knew what it was going to be doing every time, but unfortunately that is not the case. So if I'm developing this for a different user and I know nothing about their computer is this idea still possible?

[dunfiddlin]

The first thing to establish is whether the site you're using already employs absolute, pixel based measurements. If they do then the size of the application window won't make any difference to where the relevant positions are in the browser. When you discover they're not (it's never that easy, right?) you may have to consider fixed sizing for at least the browser if not the whole application. It really needs a bit of experimentation to see what your margins of error can realistically be which obviously depends on how big an area you get to register the click in.

[mm040e]

The first thing to establish is whether the site you're using already employs absolute, pixel based measurements. If they do then the size of the application window won't make any difference to where the relevant positions are in the browser. When you discover they're not (it's never that easy, right?) you may have to consider fixed sizing for at least the browser if not the whole application. It really needs a bit of experimentation to see what your margins of error can realistically be which obviously depends on how big an area you get to register the click in.

Hmm I see, well it looks like the website itself isn't fixed size. But it appears as though the java applet (which is really all that matters does stay at a fixed location in reference to the top left corner of the webbrowser. And the applet doesn't resize if I make my webbrowser larger or smaller. This is what I want, correct?

My concern now is regarding different resolution monitors. When I move the browser from my external monitor to my laptop screen the whole thing gets smaller, but I'm guessing this is due to the higher pixel density of my laptop. So my question is how do I ensure that this will work, regardless of setup?


Edit: Another question popped into my head, can you constrain the mouse movement to the webbrowser window and make all pointer locations in relation to that? Because I'll always know where my applet fields are in relation to that. However if it's not in relation to that and say I move my whole winform, then that could throw things off... Hopefully what I'm saying is making sense.

[dunfiddlin]

PointToClient(Cursor.Position) will translate to locations within the client area of the control (in this case the Webbrowser) over which the cursor is placed. There is a problem in that the webbrowser has no public mouse events though, so the position finding has to be done on a timer though and you will need to determine when you're within the webbrowser area using the control's Location and Size properties.

[mm040e]

Okay well I've discovered some good things and bad things...

The Good - It appears as though I will be able to simply cycle through all of the fields with the TAB key, meaning that the automated pointer will only have to click on one field and then the rest can be entered through keystrokes alone.

The Bad - I'm not sure how to enter text into the fields inside of the webbrowser. I tried using SendKeys.Send() but that doesn't seem to work in the webbrowser. And I can't find much information on how to make this work since the logical choice would be to simply to set the html attribute. So do you know I can input this once I have the field selected?

[dunfiddlin]

SendKeys does work in the browser so if anything is blocking it it will be the applet itself which would indeed be bad news. Trouble is I'm kinda flying blind without the applet to access (and obviously I can't go play with it and stay out of jail!!!) I'll see if I can find some examples of java forms that I can play with.

The only immediate alternative is the API key_event, literally a simulated key press, but that is restricted to one character at a time and requires a bit of jiggery pokery so it would be very much a last resort for this kind of job.

[mm040e]

SendKeys does work in the browser so if anything is blocking it it will be the applet itself which would indeed be bad news. Trouble is I'm kinda flying blind without the applet to access (and obviously I can't go play with it and stay out of jail!!!) I'll see if I can find some examples of java forms that I can play with.

The only immediate alternative is the API key_event, literally a simulated key press, but that is restricted to one character at a time and requires a bit of jiggery pokery so it would be very much a last resort for this kind of job.

Well actually this had nothing to do with the applet, it seems like my sendkeys woes were just due to the webbrowser. I was testing in a general html field.. I was able to automatically click on the field and select it but when sendkeys executed nothing happened.

And I suppose you could get access to the applet, it's through the FCC's website. It's not super secret or anything haha. However I do believe you'd have to have a login which I have already obtained which would be a lot to ask of you just to help.

[dunfiddlin]

Ah. How did you initiate the SendKeys? You need to be careful not to shift focus away from the browser and make sure that page is fully loaded.

[mm040e]

Um right after I moved the pointer over the field in question, simulated a mouse press, and simulated a mouse release, I executed SendKeys.Send("test string").

[dunfiddlin]

Ok. Then I guess we'll have to do it the hard way!

Public Class Form1
    Private Declare Auto Function VkKeyScan Lib "user32" (ByVal ch As Char) As Byte
    Private Declare Auto Sub keybd_event Lib "user32" (ByVal bVk As Byte, ByVal bScan As Byte, ByVal dwFlags As Int32, ByVal dwExtraInfo As Int32)
    Private Declare Auto Sub mouse_event Lib "user32" Alias "mouse_event" (ByVal dwFlags As UInt32, ByVal dx As UInt32, ByVal dy As UInt32, ByVal cButtons As UInt32, ByVal dwExtraInfo As IntPtr)
    Const MOUSEEVENTF_LEFTDOWN = &H2 ' left button down
    Const MOUSEEVENTF_LEFTUP = &H4 ' left button up
    Const VK_LSHIFT = &HA0
 
    Private Sub Button1_Click(sender As System.Object, e As System.EventArgs) Handles Button1.Click
        Cursor.Position = PointToScreen(New Point(100, 225))
        mouse_event(MOUSEEVENTF_LEFTDOWN, 0, 0, 0, CType(0, IntPtr))
        mouse_event(MOUSEEVENTF_LEFTUP, 0, 0, 0, CType(0, IntPtr))
 
        For Each c In "Computer 123"
            If Not Char.IsNumber(c) AndAlso Char.ToUpper(c) = c Then keybd_event(VK_LSHIFT, 0, 0, 0)
            keybd_event(VkKeyScan(c), 0, 0, 0)
            keybd_event(VkKeyScan(c), 0, 2, 0)
            If Not Char.IsNumber(c) AndAlso Char.ToUpper(c) = c Then keybd_event(VK_LSHIFT, 0, 2, 0)
        Next
    End Sub
End Class

[mm040e]

Ah I see, if I'm understanding this correctly it doesn't look quite as painful as I was thinking it might be... So will that work it's way through each character in the string and manually create a keystroke event? I'm gonna try to implement the code in a couple of hours.

[mm040e]

I implemented your code and it works beautifully

I think this a good starting point to accomplish just about everything that I was hoping to do... Do you know how I can simulate a TAB key press? So then I can just cycle through all of the fields rather than trying to pinpoint the cursor on each field and clicking.

Now my only concerns left are about resizing and/or different resolutions throwing my pointer off.

[dunfiddlin]

Const VK_TAB = &H09

Virtual Key Codes

[mm040e]

Simple enough, thank you very much!

Since I have a number of fields to input data would you recommend putting the key pressing loop in a function? That's what I was thinking..


EDIT: After some additional testing I'm running into some slight errors. Have you ever found that this tends to randomly capitalize letters at times? Because I'm observing that quite regularly... Additionally it seems as though sometimes it's chopping off one letter of a string and attaching it to another one. The following is my test code:

WebBrowser1.Focus()


While i < 14

CS = oWS.Range("G" & i + 8) 'selects the excel field to pull data from

For Each c In CS.Value2
If Not Char.IsNumber(c) AndAlso Char.ToUpper(c) = c Then keybd_event(VK_LSHIFT, 0, 0, 0)
keybd_event(VkKeyScan(c), 0, 0, 0)
keybd_event(VkKeyScan(c), 0, 2, 0)
If Not Char.IsNumber(c) AndAlso Char.ToUpper(c) = c Then keybd_event(VK_LSHIFT, 0, 2, 0)
Next

keybd_event(VK_TAB, 0, 0, 0)
keybd_event(VK_TAB, 0, KEYEVENTF_KEYUP, 0)

i += 1

End While



This theoretically should cycle through a column of items in excel and type them into separate fields. It does this for the most part, but there are the exceptions explained above...

[mm040e]

I had one more question, I was able to implement your code effectively... And as it turns out sendkeys does work, it just isn't very accurate for some reason. Or rather I should say that it enters too many keystrokes, i.e. "530.28" gets entered as "5330.222288".

But for some reason the code you provided doesn't seem to include punctuation. Since I'm mostly simulating numerical entries, it doesn't include the decimal. Do you know why this could be?

Thanks