how can I prevent this from causing an error?

[iamcpc]

I'm controlling the form1.webbrowser1 from a module so I have this loop after form1.webbrowser1.navigate:


Do While Form1.WebBrowser1.ReadyState <> WebBrowserReadyState.Complete
System.Windows.Forms.Application.DoEvents()
Loop


System.Windows.Forms.Application.DoEvents() is causing an "attempted to read or write protected memory error"

How can I correct this?

[stanav]

Why do you have to place that code in a module? And why do you need the loop and DoEvents at all? You should do it the proper way by handling the webbrowser.documentCompleted event...

[iamcpc]

Why do you have to place that code in a module? And why do you need the loop and DoEvents at all? You should do it the proper way by handling the webbrowser.documentCompleted event...

I have form1.webbrowser1 set a form1.label1 to "complete" after the document is completed.

What happens is that if form4 (or another module) navigates form1.webbrowser1 the document completed event never fires unless you say System.Windows.Forms.Application.DoEvents().

If I say on form4 navigate the form1.webbrowser1 and wait for form1.label1 to be "complete" it never happens because it's on another thread.


I put it on a module so that my form1 would only have form1 code on it and loops or jobs would be in module1

[Shaggy Hiker]

It fires. If DoEvents solves the problem, then the problem was that you are blocking somewhere and preventing the application from pumping messages. If you don't give the application the opportunity, it never gets to process the events, and they just stack up in the queue. DoEvents pauses whatever else you are doing so that the application can process the queue. Therefore, I think you ought to step through the code without DoEvents and see what is happening. Most likely you are spinning perpetually on something. For instance, if you just commented out the DoEvents in the code you showed, you'd spin forever on that loop.

As it is, the loop is a busy wait which is about the worst thing you can do. Unfortunately, I naively thought that DocumentCompleted meant that the Document really was Completed, but with the complexity of modern web pages, it isn't quite. Still, if you have to spin wait, do so in the DocumentCompleted event so that the program can get that far without clobbering the CPU.

[iamcpc]

It fires. If DoEvents solves the problem, then the problem was that you are blocking somewhere and preventing the application from pumping messages. If you don't give the application the opportunity, it never gets to process the events, and they just stack up in the queue. DoEvents pauses whatever else you are doing so that the application can process the queue. Therefore, I think you ought to step through the code without DoEvents and see what is happening. Most likely you are spinning perpetually on something. For instance, if you just commented out the DoEvents in the code you showed, you'd spin forever on that loop.

As it is, the loop is a busy wait which is about the worst thing you can do. Unfortunately, I naively thought that DocumentCompleted meant that the Document really was Completed, but with the complexity of modern web pages, it isn't quite. Still, if you have to spin wait, do so in the DocumentCompleted event so that the program can get that far without clobbering the CPU.

form4 does not fire form1 events unless i do system.application.doevents()

form4 code:

debug.print form1.webbrowser1.url.tostring()
'shows www.google.com

form1.webbrowser1.navigate("www.yahoo.com")

debug.print form1.webbrowser1.url.tostring()
'shows www.google.com after i've told form1 to navigate.

system.threading.thread.sleep(100000)

debug.print form1.webbrowser1.url.tostring()
'shows www.google.com after i've told form1 to navigate and slept the thread for 100 seconds.

[dunfiddlin]

Just to give you an idea of the complications involved, this is the record of events and Ready State.Completed from navigating (triggered from a second form) to amazon.co.uk

Navigating
Navigated
Navigating
Navigating
Navigated
Doc Completed
Doc Completed
Ready (first setting)
Navigating
Ready
Navigating
Ready
Navigating
Ready
Navigating
Ready
Doc Completed
Ready
Doc Completed
Ready
Doc Completed
Ready
Navigated
Ready
Navigating
Ready
Doc Completed
Ready
Navigating
Ready
Navigating
Ready
Navigating
Ready
Navigating
Ready
Navigating
Ready
Navigated
Ready
Navigated
Ready
Navigated
Ready
Navigating
Ready
Navigated
Ready
Navigating
Ready
Navigated
Ready
Navigated
Ready
Doc Completed
Ready
Doc Completed
Ready
Doc Completed
Ready
Navigating
Ready
Navigated
Ready
Navigated
Ready
Doc Completed
Ready
Doc Completed
Ready
Navigated
Ready
Doc Completed
Ready
Doc Completed
Ready
Doc Completed
Ready
Doc Completed
Ready

So, the conclusion is that you can't really trust the full document being loaded on any event or value of ReadyState. DocumentCompleted will almost certainly be the last event so your best bet if you're looking for a known item in a webpage is to check for it specifically. Otherwise it's all a bit in the lap of the Gods, which is why many prefer to use WebClient to download source HTML for scraping.

[DataMiser]

form4 does not fire form1 events unless i do system.application.doevents()

Yes if you are using a tight loop then it will not allow the event to fire, do events will allow the event to fire. The thing is that you should not have that loop and then you would not need the doevents in it

[dunfiddlin]

form4 does not fire form1 events unless i do system.application.doevents()

Er yes it does (see my post above). It doesn't appear to because you're stuck in the loop.

form4 code:

debug.print form1.webbrowser1.url.tostring()
'shows www.google.com

form1.webbrowser1.navigate("www.yahoo.com")

debug.print form1.webbrowser1.url.tostring()
'shows www.google.com after i've told form1 to navigate.

system.threading.thread.sleep(100000)

debug.print form1.webbrowser1.url.tostring()
'shows www.google.com after i've told form1 to navigate and slept the thread for 100 seconds.

The sleep stops all operations for the duration! You could set it for 100 years and it still wouldn't have changed!

[iamcpc]

Er yes it does (see my post above). It doesn't appear to because you're stuck in the loop.



The sleep stops all operations for the duration! You could set it for 100 years and it still wouldn't have changed!

Then how can i navigate form1.webrowser1 and wait for it to load from outside of form1 without using a form1.webbrowser1 readystate/ doevents() loop and without using the form1.documentcompleted event (because I don't know how to make it fire)?

[dunfiddlin]

Why do you need to wait for it to load? Or perhaps just, do you need to wait for it to load?

[stanav]

You should do something like this:

'In form1
 Private urlsToVisit() As String
    Private currentIndex As Integer = -1
    Public Sub NavigateTheseUrls(ByVal urls() As String)
        If urls.Length > 0 Then
            Me.urlsToVisit = urls
            Me.currentIndex += 1
            Me.WebBrowser1.Navigate(urlsToVisit(currentIndex))
        End If
    End Sub

    Private Sub WebBrowser1_DocumentCompleted(sender As Object, e As System.Windows.Forms.WebBrowserDocumentCompletedEventArgs) Handles WebBrowser1.DocumentCompleted
        If Me.WebBrowser1.ReadyState = WebBrowserReadyState.Complete Then
            Dim thisUrl As String = WebBrowser1.Url.ToString()
            'Do whatever you want with this web page here - you know what the web page is based on thisUrl
            '
            'Once you done with this page, navigate to the next
            Me.currentIndex += 1
            If Me.currentIndex < Me.urlsToVisit.Length Then
                Me.WebBrowser1.Navigate(Me.currentIndex)
            Else
                Me.currentIndex = -1
            End If
        End If
    End Sub

'And then in form4
Dim urls() As String = {"http://www.google.com", "http://www.yahoo.com"}
Form1.NavigateTheseUrls(urls)

[iamcpc]

Why do you need to wait for it to load? Or perhaps just, do you need to wait for it to load?

because the html elements my loop is trying to manipulate are not there until after the website loads.


this loop works fine on my computer when debugging but the doevents() randomly causes that protected memory error on all other computers
Do While Form1.WebBrowser1.ReadyState <> WebBrowserReadyState.Complete
System.Windows.Forms.Application.DoEvents()
Loop


so how can I wait for the form1.webbrowser1 to load from within a loop (which involves navigating multiple websites) or from outside of form1 without using application.doevents()

[stanav]

So, the conclusion is that you can't really trust the full document being loaded on any event or value of ReadyState. DocumentCompleted will almost certainly be the last event so your best bet if you're looking for a known item in a webpage is to check for it specifically. Otherwise it's all a bit in the lap of the Gods, which is why many prefer to use WebClient to download source HTML for scraping.

I agree. If all the OP wants is just the source of the page then using a webclient is probably a better choice.

[dunfiddlin]

so how can I wait for the form1.webbrowser1 to load from within a loop (which involves navigating multiple websites) or from outside of form1 without using application.doevents()

I already indicated two possible answers. Either use the DocumentCompleted event to check for the presence of a specific item in the current page so obviating the need for a wait loop, or use WebClient to download the source HTML for each page which requires no attention at all.

[Shaggy Hiker]

That loop only appears to work fine. It is actually running your CPU totally pegged. The cost is not going to be apparent as long as you are plugged into the wall, but try that while running on battery and you will see the cost. Your battery life will be dramatically shortened. You need to avoid spinning around a DoEvents in every case. The solution Stanav has would be a good one. The issue is usually that people are thinking in a linnear fashion: Do this, and when that's done, do this, and when that's done, do this other thing, etc. In event driven programming, you mostly want to be doing nothing but sitting around waiting for events to fire. You don't want to be busily spinning around checking whether an event has fired, but doing absolutely nothing (including not Sleeping). Then, when the event fires, you respond to it. So, you set the browser to navigating, but you don't eagerly wait for it to complete, instead you just do nothing. When it completes, you get an event, and that's what you act on. That does mean that the sequence of actions isn't all in one neat little function, but that's life. The neat little function starts something in motion, then you wait for the events emitted as a result of that motion.

Sleep will stop the busy wait of the CPU, but it stops the whole app, so it isn't a viable alternative, and there are very few reasons to ever use it. If you find that it solves some problem in the UI thread, then you've misdiagnosed the problem.

[iamcpc]

I already indicated two possible answers. Either use the DocumentCompleted event to check for the presence of a specific item in the current page so obviating the need for a wait loop, or use WebClient to download the source HTML for each page which requires no attention at all.

I have form1.webrowser1 with a document completed event setting a boolean to true if it's yahoo.

on form4
form1.webbrowser1.navigate("www.yahoo.com")
(document completed event does not happened yet)

wait for the form1.webrowser1 document completed event to show true for yahhoo
do while webbrowser1.url.tostring <> "www.yahoo.com"
(waits forever because form1.webbrowser1 is not navigating)

do while webbrowser1.document completed boolean <> true
(waits forever because form1.webbrowser1 is not navigating)


fill out the www.yahoo.com search box
(problem with this because the document completed event has not happened yet)

[DataMiser]

You need to forget about using the busy loops to wait. You are just hanging your program by doing that.

[iamcpc]

You need to forget about using the busy loops to wait. You are just hanging your program by doing that.

I understand that which is why i'm here asking for help.

form1.webbrowser1.navigate("www.yahoo.com")

'What code do I put here to wait for form1.webbrowser1 to finish navigating?

'after navigation fill out the yahoo search box

[stanav]

I understand that which is why i'm here asking for help.

form1.webbrowser1.navigate("www.yahoo.com")

'What code do I put here to wait for form1.webbrowser1 to finish navigating?

'after navigation fill out the yahoo search box

You need to stop thinking in linear way... Re-read post#11... No need for a busy waiting loop. In the documentcompleted event handler, you just make sure that the document is really completed, then check which url it just finised loading... And then go on to the next task right from within the documentcompleted event handler. That means you can raise an event back to form4 to let it knows that the page has done loading if you want to...

[iamcpc]

You need to stop thinking in linear way... Re-read post#11... No need for a busy waiting loop. In the documentcompleted event handler, you just make sure that the document is really completed, then check which url it just finised loading... And then go on to the next task right from within the documentcompleted event handler. That means you can raise an event back to form4 to let it knows that the page has done loading if you want to...

The problem with that is that if the user navigates form1.webbrowser1 to www.yahoo.com it will do an unwanted yahoo search. What i'm trying to do is have form4 navigate form1.webbrowser1 and then do the search while the user can still navigate form1.webrowser from form1 without running any code after the navigation is completed

[Shaggy Hiker]

So, why not create two web browsers? If you have one that you want to use in an automated fashion from form4, and another that you want the user to be able to use directly from Form1, then why not have two different ones? It kind of sounds like you may not even need to see one of them.

On the other hand, if you want the resuls to be visible to the user in either case, then using one is probably the right way to go. Technically, you could use two and just show one or the other, but that seems wasteful. If you don't want anything running after completion in some cases, then pass a Boolean to form1. Only run the code in the DocumentCompleted handler if the Boolean is true.

[iamcpc]

So, why not create two web browsers? If you have one that you want to use in an automated fashion from form4, and another that you want the user to be able to use directly from Form1, then why not have two different ones? It kind of sounds like you may not even need to see one of them.

On the other hand, if you want the resuls to be visible to the user in either case, then using one is probably the right way to go. Technically, you could use two and just show one or the other, but that seems wasteful. If you don't want anything running after completion in some cases, then pass a Boolean to form1. Only run the code in the DocumentCompleted handler if the Boolean is true.

I can try this but i'm still running into the problem of needing to stop a thread to allow the application events and threads to run but application.doevents() is causing in error.

Is there a way to pause a thread and, while that thread is paused, run the other threads of the application?

[DataMiser]

Thread.Sleep pauses the thread other threads continue on their merry way.

[Shaggy Hiker]

Are you really using threads? Everything you have shown or talked about are UI components running on the UI thread. Also, if you are really using threads, then there would be no reason to stop one to allow the others to run unless the one thread is holding a lock on a critical section, which would be a bad design to start with. Threads run simultaneously and independently unless they use shared resources or code, all of which should be locked and released as quickly as possible, and DoEvents will have nothing to do with that in any way. Nothing you have said suggests that you are really using multiple threads, and you appear to believe that DoEvents will solve the problem, which means that they aren't multiple threads at all, but just different chunks of code in the UI thread running sequentially.

The bottom line is this: If the solution requires DoEvents, then the solution could be improved. If the solution requires thread.sleep, and the solution doesn't use threading, then the solution is wrong.

How to fix either one is not so simple, though, especially once you have gone a ways down either road. We all think by forming mental models of how a process works. As long as we think we are making progress towards our goal, we feel that the mental model is correct. When we hit an obstacle, the first thought is not to throw out the mental model, but to try to solve the problem within the model. Anything else would be something close to madness. Unfortunately, this means of operating can cause us trouble when we get into a blind alley, because we will first attempt to push through the obstacles before us before we realize the route will lead us nowhere. You are in such an alley right now. The problem is that realizing that your mental model of the problem is flawed is no more than a first step, and a highly unsatisfying step, at that. The next step is to come up with a different mental model, and that isn't easy. Still, it is what you have to do.

The new model has to have these features:

1) No use of DoEvents. I'm not totally opposed to DoEvents as some folks are, but in this case you would be better off assuming that it didn't exist, since it is acting as a mental trap.

2) No spin waiting. This is true in all cases in VB.NET. There is a rare situation where spin wating is ok, but that is in pretty low level thread pool managment code. For everyone else, if you are writing a loop where you just loop waiting for some external event (not in the loop itself) to occur, then you are spin waiting (also called busy waiting), and that's really bad, especially in this modern, mobile, age.

With those constraints, you have to be willing to start processes off, such as the navigation, and move any other code to some event triggered by the process. If the process can be started off in different ways, or by different initiators, such as two different means of navigating, and if you don't want the process to be handled the same way in all cases, then you have to add some code so that when the event occurs, the event handler knows in which way it has to act. This can be done with global variables, form level variables, or it could even be done by adding something to the .Tag property of a control (I'm not sure whether the WebBrowser even HAS a .Tag property, but it might).

[iamcpc]

The new model has to have these features:

1) No use of DoEvents. I'm not totally opposed to DoEvents as some folks are, but in this case you would be better off assuming that it didn't exist, since it is acting as a mental trap.

2) No spin waiting. This is true in all cases in VB.NET. There is a rare situation where spin wating is ok, but that is in pretty low level thread pool managment code. For everyone else, if you are writing a loop where you just loop waiting for some external event (not in the loop itself) to occur, then you are spin waiting (also called busy waiting), and that's really bad, especially in this modern, mobile, age.

With those constraints, you have to be willing to start processes off, such as the navigation, and move any other code to some event triggered by the process. If the process can be started off in different ways, or by different initiators, such as two different means of navigating, and if you don't want the process to be handled the same way in all cases, then you have to add some code so that when the event occurs, the event handler knows in which way it has to act. This can be done with global variables, form level variables, or it could even be done by adding something to the .Tag property of a control (I'm not sure whether the WebBrowser even HAS a .Tag property, but it might).

When I step through a VB macro in excel I can see each step fire as I step through it. This is not the case when working with windows forms. If I use global variables and form1.webbrowser1 event handler to handle the document completed event I run into many many problems. I may want to navigate through 5 webpages to do something like verify insurance on a Medicaid website and loop through these navigations. So, in a very short period of time, my form1.webbrowser1 event handler will have hundreds or even thousands of lines of code on it (whereas some sort of a wait or pause function which allows the form to do it's thing during the wait would save hundreds of hours of programming and debugging the form1.webbrowser1 event handler).

In addition we have a browser based operating system in which the document is loaded and the webbrowser is not busy but without saying (wait a second for additional macros built into the webbrowser1 object to run) this programming is not an option. It looks like i'm just going to have to go to my boss and let him know that there is no way for a module1 loop which involes manipulating data on form1.webrowser1 to wait for the webpage to load in a manner that would work for what we're trying to accomplish (I have no idea how to manipulate form1.webbrowser1 from another thread).

[dunfiddlin]

When I step through a VB macro in excel I can see each step fire as I step through it. This is not the case when working with windows forms.

Er ... yeah it is.

browser based operating system

Say what now? How (and indeed why) are you running VB code then?

It looks like i'm just going to have to go to my boss and let him know that there is no way for a module1 loop which involes manipulating data on form1.webrowser1 to wait for the webpage to load in a manner that would work for what we're trying to accomplish

Well quite.

I have no idea how to manipulate form1.webbrowser1 from another thread

So you were picked for this job because ....... ?

[Shaggy Hiker]

A busy wait is a bad idea, even considering what you have said. That will peg your CPU, increase power usage, increase heat, drain batteries, and so forth. For that reason, it must not be done. However, that doesn't mean that anything is not possible. The fact that DocumentCompleted gets raised (potentially) many times over the course of loading a single page, is annoying, but not a deal breaker. It does make it more difficult, because you need to determine whether or not the part YOU want has loaded for any particular firing of that event.

You are able to step through most events, and even if you can't you can use breakpoints to pause in any event. You'd want to do that anyways, because the very act of stepping through the code is going to cause your deviant behavior in this asychronous case.

[iamcpc]

Er ... yeah it is.

That's funny because when I press F8 on the line of code which says form1.webbrowser1.navigate("www.yahoo.com") I don't see the webbrowser navigate....

So you were picked for this job because ....... ?

It's a long story.