[RESOLVED] Looping with WebBrowser and DocumentCompleted: Revisited

[cory_jackson]

I recently started a thread on this same topic and marked it as resolved but as I moved to the next stage I've decided it isn't. I'm not sure of the protocol and I don't see a way to remove the "Resolved" tag so I'm starting another. If this is not the proper etiquette please let me know.

Not being able to wait for the WebBrowser to load is really difficult for me to wrap my head around logically. If I have to navigate to a single page and get some text for instance I can understand how to use a sub associated with the DocumentCompleted event. And if I have to iterate thru a bunch of pages I can see how to use two subs with .navigate at the end of each to effectively create a loop. Though I don't think it elegant it works. But now in the same form I have other things I need to do with the WebBrowser control and there's only one sub associated with DocumentCompleted. Additionally the way this new routine works I'm having problems trying to figure out how to structure it. My understanding is probably incomplete but what I'm coming up with is that each part of the process needs to be broken into subs and each sub needs to end with a Navigate except the very last one. Perhaps someone can explain to me the gap in my understanding.

Let me give you a practical example. I have a WinForm with a single WebBrowser and a couple of buttons. After load the user logs into the site and hits a button and the WebBrowser loops thru a bunch of pages and extracts some data. The second function a user specifies and runs a report. Once loaded, which can take some time, they click a button and the program extracts a bunch of call IDs from a table then plows thru every call record by navigating to that page. Once done collecting all that it does a few other things and prompts the user to save a file.

According to the previous advice I don't see how I can do this all with one DocumentCompleted sub. I was thinking of a condition based on the Sender parameter but the sender will always be the WebBrowser, not the sub that called it. I suppose I could create a shared variable for condition and manually set that each time and ad a select case to the document completed. But then I'll have to put all the rest of my code in the DocumentCompleted sub? I must be missing something because this all seems incredibly complicated and confusing. I really hope there's something simple I'm missing.

If it turns out there isn't an elegant solution would it slightly less evil to just create a loop with a 100mS thread sleep to wait for the document to be done? I've seen examples from Ident for instance which used DoEvents but according to the responses that's a bad practice which makes sense to me. If this is an OK idea I was hoping someone might help me select the best Do While condition. I believe there are some scripts running on this page and I suspect that properties like DocumentCompleted might not be sufficient. IE it seems it takes a little while longer for the scripts to do their thing. I was thinking of using IsBusy and having a couple counters. One for max iterations before it gives up and another to count a few iterations of the state to make sure it's really and finally complete. Like 3 100mS iterations. Though it might not be necessary here.

As an aside I tried to go back to using WebClient but I haven't been able to figure out how to get the authentication to work. I might go back to that given it's probably an easier problem to solve.

Sorry for the lengthy post and thank you for taking the time to consider it.

[dunfiddlin]

The url of the document completed is available to you in e.url. What else do you need to provide a simple Select Case to manage which parts of the code are executed and when?

[cory_jackson]

I think that might work. The URL will be different each time so select case will not work. I could add conditions for If Contains perhaps. THis would limit me to only one procedure per page but I don't think that would be a problem.

So you're advocating that essentially all of my code for every function in this application, current and future, be contained in the one sub? IE it becomes the MC for everything that involves navigation of the WebBrowser. Perhaps I can bust some of the code out into modules to tidy it up.

I'm thinking of how this will work. User clicks the button while at the call log report page. The ButtonClick sub's will extract all the CallIDs and store them in a common DataTable. Finally it will navigate to the first call record page. Once the first page is loaded DocumentCompleted fires and executes the part of the sub that matches the partial URL. Therein a sub runs to extract the text from the page and add to the current DataTable row and determine the next CallID from the DataTable. Finally it Navigates to the next call record page. Once loaded this triggers itself to repeat with the DocumentCompleted event. But this conditional section of the DocumentCompleted event will also contain a condition to see if the last record has been reached so it will only Navigate if there are more to do. If there are no more to do the Else condition would contain all the finalizing code.

Is this what you're suggesting? Do I make sense? Should I write a quasi-code example after breakfast? It seems unintuitive to me but I just need to get over it.

[dunfiddlin]

So you're advocating that essentially all of my code for every function in this application, current and future, be contained in the one sub?

Did I say that? Or did I merely suggest that the execution management be contained within this one Sub? There's no need for a single line of actual data processing to be done in the DocumentCompleted handler if you so choose. It can be merely a 'switchboard' calling subs and functions from any source including outside the project altogether if you so desire.

[cory_jackson]

I think I understand you. See if my code below is essentially the structure you're suggesting. So in the case of the Call Log Export feature what could have been done as a single sub if WebBrowser could wait is now 4. Also more logic and some objects had to be made common. I think this will work for me but it's just hard to come around to when in my mind a simple loop will suffice with WebClient.

Public Class Test3
 
    Dim dtAdvisord As DataTable
    Dim dtCallLogs As DataTable
    Dim strDCFunction As String = ""
    Dim intCounter As Integer = 0
 
    Private Sub btnAdvisorReport_Click() Handles btnAdvisorReport.Click
        strDCFunction = "Advisor"
        wbMain.Navigate("http://company.com/advisor.html")
    End Sub
 
    Private Sub AdvisorReportExtract()
        'extract data from webpage and push into dtAdvisors
        'prompt user to select and save.
        wbMain.Navigate("http://company.com/call.html") 'except it would include a callID in the URL
    End Sub
 
    Private Sub btnCallLogExport_Click() Handles btnCallLogExport.Click
        'Extract table of data and stor in DTCallLogs
        strDCFunction = "Call Log"
        intCounter = -1
    End Sub
 
    Private Sub CallLogExtract()
        intCounter += 1 'Used as index pointer to dtCallLogs
        'extract data from webpage and push into dtCallLogs
    End Sub
 
    Private Sub CallLogWrap()
        'finalize and prompt user to save file or whatever
    End Sub
 
    Private Sub wbMain_DocumentCompleted() Handles wbMain.DocumentCompleted
        Select Case strDCFunction
            Case "Advisor"
                AdvisorReportExtract()
            Case "Call Log"
                CallLogExtract()
                If intCounter = dtCallLogs.Rows.Count - 1 Then
                    CallLogWrap()
                Else
                    wbMain.Navigate("http://company.com/call.html") 'except it woudl include a callID in the URL
                End If
            Case Else
        End Select
    End Sub
 
End Class

[cory_jackson]

To wrap this up I was able to get the logic to work and I understand Dunfiddlin's concept of using the DocumentCompleted sub as a kind of switchboard. But It's inelegant to me and I decided that one should never really do big loops with WebBrowser. It's just not what it was designed to do. So I went back and dusted off a class I made that acts much like WebCleint and after stripping out some useless code found it worked first try. Much better. I also had to change my model from using the HTML DOM to extract text to using RegEx but that wasn't too painful. Now it runs invisibly and quickly and for the 10k+ records I need to get will work much better.

So although I didn't end up using this method it helped me understand it better and I'm sure I'll be using it more effectively in the future. Thanks all but especially Dunfiddlin.