[RESOLVED] Dynamic BackgroundWorker Question

[NJDevils28]

Hello All,

In the code example below. I create a BGW for every row in a DataGridView. The code works fine and all the BGW's run concurrently, except every once in a while, a few of the BGW's just up and pause. When one of the remaining BGW's finish, then the paused one(s) will restart.
I'm a little confused. It only seems to happen when three or more BGW's are running at the same time. Is it my code or just the nature of the BGW?
Could it be a shared SUB in my code?

A little more info...
The BGW's are doing some heavy lifting with the subsystem by reading Disk Directories.

        
Dim Rowcount As Int16 = QueueDGV.Rows.Count

        Try


            For x = 1 To (Rowcount - 1)
                'Capture current row for DoWork
                RowtoProcess = x - 1
                NumWorkers = NumWorkers + 1
                ReDim Workers(NumWorkers)
                Workers(NumWorkers) = New BackgroundWorker
                Workers(NumWorkers).WorkerReportsProgress = True
                Workers(NumWorkers).WorkerSupportsCancellation = True
                AddHandler Workers(NumWorkers).DoWork, AddressOf WorkerDoWork
                AddHandler Workers(NumWorkers).ProgressChanged, AddressOf WorkerProgressChanged
                AddHandler Workers(NumWorkers).RunWorkerCompleted, AddressOf WorkerCompleted
                Workers(NumWorkers).RunWorkerAsync(RowtoProcess)

            Next

        Catch ex As Exception
            MessageBox.Show("Threading Erorr : " + vbCrLf + ex.Message)
        End Try

[topshot]

If I understand correctly, you can only have as many currently running threads as your CPU(s) will support. Thus, your limit may be 4 - the UI thread plus 3 BGW.

[NJDevils28]

If I understand correctly, you can only have as many currently running threads as your CPU(s) will support. Thus, your limit may be 4 - the UI thread plus 3 BGW.

I figured it out. It was the shared function.

Once I incorporated that code into the DoWork they all started flying.

Not sure how many I can use but I have been testing with 15 so far and they all kickoff. I may run into a ceiling when I do a production test. Thank you for the thread count suggestion!

-NJ

[digitalShaman]

        
Dim Rowcount As Int16 = QueueDGV.Rows.Count

        Try


            For x = 1 To (Rowcount - 1)
                'Capture current row for DoWork
                RowtoProcess = x - 1
                NumWorkers = NumWorkers + 1
                ReDim Workers(NumWorkers)
                Workers(NumWorkers) = New BackgroundWorker
                Workers(NumWorkers).WorkerReportsProgress = True
                Workers(NumWorkers).WorkerSupportsCancellation = True
                AddHandler Workers(NumWorkers).DoWork, AddressOf WorkerDoWork
                AddHandler Workers(NumWorkers).ProgressChanged, AddressOf WorkerProgressChanged
                AddHandler Workers(NumWorkers).RunWorkerCompleted, AddressOf WorkerCompleted
                Workers(NumWorkers).RunWorkerAsync(RowtoProcess)

            Next

        Catch ex As Exception
            MessageBox.Show("Threading Erorr : " + vbCrLf + ex.Message)
        End Try

just curious: have you ever inspected the Workers Array after the Loop?

[NJDevils28]

just curious: have you ever inspected the Workers Array after the Loop?

If your refering to the reportprogress and/or runworkercompleted, then yes. I use the sender argument for those functions. Or am I misunderstanding your question. If I am, please excuse me as the BackgroundWorker is not a strong point with my limited coding skills.

-NJ

[digitalShaman]

the Thing is that with the redim of the Array you loose all the values already assigned, so in the end, after the Loop, you should have an Array with size x where only the very last item does have a reference to a backgroundworker, all the others are Nothing.

to get around this you'd better use a list where you can add whithout knowing how many elements you will Need in the end, or, because you already know how many elements you Need before the Loop, dim once outside the Loop.

however, because you did not realize this code issue yet, the question is, why do you have the Array at all?
although i do think its a good idea to Keep a handle to each started bgw, you do not seem to really use it in your code.

[si_the_geek]

Not sure how many I can use but I have been testing with 15 so far and they all kickoff. I may run into a ceiling when I do a production test. Thank you for the thread count suggestion!

How many threads can run "at the same time" can be awkward to calculate, as there is not only the issue of how many the hardware can run simultaneously (if you open Task Manager, the number of CPU graphs is the maximum), but also what work each thread is doing... anything that means the thread needs to pause (which in your case would probably apply to accessing a disk) means that another thread can take its place for a while.

Having a situation where you have paused threads means the maximum threads increases a bit, but trying to determine how much is rather tricky, so the usual course of action is to use a bit of trial and error (if you have too many, you can see the speed drop) and then reduce the amount so that you give yourself a margin in case things change over time (and to account for unpredictable things, such as other programs using the disk).

Of course if you aren't in a rush to have all the threads finish, having some of them pausing to let others finish probably isn't a big deal.

[NJDevils28]

How many threads can run "at the same time" can be awkward to calculate, as there is not only the issue of how many the hardware can run simultaneously (if you open Task Manager, the number of CPU graphs is the maximum), but also what work each thread is doing... anything that means the thread needs to pause (which in your case would probably apply to accessing a disk) means that another thread can take its place for a while.

Having a situation where you have paused threads means the maximum threads increases a bit, but trying to determine how much is rather tricky, so the usual course of action is to use a bit of trial and error (if you have too many, you can see the speed drop) and then reduce the amount so that you give yourself a margin in case things change over time (and to account for unpredictable things, such as other programs using the disk).

Of course if you aren't in a rush to have all the threads finish, having some of them pausing to let others finish probably isn't a big deal.

Thank you for your analysis.
From what I've been tasked to do. They are going to create a VM and just let my software run for as long as it takes. Days even. So I just need to make it as streamlined as I can in order for it to NOT take days.



-NJ