-
May 4th, 2018, 02:42 AM
#1
Thread Starter
New Member
2 Background workers problem
I have 2 background workers that are running through a list of URLs.
When I start them off the first URL gets skipped and the second URL gets visited by both threads, instead of just one. The rest of the list runs fine without duplicate requests. I have a workaround for it that works but I feel like it's not the best way to fix it.
Code that ends up visiting the second URL twice, skipping the first:
Code:
Private Sub bgWorker_Click(sender As Object, e As EventArgs) Handles bgWorker.Click
theInt = 0
BackgroundWorker1.RunWorkerAsync()
System.Threading.Interlocked.Increment(theInt)
BackgroundWorker2.RunWorkerAsync()
End Sub
Workaround code I made:
Code:
Private Sub bgWorker_Click(sender As Object, e As EventArgs) Handles bgWorker.Click
theInt = 0
BackgroundWorker1.RunWorkerAsync(0)
System.Threading.Interlocked.Increment(theInt)
BackgroundWorker2.RunWorkerAsync(1)
End Sub
Private Sub BackgroundWorker_DoWork(sender As Object, e As System.ComponentModel.DoWorkEventArgs) Handles BackgroundWorker1.DoWork, BackgroundWorker2.DoWork
Dim worker As System.ComponentModel.BackgroundWorker = DirectCast(sender, System.ComponentModel.BackgroundWorker)
Dim theURL As String
If Len(e.Argument) > 0 Then
theURL = theURLs(Int(e.Argument))
Else
theURL = theURLs(theInt)
End If
Dim blh As String
'For i = 1 To 500
If worker.CancellationPending = True Then
e.Cancel = True
End If
blh = hRequest(theURL, "GET", "")
worker.ReportProgress(1)
End Sub
-
May 4th, 2018, 03:43 AM
#2
Re: 2 Background workers problem
A much better way to do this would be with a ConcurrentQueue(Of T). Because it's a queue, you remove items as you process them so you never have to worry about the index of the next item. Unlike a regular Queue(Of T), the ConcurrentQueue(Of T) is thread-safe, so you know that dequeue operations will be atomic and multiple threads can never interfere with each other. E.g.
vb.net Code:
Private itemList As New List(Of String) Private itemQueue As ConcurrentQueue(Of String) Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click 'Initialise the queue. itemQueue = New ConcurrentQueue(Of String)(itemList) BackgroundWorker1.RunWorkerAsync() BackgroundWorker2.RunWorkerAsync() End Sub Private Sub BackgroundWorker_DoWork(sender As Object, e As DoWorkEventArgs) Handles BackgroundWorker1.DoWork, BackgroundWorker2.DoWork Dim item As String Do While itemQueue.TryDequeue(item) 'Process item here. Loop End Sub
-
May 4th, 2018, 03:45 AM
#3
Re: 2 Background workers problem
Of course, you could just use a single BackgroundWorker and a Parallel.ForEach or Parallel.For call in its DoWork event handler. That would use more than just two threads at the same time so potentially speed things up further.
-
May 4th, 2018, 09:49 AM
#4
Re: 2 Background workers problem
I bet if I knew what hRequest did, I'd have a better idea. Odds are something is going wrong with it, and you have no infrastructure to tell you that.
This answer is wrong. You should be using TableAdapter and Dictionaries instead.
-
May 4th, 2018, 09:54 AM
#5
Re: 2 Background workers problem
It seems likely that the increment of theInt will happen before the first worker gets around to referencing the item in the list, such that theInt really starts at 1. It should have been incremented only in the background thread, but that doesn't seem all that safe, either. I just don't see any reason to expect that the two BGW can share an index in a way that guarantees that each location is only visited one time. The concurrent queue would be better.
My usual boring signature: Nothing
-
May 4th, 2018, 11:00 AM
#6
Thread Starter
New Member
Re: 2 Background workers problem
hRequest is just a basic HTTPWebRequest sub nothing special in there. I know theInt is getting incremented before the first HTTPWebRequest goes out, I just figured there was a better way to write that part to prevent that from happening. I'll look into concurrent queue.
-
May 4th, 2018, 01:10 PM
#7
Re: 2 Background workers problem
That's just a symptom of a larger problem. I assume that you want to use that variable to indicate which item is taken next, but that seems likely to create a race no matter how you do it. That's all that's happening, as the thread that does the initial increment is doing it before the background thread makes use of it. If the two background threads both do the incrementing themselves, there's always a chance that one item will get skipped and another will go twice. Just incrementing the index in a safe fashion isn't enough. As long as the context switches at the right time, bad things will happen.
My usual boring signature: Nothing
-
May 4th, 2018, 01:29 PM
#8
Re: 2 Background workers problem
Yeah I misunderstood, and JMC is sending you in the right direction. Let's use analogy.
Imagine an office with one worker. There is a sign on the wall that tells the worker a phone number to call to talk to a customer and lists the customer name and some other data. This works because there is one worker.
Now imagine there are two workers, but one sign. You display customer information, worker 1 starts working, then you change customer information for worker 2. Worker 1 can't get the old information anymore, and maybe they didn't have it all. This is bad! It's what happened in your first example. Sure, you started a background worker before incrementing the variable, but since it shares the variable with everything else it will get the incremented version!
A better solution is to send all of the customer information to worker 1, and all of the information to worker 2. That way they don't get confused. That's how your "workaround" was working, you just didn't know it's how it was supposed to work. The "problem" with this approach is you have to have as many workers as you have pieces of work, and that's often overwhelming to the computer.
The way JMC is describing it is actually better in many cases. To visualize it, imagine that there is one box with many customer files in it. When worker 1 needs work, he walks to the box, removes a file, then goes to do work with that file. When worker 2 needs work, she does the same thing. Worker 1 might take longer than worker 2 to finish, but that doesn't matter: it just means worker 2 will grab some more work earlier. Since no workers can grab the same file, you know each worker can't duplicate anothers' work.
This answer is wrong. You should be using TableAdapter and Dictionaries instead.
-
May 4th, 2018, 04:01 PM
#9
Thread Starter
New Member
Re: 2 Background workers problem
Thanks for your help guys, this below seems to be working fine. Do you see anything that I might have trouble with in the future?
Code:
Dim theLinks As New ConcurrentQueue(Of String)
Private Sub bgWorker_Click(sender As Object, e As EventArgs) Handles bgWorker.Click
BackgroundWorker1.RunWorkerAsync()
BackgroundWorker2.RunWorkerAsync()
End Sub
Private Sub BackgroundWorker_DoWork(sender As Object, e As System.ComponentModel.DoWorkEventArgs) Handles BackgroundWorker1.DoWork, BackgroundWorker2.DoWork
Dim worker As System.ComponentModel.BackgroundWorker = DirectCast(sender, System.ComponentModel.BackgroundWorker)
Dim theURL As String
theLinks.TryDequeue(theURL)
Dim blah As String
If worker.CancellationPending = True Then e.Cancel = True
blah = WRequest(theURL, "GET", "")
End Sub
Private Sub BackgroundWorker_RunWorkerCompleted(ByVal sender As System.Object, ByVal e As System.ComponentModel.RunWorkerCompletedEventArgs) Handles BackgroundWorker1.RunWorkerCompleted, BackgroundWorker2.RunWorkerCompleted
Dim worker As System.ComponentModel.BackgroundWorker = DirectCast(sender, System.ComponentModel.BackgroundWorker)
If theLinks.Count > 0 Then worker.RunWorkerAsync()
End Sub
-
May 4th, 2018, 07:06 PM
#10
Re: 2 Background workers problem
I think you're going to want to do something like:
If theLinks.TryDequeue(theURL)
'Do stuff here.
End if
There is a chance that you can get into that DoWork method while the queue is empty, in which case TryDequeue will return False and you don't want to go on to call WRequest in that case.
My usual boring signature: Nothing
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts
-
Forum Rules
|
Click Here to Expand Forum to Full Width
|