Results 1 to 10 of 10

Thread: 2 Background workers problem

  1. #1

    Thread Starter
    New Member
    Join Date
    May 2018
    Posts
    3

    2 Background workers problem

    I have 2 background workers that are running through a list of URLs.

    When I start them off the first URL gets skipped and the second URL gets visited by both threads, instead of just one. The rest of the list runs fine without duplicate requests. I have a workaround for it that works but I feel like it's not the best way to fix it.

    Code that ends up visiting the second URL twice, skipping the first:

    Code:
    Private Sub bgWorker_Click(sender As Object, e As EventArgs) Handles bgWorker.Click
            theInt = 0
            BackgroundWorker1.RunWorkerAsync()
            System.Threading.Interlocked.Increment(theInt)
            BackgroundWorker2.RunWorkerAsync()
        End Sub
    Workaround code I made:

    Code:
    Private Sub bgWorker_Click(sender As Object, e As EventArgs) Handles bgWorker.Click
            theInt = 0
            BackgroundWorker1.RunWorkerAsync(0)
            System.Threading.Interlocked.Increment(theInt)
            BackgroundWorker2.RunWorkerAsync(1)
        End Sub
        Private Sub BackgroundWorker_DoWork(sender As Object, e As System.ComponentModel.DoWorkEventArgs) Handles BackgroundWorker1.DoWork, BackgroundWorker2.DoWork
            Dim worker As System.ComponentModel.BackgroundWorker = DirectCast(sender, System.ComponentModel.BackgroundWorker)
            Dim theURL As String
            If Len(e.Argument) > 0 Then
                theURL = theURLs(Int(e.Argument))
            Else
                theURL = theURLs(theInt)
            End If
    
            Dim blh As String
            'For i = 1 To 500
            If worker.CancellationPending = True Then
                    e.Cancel = True
            End If
                blh = hRequest(theURL, "GET", "")
            worker.ReportProgress(1)
        End Sub

  2. #2
    .NUT jmcilhinney's Avatar
    Join Date
    May 2005
    Location
    Sydney, Australia
    Posts
    107,159

    Re: 2 Background workers problem

    A much better way to do this would be with a ConcurrentQueue(Of T). Because it's a queue, you remove items as you process them so you never have to worry about the index of the next item. Unlike a regular Queue(Of T), the ConcurrentQueue(Of T) is thread-safe, so you know that dequeue operations will be atomic and multiple threads can never interfere with each other. E.g.
    vb.net Code:
    1. Private itemList As New List(Of String)
    2. Private itemQueue As ConcurrentQueue(Of String)
    3.  
    4. Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
    5.     'Initialise the queue.
    6.     itemQueue = New ConcurrentQueue(Of String)(itemList)
    7.  
    8.     BackgroundWorker1.RunWorkerAsync()
    9.     BackgroundWorker2.RunWorkerAsync()
    10. End Sub
    11.  
    12. Private Sub BackgroundWorker_DoWork(sender As Object, e As DoWorkEventArgs) Handles BackgroundWorker1.DoWork,
    13.                                                                                     BackgroundWorker2.DoWork
    14.     Dim item As String
    15.  
    16.     Do While itemQueue.TryDequeue(item)
    17.         'Process item here.
    18.     Loop
    19. End Sub

  3. #3
    .NUT jmcilhinney's Avatar
    Join Date
    May 2005
    Location
    Sydney, Australia
    Posts
    107,159

    Re: 2 Background workers problem

    Of course, you could just use a single BackgroundWorker and a Parallel.ForEach or Parallel.For call in its DoWork event handler. That would use more than just two threads at the same time so potentially speed things up further.

  4. #4
    You don't want to know.
    Join Date
    Aug 2010
    Posts
    4,578

    Re: 2 Background workers problem

    I bet if I knew what hRequest did, I'd have a better idea. Odds are something is going wrong with it, and you have no infrastructure to tell you that.
    This answer is wrong. You should be using TableAdapter and Dictionaries instead.

  5. #5
    Super Moderator Shaggy Hiker's Avatar
    Join Date
    Aug 2002
    Location
    Idaho
    Posts
    36,280

    Re: 2 Background workers problem

    It seems likely that the increment of theInt will happen before the first worker gets around to referencing the item in the list, such that theInt really starts at 1. It should have been incremented only in the background thread, but that doesn't seem all that safe, either. I just don't see any reason to expect that the two BGW can share an index in a way that guarantees that each location is only visited one time. The concurrent queue would be better.
    My usual boring signature: Nothing

  6. #6

    Thread Starter
    New Member
    Join Date
    May 2018
    Posts
    3

    Re: 2 Background workers problem

    hRequest is just a basic HTTPWebRequest sub nothing special in there. I know theInt is getting incremented before the first HTTPWebRequest goes out, I just figured there was a better way to write that part to prevent that from happening. I'll look into concurrent queue.

  7. #7
    Super Moderator Shaggy Hiker's Avatar
    Join Date
    Aug 2002
    Location
    Idaho
    Posts
    36,280

    Re: 2 Background workers problem

    That's just a symptom of a larger problem. I assume that you want to use that variable to indicate which item is taken next, but that seems likely to create a race no matter how you do it. That's all that's happening, as the thread that does the initial increment is doing it before the background thread makes use of it. If the two background threads both do the incrementing themselves, there's always a chance that one item will get skipped and another will go twice. Just incrementing the index in a safe fashion isn't enough. As long as the context switches at the right time, bad things will happen.
    My usual boring signature: Nothing

  8. #8
    You don't want to know.
    Join Date
    Aug 2010
    Posts
    4,578

    Re: 2 Background workers problem

    Yeah I misunderstood, and JMC is sending you in the right direction. Let's use analogy.

    Imagine an office with one worker. There is a sign on the wall that tells the worker a phone number to call to talk to a customer and lists the customer name and some other data. This works because there is one worker.

    Now imagine there are two workers, but one sign. You display customer information, worker 1 starts working, then you change customer information for worker 2. Worker 1 can't get the old information anymore, and maybe they didn't have it all. This is bad! It's what happened in your first example. Sure, you started a background worker before incrementing the variable, but since it shares the variable with everything else it will get the incremented version!

    A better solution is to send all of the customer information to worker 1, and all of the information to worker 2. That way they don't get confused. That's how your "workaround" was working, you just didn't know it's how it was supposed to work. The "problem" with this approach is you have to have as many workers as you have pieces of work, and that's often overwhelming to the computer.

    The way JMC is describing it is actually better in many cases. To visualize it, imagine that there is one box with many customer files in it. When worker 1 needs work, he walks to the box, removes a file, then goes to do work with that file. When worker 2 needs work, she does the same thing. Worker 1 might take longer than worker 2 to finish, but that doesn't matter: it just means worker 2 will grab some more work earlier. Since no workers can grab the same file, you know each worker can't duplicate anothers' work.
    This answer is wrong. You should be using TableAdapter and Dictionaries instead.

  9. #9

    Thread Starter
    New Member
    Join Date
    May 2018
    Posts
    3

    Re: 2 Background workers problem

    Thanks for your help guys, this below seems to be working fine. Do you see anything that I might have trouble with in the future?

    Code:
    Dim theLinks As New ConcurrentQueue(Of String)
    Private Sub bgWorker_Click(sender As Object, e As EventArgs) Handles bgWorker.Click
            BackgroundWorker1.RunWorkerAsync()
            BackgroundWorker2.RunWorkerAsync()
    End Sub
    Private Sub BackgroundWorker_DoWork(sender As Object, e As System.ComponentModel.DoWorkEventArgs) Handles BackgroundWorker1.DoWork, BackgroundWorker2.DoWork
            Dim worker As System.ComponentModel.BackgroundWorker = DirectCast(sender, System.ComponentModel.BackgroundWorker)
            Dim theURL As String
            theLinks.TryDequeue(theURL)
            Dim blah As String
            If worker.CancellationPending = True Then e.Cancel = True
            
            blah = WRequest(theURL, "GET", "")
    End Sub
    Private Sub BackgroundWorker_RunWorkerCompleted(ByVal sender As System.Object, ByVal e As System.ComponentModel.RunWorkerCompletedEventArgs) Handles BackgroundWorker1.RunWorkerCompleted, BackgroundWorker2.RunWorkerCompleted
            Dim worker As System.ComponentModel.BackgroundWorker = DirectCast(sender, System.ComponentModel.BackgroundWorker)
            If theLinks.Count > 0 Then worker.RunWorkerAsync()
    End Sub

  10. #10
    Super Moderator Shaggy Hiker's Avatar
    Join Date
    Aug 2002
    Location
    Idaho
    Posts
    36,280

    Re: 2 Background workers problem

    I think you're going to want to do something like:

    If theLinks.TryDequeue(theURL)
    'Do stuff here.
    End if

    There is a chance that you can get into that DoWork method while the queue is empty, in which case TryDequeue will return False and you don't want to go on to call WRequest in that case.
    My usual boring signature: Nothing

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width