Use X number of threads to perform Y number of tasks

[nbrege]

I want to be able to specify the number of threads to work on a common list of tasks. So for example, if I have a list of 87 files to download & I want to use 5 threads to download those files, how would I set up such a mechanism? My initial thought is to start 5 threads & each thread would retrieve the first (or maybe last) URL from the list & then immediately remove the URL from the list so that it isn't read by another thread. Does that seem like the right approach? My concern is that the same URL might be retrieved more than once if a thread doesn't remove it fast enough before another thread tries to read the next URL from the list.

[dbasnett]

For the collection of URL's use a ConcurrentQueue. The act of removing will alleviate the concurrency concern.

[nbrege]

For the collection of URL's use a ConcurrentQueue. The act of removing will alleviate the concurrency concern.

Thank you. I was not even aware of such a thing as ConcurrentQueue...

[Niya]

I want to be able to specify the number of threads to work on a common list of tasks. So for example, if I have a list of 87 files to download & I want to use 5 threads to download those files, how would I set up such a mechanism? My initial thought is to start 5 threads & each thread would retrieve the first (or maybe last) URL from the list & them immediately remove the URL from the list so that it isn't read by another thread. Does that seem like the right approach? My concern is that the same URL might be retrieved more than once if a thread doesn't remove it fast enough before another thread tries to read the next URL from the list.

While the solution above given by dbasnett is the ideal solution here, just for completeness sake, I will also tell you how to do this yourself.

This is very easy to guard against. Wrap access to the list in a SyncLock block. Anything within a SyncLock block if done correctly is guaranteed to never be executed by more than one thread at the same time. Here's what it looks like:-

Public Class FileDownloaderMockup

    'List of URLs to download
    Private _filesURL As New List(Of String)

    'Object to lock use for locking
    Private _lock As New Object


    Public Sub DownloadFile()

        Dim url As String

        'No two threads will EVER enter this block at the same time
        SyncLock _lock

            'Retrieve the URL
            url = _filesURL.Item(0)

            'Remove it from the list so no other
            'thread will attempt to download this file
            _filesURL.RemoveAt(0)
        End SyncLock


        'Pretend this is you code to download the file

    End Sub


End Class

I could almost guarantee this is what the ConcurrentQueue is doing internally.

[Shaggy Hiker]

Is there a reason for specifying the number of threads? Usually, that's a bad idea. Multithreading is only advantageous if the CPU has the capacity to do something. For example, if you have a single core, then you may not get any advantage from two threads unless one thread has waiting for something else to happen. Downloading is an excellent situation for multiple threads, as each thread may well be waiting for a download, which frees up the hardware it was using to do something else while it waits. Generally, the OS will be better at figuring out how to optimize use. If a core is idle, and there is work left to be done, then it might launch a new thread for that work. If all cores are occupied, then it might starve a thread, or eliminate it.

For this reason, it is generally not a good idea to set the number of threads unless you are setting the number to something less than what the OS would otherwise use. That would make sense if you know that you want to devote less of the CPU to those threads than might otherwise be available.

[nbrege]

Shaggy ... in my case I am downloading files. I just assumed (maybe incorrectly) that using multiple threads would speed up the process. What would be an example of where using multiple threads would not be advantageous?

[2kaud]

For cpu-bound tasks, the number of threads shouldn't exceed the number of processing cores available - otherwise at least one task is idle at any one time waiting to be scheduled core time by the scheduler. This also involves an overhead. For non-cpu-bound threads you need to ensure that the threads aren't trying to access the same resource at the same time - otherwise you either have a race condition(s) and the program is unstable or thread(s) spend their time waiting for locks. For downloading files you have 1 resource - the internet connection (unless you have several - lucky you!). This is likely to be the limiting factor. Whilst you can download multiple files together, the capacity of the connection/network/wi-fi is fixed. So if 1 file downloads in say 10 secs then downloading 10 files together is unlikely to be quicker.

[Niya]

Shaggy ... in my case I am downloading files. I just assumed (maybe incorrectly) that using multiple threads would speed up the process. What would be an example of where using multiple threads would not be advantageous?

You absolutely should be using multiple threads for file downloads. Now 2kaud pointed out something important above which is that bandwidth is a limiting factor. However, this is true only under the condition that each download is capable of utilizing all of your available bandwidth. In practice, many sites will not utilize all of your bandwidth which leaves plenty left over for parallel downloads. Using multiple threads will always speed up such downloads provided you are downloading files from different sources. Even if each download is capable of using your full bandwidth, you will lose very little by using multiple threads.

[dbasnett]

Another possibility would be Parallel.ForEach.

    Private Async Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
        Dim URLs As New List(Of String) From {"URL1", "URL2", "URL3", "URL4", "URL5", "URL6", "URL7", "URLetc"}
        Dim procURLs As New Concurrent.ConcurrentQueue(Of String)
        Dim tsk As Task
        tsk = Task.Run(Sub()
                           Parallel.ForEach(URLs, Sub(url)
                                                      Try
                                                          'download code here
                                                          Threading.Thread.Sleep(10) 'simulate download
                                                          procURLs.Enqueue(url)
                                                      Catch ex As Exception

                                                      End Try
                                                  End Sub)
                       End Sub)
        Await tsk
    End Sub

[Shaggy Hiker]

Anything over a network is a likely candidate to benefit from threading. I was mostly curious as to why you felt 5 threads was the right number? There might be a good reason, or there might not. The thread scheduler is usually (though not always) going to be more efficient at figuring out how many threads to spin up. Therefore, it is often ideal to just thread them all and let the OS figure out how many to have active. I had an exception to that where I was doing a VERY CPU-bound operation. I was not just sure that I would lose ground if I had more threads than I had CPU cores, I was pretty sure that my number of threads shouldn't exceed the number of CPU cores minus one. With modern CPUs, there can be a whole lot of cores, so I was thinking that you might have a six core system and were choosing 6-1 threads.

[2kaud]

You absolutely should be using multiple threads for file downloads

Anything over a network is a likely candidate to benefit from threading

Not in practice in my experience. Multiple local file reads - yes, I've had improvements with threads for that. Same for multiple threads for reading a very large file in chunks. But simultaneous multiple files with threading over network - no.

[Shaggy Hiker]

With any threading, testing is warranted to see whether there is a real improvement. I'm thinking that if each download was a Task, then whether they run on different threads or not, and how many threads if they do run on threads, isn't a question that the OP need be all that involved with. My understanding is that the OS can decide where to run a Task.

However, I was really getting at, "why 5 threads?"

[2kaud]

Well if this is going to be a recurring operation, then I'd suggest trying it with different number of threads (1 - 40?) and find where the 'sweet point' is. At some point you should be seeing increasing/steady overall time taken - as opposed to decreasing overall time. Now you know for your set-up how many threads to use for optimum performance for you.

[dbasnett]

Not in practice in my experience... But simultaneous multiple files with threading over network - no.

That surprises me. I say that because a download is more than the actual transfer. But maybe a test is in order...

[dbasnett]

That surprises me. I say that because a download is more than the actual transfer. But maybe a test is in order...

Test copying remote files to local machine.

        Dim iPath As String = "\\someserver\_LRSys_\PubDocs\2018\sections"
        Dim oPath As String = IO.Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.Desktop), "testSect")

        Try
            If IO.Directory.Exists(oPath) Then
                IO.Directory.Delete(oPath, True)
            End If
            IO.Directory.CreateDirectory(oPath)
        Catch ex As Exception

        End Try

        Dim files() As String = IO.Directory.GetFiles(iPath, "S40*.*") 'approx. 1445 files

        Dim stpw As Stopwatch = Stopwatch.StartNew

        ''Test 1 - about three seconds
        'For Each iFile As String In files
        '    Dim oFile As String = IO.Path.Combine(oPath, IO.Path.GetFileName(iFile))
        '    IO.File.Copy(iFile, oFile, True)
        'Next
        'stpw.Stop()
        ''end Test 1

        'Test 2 - .5 second
        Parallel.ForEach(files, Sub(iFile)
                                    Dim oFile As String = IO.Path.Combine(oPath, IO.Path.GetFileName(iFile))
                                    IO.File.Copy(iFile, oFile, True)
                                End Sub)
        stpw.Stop()
        'end Test 2

        Debug.WriteLine(stpw.ElapsedMilliseconds)

A lot faster using threading.

[2kaud]

OK. What size are the files? I'm talking about multi-hundred megabyte files. Could be quite different experience for smallish files.

[Niya]

Not in practice in my experience. Multiple local file reads - yes, I've had improvements with threads for that. Same for multiple threads for reading a very large file in chunks. But simultaneous multiple files with threading over network - no.

I wasn't thinking in terms of LAN. On the internet it's very common for connections to under utilize bandwidth. For example, I have 100/100 Mbps internet which in practice runs at about 60 Mbps and I almost never see any download utilizing that speed. 60 Mbps is roughly 7.5 MB/sec and a typical fast single connection download for me is about 1 MB/sec which leaves 6.5 whole MB/sec of bandwidth unused. That unused bandwidth could be fully utilized by other threads downloading from other sites.

[dbasnett]

OK. What size are the files? I'm talking about multi-hundred megabyte files. Could be quite different experience for smallish files.

They are small, but overall threading is the way to go. After one thread has read a block of data and is writing it to disk another thread could be reading a block of data for another file. I agree with what Niya said about utilization.