[RESOLVED] Downloading multiple files using threads

[MadCatVB]

Hi,

I hope someone can help me with this. I'm currently writing a small app to download files from the web. At the moment i can download files individually but what i'd really like to do is to start a number of downloads simultaneously and queue the remaininfg ones to wait for download.

My thinking on this (and i could be wrong) was to use an individual thread for each download.

I'm not entirely sure how to go about setting this up. At present i have the code i want to run when the thread is initiated:

    Private Sub getFile()

        Dim wc_MyWebClient As New WebClient
        Dim sURL, sSavePath As String

        'Access property storing the file URL
        SyncLock (Me)
            sURL = propURL
        End SyncLock

        'Access property storing the save path.
        SyncLock (Me)
            sSavePath = propSavePath
        End SyncLock

        'Download file here
        wc_MyWebClient.DownloadFile(sURL, sSavePath)

        wc_MyWebClient.Dispose()

    End Sub

At present the user selects entries from a list of links and this then provides the files requiring download. When the download button is clicked i have this code so far:

    Private Sub btn_DownloadSelectedLinks_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles btn_DownloadSelectedLinks.Click

        Dim sFolderPath As String

        If FolderBrowserDialog1.ShowDialog = Windows.Forms.DialogResult.OK Then
            sFolderPath = FolderBrowserDialog1.SelectedPath
            'MsgBox(sFolderPath)


            downloadSelectedFiles(sFolderPath)
            'getFile

'loop through all selected indexes in the check list box
            For i As Integer = 0 To clb_Links.CheckedIndices.Count - 1 Step 1
                propURL = clb_Links.CheckedItems.Item(i).ToString
                propSavePath = sFolderPath & "\" & Path.GetFileName(clb_Links.CheckedItems.Item(i).ToString)
                Dim t As New Thread(AddressOf getFile)
                t.Start()
            Next

        Else
            'Dialogbox has been cancelled
            Exit Sub
        End If

    End Sub

The problem i have is here. I'm looking for a way that i could set this up so that only (say 3 for this example) threads are started and the rest have to wait until one thread finishes before another can start.

Hope this made sense.

Can anyone provide any ideas of how this may be done?

Many thanks in advance.

[jmcilhinney]

Just declare an array of Threads with three elements. When the user selects a file to download you check the array and see whether there is an element that is a null reference or refers to a thread whose IsAlive property is False. If so then create a new Thread and asign it to that element. If not, put the file in a Queue collection.

[MadCatVB]

JMcIlhinney,

Thanks once again,

So obvious now that its been pointed out

Thats exactly what i'm looking for. Just got to go code it now.

Thanks

Grant

[MadCatVB]

I fear i may have marked this post resolved a little to quickly.

I have taken on board what you suggested and come up with something that almost works perfectly. The only problem i have at present is that the threading seems to get a little mixed up.

Private Sub btn_DownloadSelectedLinks_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles btn_DownloadSelectedLinks.Click

        Dim sFolderPath As String

        If FolderBrowserDialog1.ShowDialog = Windows.Forms.DialogResult.OK Then
            sFolderPath = FolderBrowserDialog1.SelectedPath

            'Populate array with links for download
            For i As Integer = 0 To clb_Links.CheckedIndices.Count - 1 Step 1
                propLinkArray.Add(clb_Links.CheckedItems.Item(i).ToString)
            Next

            'Set up the timer and start it.
            'Timer1.Interval = 1000
            'Timer1.Enabled = True

            'test output
            txt_Log.Text = txt_Log.Text & "propLinkArray = " & propLinkArray.Count & vbCrLf
            txt_Log.Refresh()

            'Begin loop of all links in array
            For i As Integer = 0 To propLinkArray.Count - 1 Step 1

                'test output
                txt_Log.Text = txt_Log.Text & "Array element = " & i & vbCrLf
                txt_Log.Text = txt_Log.Text & "File name is: " & propLinkArray.Item(i).ToString & vbCrLf
                txt_Log.Refresh()

                'This piece of code loops until such time as an element in the
                'thread array becomes free and can be used for a new download thread
                Do Until propThreadAvailable
                    'Thread is available for use at this point. (Stored in propActiveArrayElement)
                    checkThreadArray()
                Loop

                propThreadAvailable = False
                'Believe i have to synclock these properties as they are accessed
                'within threads
                SyncLock (Me)
                    propURL = propLinkArray.Item(i).ToString
                End SyncLock

                SyncLock (Me)
                    propSavePath = sFolderPath & "\" & Path.GetFileName(propLinkArray.Item(i).ToString)
                End SyncLock

                'Create a new thread and assign it to the available element in the
                'thread array. Thread is then started.
                txt_Log.Text = txt_Log.Text & "Assigning thread to array element " & propActiveArrayElement & vbCrLf
                txt_Log.Refresh()
                Dim t As New Thread(AddressOf getFile)
                a_Thread(propActiveArrayElement) = t
                CType(a_Thread(propActiveArrayElement), Thread).Start()

            Next


            'Timer1.Enabled = False

        Else
            'Dialogbox has been cancelled
            Exit Sub
        End If

    End Sub

    Private Sub getFile()

        'Code for downloading the file (called by each thread)

        Dim wc_MyWebClient As New WebClient

        'Download file here
        Try
            SyncLock (Me)
                wc_MyWebClient.DownloadFile(propURL, propSavePath)
            End SyncLock
        Catch ex As Exception
            MsgBox(ex.Message)
        End Try

        wc_MyWebClient.Dispose()

    End Sub


    Private Function checkThreadArray() As Boolean


        For i As Integer = 0 To 2 Step 1
            'MsgBox(a_Thread.Length)

            'Check if array location is empty, if so sets the activeArrayElement to th element location,
            'and also sets propThreadAvailable to true (this is checked in the do until loop)
            If a_Thread(i) Is Nothing Then
                propActiveArrayElement = i
                propThreadAvailable = True
                'Test Output
                txt_Log.Text = txt_Log.Text & "PropActiveArrayElement = " & i & vbCrLf & "PropThreadAvailable = True" & vbCrLf
                txt_Log.Refresh()
                Return True
            End If
        Next

        For i As Integer = 0 To 2 Step 1
            If Not CType(a_Thread(i), Thread).IsAlive Then
                propActiveArrayElement = i
                propThreadAvailable = True
                txt_Log.Text = txt_Log.Text & "PropActiveArrayElement = " & i & vbCrLf & "PropThreadAvailable = True" & vbCrLf
                txt_Log.Refresh()
                Return True
            End If
        Next

        propThreadAvailable = False
        'Test Output
        txt_Log.Text = txt_Log.Text & "PropThreadAvailable = True" & vbCrLf
        txt_Log.Refresh()
        Return False

    End Function

when i come to running my app it downloads some of the files but not all. I'm a little concerned that certain variables pertaining to the savePath and the File URL are getting overwritten prior to the thread accessing them and using them to download the file.

Could some one please have a look at the above code and see if there is anything obviouskly wrong.

Thanks

Grant

[gigemboy]

I had a few problems in my only time using threads, I was able to fix my problems with using the ThreadName.Join() statement (replacing threadname with your name of the thread). I had just put it right after the Thread.start() statement, and it picks up code execution at the Join after the thread has finished. Not sure if thats the issue you are having, just wanted to put that in there...

[jmcilhinney]

If you're going to use Join then there's not much point using threads at all, because your main thread will just sit and wait until the other thread completes. Isn't the whole idea here that the file is downloaded in the background?

I'd say that your issue is the fact that you only have a single variable for things like the file that needs to be downloaded, yet you have an array of threads. You need to have more than one location to store thread data if you have more than one thread to store data for. I would suggest using a HastTable with the Thread itself as the key. That way, once inside the thread you can just use Thread.CurrentThread as the key to get the corresponding data:

Private threadData As New Hashtable
 
    Private Sub StartThread()
        Dim t As New Threading.Thread(AddressOf ThreadStart)
 
        'Set the data for the new thread.
        DirectCast(Me.threadData.SyncRoot, Hashtable).Add(t, "ThreadData")
 
        t.Start()
    End Sub
 
    Private Sub ThreadStart()
        Dim threadData As Hashtable = DirectCast(Me.threadData.SyncRoot, Hashtable)
 
        'Get the data for this thread.
        Dim data As String = CStr(threadData(Threading.Thread.CurrentThread))
 
        'Remove the data for this thread.
        threadData.Remove(Threading.Thread.CurrentThread)
 
        MessageBox.Show(data)
    End Sub

Note that collection access is not inherently thread-safe so you should use the SyncRoot property.

[MadCatVB]

Thanks for the reponse guys,

Gigemboy: I'm not entirely sure that what you have described is the problem i'm having but i appreciate the input.

JMcIlhinney:

Hashtables? I'm going to have to look them up, its been a while since i've heard about hashtables. Am i right in saying that it would be similar to a multi dimensional array. And when I assign a thread as the key i can also assign the variables unique to every thread to other elements in the hash table relating to that key?

If this is the case then I can see where its going and i'll give it a shot and see if i can get it up and running.

In this code that you provided as an example, is "ThreadData" where i would put the URL path or SavePath so that this could be accessed within the thread? Or am i so wide of the mark its untrue?

DirectCast(Me.threadData.SyncRoot, Hashtable).Add(t, "ThreadData")

Many thanks for the help, i've no doubt i'll be postig back on this topic.

Cheers

Grant

[jmcilhinney]

A HashTable could be considered to have some similarity to a multidimensional array but very little. A multidimensional array is a matrix and there is not necessarily a direct relationship between any two elements in the array, other than that they may be in the same row or column. In a HashTable there is a direct relationship between the key and value in each item. There is no concept of position in a HashTable. Each value is accessed via its key.

[MadCatVB]

That was quick, I've not used hashtables before so you'll have to bear with the slowness to pick this up and understand.

I had just edited my previous post to ask if in the following code the "ThreadData" would be replaced by my URL or SavePath? or is that not correct? If that is the case could i then pass an array with these two strings in it to the hashtable?

DirectCast(Me.threadData.SyncRoot, Hashtable).Add(t, "ThreadData")

In the example you gave, would i replace the threadArray that i have set up with 3 elements with the Hashtable stucture and use that instead.

One other thing relating to the code you supplied. I get an error on this code saying:

Unable to cast object of type 'System.Object' to type 'System.Collections.Hashtable'.

DirectCast(Me.ht_ThreadData.SyncRoot, Hashtable).Add(t, "ThreadData")

Thanks for your continued patience (hopefully it won't be stretched to far)

[MadCatVB]

Private Sub btn_DownalodViaHashtable_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles btn_DownalodViaHashtable.Click

        Dim sFolderPath As String

        If FolderBrowserDialog1.ShowDialog = Windows.Forms.DialogResult.OK Then
            sFolderPath = FolderBrowserDialog1.SelectedPath

            'Populate array with links for download
            For i As Integer = 0 To clb_Links.CheckedIndices.Count - 1 Step 1
                propLinkArray.Add(clb_Links.CheckedItems.Item(i).ToString)
            Next

            'Begin loop of all links in array
            For i As Integer = 0 To propLinkArray.Count - 1 Step 1

                propURL = propLinkArray.Item(i).ToString
                propSavePath = sFolderPath & "\" & Path.GetFileName(propLinkArray.Item(i).ToString)

                a_File_Path(0) = propURL.ToString
                a_File_Path(1) = propSavePath.ToString


                Dim t As New Threading.Thread(AddressOf getFileHashTable)
                'Set the data for the new thread.
                'DirectCast(Me.ht_ThreadData.SyncRoot, Hashtable).Add(t, a_File_Path)

                DirectCast(Me.ht_ThreadData, Hashtable).Add(t, a_File_Path)
                t.Start()

            Next

        Else
            'Dialogbox has been cancelled
            Exit Sub
        End If

    End Sub

    Private Sub getFileHashTable()

        'Dim threadData As Hashtable = DirectCast(Me.ht_ThreadData.SyncRoot, Hashtable)
        Dim threadData As Hashtable = DirectCast(Me.ht_ThreadData, Hashtable)
        'Get the data for this thread.
        Dim data() As String = DirectCast(threadData(Threading.Thread.CurrentThread), String())
        MsgBox(data(0))
        MsgBox(data(1))
        'Remove the data for this thread
        threadData.Remove(Threading.Thread.CurrentThread)

    End Sub

Ok, i've got this so far, not sure as to where i should be putting the synclock's. There is an error when trying to cast the ht_ThreadData.syncroot to a hashtable. I've removed that and the main issue i'm having is where to lock the data. i'm still getting wrong variables passed into some threads (presumably due to the data being changed in the for loop prior to it being used in the getFileHashTable() sub)

Does this sound likely? All this thread stuff is confusing

[jmcilhinney]

Sorry, I'm such an idiot. I was getting the SyncRoot and the Synchronized members mixed up. SyncRoot is an Object that you can use as an expression for a SyncLock block to synchronise access to the collection. Synchronized is a Shared function that returns a thread-safe wrapper for the collection. My code from earlier should actually look like this:

Private threadData As New Hashtable
 
    Private Sub StartThread()
        Dim t As New Threading.Thread(AddressOf ThreadStart)
 
        'Set the data for the new thread.
        Hashtable.Synchronized(Me.threadData).Add(t, "ThreadData")
 
        t.Start()
    End Sub
 
    Private Sub ThreadStart()
        Dim threadData As Hashtable = Hashtable.Synchronised(Me.threadData)
 
        'Get the data for this thread.
        Dim data As String = CStr(threadData(Threading.Thread.CurrentThread))
 
        'Remove the data for this thread.
        threadData.Remove(Threading.Thread.CurrentThread)
 
        MessageBox.Show(data)
    End Sub

You don't really need to use SyncLock because you know that no two threads will ever be accessing the same item at the same time, although you could do this if you wanted to:

Private threadData As New Hashtable
 
    Private Sub StartThread()
        Dim t As New Threading.Thread(AddressOf ThreadStart)
 
        SyncLock Me.threadData.SyncRoot
            'Set the data for the new thread.
            Me.threadData.Add(t, "ThreadData")
        End SyncLock
 
        t.Start()
    End Sub
 
    Private Sub ThreadStart()
        SyncLock Me.threadData.SyncLock
            'Get the data for this thread.
            Dim data As String = CStr(Me.threadData(Threading.Thread.CurrentThread))
 
            'Remove the data for this thread.
            threadData.Remove(Threading.Thread.CurrentThread)
        End SyncLock
 
        MessageBox.Show(data)
    End Sub

The SyncLocks go around the sections of code that access the same objects from different threads.

As for the value you store in the HashTable against the thread key, it can be whatever you want, and should be whatever data the thread will need. It could be something as simple as an Integer or as complex as an array of class instances. It really depends on what you need.

[MadCatVB]

That certainly runs through the code i have written much better.
I apologise for all the questions but its still not working as i'd expect.

If i select two links to download (in this case .jpgs) it certainly starts the two threads however:

        Dim data() As String = DirectCast(threadData(Threading.Thread.CurrentThread), String())
        MsgBox(data(0))
        MsgBox(data(1))

the code above should be returning the URL and the SavePath as the results of the messagebox.

When the message box appears both URL's relate to the second file to be downloaded and upon clicking both SavePaths are that of the second download. For some reason unknown to myself it doesn't pick up the separate variables.

It looks again as if the threads are accessing the same data and not the individual ones passed to the hashtable value.

Interestingly enough when i change the value added to the hash table from the array to the integer i (as obtained from the for loop) and change the out put to pick up this data it works fine.
Could this shed some light on what is happening. Is it maybe because the same object is passed (i.e. the array a_File_Path) to each of the hashtables only with different values in the elements each time? (just a guess)

I'm a bit slow to pick things up so thanks for replying.

[MadCatVB]

Private Sub btn_DownalodViaHashtable_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles btn_DownalodViaHashtable.Click

        Dim sFolderPath As String

        If FolderBrowserDialog1.ShowDialog = Windows.Forms.DialogResult.OK Then
            sFolderPath = FolderBrowserDialog1.SelectedPath

            For i As Integer = 0 To clb_Links.CheckedIndices.Count - 1 Step 1
                propLinkArray.Add(clb_Links.CheckedItems.Item(i).ToString)
            Next

            For i As Integer = 0 To propLinkArray.Count - 1 Step 1

                propURL = propLinkArray.Item(i).ToString
                propSavePath = sFolderPath

                Dim t As New Threading.Thread(AddressOf getFileHashTable)
                Hashtable.Synchronized(Me.ht_ThreadData).Add(t, propURL)
                t.Start()
            Next

        Else
            'Dialogbox has been cancelled
            Exit Sub
        End If

    End Sub

    Private Sub getFileHashTable()

        Dim threadData As Hashtable = Hashtable.Synchronized(Me.ht_ThreadData)
        'Get the data for this thread.
        Dim data As String = DirectCast(threadData(Threading.Thread.CurrentThread), String)

        'Code for downloading the file (called by each thread)
        Dim wc_MyWebClient As New WebClient
        wc_MyWebClient.DownloadFile(data, propSavePath & "\" & Path.GetFileName(data))
        wc_MyWebClient.Dispose()

        'Remove the data for this thread
        threadData.Remove(Threading.Thread.CurrentThread)

    End Sub

OK, a little update. After some perseverence i've found a way to get the downloads working and what seems like threaded properly.

The array that i was adding to the hashtable as the value containing the file URL and the save path was not working as it was supposed to.

I'd still like to know why this is the case if anyone knows.

I now have to try and set it up so that only a specified number of downloads (threads) are run simultaneously. It worked bfore using the array and i have got some code that initially counts the entries in the hash table to find if is possible to add any more threads.

    Private Sub checkHashTable()

        Dim threadData As Hashtable = Hashtable.Synchronized(Me.ht_ThreadData)
        'Check the length of the Hashtable first of all to see if there is space for any new
        'entries.
        If ht_ThreadData.Count < 3 Then
            'MsgBox(ht_ThreadData.Count)
            'There is an empty slot waiting to be filled. The element available
            'is the count.
            propActiveArrayElement = ht_ThreadData.Count
            propThreadAvailable = True

        End If


    End Sub

What i need to check for after any empty locations is if there are any threads that have finished running and do not flag up as IsAlive.

How would i go about accessing the threads in the hashtable entries to check this.

Thanks

[congminh6]

Hi,

where you do define ht_ThreadData in your code? I had errors in this code
Dim threadData As Hashtable = Hashtable.Synchronized(Me.ht_ThreadData)

Error 1 'ht_ThreadData' is not a member of 'A.frmMain'.

pls help, i'm new in vb.

thanks