[Resolved]Thread Pool Download Multiple Files at Once

I'm trying to use the thread pool to download more than one string at a time. But most of the tutorial on threading is in C# or C++ so my knowledge on thread pool is limited.

Here's what I got so far.

Code:

---

Imports System.Threading
Imports System.Net
Imports System.Text.RegularExpressions

Public Class Example

    Private sLocation As String
    Private sName As String
    Private GameURL As String

    Private Sub Add()
        Dim dFile As New WebClient
        Dim FileStr As String
        FileStr = dFile.DownloadString(sLocation & sName)
        If Regex.IsMatch(FileStr, "(?<=<img src="")(.*?)(?="">)") = False Then Exit Sub
        ListBox1.Items.Add(sLocation & Regex.Match(FileStr, "(?<=<img src="")(.*?)(?="">)").Value)
    End Sub

    Private Sub Example_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
        For Each FileLoc As String In IO.File.ReadAllLines(Application.StartupPath & "\Location.txt")
            For Each FileName As String In IO.File.ReadAllLines(Application.StartupPath & "\Name.txt")
                sLocation = FileLoc
                sName = FileName
                ThreadPool.QueueUserWorkItem(AddressOf Add)
            Next
        Next
    End Sub
End Class

---

I keep getting an error "Cross-thread operation not valid: Control 'Listbox1' accessed from a thread other than the thread it was created on." And it still seems to be downloading it one by one to me.

I'll take a look at your code but just to clarify the reason why you are getting that cross thread error is because you are trying to access a control (controls are created on the one and only UI thread) and controls cannot be updated from any thread other than the one they were created on. See JMC's guide to accessing controls from worker threads for info on how to work with this 'feature' - http://www.vbforums.com/showthread.php?t=498387

EDIT: Looking at your code I think you need to start learning a bit more about how threading works. You are accessing the variables sLocation and sName from more than one thread at once, which is bound to cause either 1. exceptions to be thrown or 2. unexpected results.
Basically when you call QueueUserWorkItem then that starts your Add routine on a 'worker' thread - this thread runs at the same time as the rest of your code, so while your code is still busy looping through your text file lines and setting the sLocation and sName variables, your worker threads are trying to read those same variables at the same time. The way you should do it is to declare those variables locally in your Example_Load event and then pass them in to the QueueUserWorkItem method (which in turn passes them in to your Add method). That way when your Add method executes it has its own 'private' copy of the variables that do not get modified when your loop carries on in the original thread (the UI thread).

Thanks for the reply, I fixed the access error but I still have no idea on how to download more than one at the same time.

Code:

---

Imports System.Threading
Imports System.Net
Imports System.Text.RegularExpressions

Public Class Example

    Private sLocation As String
    Private sName As String
    Private FileStr As String

    Private Sub AddItem()
        ListBox1.Items.Add(sLocation & Regex.Match(FileStr, "(?<=<img src="")(.*?)(?="">)").Value)
    End Sub

    Private Sub Add()
        Dim dFile As New WebClient
        FileStr = dFile.DownloadString(sLocation & sName)
        If Regex.IsMatch(FileStr, "(?<=<img src="")(.*?)(?="">)") = False Then Exit Sub
        SyncLock ListBox1.GetType
            ListBox1.Invoke(CType(AddressOf AddItem, MethodInvoker))
        End SyncLock
    End Sub

    Private Sub Example_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
        For Each FileLoc As String In IO.File.ReadAllLines(Application.StartupPath & "\Location.txt")
            For Each FileName As String In IO.File.ReadAllLines(Application.StartupPath & "\Name.txt")
                sLocation = FileLoc
                sName = FileName
                ThreadPool.QueueUserWorkItem(AddressOf Add)
            Next
        Next
    End Sub
End Class

---

You havent fixed the bits I mentioned though about those sLocation and sName variables being accessed by multiple threads at the same time. That could very well be what is causing you problems... and even if its not, it certainly will cause some problems or unexpected results.

Quote:

---

Originally Posted by chris128

You havent fixed the bits I mentioned though about those sLocation and sName variables being accessed by multiple threads at the same time. That could very well be what is causing you problems... and even if its not, it certainly will cause some problems or unexpected results.

---

I hope this is what you meant, sort of having a hard time grasping what you wrote.

This code still seems to download one at a time, If you could, please provide any snippets or examples, that would be greatly appreciated.

Code:

---

Imports System.Threading
Imports System.Net
Imports System.Text.RegularExpressions

Public Class TaskInfo

    Public Shared sLocation As String
    Public Shared sName As String

    Public Sub New(ByVal Loc As String, ByVal Str As String)
        sLocation = Loc
        sName = Str
    End Sub
End Class

Public Class Example

    Private FileStr As String

    Private Sub AddItem()
        ListBox1.Items.Add(TaskInfo.sLocation & Regex.Match(FileStr, "(?<=<img src="")(.*?)(?="">)").Value)
    End Sub

    Private Sub Add()
        Dim dFile As New WebClient
        FileStr = dFile.DownloadString(TaskInfo.sLocation & TaskInfo.sName)
        If Regex.IsMatch(FileStr, "(?<=<img src="")(.*?)(?="">)") = False Then Exit Sub
        SyncLock ListBox1.GetType
            ListBox1.Invoke(CType(AddressOf AddItem, MethodInvoker))
        End SyncLock
    End Sub

    Private Sub Example_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
        For Each FileLoc As String In IO.File.ReadAllLines(Application.StartupPath & "\Location.txt")
            For Each FileName As String In IO.File.ReadAllLines(Application.StartupPath & "\Name.txt")
                Dim ti As New TaskInfo(FileLoc, FileName)
                ThreadPool.QueueUserWorkItem((AddressOf Add), ti)
            Next
        Next
    End Sub
End Class

---

No thats not quite it, you have almost got it but the variables need to not be declared as Shared because that makes them accessible to everything still and not instance specific. That way you dont need a SyncLock in there (yours is in slightly the wrong place anyway though ;))
Also, you need to change the signature of the Add method so that it accepts the instance of your TaskInfo class that you are passing in and the AddItem signature also needs to be changed.

Here is how I would do it personally:

This would be the TaskInfo class:

vb Code:

Public Class TaskInfo
        Public sLocation As String
        Public sName As String
        Public FileStr As String
 
        Public Sub New(ByVal Loc As String, ByVal Str As String)
            sLocation = Loc
            sName = Str
        End Sub
End Class

and this would be your Example class:

vb Code:

Public Class Example
 
        Private Sub AddItem(ByVal info As TaskInfo)
            listbox1.Items.Add(info.sLocation & Regex.Match(info.FileStr, "(?<=<img src="")(.*?)(?="">)").Value)
        End Sub
 
        Private Sub Add(ByVal info As Object)
            Dim iTaskInfo As TaskInfo = DirectCast(info, TaskInfo)
            Dim dFile As New WebClient
            iTaskInfo.FileStr = dFile.DownloadString(iTaskInfo.sLocation & iTaskInfo.sName)
            If Regex.IsMatch(iTaskInfo.FileStr, "(?<=<img src="")(.*?)(?="">)") = False Then Exit Sub
            listbox1.Invoke(New Action(Of TaskInfo)(AddressOf AddItem), iTaskInfo)
        End Sub
 
        Private Sub Example_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
            For Each FileLoc As String In IO.File.ReadAllLines(Application.StartupPath & "\Location.txt")
                For Each FileName As String In IO.File.ReadAllLines(Application.StartupPath & "\Name.txt")
                    Dim ti As New TaskInfo(FileLoc, FileName)
                    ThreadPool.QueueUserWorkItem((AddressOf Add), ti)
                Next
            Next
        End Sub
End Class

Note that I havent tested this, its just off the top of my head mostly so you might need to make some minor changes before it will work for you... Also I have stuck to the same variable names that you used originally (FileStr etc) - you might want to change this now so that all of the variables use the same format, i.e you have sLocation and sName so I'm guessing sFile would be apt for the FileStr name.
Also its good practice to expose those Public variables in the TaskInfo class as properties rather than as fields but its not strictly necessary. An example of that would be:

vb Code:

Public Class TaskInfo
 
        Private _sFile As String
        Public Property sFile() As String
            Get
                Return _sFile
            End Get
            Set(ByVal value As String)
                _sFile = value
            End Set
        End Property
 
        Private _sLocation As String
        Public Property sLocation() As String
            Get
                Return _sLocation
            End Get
            Set(ByVal value As String)
                _sLocation = value
            End Set
        End Property
 
        Private _sName As String
        Public Property sName() As String
            Get
                Return _sName
            End Get
            Set(ByVal value As String)
                _sName = value
            End Set
        End Property
 
        Public Sub New(ByVal Loc As String, ByVal Str As String)
            sLocation = Loc
            sName = Str
        End Sub
 
End Class

That help? :)

Thanks for the help, I tried increasing the speed with Threading.ThreadPool.SetMinThreads, but that doesn't seem to help. I'll read about encapsulating data structures with properties.

Have you actually tried the code I've provided above?

Quote:

---

Originally Posted by chris128

Have you actually tried the code I've provided above?

---

Yes I did, it worked perfectly so I didn't change anything except the variable name you mentioned.

Code:

---

Imports System.Threading
Imports System.Net
Imports System.Text.RegularExpressions

Public Class TaskInfo
    Public sLocation As String
    Public sName As String
    Public sFile As String

    Public Sub New(ByVal Loc As String, ByVal Str As String)
        sLocation = Loc
        sName = Str
    End Sub
End Class

Public Class Example

    Private Sub AddItem(ByVal info As TaskInfo)
        ListBox1.Items.Add(info.sLocation & Regex.Match(info.sFile, "(?<=<img src="")(.*?)(?="">)").Value)
    End Sub

    Private Sub Add(ByVal info As Object)
        Dim iTaskInfo As TaskInfo = DirectCast(info, TaskInfo)
        Dim dFile As New WebClient
        iTaskInfo.sFile = dFile.DownloadString(iTaskInfo.sLocation & iTaskInfo.sName)
        If Regex.IsMatch(iTaskInfo.sFile, "(?<=<img src="")(.*?)(?="">)") = False Then Exit Sub
        ListBox1.Invoke(New Action(Of TaskInfo)(AddressOf AddItem), iTaskInfo)
    End Sub

    Private Sub Example_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
        For Each FileLoc As String In IO.File.ReadAllLines(Application.StartupPath & "\Location.txt")
            For Each FileName As String In IO.File.ReadAllLines(Application.StartupPath & "\Name.txt")
                Dim ti As New TaskInfo(FileLoc, FileName)
                ThreadPool.QueueUserWorkItem((AddressOf Add), ti)
            Next
        Next
    End Sub
End Class

---

Oh right, I thought you would have mentioned that it worked in your last post, rather than just saying you still have a problem... :)
If your original problem that you created this thread for is now working then you should mark this thread as Resolved (using the Thread Tools menu at the top of the page). If you have any more problems then open a new thread :)

Quote:

---

Originally Posted by chris128

Oh right, I thought you would have mentioned that it worked in your last post, rather than just saying you still have a problem... :)
If your original problem that you created this thread for is now working then you should mark this thread as Resolved (using the Thread Tools menu at the top of the page). If you have any more problems then open a new thread :)

---

Thanks, I'll see what I can work on from here.