dcsimg
Results 1 to 23 of 23

Thread: MultiThreading a LOOP

  1. #1

    Thread Starter
    Frenzied Member some1uk03's Avatar
    Join Date
    Jun 2006
    Location
    London, UK
    Posts
    1,504

    Question MultiThreading a LOOP

    I've always found the topic of Multi threading somewhat too involved in VB.

    I have a BIG loop of items that process a batch of files. Normally takes 30+ mins to complete.
    Would be nice to be able to figure out how to enable MultiThreading in VB so I could speed up this loop.

    Any templates to get me started?
    _____________________________________________________________________

    ----If this post has helped you. Please take time to Rate it.
    ----If you've solved your problem, then please mark it as RESOLVED from Thread Tools.



  2. #2
    PowerPoster
    Join Date
    Jun 2013
    Posts
    4,586

    Re: MultiThreading a LOOP

    Quote Originally Posted by some1uk03 View Post
    I've always found the topic of Multi threading somewhat too involved in VB.

    I have a BIG loop of items that process a batch of files. Normally takes 30+ mins to complete.
    Would be nice to be able to figure out how to enable MultiThreading in VB so I could speed up this loop.

    Any templates to get me started?
    Before you dive into multithreading, I'd try to optimize "the loop" in single-threaded mode first.

    And since you talk about files, a few more infos would be good as e.g.:
    - the total amount of files your current 30min-loop is touching
    - the average size of a single file (in case you "go through a files content")
    - and the type of file - or operation you perform (on said file - as e.g. "word-parsing on text-files")

    This would help, to give better advice (whether threading is helpful at all, or whether single-threaded optimization can help).

    Olaf

  3. #3
    PowerPoster
    Join Date
    Feb 2006
    Posts
    20,850

    Re: MultiThreading a LOOP

    For something that heavy you may as well use a separate process anyway.

  4. #4

    Thread Starter
    Frenzied Member some1uk03's Avatar
    Join Date
    Jun 2006
    Location
    London, UK
    Posts
    1,504

    Re: MultiThreading a LOOP

    To expand on this a little further, I have a list of 10k+ items.
    So a custom class/byteArray of 10k Items with a file size averaging 100kb to 1mb per item.

    So the loop goes on one by one to "process" these items.
    By process, I mean it goes through a custom Function to do some processing..... let's assume it's something similar to ZIPPing/Encoding the data.

    I've already tried to optimise the processing as much as I could and loop takes an average of 1.5+ secs per file.
    _____________________________________________________________________

    ----If this post has helped you. Please take time to Rate it.
    ----If you've solved your problem, then please mark it as RESOLVED from Thread Tools.



  5. #5
    Super Moderator Shaggy Hiker's Avatar
    Join Date
    Aug 2002
    Location
    Idaho
    Posts
    34,549

    Re: MultiThreading a LOOP

    It sounds like the bulk of that 1.5 seconds might be spent in the CPU. If that is the case, then threading will not gain you as much as you might expect. Threading really shines brightest when each thread will spend a fair amount of time waiting for something. Reading and writing the file may or may not count, but zipping/encoding is more of a processor dependent action. There won't be any waiting to speak of on any thread. That would mean that, if the number of threads gets higher than the number of available execution cores (which may well be less than the number of cores in the CPU), then the threads will slow down the overall time as the cost of switching contexts begins to add up.

    I don't know how well VB6 handles thread pooling, but if you have to do that yourself, you need to be careful how many threads you spin up, or you'll see total performance go down rather than up. A language with more tools for threading would allow you to fire off threads (or something like them, such as Tasks in .NET) for all the files all at once, but it won't launch all those threads. Instead, it would stage them all, but keep the number of actual threads down to whatever number is optimal for performance (or as best as can be guessed at by the language and/or OS). As each thread finished, it would get the next task. So, the number of actual threads could be one or more, and the benefit of threading would differ on different hardware.
    My usual boring signature: Nothing

  6. #6
    Frenzied Member
    Join Date
    Mar 2008
    Posts
    1,173

    Re: MultiThreading a LOOP

    You are probably wanting your single app to truly multithread multiple tasks specified within it however if you have a bunch of tasks specified in multiple batch files or other programs you can run them concurrently by just letting the (multitasking) Windows OS do it for you. e.g;

    Code:
    Option Explicit
    
    Public Declare Function OpenProcess Lib "kernel32" (ByVal dwDesiredAccess As Long, ByVal bInheritHandle As Long, ByVal dwProcessId As Long) As Long
    Public Declare Function WaitForInputIdle Lib "user32" (ByVal hProcess As Long, ByVal dwMilliseconds As Long) As Long
    Public Declare Function WaitForSingleObject Lib "kernel32" (ByVal hHandle As Long, ByVal dwMilliseconds As Long) As Long
    Public Declare Function CloseHandle Lib "kernel32" (ByVal hObject As Long) As Long
    Public Const INFINITE = &HFFFF, SYNCHRONIZE = &H100000
    
    Sub Main()
        'no Form is required
    
        Dim Pid(0 To 3) As Long, hProc As Long
        Dim i As Integer, t As Single
        
        t! = Timer
                       
        'Run the batch files concurrently, each in its own hidden Cmd window and return the Process id for each
        'Full paths to the Batch files are required of course
        Pid(1) = Shell(Environ$("COMSPEC") & " /C " & "BatchFile1.bat", vbHide)
        Pid(2) = Shell(Environ$("COMSPEC") & " /C " & "BatchFile2.bat", vbHide)
        Pid(3) = Shell(Environ$("COMSPEC") & " /C " & "BatchFile3.bat", vbHide)
            
        'Prevent THIS app from closing until the last process has completed
        For i = 1 To UBound(Pid)
            hProc = OpenProcess(SYNCHRONIZE, 0&, Pid(i))
            If hProc <> 0& Then
                WaitForInputIdle hProc, INFINITE
                WaitForSingleObject hProc, INFINITE
                CloseHandle hProc
            End If
        Next
        
        MsgBox "That took " & Timer - t & " seconds"
        
    End Sub
    Personally I use something like this to compile a bunch of about 20 .vbp files to exes. If I use a simple batch file and compile the projects in serial fashion the process takes about 25 seconds, doing it with code like the above takes about 7 seconds.

  7. #7

    Thread Starter
    Frenzied Member some1uk03's Avatar
    Join Date
    Jun 2006
    Location
    London, UK
    Posts
    1,504

    Re: MultiThreading a LOOP

    Shaggy Hiker: Essentially it is heavy CPU based. But essentially these numbers are quick with my PC and I have a pretty decent fast PC. So in comparison with an average PC, multiply my numbers by 2 or 3.
    Oh and there is no file read/write. It's all in Memory.



    Magic Ink: These are not external files that I'm dealing with. All these items are loaded within the APP/MEMORY.
    Essentially, think of it as an Asynchronous Download Manager. Where you can download multiple files with separate progress bars etc...

    It's the same methodology. I have a list of 10k Items and loop that iterates through them, processing each item through a PROCESSING Function.

    So the Idea is to be able to simultaneously pass a number of these via the Processing Function, rather than one by one.
    _____________________________________________________________________

    ----If this post has helped you. Please take time to Rate it.
    ----If you've solved your problem, then please mark it as RESOLVED from Thread Tools.



  8. #8
    PowerPoster
    Join Date
    Feb 2002
    Location
    Canada, Toronto
    Posts
    5,794

    Re: MultiThreading a LOOP

    I have posted in the CodeBank a multithreading example: http://www.vbforums.com/showthread.p...Multithreading

    Also, you can find another simple version in post #9 in that thread

  9. #9
    Frenzied Member
    Join Date
    Jun 2012
    Posts
    1,393

    Re: MultiThreading a LOOP

    Ax-DLL for easy to use Multithreading:
    http://www.vbforums.com/showthread.p...ultithreading)

    With SxS manifest can be used so no registration is needed.

  10. #10
    Frenzied Member
    Join Date
    Feb 2017
    Posts
    2,028

    Re: MultiThreading a LOOP

    As dilettante already mentioned, what about shelling several instances of the program (with some command) or of a helper program?
    It could be much easier than threading.

  11. #11

  12. #12
    Addicted Member
    Join Date
    Jan 2020
    Posts
    255

    Re: MultiThreading a LOOP

    best VB6 CreateThread by Chinese(support ocx,Support UserControl)-VBForums
    http://www.vbforums.com/showthread.p...t-UserControl)

  13. #13
    Addicted Member
    Join Date
    Jan 2020
    Posts
    255

    Re: MultiThreading a LOOP

    best VB6 CreateThread by Chinese(support ocx,Support UserControl)-VBForums
    http://www.vbforums.com/showthread.p...t-UserControl)
    Name:  exe_pic.jpg
Views: 134
Size:  52.2 KB

  14. #14
    Addicted Member
    Join Date
    Jan 2020
    Posts
    255

    Re: MultiThreading a LOOP

    if your cpu like i5 8600k(6 cpu core,Maybe you can you 6 thread or 12 thread)
    you have 10000 plan,put in arr planlist(10000) as string
    so 6 thread use 6 plan,when one thread finished,use plan 7.
    so your cpu will be used for best.
    yor also can view taskmgr.exe(if your cpu only use 30%),you can use 18 threads
    you can reply in my thread :http://www.vbforums.com/showthread.p...t-UserControl)

  15. #15
    Super Moderator Shaggy Hiker's Avatar
    Join Date
    Aug 2002
    Location
    Idaho
    Posts
    34,549

    Re: MultiThreading a LOOP

    Double posting is a problem on this forum. If you get a message saying that you have to wait N seconds before posting again, don't do it. Your post has already gone through, and if you do wait a few seconds, you'll end up with two posts.

    One alternative is to hit Go Advanced. Posting from that form is not quite as convenient as the Quick Reply, but it does not double post.
    My usual boring signature: Nothing

  16. #16

    Thread Starter
    Frenzied Member some1uk03's Avatar
    Join Date
    Jun 2006
    Location
    London, UK
    Posts
    1,504

    Re: MultiThreading a LOOP

    Quote Originally Posted by xiaoyao View Post
    if your cpu like i5 8600k(6 cpu core,Maybe you can you 6 thread or 12 thread)
    you have 10000 plan,put in arr planlist(10000) as string
    so 6 thread use 6 plan,when one thread finished,use plan 7.
    so your cpu will be used for best.
    yor also can view taskmgr.exe(if your cpu only use 30%),you can use 18 threads
    you can reply in my thread :http://www.vbforums.com/showthread.p...t-UserControl)
    Hi,

    I was just going through your Code.
    It seems to be creating a NEW Thread on EACH loop!
    Code:
        
    For i = 1 To 300
            c.CreateThread "additem", Now 
        Next
    Shouldn't there be a limit as to how many threads you want running depending on your CPU?
    Like set MAX Threads = getCPUThreadcount ?
    _____________________________________________________________________

    ----If this post has helped you. Please take time to Rate it.
    ----If you've solved your problem, then please mark it as RESOLVED from Thread Tools.



  17. #17
    Addicted Member
    Join Date
    Jan 2020
    Posts
    255

    Re: MultiThreading a LOOP

    Quote Originally Posted by some1uk03 View Post
    Hi,

    I was just going through your Code.
    It seems to be creating a NEW Thread on EACH loop!
    Code:
        
    For i = 1 To 300
            c.CreateThread "additem", Now 
        Next
    Shouldn't there be a limit as to how many threads you want running depending on your CPU?
    Like set MAX Threads = getCPUThreadcount ?
    you can see taskmgr.exe ,how much cpu used x%。。
    in fact,i like createthread(*,*,addressof sub1),without class

  18. #18
    Addicted Member
    Join Date
    Jan 2020
    Posts
    255

    Re: MultiThreading a LOOP

    Code:
    Private Sub Command3_Click()
    CreateNewThread AddressOf ThreadTest, 33&, 0&
    
    End Sub
    
    Public Sub ThreadTest(ByVal lpParam As Long)
        InitFun
        msgbox "put your code here in a *.bas file"
        ExitFun
    End Sub

  19. #19

    Thread Starter
    Frenzied Member some1uk03's Avatar
    Join Date
    Jun 2006
    Location
    London, UK
    Posts
    1,504

    Re: MultiThreading a LOOP


    The Trick
    : I've been exploring your JULIA example. Looks great. 1 Module does all that I need
    Here's a question though. Once compiled, i can see it all working fine and sped up quite fast running a 4 thread cycle.
    However, in task manager, you don't get to see the individual Threads ?

    Any reasonable reason to this?
    _____________________________________________________________________

    ----If this post has helped you. Please take time to Rate it.
    ----If you've solved your problem, then please mark it as RESOLVED from Thread Tools.



  20. #20

  21. #21
    PowerPoster Elroy's Avatar
    Join Date
    Jun 2014
    Location
    Near Nashville TN
    Posts
    5,994

    Re: MultiThreading a LOOP

    Just my two-cents.

    It's a bit of a different situation, but I had to develop a color smoothing algorithm about a year ago. I needed it to run rapidly, because the user might tweak, and run it over and over. When it was inline in my code, it took 5 or 6 seconds, which was annoying.

    I shoved it all into an ActiveX DLL, and ticked all the Advanced optimizations, and I was amazed at how much faster it ran. It decreased the time to 1/3rd, at least.

    So, there's that.

    ---

    Regarding multi-threading, I've gotten "test" projects to successfully run on multiple occasions. However, for VB6, I've never gotten comfortable enough with it to trust it for distribution to users. Also, I realized that there's little downside to just spawning an entirely new process (i.e., a separate program). This (sort of) solves all the marshaling issues. However, you've still got to work out a fast mode of inter-process communication. I typically find (or create) some window (form) to subclass to receive the messages. Sending messages to another form doesn't require subclassing, just the use of SendMessageTimeout. I can post code I use to do that, but it's not difficult.

    Then, all you've got to work out is the handshaking protocols between the two processes, and that's entirely up to you.

    And, as a check on someone running this secondary process (program) from the command line, you might put some quasi-secret command line argument in it, and not load the inter-process-communications-form if it's not there.

    ---

    Anyway, that's my two-cents.
    Any software I post in these forums written by me is provided “AS IS” without warranty of any kind, expressed or implied, and permission is hereby granted, free of charge and without restriction, to any person obtaining a copy. Please understand that I’ve been programming since the mid-1970s and still have some of that code. My contemporary VB6 project is approaching 1,000 modules. In addition, I have a “VB6 random code folder” that is overflowing. I’ve been at this long enough to truly not know with absolute certainty from whence every single line of my code has come, with much of it coming from programmers under my employ who signed intellectual property transfers. I have not deliberately attempted to remove any licenses and/or attributions from any software. If someone finds that I have inadvertently done so, I sincerely apologize, and, upon notice and reasonable proof, will re-attach those licenses and/or attributions. To all, peace and happiness.

  22. #22
    Addicted Member
    Join Date
    Jan 2020
    Posts
    255

    Re: MultiThreading a LOOP

    more times,you can use only one standard exe for simple createthread,no need activex dll,no need activex.exe.

  23. #23
    Frenzied Member
    Join Date
    Feb 2015
    Posts
    1,488

    Re: MultiThreading a LOOP

    Regarding multi-threading, I've gotten "test" projects to successfully run on multiple occasions. However, for VB6, I've never gotten comfortable enough with it to trust it for distribution to users. Also, I realized that there's little downside to just spawning an entirely new process (i.e., a separate program). This (sort of) solves all the marshaling issues. However, you've still got to work out a fast mode of inter-process communication. I typically find (or create) some window (form) to subclass to receive the messages. Sending messages to another form doesn't require subclassing, just the use of SendMessageTimeout. I can post code I use to do that, but it's not difficult.

    Then, all you've got to work out is the handshaking protocols between the two processes, and that's entirely up to you.

    And, as a check on someone running this secondary process (program) from the command line, you might put some quasi-secret command line argument in it, and not load the inter-process-communications-form if it's not there.
    You just can create an instance of a COM class and call the methods directly between processes. Moreover you can call the methods between different machines or cross-bitness applications, scripts like vbs etc. The COM does all the job.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Featured


Click Here to Expand Forum to Full Width