Results 1 to 6 of 6

Thread: How to download multiple URLs at the same time?

  1. #1

    Thread Starter
    New Member
    Join Date
    Aug 2021
    Posts
    2

    How to download multiple URLs at the same time?

    Code:
    For i As Integer = 0 To RichTextBox1.Lines.Length - 1
        wreq=System.Net.WebRequest.Create("http://www.....id(i)")
        wreq.AutomaticDecompression = Net.DecompressionMethods.GZip
        wres = wreq.GetResponse
        Dim s As System.IO.Stream = wres.GetResponseStream
        Dim sr As New System.IO.StreamReader(s)
        html = sr.ReadToEnd
        s = html.Split(";")
        'here is other codes
    Next
    this is part of my program.
    When I use this, it takes a long time for everyone to download. How can I download all the addresses at the same time?
    thank you
    Last edited by dday9; Aug 3rd, 2021 at 08:48 AM. Reason: Formatted code and added code tags

  2. #2
    Angel of Code Niya's Avatar
    Join Date
    Nov 2011
    Posts
    8,598

    Re: How to download multiple URLs at the same time?

    Use threads:-
    Code:
            For i As Integer = 0 To RichTextBox1.Lines.Length - 1
    
                'Runs each download on it's own thread...
                Threading.ThreadPool.QueueUserWorkItem(Sub()
                                                           Dim wreq As Net.HttpWebRequest
                                                           Dim wres As Net.HttpWebResponse
                                                           Dim html As String
                                                           Dim s As String()
    
                                                           wreq = System.Net.WebRequest.Create("http://www.....id(i)")
                                                           wreq.AutomaticDecompression = Net.DecompressionMethods.GZip
                                                           wres = wreq.GetResponse
                                                           Dim s As System.IO.Stream = wres.GetResponseStream
                                                           Dim sr As New System.IO.StreamReader(s)
                                                           html = sr.ReadToEnd
                                                           s = html.Split(";")
                                                           'here is other codes
                                                       End Sub)
    
            Next
    Treeview with NodeAdded/NodesRemoved events | BlinkLabel control | Calculate Permutations | Object Enums | ComboBox with centered items | .Net Internals article(not mine) | Wizard Control | Understanding Multi-Threading | Simple file compression | Demon Arena

    Copy/move files using Windows Shell | I'm not wanted

    C++ programmers will dismiss you as a cretinous simpleton for your inability to keep track of pointers chained 6 levels deep and Java programmers will pillory you for buying into the evils of Microsoft. Meanwhile C# programmers will get paid just a little bit more than you for writing exactly the same code and VB6 programmers will continue to whitter on about "footprints". - FunkyDexter

    There's just no reason to use garbage like InputBox. - jmcilhinney

    The threads I start are Niya and Olaf free zones. No arguing about the benefits of VB6 over .NET here please. Happiness must reign. - yereverluvinuncleber

  3. #3
    King of sapila
    Join Date
    Oct 2006
    Location
    Greece
    Posts
    6,597

    Re: How to download multiple URLs at the same time?

    With the exception of handling errors when they occur (and not waiting for task.await), isn't it preferred to use task.run as the QueueUserWorkItem is very old?
    I admit I haven't used QueueUserWorkItem so I don't have an opinion on the performance but it would seem more logical to use newer stuff.
    ἄνδρα μοι ἔννεπε, μοῦσα, πολύτροπον, ὃς μάλα πολλὰ
    πλάγχθη, ἐπεὶ Τροίης ἱερὸν πτολίεθρον ἔπερσεν·

  4. #4
    Angel of Code Niya's Avatar
    Join Date
    Nov 2011
    Posts
    8,598

    Re: How to download multiple URLs at the same time?

    I could be wrong but I see no point in using Task objects if they're not intended to be used with Await. I'd just go with normal thread pool threads. Thread pools are meant for exactly this kind of thing. The ThreadPool class has internal logic for assigning threads to workloads efficiently which is what you would want if you think your code is going to spin up a lot of threads. For all I know his code is trying to download 1000 things in which case, you definitely want to use the ThreadPool class. There is no reason for me to assume that Task.Run threads would have the same internal logic since it was made for an entirely different purpose.

    EDIT:

    I just decided to look it up. Task objects indeed use thread pool threads internally so disregard what I said above. You're correct. He can use Task.Run and it would be the better practice.
    Last edited by Niya; Aug 3rd, 2021 at 04:23 PM.
    Treeview with NodeAdded/NodesRemoved events | BlinkLabel control | Calculate Permutations | Object Enums | ComboBox with centered items | .Net Internals article(not mine) | Wizard Control | Understanding Multi-Threading | Simple file compression | Demon Arena

    Copy/move files using Windows Shell | I'm not wanted

    C++ programmers will dismiss you as a cretinous simpleton for your inability to keep track of pointers chained 6 levels deep and Java programmers will pillory you for buying into the evils of Microsoft. Meanwhile C# programmers will get paid just a little bit more than you for writing exactly the same code and VB6 programmers will continue to whitter on about "footprints". - FunkyDexter

    There's just no reason to use garbage like InputBox. - jmcilhinney

    The threads I start are Niya and Olaf free zones. No arguing about the benefits of VB6 over .NET here please. Happiness must reign. - yereverluvinuncleber

  5. #5
    King of sapila
    Join Date
    Oct 2006
    Location
    Greece
    Posts
    6,597

    Re: How to download multiple URLs at the same time?

    Ye man, as I've said I never used QueueUserWorkItem so this was more of a question than a certainty.
    ἄνδρα μοι ἔννεπε, μοῦσα, πολύτροπον, ὃς μάλα πολλὰ
    πλάγχθη, ἐπεὶ Τροίης ἱερὸν πτολίεθρον ἔπερσεν·

  6. #6
    Angel of Code Niya's Avatar
    Join Date
    Nov 2011
    Posts
    8,598

    Re: How to download multiple URLs at the same time?

    QueueUserWorkItem would be the more "old school" way of doing things. I still use it to this day when I don't really need awaitable tasks.
    Treeview with NodeAdded/NodesRemoved events | BlinkLabel control | Calculate Permutations | Object Enums | ComboBox with centered items | .Net Internals article(not mine) | Wizard Control | Understanding Multi-Threading | Simple file compression | Demon Arena

    Copy/move files using Windows Shell | I'm not wanted

    C++ programmers will dismiss you as a cretinous simpleton for your inability to keep track of pointers chained 6 levels deep and Java programmers will pillory you for buying into the evils of Microsoft. Meanwhile C# programmers will get paid just a little bit more than you for writing exactly the same code and VB6 programmers will continue to whitter on about "footprints". - FunkyDexter

    There's just no reason to use garbage like InputBox. - jmcilhinney

    The threads I start are Niya and Olaf free zones. No arguing about the benefits of VB6 over .NET here please. Happiness must reign. - yereverluvinuncleber

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width