Results 1 to 23 of 23

Thread: [RESOLVED] WebClient downloads only 1 GB, then fails & Round number to 1 decimal place

  1. #1

    Thread Starter
    Junior Member
    Join Date
    Jul 2015
    Posts
    20

    Resolved [RESOLVED] WebClient downloads only 1 GB, then fails & Round number to 1 decimal place

    Hi! I wanna download 2 GB file with WebClient, but after it downloads 1 GB it stops without any error.
    Thanks for any help!

    Code:
    Sub download(ByVal file As String, ByVal uri As String, ByVal ffile As String)
            la_dl.Text = file
            SW = Stopwatch.StartNew
            Dim client As WebClient = New WebClient
            AddHandler client.DownloadProgressChanged, AddressOf download_change
            AddHandler client.DownloadFileCompleted, AddressOf client_DownloadCompleted
            If My.Computer.FileSystem.DirectoryExists(Application.StartupPath & "/dl") = False Then
                My.Computer.FileSystem.CreateDirectory(Application.StartupPath & "/dl")
            End If
            client.DownloadFileAsync(New Uri(uri), Application.StartupPath & "/dl/" & ffile)
        End Sub
    Last edited by henryolik; Dec 29th, 2017 at 06:01 AM. Reason: Deleted second issue

  2. #2
    Super Moderator jmcilhinney's Avatar
    Join Date
    May 2005
    Location
    Sydney, Australia
    Posts
    110,297

    Re: WebClient downloads only 1 GB, then fails & Round number to 1 decimal place

    Quote Originally Posted by henryolik View Post
    And the second thing...
    Belongs in a thread of its own. One thread per topic and one topic per thread.

  3. #3

    Thread Starter
    Junior Member
    Join Date
    Jul 2015
    Posts
    20

    Re: WebClient downloads only 1 GB, then fails & Round number to 1 decimal place

    Okay, thanks for info!

  4. #4
    You don't want to know.
    Join Date
    Aug 2010
    Posts
    4,578

    Re: WebClient downloads only 1 GB, then fails & Round number to 1 decimal place

    It probably doesn't stop without error. You maybe didn't choose to do anything to get the error if an error happens. I can't tell you because you didn't show your DownloadCompleted error handler. Are you checking the AsyncCompletedEventArgs.Error property?

    A slight possibility is that in the time it takes the download to complete, the .NET Garbage Collector runs and notices your WebClient isn't being referenced by anything. Then it decides to "help" by destroying it for you. To solve this, you should store the WebClient in a field that has a lifetime at least as long as the download.

    The pattern for downloading a file with this clunky event-based technique should look something like this:
    Code:
    Private _activeClient As WebClient
    
    Public Sub DoDownload()
        _activeClient = New WebClient()
        AddHandler client.DownloadFileCompleted, AddressOf WhenFileDownloadCompletes
        _activeClient.DownloadFileAsync(...)
    End Sub
    
    Private Sub WhenFileDownloadCompletes(...)
        If e.Error Then
            MessageBox.Show("There was an error downloading the file.")
        Else
            MessageBox.Show("The file was downloaded.")
        End If
    
        _activeClient.Dispose()
        _activeClient = Nothing
    End Sub
    Try using something that looks more like that and see if you either get an error or if it starts finishing the job.
    This answer is wrong. You should be using TableAdapter and Dictionaries instead.

  5. #5

    Thread Starter
    Junior Member
    Join Date
    Jul 2015
    Posts
    20

    Re: WebClient downloads only 1 GB, then fails & Round number to 1 decimal place

    I tried it, I'm now getting "An exception occurred during a WebClient request." error right on 1 GB downloaded. Here is the rest of code (download change + download completed):

    Code:
    Private Sub download_change(ByVal sender As Object, ByVal e As DownloadProgressChangedEventArgs)
            Dim mbytesIn As Double = Double.Parse(e.BytesReceived.ToString()) / 1000000
            Dim totalmBytes As Double = Double.Parse(e.TotalBytesToReceive.ToString()) / 1000000
            Dim percentage As Double = mbytesIn / totalmBytes * 100
            pb_load.Value = Int32.Parse(Math.Truncate(percentage).ToString())
            percentage = Math.Round(percentage, 0)
            la_perc.Text = percentage & " %"
            Dim unita As String
            If totalmBytes < 1 Then
                totalmBytes = totalmBytes * 1000
                unita = " kB"
                If totalmBytes < 1 Then
                    totalmBytes = totalmBytes * 1000
                    unita = " B"
                End If
            ElseIf totalmBytes > 1000 Then
                totalmBytes = totalmBytes / 1000
                unita = " GB"
            Else
                unita = " MB"
            End If
            Dim unitb As String
            If mbytesIn < 1 Then
                mbytesIn = mbytesIn * 1000
                unitb = " kB"
                If mbytesIn < 1 Then
                    mbytesIn = mbytesIn * 1000
                    unitb = " B"
                End If
            ElseIf mbytesIn > 1000 Then
                mbytesIn = mbytesIn / 1000
                unitb = " GB"
            Else
                unitb = " MB"
            End If
            mbytesIn = Math.Round(mbytesIn, 1)
            totalmBytes = Math.Round(totalmBytes, 2)
            la_size.Text = mbytesIn & unitb & " / " & totalmBytes & unita
            Dim speed As Double = e.BytesReceived / SW.ElapsedMilliseconds / 1000
            Dim unit As String
            If speed < 1 Then
                unit = " kB/s"
                speed = speed * 1000
                If speed < 1 Then
                    unit = " B/s"
                    speed = speed * 1000
                End If
            Else
                unit = " MB/s"
            End If
            speed = Math.Round(speed, 1)
            la_speed.Text = "Download speed: " & speed & unit
        End Sub
    
        Private Sub client_DownloadCompleted(ByVal sender As Object, ByVal e As System.ComponentModel.AsyncCompletedEventArgs)
            If Not e.Error Is Nothing Then
                MessageBox.Show("There was an error downloading the file. Error: " & e.Error.Message)
                SW.Stop()
                predl()
            Else
                dl = True
                SW.Stop()
                predl()
            End If
        End Sub

  6. #6
    Super Moderator Shaggy Hiker's Avatar
    Join Date
    Aug 2002
    Location
    Idaho
    Posts
    38,988

    Re: WebClient downloads only 1 GB, then fails & Round number to 1 decimal place

    Was there any other information included with that exception? As it stands, that's a pretty minimalist error message, and MS usually gives you more than that....though not always USEFUL information.
    My usual boring signature: Nothing

  7. #7

    Thread Starter
    Junior Member
    Join Date
    Jul 2015
    Posts
    20

    Re: WebClient downloads only 1 GB, then fails & Round number to 1 decimal place

    Name:  err.PNG
Views: 1520
Size:  14.1 KB
    In English - "An exception occurred during a WebClient request".
    No further information :/.

  8. #8
    You don't want to know.
    Join Date
    Aug 2010
    Posts
    4,578

    Re: WebClient downloads only 1 GB, then fails & Round number to 1 decimal place

    The feeling I get from some search results is WebClient makes the curious choice to store the downloaded data in a MemoryStream until it completes, then tries to write that whole MemoryStream. That goes fine for (relatively) small files, but can get nasty for large files. MemoryStream is backed by an array, and if data exceeds capacity it re-allocates a new array at twice the size. For files that exceed 1GB, that might involve trying to allocate a greater-than-2GB array and for various reasons that can be a problem in .NET unless you write very special code to handle it. Not to mention on many machines "please allocate 3GB of total memory" is a tall order.

    So you should probably just try using WebRequest to download bytes in "blocks" and write each "block" to the file as it comes in. C# will soon have a nice feature to help with this but it's not on the VB roadmap so I'm not sure why I mentioned it. This is a C# example of the technique.
    This answer is wrong. You should be using TableAdapter and Dictionaries instead.

  9. #9
    Angel of Code Niya's Avatar
    Join Date
    Nov 2011
    Posts
    8,598

    Re: WebClient downloads only 1 GB, then fails & Round number to 1 decimal place

    Quote Originally Posted by Sitten Spynne View Post
    This is a C# example of the technique.
    I see no reason why that can't be implemented in VB.
    Treeview with NodeAdded/NodesRemoved events | BlinkLabel control | Calculate Permutations | Object Enums | ComboBox with centered items | .Net Internals article(not mine) | Wizard Control | Understanding Multi-Threading | Simple file compression | Demon Arena

    Copy/move files using Windows Shell | I'm not wanted

    C++ programmers will dismiss you as a cretinous simpleton for your inability to keep track of pointers chained 6 levels deep and Java programmers will pillory you for buying into the evils of Microsoft. Meanwhile C# programmers will get paid just a little bit more than you for writing exactly the same code and VB6 programmers will continue to whitter on about "footprints". - FunkyDexter

    There's just no reason to use garbage like InputBox. - jmcilhinney

    The threads I start are Niya and Olaf free zones. No arguing about the benefits of VB6 over .NET here please. Happiness must reign. - yereverluvinuncleber

  10. #10

    Thread Starter
    Junior Member
    Join Date
    Jul 2015
    Posts
    20

    Re: WebClient downloads only 1 GB, then fails & Round number to 1 decimal place

    I don't think WebClient downloads it to memory first. When downloading it writes every single bit straight to destination file on disk. So I don't think that's the reason. I'll probably split that file into 3 parts, download it and join it together afterwards

  11. #11

    Thread Starter
    Junior Member
    Join Date
    Jul 2015
    Posts
    20

    Re: WebClient downloads only 1 GB, then fails & Round number to 1 decimal place

    Splitted, again, at 1 GB total downloaded it throws error. I'm really desperate now...

  12. #12
    Lively Member
    Join Date
    Jun 2017
    Posts
    77

    Re: WebClient downloads only 1 GB, then fails & Round number to 1 decimal place

    A few observations regarding your initial post:

    1. You are using DownloadFileAsync without awaiting it. You should look up async and await.
    2. You are using WebClient. A better alternative would be HttpClient.

  13. #13
    Super Moderator jmcilhinney's Avatar
    Join Date
    May 2005
    Location
    Sydney, Australia
    Posts
    110,297

    Re: WebClient downloads only 1 GB, then fails & Round number to 1 decimal place

    Quote Originally Posted by BlackRiver1987 View Post
    1. You are using DownloadFileAsync without awaiting it. You should look up async and await.
    The WebClient.DownloadFileAsync predates async/await and is not awaitable. It follows an older async pattern where it is paired with the DownloadFileCompleted event, much as the RunWorkerAsync method and RunWorkerCompleted event of the BackgroundWorker class are paired.
    Quote Originally Posted by BlackRiver1987 View Post
    2. You are using WebClient. A better alternative would be HttpClient.
    Why so? A WebClient can handle downloading a file just fine in general. What would an HttpClient add in this case? There may be something that I'm not aware of but I suspect that this issue would not be solved by using an HttpClient.

  14. #14
    Lively Member
    Join Date
    Jun 2017
    Posts
    77

    Re: WebClient downloads only 1 GB, then fails & Round number to 1 decimal place

    Quote Originally Posted by jmcilhinney View Post
    The WebClient.DownloadFileAsync predates async/await and is not awaitable. It follows an older async pattern where it is paired with the DownloadFileCompleted event, much as the RunWorkerAsync method and RunWorkerCompleted event of the BackgroundWorker class are paired.
    Thank you for that info. Did not know that.

    Quote Originally Posted by jmcilhinney View Post
    Why so? A WebClient can handle downloading a file just fine in general. What would an HttpClient add in this case? There may be something that I'm not aware of but I suspect that this issue would not be solved by using an HttpClient.
    Just a personal preference, since it's easier to use when going full async.

  15. #15
    Super Moderator jmcilhinney's Avatar
    Join Date
    May 2005
    Location
    Sydney, Australia
    Posts
    110,297

    Re: WebClient downloads only 1 GB, then fails & Round number to 1 decimal place

    Quote Originally Posted by BlackRiver1987 View Post
    Just a personal preference, since it's easier to use when going full async.
    I'd tend to agree if you were using async/await throughout your code but that is probably not the case here, which you probably didn't realise.

  16. #16
    You don't want to know.
    Join Date
    Aug 2010
    Posts
    4,578

    Re: WebClient downloads only 1 GB, then fails & Round number to 1 decimal place

    Actually, HttpClient is recommended going forward.

    WebClient is older and potentially based on a different HTTP stack. HttpClient supports the newer async patterns and is what MS supports in terms of UWP and newer .NET code.

    But the main reason I'd suggest trying it is that "potentially different" aspect. Obviously we've found some mysterious problem with WebClient. I'd love to get to the bottom of it, but if we try HttpClient and it works that solves the problem. If it doesn't, we can revert back to WebClient and start looking for more answers.

    If we can't (It's possible HttpClient is newer than VS 2010 and might not work at all), then my next question is, "Is there an InnerException in that exception that gets thrown? Usually if a .NET Exception says "An exception was thrown..." it puts that exception in the InnerException property."
    This answer is wrong. You should be using TableAdapter and Dictionaries instead.

  17. #17
    Super Moderator jmcilhinney's Avatar
    Join Date
    May 2005
    Location
    Sydney, Australia
    Posts
    110,297

    Re: WebClient downloads only 1 GB, then fails & Round number to 1 decimal place

    Quote Originally Posted by Sitten Spynne View Post
    It's possible HttpClient is newer than VS 2010 and might not work at all
    According to the documentation, HttpClient was introduced in .NET 4.5. VS 2010 was released with .NET 3.5, if I remember correctly. I'm not sure whether it can target .NET 4.5 or not but, even if it can, there's every chance that the project was created against an earlier version.

  18. #18
    You don't want to know.
    Join Date
    Aug 2010
    Posts
    4,578

    Re: WebClient downloads only 1 GB, then fails & Round number to 1 decimal place

    Yeah, given the context of this thread I'm thinking HttpClient won't really be available anyway.

    Given that, taking the code in #5, I'd make this change:
    Code:
    Private Sub client_DownloadCompleted(...)
        If Not e.Error Is Nothing Then
            Dim actualException = e.Error
            While actualException.InnerException IsNot Nothing
                actualException = actualException.InnerException
            End While
            
            MessageBox.Show("There was an error downloading the file. Error: " & actualException.Message)
            SW.Stop()
            ...
    This will make sure it reports the lowest-level exception. If there's an InnerException at all, it's more likely to have useful information.

    The next steps if that isn't illuminating are to try WebRequest/WebResponse and see if when you take lower-level control of the process you bypass the problem. Writing lower-level code is tougher, but usually gives you more insight into specific problems.
    This answer is wrong. You should be using TableAdapter and Dictionaries instead.

  19. #19
    Bad man! ident's Avatar
    Join Date
    Mar 2009
    Location
    Cambridge
    Posts
    5,398

    Re: WebClient downloads only 1 GB, then fails & Round number to 1 decimal place

    Just to satisfy my own curiosity have you tried not downloading the file to Temporary Projects, which I assume you are if running inside the IDE.

    Code:
        Private _activeClient As WebClient
    
        Private Sub DoDownload()
            Dim url As String = "http://...."
            Dim directory As String = Environment.GetFolderPath(Environment.SpecialFolder.Desktop)
            Dim fileName As String = IO.Path.GetFileName(url)
            Dim path As String = IO.Path.Combine(directory, "dl", fileName)
    
            IO.Directory.CreateDirectory(...
    
            _activeClient = New WebClient()
    
            AddHandler _activeClient.DownloadProgressChanged, ...
            AddHandler _activeClient.DownloadFileCompleted, ...
    
    
            _activeClient.DownloadFileAsync(.....
        End Sub

  20. #20
    Bad man! ident's Avatar
    Join Date
    Mar 2009
    Location
    Cambridge
    Posts
    5,398

    Re: WebClient downloads only 1 GB, then fails & Round number to 1 decimal place

    Just Once again for my own curiosity. I admit I am grasping...

    Code:
    Imports System.Net
    
    
    Public Class Form1
    
    
        Private _activeClient As WebClient
    
    
        Private Sub DoDownload()
            .....
            _activeClient = New WebClient()
            _activeClient.Headers.Add(
                "user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)")
        End Sub
    
    
    End Class

  21. #21

    Thread Starter
    Junior Member
    Join Date
    Jul 2015
    Posts
    20

    Re: WebClient downloads only 1 GB, then fails & Round number to 1 decimal place

    Quote Originally Posted by Sitten Spynne View Post
    Code:
    Private Sub client_DownloadCompleted(...)
        If Not e.Error Is Nothing Then
            Dim actualException = e.Error
            While actualException.InnerException IsNot Nothing
                actualException = actualException.InnerException
            End While
            
            MessageBox.Show("There was an error downloading the file. Error: " & actualException.Message)
            SW.Stop()
            ...
    I added this to my code and now it's working! Maybe that error doesn't want to be revealed and rather works Otherwise, thanks for everybody's help!

  22. #22
    You don't want to know.
    Join Date
    Aug 2010
    Posts
    4,578

    Re: WebClient downloads only 1 GB, then fails & Round number to 1 decimal place

    Quote Originally Posted by Niya View Post
    I see no reason why that can't be implemented in VB.
    I missed this when you posted it but you're right, the link was too close to the sentence about the new C# feature. I meant it to mean "Here's someone that did what I'm talking about in C#, a translation to VB should work"!
    This answer is wrong. You should be using TableAdapter and Dictionaries instead.

  23. #23
    Angel of Code Niya's Avatar
    Join Date
    Nov 2011
    Posts
    8,598

    Re: WebClient downloads only 1 GB, then fails & Round number to 1 decimal place

    Quote Originally Posted by Sitten Spynne View Post
    I missed this when you posted it but you're right, the link was too close to the sentence about the new C# feature. I meant it to mean "Here's someone that did what I'm talking about in C#, a translation to VB should work"!
    Ah....Fair enough.
    Treeview with NodeAdded/NodesRemoved events | BlinkLabel control | Calculate Permutations | Object Enums | ComboBox with centered items | .Net Internals article(not mine) | Wizard Control | Understanding Multi-Threading | Simple file compression | Demon Arena

    Copy/move files using Windows Shell | I'm not wanted

    C++ programmers will dismiss you as a cretinous simpleton for your inability to keep track of pointers chained 6 levels deep and Java programmers will pillory you for buying into the evils of Microsoft. Meanwhile C# programmers will get paid just a little bit more than you for writing exactly the same code and VB6 programmers will continue to whitter on about "footprints". - FunkyDexter

    There's just no reason to use garbage like InputBox. - jmcilhinney

    The threads I start are Niya and Olaf free zones. No arguing about the benefits of VB6 over .NET here please. Happiness must reign. - yereverluvinuncleber

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width