Asynchronous File Downloading Queue System

[formlesstree4]

Update (1/16/2010): Rewrote the entire thing cause I wasn't satisfied with the use of Queue. I had no real control over it and couldn't really do much other than dequeue, enqueue, and peek at the list. So I changed everything. I also fully commented it this time, and ended up creating a Structure for the Download Items.

Update (3/3/2010): Added in a few new items here and there. Included Transfer Rate calculations (in kilobytes per second, rounded to two decimal places). The code is attached because it's to long for this post.

Update (3/19/2010): Fixed the transfer rate calculations so they actually work properly. Added in a few properties to easily remove items and some more events so you know when a file is added.

Update (5/13/2010): Applied DJ PIP's fix to the speed calculations (you can see his post down below if you don't want to redownload the file).

Update (7/18/2010): Adding the beta ASQS to this post now. I did a major rework on the events, so it will break existing implementations of this class (sorry djpip). I changed them to look similar to the framework (ByVal sender As Object, ByVal e As someargtype), so all data is now grouped in a single structure for each event rather than a long signature for each event. This also means you can Override three new subs:
OnProgressChange, OnFileCompletion, and OnFileError. I do plan to add in more events as I work on this guy some more to flush him out and give him a few more features. I also did a few bug fixes here and there as well as some code changes.

Update (10/21/2010): Added a trashy Demo App that I'm gonna work on some more. Threw it together real fast due to get an example out and cause I'm low on time.

Questions, comments, and criticism are welcome.

[formlesstree4]

The 2nd update brings a bit more to the table, with the ability for it to calculate the transfer rate and pass it through. It updates literally every time the progress changes, so it's accuracy is quite spot on, but because it updates so often, it's not always consistent (like how Firefox/google chrome have more consistent ones). It still gets the job done however.

More functionality is being worked on so it's much easier to stop and skip downloads (not pause because the WebClient doesn't have a pause method, and I don't know how I'll get one to work.)

[formlesstree4]

Did a 3rd update today, let me know how things work out. (changes in first post).

[djpip27]

Hi there,

First of all can I say a big thank you.
I too am using a webclient in my software which downloads multiple files from the internet. Your class is clearly organised, easy to use and I can't wait to get it working fully in my program!

I am having a little trouble with the Percentage Event.
Basically, this is a really daft question but ...

How can I get the percentage values?
I created this sub in my program to handle the event, but how do I get the data from it?

Public Sub ProgressChange() Handles downloadersystem.Percentage

End Sub

Many Thanks.
DJ PIP

[djpip27]

Don't worry.
Figured it out!

Straightforward.

Public Sub ProgressChange(ByVal currentFile As String, ByVal prog As Integer, ByVal Rate As String) Handles downloadersystem.Percentage

End Sub

Thank you.
DJ PIP

[formlesstree4]

Glad you figured it out! If any problems arise, let me know.

[djpip27]

Fix for Download Speeds.

Amazing Class as I said earlier.
Got it working really well in my software.

I've noticed when downloading the speed will suddenly jump after around 60 seconds to several GB's per second. I'd love this to be true, but unfortunately I think this is caused by a small error in your code.


In this section ...
Private Sub ChangeInProgress(ByVal sender As Object, ByVal e As DownloadProgressChangedEventArgs)

I believe this should be the correct formula to calculate the speed.

Speed = CInt((CurBytes / SpeedCalculate.Elapsed.TotalSeconds))


This is because when using ...
Speed = CInt((CurBytes / SpeedCalculate.Elapsed.Seconds))
... after 59 seconds, it then resets to 01, 02, 03 again, when actually the download is still happening.

See what you think.

DJ PIP

[formlesstree4]

Thank you for noticing that. I hadn't had a download last that long and I never really figured it would do that to me. If it's working for you and you haven't had an issue, I'll gladly fix my code.

[djpip27]

Hi,

Just wanted to update this thread slightly.
I've been using your code for a month now, works perfectly and I have experienced no problems after making that small change I suggested earlier so that download speeds are more accurate.

Downloads have been working wonderfully and I can't thank you enough.

Thank you for your hard work!

DJ PIP

[macbrutal]

Hi, Sorry to bother you guys but I can't understand how to use this module!
I would love a tiny explaination for this one like, what do I have to do to make those functions work?
I tried to call a few functions but that just gave me "It's not declared" etc.

I've been reading about the basics about modules in VB.net and all I could found was like this:
-------------
(Module)
Public sub example()
...code...
(Form1.vb)
Call example
--------------

[formlesstree4]

What's wrong exactly? This isn't a module, it's a class, so things are done differently. You need to declare a New Class, and call the variables from that.

Example:

Dim x As New DS.ASQS
x.AddToQueue("url", "savelocation")

[formlesstree4]

Another update, hopefully the final one. The update uses DJ PIP's fix for the Download Speeds (thanks again for spotting that), so I hope the class is virtually bug free now.

[formlesstree4]

This is a beta of a rewrite I'm doing.

Basically, it allows for simultaneous downloads rather than just one at a time. I have yet to see how it actually will flow, and I'm thinking it will give me problems.

I would like to have some beta testers give it a spin since I can't really easily test it right now. It's attached (Visual Studio 2010 project), and if you all can't compile it due to using 2008 or some other reason, let me know and I'll send you a compiled version.

Update 1 (May 20th, 2010): Added in 3 lines to enable the missing handlers. Helps if all events are properly raised.

Update 2 (May 22nd, 2010): Added some new stuff in. If you reduce the maximum downloads below your current downloads, it starts stopping the latest downloads, nice little thing. Few other misc. bugfixes, things of that nature.

Update 3 (July 18, 2010): See main post for new attachment and update notes.

[djpip27]

I have placed this new code in my software.
The downloads themselves work and I can get the simultaneous downloads working as designed.
Unfortunately I can't seem to get the events such as progress changed to work.

I'm using it like this ...

Public WithEvents downloadersystem As New ManagerClass
...
'I add some downloads, start them and then use this code to report the progress.

Public Sub ProgressChange(ByVal file As String, ByVal speed As String, ByVal percent As Integer) Handles downloadersystem.ProgressChanged

When I debug my application, after placing a breakpoint at ProgressChange, the breakpoint is marked stating that it will never be executed.
Am I doing this in the right way?

DJ PIP

P.S: It looks fantastic so far, and I can't wait until we get this fully working.

[formlesstree4]

OK, I found the error.

In ManagerClass.vb, find the Add() Sub. Replace it with this:

    Public Sub Add(ByVal FD As FileData)
        Dim DC As New DownloadClass(FD)
        If Not CheckForItemInList(FD) Then
            DownloadList.Add(DC)
            AddHandler DC.FileError, AddressOf FileErrorSub
            AddHandler DC.ProgressChanged, AddressOf ProgressChangedSub
            AddHandler DC.ProgressFinished, AddressOf ProgressFinishedSub
            If AutoDownload Then RunList()
        End If
    End Sub

I'll put an updated zip in the post of the beta, including a change log.

[djpip27]

Great getting the progress of downloads works fine now.
One other small thing I'd noticed.
Some-times the download complete event fires twice.

Don't know why, but I'd noticed this when using the webclient before.
There's nothing you can do about this I don't think, although a small change here....

Download Class.

Private Sub WC_DownloadFileCompleted(ByVal sender As Object, ByVal e As System.ComponentModel.AsyncCompletedEventArgs) Handles WC.DownloadFileCompleted
If e.Error IsNot Nothing Then
RaiseEvent FileError(Data.FileName, e.Error)
End If
RaiseEvent ProgressFinished(Data.FileName)
If SpeedCalculate IsNot Nothing Then
SpeedCalculate.Stop()
SpeedCalculate = Nothing
End If
GC.Collect()
End Sub

If SpeedCalculate IsNot Nothing Then ... Stops a Null-Reference exception.

Will integrate it some more into my Application and will report back soon.

DJ PIP

[djpip27]

I'm loving the new system. Really nice work.

I have a question.
I'm currently allowing 5 downloads at a time, and sometimes adding about 8/9 downloads into the list and then starting it.
After a download finishes, another slot of my original 5 is now 'free'.
How can I make it download the next one in the list, in this case number 6?

I hope this makes some sense.
Eg. Downloads 1-5 start.
Downloads 6-8 are paused (max limit of 5 reached.)
Download 1 finished.
Download 6 can start.
Download 4 finishes.
Download 7 can start.

[formlesstree4]

For some reason, I forgot to add a line of code in the ProgressFinishedSub. Whenever the event is raised, at least for now, call RunList() to start the next download, or you can edit the ProgressFinishedSub yourself (it's in the ManagerClass). Just add "Call RunList()" at the end of it:

    Private Sub ProgressFinishedSub(ByVal file As String)
        Dim DC As DownloadClass = FindDownloader(file)
        Call Remove(file)

        RemoveHandler DC.ProgressChanged, AddressOf ProgressChangedSub
        RemoveHandler DC.FileError, AddressOf FileErrorSub
        RemoveHandler DC.ProgressFinished, AddressOf ProgressFinishedSub

        RaiseEvent ProgressFinished(file)
        DC.Dispose()

        Call RunList()

    End Sub

[djpip27]

Great Work.
You also need to add 'CurrentDownloads -= 1' before you Call RunList().
If you don't do this then the CurrentDownloads value remains at your maximum and then no more start.

DJ PIP

[formlesstree4]

Ah yes, I forgot that too. Thanks for the help. Hopefully it won't cause many issues and should be relatively bug free (now). I'll upload a fixed zip file later tonight.

[formlesstree4]

Updated to fix a few other bugs I had missed when I originally coded the system.

[Grebo]

Hello,

I would like to use this class for a project I have that is being converted to .NET 2010. I compiled the ASQS class and made reference to the dll in a new project but am having trouble with the first step in adding a download. The .data property from downloadclass isn't listing the saveto, URL, etc. I'm new to .NET and must be something I'm missing..

Imports ASQS
Imports ASQS.FileData
Imports System

Public Class Form1
    Public WithEvents dclass As ASQS.DownloadClass
    Public WithEvents s As New ASQS.ManagerClass
    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        Dim wmp3 As New ASQS.FileData
        wmp3.Equals?
        dclass.Data.Equals?

    End Sub
End Class

Thanks

Grebo

[formlesstree4]

Open up the FileStructure.vb file and all the Properties from "Friend" to "Public", then recompile the DLL.

[Grebo]

Thank you, but I am now having a problem starting a download. I had to make changes to WC_DownloadFileCompleted in the DownloadClass just like djpip27 did so it wouldn't give null object reference error. I get no error in the code now and no errors while running the compiled application/clicking Button1 but nothing is getting downloaded.

I'm not sure if I am adding a download correct..

Imports System
Imports ASQS
Imports ASQS.FileData

Public Class Form1
    Public WithEvents dclass As DownloadClass
    Public WithEvents s As New ManagerClass

    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        Dim df As New ASQS.FileData
        's.AutoDownload = True
        s.MaxDownloads = 2
        df.SaveTo = "C:\Downloads"
        df.URL = "http://www.site.com/song.mp3"
        s.Add(df)
        s.RunList()
    End Sub
End Class

Thanks

Grebo

[formlesstree4]

What OS are you running and do you have permissions to download to the location as well as retrieve it from the website?

[Grebo]

I'm running Windows XP and yes I've downloaded the same files just fine using the just the QueueSystem.vb class.

[formlesstree4]

Try stepping through the code and make sure that no exceptions are being thrown but ignored by the code, it's happened before. I might have placed a careless Try Catch without actually handling the error.

[Grebo]

Wow, I forgot to actually set a file to save as in the saveto, should be df.SaveTo = "C:\Downloads\thesong.mp3". Everything works fine, thank you for the downloading class. If you are wondering, I'm making a mp3 ripper.

[formlesstree4]

Well, glad you figured it out

[vixez]

I can't quite figure out how to use this properly.

I have this code:

    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        Try
            Dim df As New FileData
            's.AutoDownload = True
            s.MaxDownloads = 2
            df.SaveTo = "D:\Users\Glenn Ruysschaert\Documents\Visual Studio 2008\Download\t.txt"
            df.URL = "http://badboy70.cwahi.net/welcome.txt"
            s.Add(df)
            s.RunList()
        Catch ex As Exception
            MsgBox(ex.Message)
        End Try
      
    End Sub

    Private Sub dclass_ProgressChanged(ByVal file As String, ByVal speed As String, ByVal percent As Integer) Handles dclass.ProgressChanged
        Label1.Text = percent
    End Sub

    Private Sub dclass_ProgressFinished(ByVal file As String) Handles dclass.ProgressFinished
        MsgBox("done")
    End Sub

It downloads the file, but the label doesn't change and the msgbox doesn't show up either.

What am I doing wrong?

[formlesstree4]

I'm surprised you're not getting some sort of cross threading error. Those events are raised on a different thread so you need to use delegates to bring them back over.

[JuggaloBrotha]

Hey formless, I have a suggestion: When you raise events, give them the normal signature with sender being the object that raised it and e having the properties of the other information. So instead of:

Private Sub dclass_ProgressChanged(ByVal file As String, ByVal speed As String, ByVal percent As Integer) Handles dclass.ProgressChanged

You have:

Private Sub dclass_ProgressChanged(ByVal sender As Object, ByVal e As ProgressArgs) Handles dclass.ProgressChanged

where e has these properties of your ProgressArgs class: file As String, speed As String, percent As Integer

Also I would make percent a Double (between 0.0 and 1.0) instead of an Integer.

[formlesstree4]

I'll try doing that on the next update, thanks for the tip.

[Grebo]

I'm having trouble as well with trying to set a handler for progresschanged event in the download class. I've been trying to follow tutorials on this but no luck and here what I have but it is throwing an error on the AddHandler dclass.ProgressChanged, AddressOf OnProgress line with this "Object reference not set to an instance of an object."

Option Explicit On
Option Strict On

Imports System
Imports ASQS
Imports ASQS.FileData
Imports ASQS.DownloadClass
Imports ASQS.ManagerClass
Public Class Form1
    Public WithEvents dclass As DownloadClass
    Public WithEvents sas As New ManagerClass

    Public Delegate Sub ProgressChangedEventHandler(ByVal file As String, ByVal speed As String, ByVal percent As Integer)

    Public Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        Dim where As New FileData
        AddHandler dclass.ProgressChanged, AddressOf OnProgress

        sas.AutoDownload = True
        sas.MaxDownloads = 2
        where.SaveTo = "C:\Downloads\test.mp3"
        where.URL = "http://www.site.com/a.mp3"
        'MsgBox(where.FileName)
        'MsgBox(where.FilePath)
        sas.Add(where)
        'sas.RunList()
    End Sub
    Public Sub OnProgress(ByVal file As String, ByVal speed As String, ByVal percent As Integer) Handles dclass.ProgressChanged

        If Me.InvokeRequired Then
            Me.Invoke(New ProgressChangedEventHandler( _
            AddressOf OnProgress _
            ), New Object() {percent})
        Else
            Me.ProgressBar1.Value = percent
        End If
    End Sub
End Class

Here is the detailed exception:

System.NullReferenceException was unhandled
  Message=Object reference not set to an instance of an object.
  Source=pmr
  StackTrace:
       at WindowsApplication1.Form1.Button1_Click(Object sender, EventArgs e) in C:\Documents and Settings\Administrator\My Documents\Visual Studio 2010\Projects\pmr\pmr\Form1.vb:line 17
       at System.Windows.Forms.Control.OnClick(EventArgs e)
       at System.Windows.Forms.Button.OnClick(EventArgs e)
       at System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent)
       at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
       at System.Windows.Forms.Control.WndProc(Message& m)
       at System.Windows.Forms.ButtonBase.WndProc(Message& m)
       at System.Windows.Forms.Button.WndProc(Message& m)
       at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
       at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
       at System.Windows.Forms.NativeWindow.DebuggableCallback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
       at System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG& msg)
       at System.Windows.Forms.Application.ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(IntPtr dwComponentID, Int32 reason, Int32 pvLoopData)
       at System.Windows.Forms.Application.ThreadContext.RunMessageLoopInner(Int32 reason, ApplicationContext context)
       at System.Windows.Forms.Application.ThreadContext.RunMessageLoop(Int32 reason, ApplicationContext context)
       at System.Windows.Forms.Application.Run(ApplicationContext context)
       at Microsoft.VisualBasic.ApplicationServices.WindowsFormsApplicationBase.OnRun()
       at Microsoft.VisualBasic.ApplicationServices.WindowsFormsApplicationBase.DoApplicationModel()
       at Microsoft.VisualBasic.ApplicationServices.WindowsFormsApplicationBase.Run(String[] commandLine)
       at WindowsApplication1.My.MyApplication.Main(String[] Args) in 17d14f5c-a337-4978-8281-53493378c1071.vb:line 81
       at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
       at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
       at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
       at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading.ThreadHelper.ThreadStart()
  InnerException:

Thanks

[formlesstree4]

Change

Public WithEvents dclass As DownloadClass

to

Public WithEvents dclass As New DownloadClass

[Grebo]

I've been trying to figure out why I couldn't do that as well because when I try it throws this error:
Overload resolution failed because no accessible 'New' accepts this number of arguments.

[formlesstree4]

Well, what does it want from you? Did you read the Intellisense? I commented on the entire class pretty much with XML comments.

[Grebo]

d
I set it as Public WithEvents dclass As New DownloadClass(vbNullString, vbNullString) or
Public asdf As FileData
Public WithEvents dclass As New DownloadClass(asdf)

which it seems to accept but I can't actually use the downloadclass because I tried doing an dclass.abort and dclass.dispose but nothing happens trying both. I'm assuming that is why the ProgressChanged event handler isn't working as well..

[formlesstree4]

I just realized something, you only should be using the ManagerClass. It is the one that controls all the stuff on the system. I'm sorry about not remembering sooner, it's been a long day.


Try Handling the ManagerClass events instead.

[Grebo]

Can someone please tell me what I'm missing in getting the event to actually fire using delegates..

This is what I have.

Option Explicit On
Option Strict On

Imports System
Imports ASQS
Imports ASQS.FileData
Imports ASQS.DownloadClass
Imports ASQS.ManagerClass
Public Class Form1

    Public WithEvents sas As New ManagerClass
    Public EventInvoker As prog_del
    Public Delegate Sub prog_del(ByVal file As String, ByVal speed As String, ByVal percent As Integer)

    Public Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        Try
            Dim where As New FileData

            sas.AutoDownload = True
            sas.MaxDownloads = 2
            where.SaveTo = "C:\Downloads\test.mp3"
            where.URL = "https://www.site.com/asd.mp3"
            'MsgBox(where.FileName)
            'MsgBox(where.FilePath)

            sas.Add(where)
            'sas.RunList()
        Catch ex As Exception
            MsgBox(ex.Message & vbCrLf & ex.StackTrace)
        End Try

    End Sub

    Private Sub sas_DownloadProgressChanged(ByVal file As String, ByVal speed As String, ByVal percent As Integer) Handles sas.ProgressChanged
        If Me.Label1.InvokeRequired Then
            EventInvoker = New prog_del(AddressOf Me.sas_DownloadProgressChanged)
            Me.Label1.Invoke(EventInvoker, speed)
            'Exit Sub
        End If
        Me.Label1.Text = speed
    End Sub
End Class