VB6 Download multiple files at once: no API, no extra dependency

[Merri]

You can actually download files in VB without extra dependancy to anything. You don't even need a single API call. The only requirement is a user control.

Below is a simple sample that has the ability to download multiple files at once, and your application remains responsive as well. It keeps track of the given keys (or "PropertyNames") so you won't run into an error condition if you try to download a file while it is already being downloaded. The control will cancel the existing download when that same file is being tried to download again.

' Downloader.ctl
Option Explicit

Public Event Complete(ByRef URL As String, ByRef Data As String, ByRef Key As String)
Public Event Progress(ByRef URL As String, ByRef Key As String, ByVal BytesDone As Long, ByVal BytesTotal As Long, ByVal Status As AsyncStatusCodeConstants)

Public Enum DownloaderCache
    [Always download] = vbAsyncReadForceUpdate
    [Get cache copy only] = vbAsyncReadOfflineOperation
    [Update cached copy only] = vbAsyncReadResynchronize
    [Use cache if no connection] = vbAsyncReadGetFromCacheIfNetFail
End Enum

Private m_Keys As String

Private Function Private_AddKey(ByRef Key As String) As Boolean
    ' see if we do not have the key
    Private_AddKey = InStr(m_Keys, vbNullChar & Key & vbNullChar) = 0
    ' we can add it
    If Private_AddKey Then
        m_Keys = m_Keys & Key & vbNullChar
    End If
End Function

Private Sub Private_RemoveKey(ByRef Key As String)
    ' remove the key
    m_Keys = Replace(m_Keys, vbNullChar & Key & vbNullChar, vbNullChar)
End Sub

Public Sub Start(ByRef URL As String, Optional ByVal CacheMode As DownloaderCache = [Always download], Optional ByVal Key As String)
    ' use URL as key if no key is given
    If LenB(Key) = 0 Then Key = URL
    ' do we already have this key?
    If Not Private_AddKey(Key) Then
        ' cancel the old one
        CancelAsyncRead Key
    End If
    ' begin download process
    AsyncRead URL, vbAsyncTypeByteArray, Key, CacheMode
End Sub

Private Sub UserControl_AsyncReadComplete(AsyncProp As AsyncProperty)
    Dim strData As String
    ' get Variant byte array to byte string (needs StrConv to Unicode for displaying in a textbox)
    If AsyncProp.BytesRead Then strData = AsyncProp.Value Else strData = vbNullString
    ' redirect information
    RaiseEvent Complete(AsyncProp.Target, strData, AsyncProp.PropertyName)
    ' remove the key
    Private_RemoveKey AsyncProp.PropertyName
End Sub

Private Sub UserControl_AsyncReadProgress(AsyncProp As AsyncProperty)
    With AsyncProp
        ' redirect event information
        If LenB(.PropertyName) Then
            RaiseEvent Progress(.Target, .PropertyName, .BytesRead, .BytesMax, .StatusCode)
        Else
            RaiseEvent Progress(.Target, vbNullString, .BytesRead, .BytesMax, .StatusCode)
        End If
    End With
End Sub

Private Sub UserControl_Initialize()
    m_Keys = vbNullChar
End Sub

A sample application: you need two command buttons, one multiline textbox and a listbox. You need the user control as well.

Option Explicit

Private Sub Command1_Click()
    Downloader1.Start "http://merri.net/"
End Sub

Private Sub Command2_Click()
    Downloader1.Start "http://www.vbforums.com/"
End Sub

Private Sub Downloader1_Complete(URL As String, Data As String, Key As String)
    Me.Caption = URL
    Text1.Text = StrConv(Data, vbUnicode)
End Sub

Private Sub Downloader1_Progress(URL As String, Key As String, ByVal BytesDone As Long, ByVal BytesTotal As Long, ByVal Status As AsyncStatusCodeConstants)
    List1.AddItem URL & " @ " & (BytesDone \ 1024) & " kB, status code: " & Status, 0
End Sub

You can also download images directly as a Picture object. To see other possible values, open up Object Browser with F2 and type Async into the search field.


Possible issues
Showing a message box seems to entirely prevent events from triggering under the IDE. I haven't tested compiled, but I believe it works just fine there: timer events don't run in the IDE when a messagebox is shown, but compiled it works just fine.


Updates!
2008-09-14: changed Data to string from byte array and made a check for if any data was received to avoid an error.

[guitar]

It is indeed the easiest way I've ever known to do it but I prefer to use the Winsock API or the control, mainly because:

1.) Most cases, you don't need large file downloads, but depending on what you are making (like a download manager, for example), you are limited to 2GB.

2.) I've had problems with this launching the user's default download manager (ie: Download Accelerator Pro). But it only happened in a few cases, I was never entirely sure why.

But if these 2 things aren't a problem then it is really the easiest and cleanest way to do it I think...

[Merri]

I guess one of the favorite uses would be an automatic updater.

On the first point, have you tried to let it download to file to see whether that limitation is true? You can't load 2 GB to byte array anyway as in most cases people run out of memory before that. However, these days I wouldn't download anything 2 GB of size if it isn't a torrent.

[guitar]

I'm assuming it's limited to 2GB by the fact that the AsyncProp's "BytesMax" and other members are of type Long. Maybe I am wrong about that?

Works perfect for an auto updater and other things like that.

Maybe you can try to see if it starts the user's default download manager, that's the only reason I stopped using it.

I also like how you made the single control do multiple downloads and also have the Cache options in there.

[Merri]

The datatype is a limitation (you can hack that up to 4 GB if you know a bit about conversions just like you can do that with FileLen), but it still doesn't mean you couldn't download to a file - it just depends on how the file downloading works since I didn't try that out (when a file is created and who is responsible of creating it? Is it a return value of Value?)

Personally I don't understand why one would like to use a download manager, I tried a few some years ago and they didn't really make things better at all. The opposite, some sites prevent you to access their files if you use a download manager (if you use one that tries to boost the download speed). Firefox 3 lets you even continue downloads, a long waited improvement for any browser.

[dilettante]

You can use AsyncRead to fetch to a temporary file instead of the a Byte array as well. It has lots of uses, but the primary limitation is that it can only do GET requests.

It was really meant for ActiveX controls you'd write for hosting in a browser page, but it works fine as shown above.

[dimicamp]

Hi Merri! Please, see if you can help me: I'm using VBS embedded in a SCADA system (WinCC). I've got the following code that's working well when I want to download a simple file. What can I do to download all the files (small ones) inside a folder?

obs 1: just posted the code to ilustrate my environment, I don't have to use a line from it.

obs 2: the download calls are made just once (initialization event), so we don't need to worry about duplicated calls for the same files.

Const adTypeBinary = 1
Const adSaveCreateOverWrite = 2

Dim strFileURL
Dim strHDLocation
Dim objXMLHTTP
Dim objStream

strFileURL = "http://localhost/Simatec/PrGe_MsgLog.Txt"
strHDLocation = "E:\PrGe_MsgLog.Txt"
 
Set objXMLHTTP = CreateObject("MSXML2.XMLHTTP")
objXMLHTTP.open "GET", strFileURL, False
objXMLHTTP.send()
 
If objXMLHTTP.Status = 200 Then
	Set objStream = CreateObject("ADODB.Stream")
	objStream.Open
	objStream.Type = adTypeBinary

	objStream.Write objXMLHTTP.ResponseBody
	objStream.Position = 0 'Set the stream position to the start
       
	objStream.SaveToFile strHDLocation, adSaveCreateOverWrite
	objStream.Close
	Set objStream = Nothing
End If
 
Set objXMLHTTP = Nothing

Thanks in advance,
Dimitri.

[Merri]

VBS unfortunatenaly goes out of the range of this code snippet. However, to get the file listing you have to do parsing. That would also go out of the range of this thread. It would be better to ask the question at ASP, VB Script forum.

[Edgemeal]

Is there something wrong with the way I'm using the code below casue every once in great while it returns nothing, I was using API code (sync mode) instead of this and that never failed. btw I'm just using it to download like a 127K sized html web page.

Option Explicit
Public Event Complete(ByRef URL As String, ByRef Data() As Byte)

Public Sub Start(ByRef URL As String)
    AsyncRead URL, vbAsyncTypeByteArray, "none", vbAsyncReadForceUpdate
End Sub

Private Sub UserControl_AsyncReadComplete(AsyncProp As AsyncProperty)
    Dim bytData() As Byte
    On Error GoTo DownloadFailed
    bytData = AsyncProp.Value
    RaiseEvent Complete(AsyncProp.Target, bytData)
    Exit Sub
DownloadFailed:
    Erase bytData()
    RaiseEvent Complete(AsyncProp.Target, bytData)
End Sub

[Merri]

Make UserControl_AsyncReadProgress print out the status code so you'll see a reason for why it isn't working sometimes. Also, you can check whether you are getting data by checking AsyncProp.BytesRead before assigning to byte array. This way you won't get an error when trying to retrieve data that does not exist.

[coolcurrent4u]

Merri what should i use for a program that download several links from the web at once

1 winsock control?
2 winsock api (dll)?
3 Urlmon?

i need to do a progress of each download

[Merri]

If you try out the AsyncProperty of a user control you'll notice it performs rather well. In the background it uses Internet Explorer's downloading routines, which seem to often perform better than people are able to do themselves using Winsock.

The only issue I guess is that you may not always get the full length information for the download, making it impossible to make an actual progress bar. You can however make some alternative methods, such as make an average of download speed.

[coolcurrent4u]

and how do you print the status

async read give error of unkown url if you pass a complex url with parameters like

http://www.vbforums.com/showthread.p...hlight=winsock

pls help with download resuming. how do i implement this

[coolcurrent4u]

ok pls help me with some code on how to do it

and how to download multiple files simultaneuosly with your control

[coolcurrent4u]

from the documetation of the AsyncRead it seems i can add your code or convert it to Class instead of user control.

Class will make it faster

am i right

[Merri]

i want to download only a particular no. of bytes, means like 100 to 1000th byte or generally mth to nth byte...how to do...
this refers to your post at http://www.vbforums.com/showthread.php?t=532597

Not possible using AsyncRead, you have to use another kind of implementation such as a Winsock. Also, just to let you know: some servers do not support reading files from anywhere.

In future please ask questions in the thread and not via email or PM if it directly relates to the thread in question.


coolcurrent4u: can't make it a class, AsyncRead is only available in a user control.

[Abu Badr]

nice downloader (Y) thanks

ok if I download a text file where does it go??
I can see the content of that text file in the textbox but I need the file itself.. and also the Arabic text shown as quistion marks!! like this "???? ??????? ??? ??????? ?????????"
how can i solve this problem?

[Jim Davis]

However, these days I wouldn't download anything 2 GB of size if it isn't a torrent

You dont have to download 2Gb of data into one single byte array becuse,

1. torrents are download small chunks/partials about 256K-2/5mbytes per piece (based on the torrent size/settings in client etc..) , on multiple threads. As soon as the chunk is completed, you can save the content into the file, then regenerate the array, and ask for an another piece on the same chunk size.

2. file downloaders (see flashget for example) that use httpd to retrieving the content can slice up the whole file into smaller chunks, and fire multiple threads to use the Resume feature of the httpd, so they can download these chunks simulteanously, that they save into the file using pointers.

Regarding your code in this post... Merri i have to say you are a genius. This is simple as much as just it can be, but still; it looks like the best downloader example i've see so far.

A very big up goes here
Thanks for the example as well!

edit:
What do you think, can it be used for retrieving partials of a given file by using the Resume feature on a certain website, or is it more like suitable for retrieving html files, then retrieve images and such things in a pretty much same way as the application Teleport does (that app will makes local backups of the site, by walking thru each links, then retrieving every html/image/etc content recursively)?

[Abu Badr]

any updates?!

[Merri]

Abu Badr: instead of displaying in a textbox you save a file

Private Sub Downloader1_Complete(URL As String, Data As String, Key As String)
    Dim intFF As Integer, bytData() As Byte
    Me.Caption = URL
    ' get a file number
    intFF = FreeFile
    ' place in a byte array
    bytData = Data
    ' open for binary access so every byte is saved the way it is
    Open "C:\test.txt" For Binary Access Write As #intFF
        Put #intFF, , bytData
    Close #intFF
End Sub

This of course always saves to just one file, you need to write your own code to allow saving anywhere you want. Note that this method is not the shortest for "just saving files", there are other ways of doing that, but they're out of the scope of this thread.


Jim Davis: I wasn't talking about saving 2 GB of data to a byte array, I speaked more generally on file downloading. I never download anything that big unless it is a torrent. Also, this method is very simple and does not allow chunks. It is just the same mechanism that Internet Explorer uses to download files, I think.

[Abu Badr]

Thanks Merri that solved the Arabic issue also 2 in 1

[coolcurrent4u]

hello Merri, wheni saw the comments, i thought there was an update. any update for this code.

good work.

[Merri]

No update; I guess any update would make the project only bigger and that would ruin the concept of a simple starting point component.

[Chris001]

Is there a way to get the size of the file before downloading starts, so I can use a progressbar?

Does this also work on Vista and Windows 7?

[Merri]

Afaik it works, but I can't recall whether I have tested it or not. As for final file size, I guess it depends on server. Some servers may not report file size when deliverying a file. I didn't do extensive tests on different sites to see if it ever gives the total file size during progress.

[Chris001]

Thank you, Merri.

[adw1104]

Thanks for this code snippet Merri. I am quite inexperienced and I need some help applying this to my application.

It appears that downloader returns the data in a string.

I want to download .jpg images from a web based product catalogue.

I will pick up the URL from the Change event of a textbox and pass it to the "Start" sub in your downloader.

Then I want to pass the result from downloader into an Image control on my form.

How do I handle .jpg image files using your downloader?

[Merri]

That is a separate issue. You may wish to look for topics that deal with opening a JPEG image directly from memory, I think I've seen some of those sometime. The easiest to do is to save the image directly in a file, but I assume it would kind of defeat the purpose of using a memory downloader.

[adw1104]

Do you mean I can just save the "Data" returned in the downloader string as a *.jpg and then pass it to the picture property of the image control?

[Merri]

Yes. For saving the file you can refer to post #20 of this thread to see how to save the data so that no byte is changed. Just as a FYI, when you normally save or load a file involving a string in VB6 the data is processed "intelligently", you may get a different result than you expect.

Note however that there are also one line API solutions for saving a file to disk, which may or may not be better. It all depends on what you try to do.

[Chris001]

I've got two more questions.

1) Is there a proper way of stopping the download?

I added the following sub to the control to cancel the download and that works fine, but when I start the download again I get an error.

Run-time error '-2147024882 (8007000e)':
System Error &H80070057 (-2147024809).

Public Sub CancelDownload(URL As String)
    CancelAsyncRead URL
End Sub

2) Is there a way to save the data while it is downloading? With the current code, the data is only saved to the hard drive when it has been fully downloaded.

[Chris001]

This is frustrating.

I just finished my application and I'm using this method to download a webpage. Everything works fine on my computer. But when I run my app on an other computer, the "Downloader1_Complete" event fires but 'Data' is empty.


Edit::

Fixed the problem. Make sure you have at least Internet Explorer 7 installed. With IE6 it sometimes fails to download the file/webpage.

[Chris001]

Is there a way to calculate the download speed and remaining download time when using AsyncRead?

[batori]

or a way to get the filesize ... in order to make a progress bar out of it ...

very nice , thanks

[batori]

managed to get the filesize...

is there a way of adding custom cookie??

cheers

[coolcurrent4u]

managed to get the filesize...

is there a way of adding custom cookie??

cheers

do you mind to share you code?

[batori]

of course, sure.

This is my usercontrol, thanks to Merri! I added the cancel procedure, changed in the Event Progress, BytesDone as Double, since i use a procedure to convert bytest to MB , ecc.

Option Explicit

Public Event Complete(ByRef URL As String, ByRef data As String, ByRef Key As String)
Public Event Progress(ByRef URL As String, ByRef Key As String, ByVal BytesDone As Double, ByVal bytesTotal As Double, ByVal Status As AsyncStatusCodeConstants)

Public Enum DownloaderCache
    [Always download] = vbAsyncReadForceUpdate
    [Get cache copy only] = vbAsyncReadOfflineOperation
    [Update cached copy only] = vbAsyncReadResynchronize
    [Use cache if no connection] = vbAsyncReadGetFromCacheIfNetFail
End Enum

Private m_Keys As String

Private Function Private_AddKey(ByRef Key As String) As Boolean
    ' see if we do not have the key
    Private_AddKey = InStr(m_Keys, vbNullChar & Key & vbNullChar) = 0
    ' we can add it
    If Private_AddKey Then
        m_Keys = m_Keys & Key & vbNullChar
    End If
End Function

Private Sub Private_RemoveKey(ByRef Key As String)
    ' remove the key
    m_Keys = Replace(m_Keys, vbNullChar & Key & vbNullChar, vbNullChar)
End Sub

Public Sub Start(ByRef URL As String, Optional ByVal CacheMode As DownloaderCache = [Always download], Optional ByVal Key As String)
   On Error Resume Next
    ' use URL as key if no key is given
    If LenB(Key) = 0 Then Key = URL
    ' do we already have this key?
    If Not Private_AddKey(Key) Then
        ' cancel the old one
        CancelAsyncRead Key
    End If
    ' begin download process
    AsyncRead URL, vbAsyncTypeByteArray, Key, CacheMode
End Sub
 
Public Sub CancelDownload(URL As String)
On Error Resume Next
    CancelAsyncRead URL
End Sub
Private Sub UserControl_AsyncReadComplete(AsyncProp As AsyncProperty)
    Dim strData As String
    ' get Variant byte array to byte string (needs StrConv to Unicode for displaying in a textbox)
    If AsyncProp.BytesRead Then strData = AsyncProp.Value Else strData = vbNullString
    ' redirect information
    RaiseEvent Complete(AsyncProp.Target, strData, AsyncProp.PropertyName)
    ' remove the key
    Private_RemoveKey AsyncProp.PropertyName
End Sub

Private Sub UserControl_AsyncReadProgress(AsyncProp As AsyncProperty)
    With AsyncProp
        ' redirect event information
        If LenB(.PropertyName) Then
            RaiseEvent Progress(.Target, .PropertyName, .BytesRead, .BytesMax, .StatusCode)
        Else
            RaiseEvent Progress(.Target, vbNullString, .BytesRead, .BytesMax, .StatusCode)
        End If
    End With
End Sub

Private Sub UserControl_Initialize()
    m_Keys = vbNullChar
End Sub

This is how i start the download :

Downloader1.Start "dl_link", [Always download]

My question would be, how to add a custom cookie since like "myCookie=sddfasdezrzeew23213" ?

Cheers

[Ravenshade]

Okay, I'm ashamed to be reviving post so old, but this is the only thread which I've found to be useful. I'm using vb6, but I'm having problems with the code above.

Here's my code that I'm using

Option Explicit

Private Sub Cmd_Update_Click()
    Dim UserControl1 As UserControl
    UserControl1.Start ("Bahamut/launcherpad")
End Sub

I had to put in the definition as using the default code all I was getting was that it couldn't find the User Control. I'm new to vb6 but I'm working at it.

Everything is is pretty much as it should be, but I'm getting an error even with this

Runtime Error 91 Object Variable or With block variable not set.

Could someone assist me with what I'm doing wrong?

[Ravenshade]

Sorry for the dupe post. Can't find the edit button.

I have tried using the call function to no avail. Defining it this way is the only way I can get it to compile.

[Zeev]

I dont get it, where do I download the Downloader.ctl file?