[RESOLVED]Is there a faster alternative for HTTP download? (URLDownloadToFile) urlmon

[gabba]

I am making an application that is speed-critical, and needs to download a file over HTTP and parse the download within 0.3 seconds.
The program needs to be able to complete at least 3 cycles per second.

The actual page being downloaded is actually a very modest 5900 bytes, but I cannot seem to get my VB application to download it any faster than 400ms, even on an 8MBps connection. Our work has an 8MB connection, so the actual environment is the same, and I am using the same disk speed (the benchmarks are around the same too).

Can I cut the download time of 400ms to to maybe 50 - 200ms? The parser takes about 20ms, and that is simplified to a stick-figure.

Here is my download module:

Public Declare Function URLDownloadToFile Lib "urlmon" Alias "URLDownloadToFileA" (ByVal pCaller As Long, ByVal szURL As String, ByVal szFileName As String, ByVal dwReserved As Long, ByVal lpfnCB As Long) As Long

Public Function DownloadFile() As String
Dim lngRetVal As Long
Dim hfile As Long
LocalFilename = "C:\jEdit.working.temp"
lngRetVal = URLDownloadToFile(0, strCurrentURL, LocalFilename, 0, 0)
      hfile = FreeFile
      Open LocalFilename For Input As #hfile
         DownloadFile = Input$(LOF(hfile), hfile)
      Close #hfile
End Function

Are there any FAST alternatives to URLDownloadToFile?

I have tried Winsock, but the average (over 50+ tries.. I lost count) is about 800ms.

Many thanks in advance,

Gab

[si_the_geek]

I think the biggest issue in terms of speed is the saving the file, and loading it again. Ideally you should just be downloading directly to a variable.

I haven't done much of this kind of thing myself, but I think this example from API-Guide will be faster:

Const sURL = "http://www.microsoft.com/index.htm"
Const scUserAgent = "API-Guide test program"
 
Const INTERNET_OPEN_TYPE_DIRECT = 1
Const INTERNET_OPEN_TYPE_PROXY = 3
Const INTERNET_FLAG_RELOAD = &H80000000
Private Declare Function InternetOpen Lib "wininet" Alias "InternetOpenA" (ByVal sAgent As String, ByVal lAccessType As Long, ByVal sProxyName As String, ByVal sProxyBypass As String, ByVal lFlags As Long) As Long
Private Declare Function InternetCloseHandle Lib "wininet" (ByVal hInet As Long) As Integer
Private Declare Function InternetReadFile Lib "wininet" (ByVal hFile As Long, ByVal sBuffer As String, ByVal lNumBytesToRead As Long, lNumberOfBytesRead As Long) As Integer
Private Declare Function InternetOpenUrl Lib "wininet" Alias "InternetOpenUrlA" (ByVal hInternetSession As Long, ByVal lpszUrl As String, ByVal lpszHeaders As String, ByVal dwHeadersLength As Long, ByVal dwFlags As Long, ByVal dwContext As Long) As Long
 
Private Sub Form_Load()
    'KPD-Team 1999
    'URL: [url]http://www.allapi.net/[/url]
    'E-Mail: [email][email protected][/email]
 
    Dim hOpen As Long, hFile As Long, sBuffer As String, Ret As Long
    'Create a buffer for the file we're going to download
    sBuffer = Space(1000)
    'Create an internet connection
    hOpen = InternetOpen(scUserAgent, INTERNET_OPEN_TYPE_DIRECT, vbNullString, vbNullString, 0)
    'Open the url
    hFile = InternetOpenUrl(hOpen, sURL, vbNullString, ByVal 0&, INTERNET_FLAG_RELOAD, ByVal 0&)
    'Read the first 1000 bytes of the file
    InternetReadFile hFile, sBuffer, 1000, Ret
    'clean up
    InternetCloseHandle hFile
    InternetCloseHandle hOpen
    'Show our file
    MsgBox sBuffer
End Sub

..the only thing that worries me is how to detect the size of the file.

By the way, are you doing your timing in VB, or with the compiled project? (the compiled project will be faster)

[VBAhack]

You might try HTTPrequest:

http://www.vbforums.com/showthread.php?t=433435

[gabba]

URLDownloadToFile seems to prevail as fastest, being 2-4 times faster (on average) that the other two methods (and many more) suggested. I've tested many, using winsock and inetcontrol.

Are there any APIs that people are aware of, that can shift binary, fast?

I have exhausted most Referenced objects and non-commercial controls.

Would I be looking to buy an OCX or DLL for fast data transfer?

[DigiRev]

You could try the Winsock API but I doubt you would get any speed increase.

Downloading the file to a byte variable would be faster than a String, so maybe you could try that with Winsock/Winsock API.

What is your parsing routine?

Also, I don't see how URLToDownload() would be faster than si_the_geek's method because you're saving to a temporary file first, when you should be storing the file directly to memory.

[gabba]

It's true... I was also intrigued as to why the difference, but I'm cracking on with the rest of the program, and using URLDownloadToFile.

Thank you for your help Si, DigiRev and VBAhack.

[rory]

Problem with UrlDownloadToFile API is it hangs the app while it downloads ..
But this is how i did it ..

Private Declare Function URLDownloadToFile Lib "urlmon" Alias _
"URLDownloadToFileA" (ByVal pCaller As Long, _
ByVal szURL As String, _
ByVal szFileName As String, _
ByVal dwReserved As Long, _
ByVal lpfnCB As Long) As Long

Private Function DownloadFile(URL As String, LocalFilename As String) As Boolean
    Dim lngRetVal As Long
    lngRetVal = URLDownloadToFile(0, URL, LocalFilename, 0, 0)
    If lngRetVal = 0 Then DownloadFile = True
End Function

Private Sub Command1_Click()
    If DownloadFile("http://www.mysite.com/setup.exe", "c:\setup.exe") Then
        Debug.Print "Download Complete"
    End If
End Sub

And here is a version with progress .. uses the Inet API and works differently.
you could just cut out all the progress stuff for your own app ..
http://www.vbforums.com/showthread.p...19#post2576619


Here is a winsock version ..
http://www.vbforums.com/attachment.p...chmentid=52319

Attached is a VBscript version using MSXML ..
can easily convert to VB6.

[rory]

try this Inet API (its minus that progress code I posted above)
its pretty fast ... may also want to add HTTP Response Error codes perhaps.

Form: (requires command1 for sample)

Option Explicit

Private Sub Command1_Click()
    Dim sURL As String
    sURL = "http://www.mysite.com/setup.exe"
    '// START DOWNLOAD
    Command1.Enabled = False
    Command1.Caption = "Downloading.."
    If DownloadInetAPI(sURL, True) Then
        '// DOWNLOAD SUCCEEDED
        Command1.Caption = "Download Complete"
    Else
        '// DOWNLOAD FAILED
        Command1.Caption = "Download Failed"
    End If
    Beep
End Sub

Module:

Option Explicit

Private Const MAX_BUFFER_LENGTH = 8162
Private Const API_AGENT_NAME As String = "VB Program"
Private Const INTERNET_OPEN_TYPE_DIRECT = 1
Private Const INTERNET_FLAG_NO_CACHE_WRITE = &H4000000

Private Declare Function InternetCloseHandle Lib "wininet" (ByRef hInet As Long) As Long
Private Declare Function InternetOpen Lib "wininet" Alias "InternetOpenA" (ByVal sAgent As String, ByVal lAccessType As Long, ByVal sProxyName As String, ByVal sProxyBypass As String, ByVal lFlags As Long) As Long
Private Declare Function InternetReadFile Lib "wininet" (ByVal hFile As Long, ByVal sBuff As String, ByVal lNumBytesToRead As Long, lNumberOfBytesRead As Long) As Integer
Private Declare Function InternetOpenUrl Lib "wininet" Alias "InternetOpenUrlA" (ByVal hInternetSession As Long, ByVal lpszUrl As String, ByVal lpszHeaders As String, ByVal dwHeadersLength As Long, ByVal dwFlags As Long, ByVal dwContext As Long) As Long

Public Function DownloadInetAPI(ByVal sURL As String, ByVal DEBUG_ON As Boolean) As Boolean
    Dim hOpen As Long, hFile As Long, Ret As Long
    Dim sBuff As String * MAX_BUFFER_LENGTH, sData As String
    Dim sFileName As String, iFile As Integer, dData As Double
    If InStr(1, sURL, "/") Then
        sFileName = App.Path & "\" & Right$(sURL, Len(sURL) - InStrRev(sURL, "/"))
    Else
        sFileName = "Download" & Format(Now, "hhmmssmmddyy") & ".tmp"
    End If
    If DEBUG_ON Then Debug.Print "Connecting ... "
    hOpen = InternetOpen(API_AGENT_NAME, INTERNET_OPEN_TYPE_DIRECT, vbNullString, vbNullString, 0)
    If hOpen = 0 Then
        MsgBox "Error opening Internet connection"
        DownloadInetAPI = False
        Exit Function
    End If
    hFile = InternetOpenUrl(hOpen, sURL, vbNullString, 0, INTERNET_FLAG_NO_CACHE_WRITE, 0)
    If hFile = 0 Then
        dData = 0
    Else
        If DEBUG_ON Then Debug.Print "Downloading ... " & Format(Len(sData) / 1024, 0) & " KB"
        InternetReadFile hFile, sBuff, MAX_BUFFER_LENGTH, Ret
        sData = sBuff
        Do While Ret <> 0
            If DEBUG_ON Then Debug.Print "Downloading ... " & Format(Len(sData) / 1024, 0) & " KB"
            InternetReadFile hFile, sBuff, MAX_BUFFER_LENGTH, Ret
            sData = sData + Mid(sBuff, 1, Ret)
        Loop
        dData = Len(sData): iFile = FreeFile
        Open sFileName For Binary Access Write Lock Write As #iFile
        Put #iFile, , sData: Close #iFile
    End If
    InternetCloseHandle hFile
    InternetCloseHandle hOpen
    sData = ""
    If dData Then
        DownloadInetAPI = True
    End If
End Function

[gabba]

Rory, I thank you greatly for your time and input to this discussion.

I downloaded you code, and I still concur that URLDownloadToFile is faster than InternetOpen. Strange, but true.

To attempt to make it even faster, I am going to buy SpeedSoft RAMDisk Plus and set the URLDownloadToFile :: szFileName a file on the RAMDisk... hopefully my PC2-5400 memory will be slightly faster than my SATA-150 laptop disk... don't know yet, so I am going to try... I'm sure it will be faster, because of the lookup rate of the disk is 8ms alone... Access must be another 10ms, and transfer ontop of that...

Gab

[DigiRev]

Are you sure your test results are accurate? What are you using to time the results?

Also, there's so many variables in downloading files over the internet, that it will be hard to get it to download in 50-200ms every single time. An HTTP request can take longer than that, not to mention transferring the file.

Also, the HTTP server could be busy, your internet connection could be a little slower sometimes depending on your ISP...etc.

I would recommend you time your results with the GetTickCount() API function, and not a timer, if that's what you're using.

[gabba]

Test results are accurate as far as I know;
I'm using GetTickCount
HTTP Webserver is private, and isn't accessed by anyone during weekends, except me (hardware firewall report agrees with me)

Just spent $50 on a copy of RAMDisk, I've got to say, it's instant disk access... really really fast. Haven't benched it yet though, expecting great things.

[intothegreatwideopen]

Hi,
allow me one correction. The URLDownloadToFile method has one great disadvantage. It delivers a cached file so this method seems to be very fast beeing frequently called. But that's not what you wanted. So you have to empty the cache before. And then the Wininet-methods are beeing the faster solution !

Public Declare Function DeleteUrlCacheEntry Lib "Wininet.dll" Alias "DeleteUrlCacheEntryA" (ByVal lpszUrlName As String) As Long

Cheers Claus

[DataMiser]

Have you tried using FTP? Would that be an option? I would think that part of your speed issue is getting the connection and making the request for the file. This may be faster using FTP once you are connected to the server.

[Tilator]

Have you tried using FTP? Would that be an option? I would think that part of your speed issue is getting the connection and making the request for the file. This may be faster using FTP once you are connected to the server.

B.T.W

URLDownloadToFileA is affected with webbrowser emulation reg key too.

[Zahl]

FTP still needs to establish another connection for the file transfer itself, not much is gained.

Afaik URLDownloadToFileA does a lot of magic in the background, caching was mentioned before. It probably also uses keep-alive connections which would save you some time on consecutive requests to the same server. If it doesn't do that, that would definitely be something to look into. If you're running this against a server under your control, you could configure it to allow really long keep alive times, so your program basically just establishes a connection once and keeps using it for as long as it's running.

And, as others said, try looking into solutions that directly return the data as a String or Byte(). That kludge with the RAMdisk sounds really unnecessary.

EDIT: Great, just noticed everything but the last reply was from 2013.

[wqweto]

EDIT: Great, just noticed everything but the last reply was from 2013.

This happens sometimes in these forums :-))

For posterity:

For large files download managers (used to) split these and use range requests with http protocol which will beat URLDownloadToFileA every time. Another (better) option is torrent download -- this is like splitting the download into hundreds of simultaneous connections.

For small 5KB files nothing can be done except upgrading internet connection to something with less latency (not satelite links) and/or moving physically to the server which is providing the files (as the speed of light is constant).

cheers,
</wqw>

[The trick]

URLDownloadToFile by default is just wrapper around URL-moniker (if you don't use additional parameters). You could use Usercontrol.AsyncRead instead.

[xiaoyao]

If the download speed is still not fast enough, you can try downloading in chunks. For example, a ZIP file, divided into 10 multi-threaded or controlled downloads.

[wqweto]

If the download speed is still not fast enough, you can try downloading in chunks. For example, a ZIP file, divided into 10 multi-threaded or controlled downloads.

How to do this? Show us some VB6 code.

cheers,
</wqw>