Question about WinHttpRequest object

[Ben321]

Ok so I'm writing this downloader application that will download a video from Dailymotion. Now I have 2 options, when using WinHttpRequest. I can either wait until the entire file is downloaded by using the OnResponseFinished event, and then save to a file the entire content of the video all at once, or I can save it one block at a time by using the OnResponseDataAvailable event. The problem with the first technique is it has to save that data somewhere so that it can be retrieved with the ResponseBody property, and guess where it goes. It goes into memory. So that poses the problem of the 1GB limit available to programs written in VB6. Downloading a 20 minute 1080p video from Dailymotion was over 1GB, and my program crashed. So I decided to try the "save one block at a time" approach. Problem is, that even when I'm using the Data() array available in the OnResponseDataAvailable event (even when I use the command Erase Data() to try to clear that array after it's done being used), that does NOT clear out the memory being stored by the WinHttpRequest object, in preparation for making it available in the ResponseBody property when the OnResponseFinished event is fired. I can find no way to clear out the memory being used for the ResponseBody property, even when I'm using the Data() byte array in the OnResponseDataAvailable event. And the end result is still my program crashes. I need help from somebody here who knows all the tricks for using the WinHttpRequest object. How can I clear that memory to prevent my program from crashing? Is it not possible?

[fafalone]

WinHttpRequest keeps the entire response in memory by design; really doubt there's any getting around that.

[Bonnie West]

You may want to try the WinINet APIs instead. Here's an example:

Private Declare Function CloseHandle Lib "kernel32.dll" (ByVal hObject As Long) As Long
Private Declare Function CreateFileW Lib "kernel32.dll" (ByVal lpFileName As Long, ByVal dwDesiredAccess As Long, ByVal dwShareMode As Long, ByVal lpSecurityAttributes As Long, ByVal dwCreationDisposition As Long, Optional ByVal dwFlagsAndAttributes As Long, Optional ByVal hTemplateFile As Long) As Long
Private Declare Function GetQueueStatus Lib "user32.dll" (ByVal Flags As Long) As Long
Private Declare Function InternetCloseHandle Lib "wininet.dll" (ByVal hInternet As Long) As Long
Private Declare Function InternetOpenW Lib "wininet.dll" (ByVal lpszAgent As Long, ByVal dwAccessType As Long, ByVal lpszProxyName As Long, ByVal lpszProxyBypass As Long, ByVal dwFlags As Long) As Long
Private Declare Function InternetOpenUrlW Lib "wininet.dll" (ByVal hInternet As Long, ByVal lpszUrl As Long, ByVal lpszHeaders As Long, ByVal dwHeadersLength As Long, ByVal dwFlags As Long, ByVal dwContext As Long) As Long
Private Declare Function InternetReadFile Lib "wininet.dll" (ByVal hFile As Long, ByVal lpBuffer As Long, ByVal dwNumberOfBytesToRead As Long, ByRef lpdwNumberOfBytesRead As Long) As Long
Private Declare Function SysReAllocStringLen Lib "oleaut32.dll" (ByVal pBSTR As Long, Optional ByVal pszStrPtr As Long, Optional ByVal Length As Long) As Long
Private Declare Function WriteFile Lib "kernel32.dll" (ByVal hFile As Long, ByVal lpBuffer As Long, ByVal nNumberOfBytesToWrite As Long, Optional ByRef lpNumberOfBytesWritten As Long, Optional ByVal lpOverlapped As Long) As Long

'Downloads the remote file specified by the sURL argument to the local file pointed
'by the sFileName parameter. The optional Chunk parameter determines the number
'of bytes to be downloaded at a time. Bigger chunks download faster while smaller
'ones enables the GUI to be more responsive. Returns the total number of bytes
'successfully written to disk. Maximum download size of 2,047.99 MB only.

Public Function DownloadURL2File(ByRef sURL As String, ByRef sFileName As String, Optional ByVal Chunk As Long = 1024&) As Long
    Const INTERNET_OPEN_TYPE_DIRECT = 1&, INTERNET_FLAG_DONT_CACHE = &H4000000, INTERNET_FLAG_RELOAD = &H80000000
    Const GENERIC_WRITE = &H40000000, FILE_SHARE_NONE = 0&, CREATE_ALWAYS = 2&, QS_ALLINPUT = &H4FF&
    Const INVALID_HANDLE_VALUE = -1&, ERROR_INSUFFICIENT_BUFFER = &H7A&
    Dim hInternet As Long, hURL As Long, hFile As Long, nBytesRead As Long, nBytesWritten As Long
    Dim bSuccess As Boolean, sBuffer_Ptr As Long, sBuffer_Size As Long, sBuffer As String

    Select Case True
        Case LenB(sURL) = 0&, LenB(sFileName) = 0&, Chunk < 2&:  Exit Function
    End Select

    hInternet = InternetOpenW(StrPtr(App.Title), INTERNET_OPEN_TYPE_DIRECT, 0&, 0&, 0&)
    If hInternet Then
        hURL = InternetOpenUrlW(hInternet, StrPtr(sURL), 0&, 0&, INTERNET_FLAG_DONT_CACHE Or INTERNET_FLAG_RELOAD, 0&)
        If hURL Then
            hFile = CreateFileW(StrPtr(sFileName), GENERIC_WRITE, FILE_SHARE_NONE, 0&, CREATE_ALWAYS) 'Overwrite existing
            If hFile <> INVALID_HANDLE_VALUE Then
                Do: SysReAllocStringLen VarPtr(sBuffer), , (sBuffer_Size + Chunk) * 0.5!
                    sBuffer_Size = LenB(sBuffer):   sBuffer_Ptr = StrPtr(sBuffer)
                    Do While InternetReadFile(hURL, sBuffer_Ptr, sBuffer_Size, nBytesRead)
                        If nBytesRead Then
                            bSuccess = (WriteFile(hFile, sBuffer_Ptr, nBytesRead, nBytesWritten) <> 0&) _
                                        And (nBytesWritten = nBytesRead): Debug.Assert bSuccess
                            If bSuccess Then DownloadURL2File = DownloadURL2File + nBytesWritten
                            If GetQueueStatus(QS_ALLINPUT) And &HFFFF0000 Then DoEvents
                        Else
                            Exit Do
                        End If
                    Loop
                Loop While Err.LastDllError = ERROR_INSUFFICIENT_BUFFER
                hFile = CloseHandle(hFile):                               Debug.Assert hFile
            End If
            hURL = InternetCloseHandle(hURL):                             Debug.Assert hURL
        End If
        hInternet = InternetCloseHandle(hInternet):                       Debug.Assert hInternet
    End If
End Function

[Ben321]

You may want to try the WinINet APIs instead. Here's an example:

Does the WinINet API use caching like the Internet Explorer browser does? Often times different browsers have their own cache (Firefox's cache is independent of the Windows internet cache setting that Internet Explorer uses, for example). Does this WinINet use the these same Windows settings that Internet Explorer uses, or does it have its own (in effect, behaving as its own separate web browser). The last thing I want to do is have a bunchof large files wasting harddrive space because I forgot to clear my IE cache. I don't often worry about clearing the IE cache out, because I always use Firefox for browsing. So if I use WinINet, and it uses the IE cache, then it would be easy to forget to clear it out, and get a bunch of big files eating up harddrive space.

[wqweto]

@Ben321: You can use `INTERNET_FLAG_NO_CACHE_WRITE` to not fill up IE cache with large AVIs

cheers,
</wqw>

[Ben321]

You may want to try the WinINet APIs instead. Here's an example:

I just noticed your last parameter (the flags parameter) in your use of InternetOpen is set to 0 instead of &h10000000 (INTERNET_FLAG_ASYNC). Why? That means if the URL you type in is invalid it will try forever to go there, and lock up the program, causing you to need to use the task manager to close it. Whenever there is an ASync parameter in an API call, I always enable it, so that the program doesn't lock up. I then use my own methods to determine if the procedure that was initiated had completed successfully and a timeout condition to exit that procedure after a reasonable time.

[Ben321]

@Ben321: You can use `INTERNET_FLAG_NO_CACHE_WRITE` to not fill up IE cache with large AVIs

cheers,
</wqw>

Does the use of INTERNET_FLAG_DONT_CACHE in the InternetOpenURL function also work? I notice "Bonnie West" used that flag in her sample code.

[Ben321]

Just looked at some of the flags for the InternetOpen and InternetOpenUrl functions and noticed that 2 of the flags appear to be described as having the same effect. Can you let me know what the difference is?
According to MSDN:

INTERNET_FLAG_PRAGMA_NOCACHE
0x00000100
Forces the request to be resolved by the origin server, even if a cached copy exists on the proxy. The InternetOpenUrl function (on HTTP and HTTPS requests only) and HttpOpenRequest function use this flag.

INTERNET_FLAG_RELOAD
0x80000000
Forces a download of the requested file, object, or directory listing from the origin server, not from the cache. The FtpFindFirstFile, FtpGetFile, FtpOpenFile, FtpPutFile, HttpOpenRequest, and InternetOpenUrl functions use this flag.

It sounds like they do the same thing, in both cases, it sounds like it tells the browser to ignore the cache, and instead get the intended resource from the server. Can you explain what the difference is?

[fafalone]

One works with both ftp and http, the other only http?

[Ben321]

You may want to try the WinINet APIs instead. Here's an example:

I'd like to point out that according to the MSDN page here https://msdn.microsoft.com/en-us/lib...=vs.85%29.aspx the InternetCloseHandle function is designed to close an hInternet, not an hURL. So there must be some other handle closing function for an hURL.

[Ben321]

You may want to try the WinINet APIs instead. Here's an example:

I am finding that InternetCloseHandle(hURL) and InternetCloseHandle(hInternet) always returns 0, even if I'm not making the mistake of trying to close the handle while while data is being transferred.

[Ben321]

I am finding that InternetCloseHandle(hURL) and InternetCloseHandle(hInternet) always returns 0, even if I'm not making the mistake of trying to close the handle while while data is being transferred.

Nevermind. I found that I was using the incorrect definition of one of the parameters (The software "API Viewer 2004" had it "ByRef hInternet as Long" when it should have been "ByVal hInternet as Long"), so I had to correct it. It works now, and returns 1 instead of 0, indicating that it is now working.

[Bonnie West]

Does the WinINet API use caching like the Internet Explorer browser does? ... Does this WinINet use the these same Windows settings that WinINet uses, or does it have its own (in effect, behaving as its own separate web browser).

See the WinINet's Caching subtopic.

I just noticed your last parameter (the flags parameter) in your use of InternetOpen is set to 0 instead of &h10000000 (INTERNET_FLAG_ASYNC). Why? That means if the URL you type in is invalid it will try forever to go there, and lock up the program, causing you to need to use the task manager to close it.

When I tried passing a bogus URL to the InternetOpenUrl function, it froze momentarily but eventually returned with a value of 0. The process didn't hang indefinitely, so I don't think there's any risk of specifying inaccessible URLs.

I chose not to use the INTERNET_FLAG_ASYNC flag because my strategy was to make InternetReadFile download small chunks of data at a time and then call DoEvents if there's any input to process.

Does the use of INTERNET_FLAG_DONT_CACHE in the InternetOpenURL function also work? I notice "Bonnie West" used that flag in her sample code.

INTERNET_FLAG_DONT_CACHE has the same meaning & value as the preferred constant name INTERNET_FLAG_NO_CACHE_WRITE. At the time I wrote that function, the documentation was still using the old constant name.

It sounds like they do the same thing, in both cases, it sounds like it tells the browser to ignore the cache, and instead get the intended resource from the server. Can you explain what the difference is?

I found this blog post that provides some info regarding the INTERNET_FLAG_PRAGMA_NOCACHE and INTERNET_FLAG_RELOAD flags:

• If INTERNET_FLAG_PRAGMA_NOCACHE is set
  
  • If the request is going through a proxy or is HTTP/1.0, Pragma: no-cache is added. If the request is not going through a proxy and is HTTP/1.1, then Cache-Control: no-cache is added.
• If INTERNET_FLAG_RELOAD is set:
  
  • WinINET will bypass the cache (redownloading all entries)
  • WinINET will not send an If-Modified-Since or If-None-Match request header on these requests (Unconditional request; server cannot return a HTTP/304).
  • WinINET will add a request header to help ensure that an intermediary does not return a previously-cached result.
    
    • If the request is going through a proxy or is HTTP/1.0, Pragma: no-cache is added. If the request is not going through a proxy and is HTTP/1.1, then Cache-Control: no-cache is added.

I'd like to point out that according to the MSDN page here https://msdn.microsoft.com/en-us/lib...=vs.85%29.aspx the InternetCloseHandle function is designed to close an hInternet, not an hURL. So there must be some other handle closing function for an hURL.

The InternetOpenUrl API function actually returns a handle of type HINTERNET as well. I only named that variable as hURL because I was already using the hInternet name for the other variable.

After the calling application has finished using the HINTERNET handle returned by InternetOpenUrl, it must be closed using the InternetCloseHandle function.

[Jonney]

Look like what approach of "save one block at a time" is somewhat streaming? Many on-line video players have such "downloading and playing" feature.

[Ben321]

See the WinINet's Caching subtopic.



When I tried passing a bogus URL to the InternetOpenUrl function, it froze momentarily but eventually returned with a value of 0. The process didn't hang indefinitely, so I don't think there's any risk of specifying inaccessible URLs.

I chose not to use the INTERNET_FLAG_ASYNC flag because my strategy was to make InternetReadFile download small chunks of data at a time and then call DoEvents if there's any input to process.



INTERNET_FLAG_DONT_CACHE has the same meaning & value as the preferred constant name INTERNET_FLAG_NO_CACHE_WRITE. At the time I wrote that function, the documentation was still using the old constant name.



I found this blog post that provides some info regarding the INTERNET_FLAG_PRAGMA_NOCACHE and INTERNET_FLAG_RELOAD flags:





The InternetOpenUrl API function actually returns a handle of type HINTERNET as well. I only named that variable as hURL because I was already using the hInternet name for the other variable.

Thanks for the info. That really helped.

[Ben321]

You may want to try the WinINet APIs instead. Here's an example:

I know this is a REALLY REALLY REALLY OLD thread, but I'm revisiting internet programming now, and wondering if the same thing that your above code does in wininet.dll can also be accomplished with the winhttp.dll (just not the winhttp ActiveX component). Do the normal winhttp functions store the full response buffer like the winhttp ActiveX component does? I was thinking about using winhttp instead of wininet, due to me thinking that maybe it doesn't cache the server responses to the harddrive. I know that with the right flags wininet shouldn't either, but I read that those flags for no caching to the harddrive are only for http connections. Https connections in wininet are automatically cached to the harddrive unless the Windows internet control panel advanced settings are configured to not do https caching.

To make my program more user friendly, I was thinking that maybe instead of asking the user to change their control panel settings to disable caching, I could simply use winhttp instead of wininet, as I don't think the winhttp functions do any caching (at least I haven't read that they do). Can you confirm 2 things for me? These are:
Whether or not the winhttp.dll functions avoid caching completely for both http and https (no need to change Windows control panel settings).
Whether or not the winhttp.dll functions avoid storing the complete response in a single large buffer.

[wqweto]

I'm not sure Bonnie West will ever answer these questions but IMO you'll have yes on both of these.

The only way WinHTTP might get cached results is by intermediate proxies so you have to set cache control headers to prevent this.

For chunked response handling you can take a look at WinHttpReadData API function sample code at the bottom of the page.

cheers,
</wqw>

[Ben321]

I'm not sure Bonnie West will ever answer these questions but IMO you'll have yes on both of these.

The only way WinHTTP might get cached results is by intermediate proxies so you have to set cache control headers to prevent this.

For chunked response handling you can take a look at WinHttpReadData API function sample code at the bottom of the page.

cheers,
</wqw>

Does Wininet always do chunked? Does Winhttp only do chunked if specified? I just want to make sure that one other requirement of my program is met, and that is that it's WinXP compatible. Enough people I think still use it, that for the time being, I want my programs to be XP compatible. And the feature you talked about on the Microsoft WinHttpReadData page indicates that it only became available in Windows Vista. That's a no-go for me right there.

Also it mentions in "Starting in Windows Vista and Windows Server 2008, WinHttp enables applications to perform chunked transfer encoding on data sent to the server" that is referring to data sent TO the server such as in an HTTP POST request (not received from it).

[wqweto]

I meant "chunked" like in parts, not like in "Transfer-Encoding: chunked" header attribute.

The comment about sending POST requests using "Transfer-Encoding: chunked" I'm not sure what the benefit of this would be but since they did it -- good for them, it's transparent and does not bother client code (your code) using the API.

You can build a small test application and run it under XP to see if API working ok -- it should be when the docs say "Minimum supported client: Windows XP".

The only thing problematic thing under XP would be TLS in https requests -- there is no version 1.2 (or 1.3) support under XP which currently the sites are converging to as a minimum supported version -- but if you keep downloads to plain http there shouldn't be any problems with this.

cheers,
</wqw>