Problem with InternetReadFile

[Ellis Dee]

I've been using this library function to download pages from the internet, and it's worked great for months.

Private Declare Function InternetCloseHandle Lib "wininet.dll" (ByVal hInet As Long) As Integer
Private Declare Function InternetOpenUrl Lib "wininet.dll" Alias "InternetOpenUrlA" (ByVal hInternetSession As Long, ByVal lpszUrl As String, ByVal lpszHeaders As String, ByVal dwHeadersLength As Long, ByVal dwFlags As Long, ByVal dwContext As Long) As Long
Private Declare Function InternetOpen Lib "wininet.dll" Alias "InternetOpenA" (ByVal sAgent As String, ByVal lAccessType As Long, ByVal sProxyName As String, ByVal sProxyBypass As String, ByVal lFlags As Long) As Long
Private Declare Function InternetReadFile Lib "wininet.dll" (ByVal hFile As Long, ByVal sBuffer As String, ByVal lNumBytesToRead As Long, lNumberOfBytesRead As Long) As Integer

Public Function DownloadURL(pstrURL As String) As String
    Const INTERNET_OPEN_TYPE_PRECONFIG = 0
    Const INTERNET_FLAG_RELOAD = &H80000000
    Dim strBuffer As String * 4096
    Dim strReturn As String
    Dim lngLen As Long
    Dim lngSession As Long
    Dim lngFile As Long
    Dim blnSuccess As Boolean
    
    lngSession = InternetOpen("GSInternet", INTERNET_OPEN_TYPE_PRECONFIG, vbNullString, vbNullString, 0)
    lngFile = InternetOpenUrl(lngSession, pstrURL, vbNullString, 0, INTERNET_FLAG_RELOAD, 0)
    If lngFile Then
        Do
            blnSuccess = InternetReadFile(lngFile, strBuffer, Len(strBuffer), lngLen)
            If lngLen Then strReturn = strReturn & Left$(strBuffer, lngLen)
        Loop While blnSuccess And lngLen > 0
        InternetCloseHandle (lngFile)
        DownloadURL = strReturn
    End If
End Function

Now it doesn't work anymore. The problem is that the server seems to have a delay in it. When browsing manually, much of the page gets drawn, then there's a second or two delay, then the rest of the page gets drawn.

That second or two delay seems to be killing the above utility function. What happens now is the function only returns the half of the page that gets sent before the second or two delay.

Does anyone know how or why this would happen, and any way to fix it?

[Ellis Dee]

Sadly, adding a delay and running the loop again does nothing.

I'm basically unable to download the entire contents of a page. This program doesn't do anything except download that page and present the data in a more useful manner to the user, meaning this is a complete dealbreaker.

Any assistance would be appreciated.

[Ellis Dee]

Switching out my utility function with the OpenURL method of an inet control yields the same results.

It would appear that the page is waiting for some kind of feedback from the browser to tell it to send the rest of the page. It didn't use to have this feature; my program worked great up until yesterday.

Is there any way I can watch the communication with my browser to find out what that feedback is?

[vb5prgrmr]

Maybe you can watch with a network sniffing tool...

Good Luck

[Ellis Dee]

Using a webbrowser control I get the full page.

Is there any way to mimic the behavior of the webbrowser.Navigate method without using any components or references?

The problem is that I can't ask the users of this (guild mates in an online game) to run any kind of installation. All my utilities are therefore native VB. To that end, all advanced GUI (statusbar, progressmeter, etc...) is implemented as uderdrawn controls inside pictureboxes. Even more hardcore is that I use udt arrays in lieu of ADO to keep it installation-free.

Any suggestions are welcome.

[DigiRev]

Ellis, I had the same problem which is why I stopped using that for downloading web pages, and no longer recommend it... I could be mistaken, but I think they don't work on Windows 98 either?

You can download files using a UserControl's AsyncRead, without any references or controls.

There is one in the CodeBank, and here's one I just found, not sure how good it is, but should show how it's done:

http://www.pscode.com/vb/scripts/Sho...36935&lngWId=1

Also, there is a downloader class made in VB that uses the Winsock API functions, I don't think the link is working right now, but maybe you can try later:
http://allapi.mentalis.org/vbexample...ategory=SOURCE

If neither of those work, I would probably just write one from scratch to download files using the Winsock API.

Edit: Here is an example from AllAPI for downloading a file via Winsock API:
http://allapi.mentalis.org/apilist/B...D9CC47087.html

[Ellis Dee]

Hmmm. I tried the project from AllAPI, but after plugging in the url I want to get, it retrieves even less than the InternetReadFile method gets.

The full download is around 750k, InternetReadFile grabs around 150k, and this initial attempt at WinSock nabs only 6k.

The problem with the WinSock project may be that it's expecting to download a file while I'm just trying to open a web page. The other example uses subclassing, which I try to avoid unless it's absolutely necessary. Is it?

[DigiRev]

Hm, try this:

http://www.vbforums.com/showthread.php?t=532597

[Ellis Dee]

Hm, try this:

http://www.vbforums.com/showthread.php?t=532597

That's way cool, but it has the same issue as InternetReadFile and the Inet control.

The frustrating part is that the status codes show that there is 750k to download but it only manages to get 65k.

[Milk]

I don't know if it's any help but if I pop that url in my browser I get a login page of just under 6KB (5779) which is of a similar size to your initial Winsock attempt. Did you check the data or just the size? Could it be that Winsock was working properly after all, just not logging on?

[Ellis Dee]

That's probably it, but I was logged in at the time. I'd really rather not log in programmatically, so if WinSock will only work if it can log in itself, I'll look for an alternative.