[RESOLVED] Download a web page into a text file

[Karl-Erik]

Hello !

In an application I declare:

Public Declare Function URLDownloadToFile Lib "urlmon" _
    Alias "URLDownloadToFileA" (ByVal pCaller As Long, _
    ByVal szURL As String, ByVal szFileName As String, _
    ByVal dwReserved As Long, ByVal lpfnCB As Long) As Long

use the function:

Public Function DownloadFile(URL As String, LocalFilename As String) As Boolean
    Dim lngRetVal As Long
    lngRetVal = URLDownloadToFile(0, URL, LocalFilename, 0, 0)
    If lngRetVal = 0 Then DownloadFile = True
End Function

and:

URLDownloadToFile 0, "https://some_url.com", "TextFile.txt", 0, 0
.....
.....

to download a, not so large, web page into a text file.

When I start the application the first time often nothing happens. (Nothing is downloaded !)

But if I first manually, with a web browser, open the same website, close it again and thereafter re-run the application, everything works as it should and then every attempt !

At least that's the case in design mode....

What can I add to the application to make it download the web page at the first start, without having to preopen the website ?

Best regards and thanks in advance.

/Kalle

[fafalone]

Try the BINDF_GETNEWESTVERSION flag.

If that doesn't work, the function I usually use for simple http text grabs is this:

Public Declare Function InternetCloseHandle Lib "wininet.dll" (ByVal hInet As Long) As Integer

Public Declare Function InternetOpen Lib "wininet.dll" Alias "InternetOpenA" _
                                                                            (ByVal lpszAgent As String, _
                                                                            ByVal dwAccessType As Long, _
                                                                            ByVal lpszProxyName As String, _
                                                                            ByVal lpszProxyBypass As String, _
                                                                            ByVal dwFlags As Long) As Long

Public Declare Function InternetOpenUrl Lib "wininet.dll" Alias "InternetOpenUrlA" (ByVal hInternetSession As Long, _
                                                                                    ByVal lpszUrl As String, _
                                                                                    ByVal lpszHeaders As String, _
                                                                                    ByVal dwHeadersLength As Long, _
                                                                                    ByVal dwFlags As Long, _
                                                                                    ByVal dwContext As Long) As Long

Public Declare Function InternetReadFile Lib "wininet.dll" (ByVal hFile As Long, _
                                                            ByVal lpBuffer As String, _
                                                            ByVal dwNumberOfBytesToRead As Long, _
                                                            lNumberOfBytesRead As Long) As Integer


Private Sub CopyURLToFile(ByVal URL As String, ByVal sFilename As String)
    Dim hInternetSession As Long
    Dim hUrl As Long
    Dim hFile As Integer
    Dim hr As Boolean
    Dim lBytes As Long
    Dim sBuffer As String
    Dim fileIsOpen As Boolean

    On Error GoTo e0

    If (Len(URL) = 0) Or (Len(sFilename) = 0) Then Exit Sub

    hInternetSession = InternetOpen(App.EXEName, INTERNET_OPEN_TYPE_PRECONFIG, "", "", 0)
    If hInternetSession = 0 Then Err.Raise vbObjectError + 1000, , _
        "An error occurred calling InternetOpen function"

    hUrl = InternetOpenUrl(hInternetSession, URL, vbNullString, 0, _
        INTERNET_FLAG_EXISTING_CONNECT, 0)
    If hUrl = 0 Then Err.Raise vbObjectError + 1000, , _
        "An error occurred calling InternetOpenUrl function"

    On Error Resume Next
    Kill sFilename

    On Error GoTo e0
    
    hFile = FreeFile
    Open sFilename For Binary As hFile
    fileIsOpen = True

    sBuffer = Space(4096)
    
    Do
        hr = InternetReadFile(hUrl, sBuffer, Len(sBuffer), lBytes)
        If lBytes = 0 Or Not hr Then Exit Do
        Put #hFile, , Left$(sBuffer, lBytes)
    Loop

e0:
    Close #hFile
    If hUrl Then InternetCloseHandle hUrl
    If hInternetSession Then InternetCloseHandle hInternetSession
    If Err Then
        Debug.Print "CopyURLToFile.Error->" & Err.Description & " (" & Err.Number & ")"
    End If
End Sub

[Karl-Erik]

Thanks fafalone !

I've tested different variations of your suggestion !

The problem seems to be 'InternetOpenUrl' which mostly results in = 0, but for unknown reasons suddenly shows an eight digit numbers, which means that contact has been established with the URL and the application can continue.
Repeated stops and runs will remain positive, with differentiated eight-digit number as long as I don't unload my program.

But if I close down the program ( Unload it ! Still in design mode) and then load it up and restart again, the result will be back to = 0 - until it suddenly and magically become high again after a number of attempts ...

I can't see why this is happening, because I don't change any parameters ! My wish is to find this line work every time and from start !

Maybe it's different and depending of what URL I try to connect ? ? !
(It's a big difference if I try on "http://wwwmicrosoft.com" ! There it works from the very start and keep on...)

My attempts are to localize different IP addresses from 'https://www.db-ip.com/IP"


Have some explanations ? ?
Why it still remains high during the same session ?
Why it from start remains zero so many attempts....

/Kalle

[westconn1]

try deleting the cache entry for the url first

[Karl-Erik]

try deleting the cache entry for the url first

Yes ! I sure will..

But how ? Programmatically or....

How can it affect solution of the problem ?

[MarkT]

I see in your sample code the URL is https. Do you need to log into the site before accessing content? If so this may be a problem with your approach.

[Karl-Erik]

I see in your sample code the URL is https. Do you need to log into the site before accessing content? If so this may be a problem with your approach.

Access to the website works with and without "s", after "http". But the "s" is automatically added at the opening !

Just to try (again !);

I opened the site with my web browser (Mozilla Firefox).
Then I started to use my program once again to open the same site, and failed - until I closed the web browser. Then the program worked !
Opened the web browser again at the site, and the program keept on to work.
Closed the VB design program and the browser

Started VB design program. Result = 0. Opened the site again with web browser. Result = 0, even with repeated close web browser....

Wrote those last lines... Tried again and the reult was 8 digits and high.....

There is no pattern to lean on....

/Kalle

[LaVolpe]

What about accessing very common sites, i.e., google, cnn, foxnews, etc? Same issues?

If not, maybe that site is restricting how fast you can access it between hits or how many simultaneous connections to the site from the same IP? Just a guess. Also that InternetOpenURL function has several optional flags

Ensure you are closing all handles created by those APIs before you exit your routine.

[Karl-Erik]

What about accessing very common sites, i.e., google, cnn, foxnews, etc? Same issues?

If not, maybe that site is restricting how fast you can access it between hits or how many simultaneous connections to the site from the same IP? Just a guess. Also that InternetOpenURL function has several optional flags

Ensure you are closing all handles created by those APIs before you exit your routine.

As I wrote earlier: "It's a big difference if I try on "http://www.microsoft.com" ! There it works from the very start and keep on...

Also, when in "high 8 digits mode" I can access how many times I want on this site, as well as miss if not in the mode...

I'm still in design mode in VB, so it's actually not a finished program. Therefor I've put a breakpoint just after hitting 'InternetOpenURL', so that the program halt.
Then I end it after check the result of 'InternetOpenURL'. So I think all handles are closed then....

Am I wrong some how ??

I will take a closer look at the flags you mention. Any special ? ?

[LaVolpe]

Do not recommend ending it after that check. You can add a couple lines, for debugging, that goes to the clean-up routine (in blue below). Modifications below are using the sample code in post #2 above.

...
   hUrl = InternetOpenUrl(hInternetSession, URL, vbNullString, 0, _
        INTERNET_FLAG_EXISTING_CONNECT, 0)

    If True = True Then   ' debugging only. When not needed, either remove IF/EndIF code or change 1 True to False
       Stop
        GoTo e0
    End If

...

e0:
    Close #hFile
    If hUrl Then InternetCloseHandle hUrl
    If hInternetSession Then InternetCloseHandle hInternetSession
    If Err Then
        Debug.Print "CopyURLToFile.Error->" & Err.Description & " (" & Err.Number & ")"
    End If
...

As for the flags, maybe this one: INTERNET_FLAG_NO_CACHE_WRITE. But first see if not closing your handles was the problem

Do recommend saving project & closing all instances of VB first. Then start fresh. If you had any open handles, they should be released once VB is completely unloaded.

[westconn1]

But how ? Programmatically or....

deleteurlcacheentry API

[Bonnie West]

In another recent thread, I posted a variation of the following. This might also work for you:

Public Sub SaveWebPageToFile(ByRef URL As String, ByRef FileName As String, Optional ByRef Charset As String = "utf-8")
    Const adSaveCreateOverWrite = 2&
    Dim oHttpReq As Object

    Set oHttpReq = CreateObject("WinHttp.WinHttpRequest.5.1")
    oHttpReq.Open "GET", URL
    oHttpReq.Send

    With CreateObject("ADODB.Stream")
        .Open
        .Charset = Charset
        .WriteText oHttpReq.ResponseText
        .SaveToFile FileName, adSaveCreateOverWrite
        .Close
    End With
End Sub

Private Sub Main()
    SaveWebPageToFile "https://www.db-ip.com/", "index.htm"
End Sub

[Karl-Erik]

Many thanks, Bonnie West !

I'm now a happy man !
With your code, it worked perfectly from the start and repeatedly with different IP numbers !
Great ! ! You're worth a golden star !

But also thanks to all the others for your efforts : fafalone, Westconn1, MarkT and LaVolpe !
You gave me lots of inspiration (and some transpiration !) !

/ Karl-Erik in Sweden