HTML download problem

[Jon12345]

I some code that is supposed to download a whole webpage but it seems to only download the first portion of it, maybe 50%. Why is that? Or does MsgBox only display a limited amount of characters?

Dim Data as String
Dim strKeywords  as String
strKeywords="Dog food"
Data = Inet1.OpenURL("http://mystite.com/cgi-bin/script/script.cgi/?skin=jonclean1&length=10&keywords=" & strKeywords & "'")
MsgBox Data

Webpage html:

      <span class="class1">
      <p>  <a href="http://www.5stardog.com/">5 Star Dog</a>
        Online resource for dog owners that includes information on pet selection, training, breeders, and food along with breed descriptions, pictures, forums, and links.<br>
         <a href="http://www.purina.com/intl/default.asp">Nestlé Purina PetCare Company</a>
        International manufacturer and distributor of food, treats and litter for dogs and cats. Portal to specific national sites.<br>
         <a href="http://www.amazon.com/exec/obidos/tg/detail/-/0439110165?v=glance">Amazon.com: Dog Food: Books: Joost Elffers,Saxton Freymann</a>
        ... Dog Food (Hardcover) by Joost Elffers, Saxton Freymann "Good dog ... a list of ingredients included on the back cover, Dog Food is a captivating and humorous feast for the eyes ...<br>
         <a href="http://www.therustydog.com/">The Rusty Dog</a>
        Carries natural foods as well as pet furniture and gifts.<br>
         <a href="http://www.vegetariandogs.com/">Vegetarian Dog Food</a>
        This beautiful and easy to use site is dedicated to vegetarian dog information. Find out where to purchase commercial vegetarian dog food or how you can learn to make your own. ... If you would rather purchase a commercial vegetarian dog food (see resources), you will gain confidence and the ... 90%."3 Overall, there are no major obstacles to vegetarian dog food ...<br>
         <a href="http://www.vetref.net/">VetRef.net</a>
        Resources for owners and veterinarians; includes articles on pet health, whole foods diets, electron microscopy, anti-roaming dog downloads, and links to other sites of veterinary interest.<br>
         <a href="http://en.wikipedia.org/wiki/Dog_food">Wikipedia: Dog food</a>
        Dog food is plant or animal material intended for consumption by dogs or other canids. Special dog foods given as a reward, and not as a staple, are known as dog treats. ... Some people make their own dog food or feed their dogs meals made from ingredients ...<br>
         <a href="http://www.ecopet.com.au/">Ecopet</a>
        Formulating natural food for dogs and cats.<br>
         <a href="http://www.dogchow.com/">Purina Dog Chow</a>
        Get information on nutrition for your best friend and even create a homepage for your dog.<br>
         <a href="http://www.yorkie.org/supplies/pet_food.html">Rhapsody Yorkshire Terriers: Dog Food and Treats</a>
        Offers treat and food recipes targeted for Yorkshire Terriers.<br>
           
        </p>
	  </span>

[Static]

Try

Debug.Print Data

then look in your immediate (debug) window... see if its all there.

[Jon12345]

Ok, tried that and all I got was this...

      <span class="class1">
      <p>  <a href="http://www.5stardog.com/">5 Star Dog</a>
        Online resource for dog owners that includes information on pet selection, training, breeders, and food along with breed descriptions, pictures, forums, and links.<br>
         <a href="http://www.purina.com/intl/default.asp">Nestlé Purina PetCare Company</a>
        International manufacturer and distributor of food, treats and litter for dogs and cats. Portal to specific national sites.<br>
         <a href="http://www.amazon.com/exec/obidos/tg/detail/-/0439110165?v=glance">Amazon.com: Dog Food: Books: Joost Elffers,Saxton Freymann</a>
        ... Dog Food (Hardcover) by Joost Elffers, Saxton Freymann "Good dog ... a list of ingredients included on the back cover, Dog Food is a captivating and humorous feast for the eyes ...<br>
         <a href="http://www.therustydog.com/">The Rusty Dog</a>
        Carries natural foods as well as pet furniture and gifts.<br>
         <a href="http://www.vegetariandogs.com/">Vegetarian Dog Food</a>
        This beautiful and easy to use site is dedicated to vegetarian dog information. Find out where to purchase commercial vegetarian dog food or how you can learn to make your own. ... If you would rather purchase a commerc

[smUX]

inet and webbrowser are very similar, which is why I am posting this. When I downloaded websites with webbrowser and grab the source HTML from it, I would find that the original source would be missing certain elements that should be there. Now someone ELSE will have to help you here, but try setting HEADER data to be sent when you request the URL. Header data would include shizzle like the browser you're using, the system you're on and all-sorts like that (even cookie data if you were using a site that needed it)...without the browser information, some sites would return partial data...hope that helps :-)

Edit: In fact, I had one site that REFUSED flat out to let me send get data (form stuff) without putting valid header data in to tell it referer and all that :-)

[shirazamod]

Debug.Print sometimes doesn't print all data. If you want to see all the data in the string, print it to a file & then open the file manually (use 'Open "C:\debug.txt" For Output As #1...)

[Jon12345]

A bit later in my code it updates a mySQL database anyway so I can check. Is webbrowser better to use than inet?

[Static]

Debug.Print sometimes doesn't print all data. If you want to see all the data in the string, print it to a file & then open the file manually (use 'Open "C:\debug.txt" For Output As #1...)

Debug.print doesnt print all the data??

[smUX]

Is webbrowser better to use than inet?

The problem with webbrowser is that (unless you write in code to stop it) it loads everything on a page...images...background music (if you have it)...everything...Inet just grabs the HTML and that's it, and Inet is a hell of a lot faster in that respect because of it. However, it's a lot more work to get it working if you need specialised stuff like cookies or header info :-)

[Static]

whats the real URL???.. using Webbrowser & HTML object you can pull any part of the page...

[TheBigB]

Dim Data As String
Data = Inet1.OpenURL("http://www.vbforums.com")
While Inet1.StillExecuting = True
    DoEvents
Wend
MsgBox Data
Debug.Print Data

i used vbforums as example (couldn't reach your given page). When the messagebox appears, a big part of the text is missing. if you look at debug now, it has the full code.

[Jon12345]

Since there seems to be a bug in the inet component, I would like to try webbrowser. So I have the following code:

Dim hDoc
    Dim strKeywords as String
    Dim HTMLSource As String
    strKeywords = "dog%20food"
    WebBrowser1.Navigate ("http://mysite.com/cgi-bin/script/script.cgi/?skin=jonclean1&length=10&keywords=" & strKeywords & "'")
    hDoc = WebBrowser1.Document
    HTMLSource = hDoc.body.innerhtml

But when I run this I get this error:

"Object variable or With block variable not set"

What is wrong with the code?

[Static]

Private Sub Form_Load()
 
    strKeywords = "dog%20food"
    WebBrowser1.Navigate "http://mysite.com/cgi-bin/script/script.cgi/?skin=jonclean1&length=10&keywords=" & strKeywords & "'"
 
End Sub
 
 
Private Sub WebBrowser1_DocumentComplete(ByVal pDisp As Object, URL As Variant)
    If (pDisp Is WebBrowser1.Application) Then
        Dim HTMLSource As String
        HTMLSource = WebBrowser.Document.documentElement.innerHTML
        
    End If
End Sub

[Jon12345]

Static, I am a bit confused on how I would use your code. Currently, my code runs if someone clicks on a command button. It then cycles through a list of urls.

[Static]

add the webbrowser control

then paste in the document complete code.

put the navigate to behind your button click

I cant test it because that url is no good

[Jon12345]

To make things simpler, I am just using www.yahoo.com

So, my code is as follows:

Private Sub Command1_Click()
strKeywords = "dog%20food"
WebBrowser1.Navigate "www.yahoo.com"
End Sub
 
Private Sub WebBrowser1_DocumentComplete(ByVal pDisp As Object, URL As Variant)
    If (pDisp Is WebBrowser1.Application) Then
        Dim HTMLSource As String
        HTMLSource = WebBrowser.Document.documentElement.innerHTML
        
    End If
End Sub

When I click the button, it loads Yahoo into the WebBrowser control and then I get the error Object required. Run-time error '424'. The following line of code is then highlighted:

HTMLSource = WebBrowser.Document.documentElement.innerHTML

[Static]

oops WebBrowser

WebBrowser1

[Jon12345]

No error message now. But I can't see how the WebBrowser1_DocumentComplete subroutine is called. From what I can see, you click the button and it just runs the two lines of code.

What am I missing here?

[shirazamod]

When the website has been fully loaded (including images etc) it will call the DocumentComplete event

[Static]

you are also not displaying it anywhere

add

Debug.Print HTMLSource

[Jon12345]

What I am getting confused about is that I want to run a loop with changing urls. This will happen when I click the command button. But it seems that "Private Sub Command1_Click()" subroutine loses control and another subroutine is triggered.

Therefore, I cannot loop through the code for changing urls, correct?

[Static]

Yes.. I would do it like this
(assuming u have URLs listed in Listbox (List1))

Dim lIndex As Integer
Private Sub Command1_Click()
    lIndex = 0
    LoopURLS
End Sub
 
Private Sub WebBrowser1_DocumentComplete(ByVal pDisp As Object, URL As Variant)
    If (pDisp Is WebBrowser1.Application) Then
        Debug.Print WebBrowser.Document.documentElement.innerHTML
        LoopURLS
    End If
End Sub
 
Private Sub LoopURLS()
    Dim tmp As String
    If lIndex >= List1.ListCount Then Exit Sub
    tmp = List1.List(lIndex)
    WebBrowser1.Navigate tmp
    lIndex = lIndex + 1
End Sub

[Jon12345]

Thanks Static! I will give that a whirl.