Capturing data from a website

[Schmalz]

OK, this is the age-old question. Does anyone know how (or can refer me to any source of information) I can use VB6 to extract data from a website to use in a VB6 application? I will use the usual example of extracting stock and share prices from a "live" site, such as the Australian Stock Exchange (which, coincidentally, is exactly what I am trying to do). As an example, say I was creating an application for my desktop that accessed the internet to extract the live share prices from the remote website to display them in my application. Anyway, any advice and/or assistance would be gratefully received.

Schmalz

[RobDog888]

Welcome to the Forums.

Check out my code examples in this thread - http://www.vbforums.com/showthread.php?t=330341

[shirazamod]

Welcome to the forum Schmalz!
You could use the Inet control if you want a simple way to get the html source from a website.
The Inet control is called the 'Internet Transfer Control' and once you've added it to your form all you need to do to get the html for a site is use

Dim HTML as String
HTML = Inet1.OpenURL("http://www.google.com")

[RobDog888]

The only thing with the Inet control is that the site in question probably has security in place to prohibit this type of interaction. so by using a web browser control and parse it out you will get much further easier.

[shirazamod]

@RobDog888, what kind of security prevents the Inet's connection?
As far as I know it acts exactly as a webbrowser except its stripped down

[RobDog888]

Like if your trying to automate a search query on google, it prevents automation somehow.

They dont want you using their search engine behind the scenes without credit given to googlem,etc

[shirazamod]

WOW, I've never encountered that before!
All the websites that I've used the Inet for have worked fine.
I'm going to investigate how they worked out I wasn't using regular IE.

[RobDog888]

They have some major security in place to prevent attacks, automation, and hacking. If you think about it, they would be really foolish if they didnt because then we all could take advantage of their services for free or without acknolowging them.

[shirazamod]

Ok, I did some tests on my webserver and I think the way they validate you is by the User-Agent header, so if you spoof it you will be able to get unrestricted access

[RobDog888]

such an easy workaround. Did it work on google?

[shirazamod]

Just finished working on it.
I managed to fool Google!
Only problem is that I don't know how to change the headers when using .OpenURL so I had to use .Execute instead.

Inet1.Execute "http://www.google.com/search?q=fly", "GET", , "User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

If you know how to edit headers while using .OpenURL please let me know

[RobDog888]

If you use the InternetOpen API it has an argument for the Agent.

hOpen = InternetOpen("Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)", INTERNET_OPEN_TYPE_PRECONFIG, vbNullString, vbNullString, 0)
hConnection = InternetConnect(hOpen, "google.com", INTERNET_DEFAULT_HTTP_PORT, vbNullString, vbNullString, INTERNET_SERVICE_HTTP, IIf(PassiveConnection, INTERNET_FLAG_PASSIVE, 0), 0)
'...
'...

[rory]

and in MSXML

Dim objHTTP As New MSXML.XMLHTTPRequest
    Dim strData As String
    objHTTP.Open "get", "http://www.google.com/search?q=fly", False
    objHTTP.setRequestHeader "Content-Type", "text/html"
    objHTTP.setRequestHeader "User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
    objHTTP.send
    strData = objHTTP.responseText

and winsock ..

Dim strData As String
    strData = "GET [url]http://www.google.com/search?q=fly[/url] HTTP/1.0" & vbNewLine
    strData = strData & "Host: www.google.com" & vbNewLine
    strData = strData & "User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
    strData = strData & vbNewLine & vbNewLine
    Winsock1.SendData strData

[Schmalz]

Thanks for your help so far guys (and gals?). I am still trying to come to grips with this Inet thing. Can you confirm if it works with "live" data such as a website created "on the fly" by php etc.? Also, how can I, say, access a site and fill in a form using a VB6 application?

[rory]

see post 11 .. send the querystring as well ... wont 'fill in a form' but you can send it to the page that processes the form .. i dont know php but dont use .form use .querystring and get for the form instead ..

http://www.myurl.com/default.php?myvar=this

[vivek_master146]

Try www.planet-source-code.!!!!!!!