Help Me! - How to extract website's contents using VB - Help Me!

[maxcoder]

Hi there!
I am a vb programmer. I have a problem in my last project and welcome any solution from you.

I am developing a program to connect to internet and extract some information from some websites. I am able to code extraction using winsock, API or inet control. These websites uses login and cookies and check it in each HTTP request. Websites uses POST method for login so I can't include info in URL as a GET action. I also able to code HTTP POSTs using either Winsock or Inet. So I can POST a login username and password.

Problem:
1.Some websites use session, When I login and request next page it lose the session so request login again! Websites I said about are ASP and JSP technologies based websites. ASPs maybe uses some built-in variables or cookies but I see something like "jsession=" in JSP URLs, However I try to include it in next call but problem continues.

2.Some websites use cookies, (maybe problem comes from cookies). I don't know how I can save, get and post cookies for next call.

3.Some websites also check the browser for cookies and java compatiblity, so they reject login request from Winsock and Inet. I don't know how I can set browser-agent-name in Inet but It's OK for Winsock control.

However I thing that my problem based in sessions or cookies. I need a solution (source code, example project or reference about this problem) that show me how I can LIVE a session or cookie during several visits of a website? How I can import session and login info from one page visit to another? Or my solution is completely worng and I have to change the way?

It's a program that have to login to websites, go to next page after login page (a search page) and fill and submit a form (search query) and exract some information from results page's HTML tags. All my problem is for logon and borwser checkups.

Thank you...
Max

[penagate]

Welcome to the forums
I would say your best bet is to use the WebBrowser control, which will handle things like cookies, etc. Are you using VB6 or VB.NET?

[plenderj]

I'll post my scraper in a minute....

[MalcolmCarmen]

It might make it easier to take a look at what your browser sends to X site using a packet sniffer like Ethereal, or something known as "appsniff" which does quite the same thing on a higher level - with winsockets that is.

Coding this sort of stuff isn't fun

[maxcoder]

I'll post my scraper in a minute....

can you show me download URL? thanks.

[maxcoder]

How I can change agent name of browser? I thing that website find what I am using and say that I need an IE browser.

[maxcoder]

Welcome to the forums
I would say your best bet is to use the WebBrowser control, which will handle things like cookies, etc. Are you using VB6 or VB.NET?

I use VB6. I did not work with webbrowser. can you show me some reference or useful examples? thanks,

[plenderj]

can you show me download URL? thanks.

Here, but it's in .NET:
http://www.vbforums.com/showthread.p...87#post2114287