Program advice

Hey everyone,

I'm a novice vb programmer looking to create a search program but don't know where to start. I know that 'search program' is a bit ambiguous so perhaps an example might make more sense.

ex) take a website like newegg.com which lists many products but whose specifications page follow a relatively uniform layout. Then enter into the program parameters I want it to find. For example, if I was looking for a cpu, I would enter 4 core, amd, 6mb l3 cache, etc. The program would search all the products on newegg.com and report back cpu's that meet the parameters I set.

I hope that helps explain it a little.

The problem is, being a novice, I have no idea where to start. I have read some tutorials online and a few textbooks about vb.6 but most of them just talk about the gui of vb rather than the code and how to use it to create programs you want. Basically, I'm open to any and all opinions and advice.

Thanks anyone and everyone who can lend a helping hand.

Commercial web sites really, really don't like to be gratuitiously spidered. They're likely to detect heavy activity and blacklist you.

Bandwidth and server resources cost money.

You might get away with doing this gradually a few times, allowing you to cache the info locally for subsequent use.

Thank you for the info dilettante. I did a little research on spiders/crawlers and they seem very much like my objective. Do you have any advice on how to go about creating crawlers based in vb.

Also, my target site is actually local government. Would they react in much the same way as a commercial site? Why do commercial sites not like being spidered. Doesn't it mean more traffic for them?

Sorry for all the noob questions. Thanks again.

Quote:

---

Originally Posted by kupo kupo

Thank you for the info dilettante. I did a little research on spiders/crawlers and they seem very much like my objective. Do you have any advice on how to go about creating crawlers based in vb.

Also, my target site is actually local government. Would they react in much the same way as a commercial site? Why do commercial sites not like being spidered. Doesn't it mean more traffic for them?

Sorry for all the noob questions. Thanks again.

---

Traffic isnt always good and If its goverment then its likly they may react worse ;)

What are you trying to do?

It's to look through a county real estate auction site. The site is arranged in the most unuser friendly way so finding relevant information is very tedious. Is it really that bad an idea?

Quote:

---

Originally Posted by kupo kupo

It's to look through a county real estate auction site. The site is arranged in the most unuser friendly way so finding relevant information is very tedious. Is it really that bad an idea?

---

I'd say so, if the sites that bad write a competing one.

Agreed. Best just to make a competing site. Trying to make a crawler for a specific site is like herding cats. All they need to do is make ONE little change and next thing you know, your crawler is broken, or worse, pulling bad data.

Sites don't like crawlers because they generate unnecessary traffic and that bandwidth costs them money. Granted, crawlers aren't nearly as bad as leeches (sites that link to resources on other sites, like product pictures or files), but they're still universally disliked only for the fact that a human isn't on the other end being exposed to web-based advertising and marketing.