Making my own search engine 'spider'?

Hi

I think that this question will be very difficult to pursue but the benefits of it would be immense. I'd like to somehow put a few websites into my database and then activate a search engine spider to look for certain links and add them to my library available for searching. At the moment I am adding links by hand. These are relevant links to my site, not stealing content or anything like that. However, I have tons of them! It would be so much better if I could somehow search the sites, get the relevant information and bookmark it to my site automatically.

How complex is this? How do I get started? Thanks

If you have your own search engine, how nice. It is the best way to catch people on your site. If your search engine quick, as well as correct(I mean the similarity between user typed words and engine results), you are in right target.

In the university I've learned a subject on "Data mining and machine learning." Basically it is about developing algorithms on search engine. Specially on mail servers. It explains how you should store data links on database and many more.

So my final conclusion is that, as much as possible try to develop a strong algorithm. Take your time and read at least one book.

thanks for the advice! i see its a very complex subject and ill do my best to learn more!

Actually it is not much difficult, if you with best in AI. Even you don't know anything on AI, just think the way you can handle data here and there. Start from a simple step and move to advanced level.

Google Custom Search Engine

Maybe this is what you're looking for?

Ya, that is interesting for me. Actually nothing to do much there.