How to create spiders and crawlers

I want to create spiders or crawlers , which helps us to optimise search engine on the web.Pl help me to save my time on developing of these things.

Thanks