Distributed Computing Idea

Whenever somebody asks me if there's like a "map" to the internet I always rattle off the same bull**** about it being too big and blah blah blah.

But !
If you were to write an app that, given a set number of starting links, would scour through them finding links and following those links and then those links (ad nauseum), you would end up with a lot of links.
Not all though ; because the net is so big.

If then you were to put that app onto hundreds of pcs around the world, and have them all sending back their results every week or so, then you'd end up with a pretty big map pretty quick :)

A variant of the application would be one which take a dictionary list (ie. just a list of words), and then supplies each word to a number of different search engines. That would producs millions of links.
That app itself could then following those links, or allow the clients to download those links for further processing.


A tall task I'll admit, but easily within the boundaries of sense !