|
-
Oct 6th, 2010, 04:00 AM
#8
Re: Dealing with Screen Scrappers and Bots
jakkjakk,
Gary has hit the nail on the head so to speak. If you put information in the public domain then it's open to anyone/thing to consume and you almost can't change that.
If you google something like "stop screen scraping" you'll see this problems is mainly resolved by analyzing log files for potential offenders and creating a blacklist on requesting useragents,IP's etc.
Stopping/routing blacklist requests requires examining every request and looking up your blacklist thus must be efficient. I've not tried to do it but on shared hosting maybe a http module could be a good approach. OR just make your pages hard for scrapers to read but easy for search engine bots yer right.
Let us know if you come up with a good compromise. I sympathize with you because I've been ask to stop scraping of data and I couldn't stop myself doing it without comprimising SEO or app scalability.
The problem with computers is their nature is pure logic. Just once I'd like my computer to do something deluded. 
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts
-
Forum Rules
|
Click Here to Expand Forum to Full Width
|