Why don't most web spiders handle gzip?

Friday, August 27. 2004

Why don't most web spiders handle gzip?

Lately the web spider Pompos has been eating up bandwidth on my site. Watching the logs, I noticed that many of the major bots seem to ignore the fact that I am using mod_gzip to compress my data and use less web server bandwidth. So far this month Pompos has eaten 20MB (I'm thinking about blocking it) and Googlebot has downloaded 8MB. If they'd accept compressed data, it would be a lot less.

Thanks to Yahoo Slurp, which does accept gzip-encoded content.

Posted by Mike Boone in Tech at 22:00 | Comments (0) | Trackbacks (0)

Trackbacks

Trackback specific URI for this entry

No Trackbacks

Comments

Display comments as (Linear | Threaded)

No comments

Add Comment

Name
Email
Homepage
In reply to
Comment	Enclosing asterisks marks text as bold (word), underscore are made via _word_. Standard emoticons like :-) and ;-) are converted to images. To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly. Enter the string from the spam-prevention image above:
	Remember Information? Subscribe to this entry