Tonight I blocked the msnbot in my robots.txt file. I’ve noticed hits from that bot in my logfile occasionally, but recently it’s gone nuts. So far this month it has eaten 41 MB of bandwidth and hit 18000+ pages. By comparison, Google has only hit 4 MB (~1300 pages).

I don’t have 18000 different pages of content on this site, so msnbot is hitting some pages over and over again. From what I’ve read, the msnbot results aren’t even being used yet in MSN’s search. I’m not going to waste my bandwidth on their experiment.

I’ll reconsider if they actually start using the msnbot data for search results.