I have a load balancer using apache: http://httpd.apache.org/docs/2.2/mod/mod%5Fproxy%5Fbalancer.html
The problem is our bandwidth. We're trying to get more, but the ISP has to run new lines and keeps putting us off, so I'd like to throttle down the spiders to conserve bandwidth until we can get more. I tried mod cband, but it won't work on load balanced virtual hosts.
Is there any apache modules that can throttle traffic on load balanced virtual hosts?
Best Answer
Not an apache module, but you can try using the "Crawl delay" directive in robots.txt to slow down well behaved spiders.
http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive