I don’t mind search engine spiders. Without them it would be hard to find anything on the Internet. But some spiders don’t know how to behave. So far this month 21.36% of the kilobyte traffic on my site is attributed to the Gigabot spider. It’s hit my site from 7 IP addresses for a total of 6808 hits. That’s 296 times a day on average. That’s a bit aggressive. The spider seems to be downloading my site every few minutes. That’s not necessary.
I’ve banned only one other search spider in my life and that would be the Alexa bot. That spider wasn’t causing a problem or being annoying, I’m just opposed to Alexa’s business practices and its use of spyware/adware. I’ll have nothing to do with ripping people off or annoying them with ads. But Gigabot can join the ranks of Alexa. I know nothing about Gigabot or its search site Gigablast. In fact I’d never heard of Gigablast before today. It’s probably just another upstart that’s trying to archive the entire Internet quickly without regards for manners.