Articles Tagged googlebot

Googlebot's going in circles

Posted on June 4th, 2008 18:35:31 by landon
Comments

Today I noticed that irssi was displaying a spike in network activity randomly. Wondering who it could be that was hammering my little section of the internet, I checked the lighttpd logs only to find out that the Googlebot had found my test spider trap! A bit more digging and it looks like it has been accessed over 35,000 times by Googlebot alone and is featured on over 40 pages of Google search results for listings from my site.

I guess it's about that time to set up a robots.txt :)

(The spider trap is located at http://lfowles.org/loop.py ... don't worry, it's safe to a normal browser ;) )