[gopher] Re: New Gopher Wayback Machine Bot
[Top] [All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index] [Thread Index]
On Wed, Oct 12, 2005 at 01:01:32PM -0500, John Goerzen wrote:
> This bot is now running on my laptop and is spidering away. The IP that
> you see connections from will vary depending on where my laptop is at a
> given time ;-)
Speaking of this, do we have any equivolent of robots.txt? Cameron, do
you have some sort of exclude list or anything?
--
John Goerzen
Author, Foundations of Python Network Programming
http://www.amazon.com/exec/obidos/tg/detail/-/1590593715
|
|
