Complete.Org: Mailing Lists: Archives: gopher: October 2005:
[gopher] Re: New Gopher Wayback Machine Bot
Home

[gopher] Re: New Gopher Wayback Machine Bot

[Top] [All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index] [Thread Index]
To: gopher@xxxxxxxxxxxx
Subject: [gopher] Re: New Gopher Wayback Machine Bot
From: John Goerzen <jgoerzen@xxxxxxxxxxxx>
Date: Wed, 12 Oct 2005 13:51:41 -0500
Reply-to: gopher@xxxxxxxxxxxx

On Wed, Oct 12, 2005 at 01:01:32PM -0500, John Goerzen wrote:
> This bot is now running on my laptop and is spidering away.  The IP that
> you see connections from will vary depending on where my laptop is at a
> given time ;-)

Speaking of this, do we have any equivolent of robots.txt?  Cameron, do
you have some sort of exclude list or anything?

-- 
John Goerzen
Author, Foundations of Python Network Programming
http://www.amazon.com/exec/obidos/tg/detail/-/1590593715



[Prev in Thread] Current Thread [Next in Thread]