[gopher] Re: Veronica-2 and robot exclusion

[John Goerzen <jgoerzen@xxxxxxxxxxxx>]

I'm on a train just now and away from the net (horrors!) so I can't check
your URLs, but a quick question:

Is this the same robots.txt format that is used for the Web?  If so, could
one potentially use a single robots.txt file in the root of a site that is
served up both as Gopher and over HTTP?  (As quux.org is, for instance)
That would be a very nice feature.

On Wed, Jun 18, 2003 at 06:24:18AM -0700, Cameron Kaiser wrote:
> Also, I was thinking of making the file ".robots.txt" since many Unix
> gophers don't serve dot-files, although there are a growing number of
> Windows-hosted gophers and I don't know if it will break these (I don't
> do x86 myself).

AFAIK, all Unix gophers *will* serve dot files, many just won't include them
in directory listings.  (Which I think is what you meant anyway)