[gopher] Re: Gopher "robots.txt" (was Re: New V-2 WAIS database)

[Marco d'Itri <md@xxxxxxxx>]

On Jan 15, Cameron Kaiser <spectre@xxxxxxxxxxxxxxxxxxxx> wrote:

I agree server administrators need something like a /robots.txt file
which can be downloaded once and cached by the robot. The only other
pratical solution is to modify the server and make it *automatically*
add robots information in the relevant menu.
I can't put other files in the mirror directories, and excluding only
the top level entry in the hope nobody will reference anything below
it is not an acceptable solution.

 >I'm still waiting for Marco to tell me those regexes so I can hard-code them.
You should look at the root menu of gopher.bofh.it. There are two
entries for the debian archive and more entries for the NNTP gateway.

All gateway entries execute the gateway program, so I think it would be
a good idea to exclude *all* program-generated files on all hosts.

E.g.:

Path=exec:-T it.fan.marco-ditri:/gonnrp/gonnrp.pl

-- 
ciao,
Marco