[gopher] Re: Gopher "robots.txt" (was Re: New V-2 WAIS database)

[David Allen <s2mdalle@xxxxxxxxxxxxx>]

On Sat, Jan 13, 2001 at 04:28:12PM -0800, Cameron Kaiser wrote:
> 
> >>Good point. I am actually trying to think of a way like the HTTP robots.txt
> >>that can more or less transparently tell V-2 what to stay out of. 
> >>Suggestions?
> 
> > I don't know well gopher yet, but you'd better find one before starting
> > to index my site, or your database will be filled with crap.
> 
> I'm sure Dave and John will have some ideas, but for now just mail me offlist
> with some regexes that are off-limits and I'll hardcode them for the present.
> I appreciate it :-)

Personally, I don't see any reason not to just lift the robots.txt
verbatim and add it.  Minus the User-Agent part, which gopher doesn't
really support.  (Or, it could always be '*' in case web agents for
some reason ended up reading the file) So maybe we could do something
like this: 

Disallow: some_directory
Disallow: another_directory

Questions on these items though:

1.) Should "some_directory" be a selector string, or the portion of the URL
  after the host?  I.e. on my system, I have a selector, 
"1/Python Stuff".  Should it be listed as that, or as the portion of
  the URL after the host, which would be: "/11/Python%20Stuff"?

2.) Should gopher servers hide files by the name of robot.txt from the
  view of the client?  (i.e. should it be possible for a human user to
  come into a directory and see a robots.txt entry, or should it be
  automagically hidden?)

-- 
David Allen
http://opop.nols.com/
----------------------------------------
Atlee is a very modest man. And with reason. 
- Winston Churchill