[gopher] Re: Gopher "robots.txt" (was Re: New V-2 WAIS database)

[David Allen <s2mdalle@xxxxxxxxxxxxx>]

On Sat, Jan 13, 2001 at 04:56:21PM -0800, Cameron Kaiser wrote:
> I'd like to see an approach on a per-directory (menu) basis, maybe a selector
> like this in menus not to be indexed:
> 
> iFFerror.hostF909
> 
> F, of course, is Gopherese for <TAB>. It's a null display string of item
> type i, so normal clients just display a blank line. You can encode any
> sort of pragma this way; I just encoded it in the port number for poor sods
> who use IE so that clicking on the link doesn't do anything useful. 909 as
> the port number is the pragma for the robot to not index anything in this
> directory/menu, so it leaves it alone.
> 
> Such a pragma, like I said, would be wide open for other applications and
> best of all is very simple to implement on both robot (my) side and servers.

That's not a bad idea...I'd prefer personally to put whatever
information that this null statement has in the gopher+ fields to
avoid overloading the meaning of the other fields (type, name, host,
port).  So maybe something like this to disallow the debian
subdirectory:

iFFerror.hostF909FDisallow: selector_for_debian_directory

I like the idea of using these null statements as pragmas of a sort,
so it may be better to use a keyword like Disallow or deny or
something like that, so that multiple pragmas can be used.  Just as a
random suggestion, you could use this to even define new types
previously unknown like:
iFFerror.hostF909FType "~" mimetype
to allow for the use of the ~ type to represent things of <mimetype>

For display purposes, just as a convention, it may be best to also put
these at the bottom of documents rather than the top or middle, so if it is
rendered as a blank line (older clients and clients that don't know
about these statements), it won't vertically indent the actual
content of the directory. 

-- 
David Allen
http://opop.nols.com/