[gopher] Re: Gopher "robots.txt" (was Re: New V-2 WAIS database)

[Cameron Kaiser <spectre@xxxxxxxxxxxxxxxxxxxx>]

> This proposal would be extremely difficult to implement for gopher
> sites that carry mirrors, especially mirrors of other gopher sites.

Well, it's just a framework for a pragma to contain such information. 
Consider possibly a menu that had, as one of its offerings, such a mirror,
under 1/stayout, and maybe another resource under 1/dontindex also.

You could instead put in something like

iF1/stayout;1/dontindexFerror.hostF909

in the menu that references these selectors.

Mind you, I'd be happy with any approach that works on a per-menu level,
just as long as the bot doesn't have to cache every server's particular
robot policy and can determine the policy for a selector from the menu(s)
that reference that selector. This is just one way I can think of.

-- 
----------------------------- personal page: http://www.armory.com/~spectre/ --
 Cameron Kaiser, Point Loma Nazarene University * ckaiser@xxxxxxxxxxxxxxxxxxxx
-- if (you.canRead(this)) you.canGet(new job(!problem)); -- Seen at JavaOne ---