[gopher] Re: Gopher "robots.txt"

[David Allen <s2mdalle@xxxxxxxxxxxxx>]

On Sun, Jan 14, 2001 at 10:03:56PM -0800, Cameron Kaiser wrote:
> > Hm.  Well, I can agree with this, but at the same time, what is to
> > become of gopher+?  You can either merge it with the original gopher
> > and say "this is what gopher is, and we'll write software
> > accordingly", or you can throw out gopher+ and stick only with gopher,
> > but having gopher+ sitting around as something that might be supported
> > and might not be seems like a pain.  Gopher+ has some decent
> > abilities, and I don't see why you shouldn't use them in this
> > situation.  So maybe we should be asking if gopher+ is something that
> > should continue to be sometimes-supported, or if it should be taken in
> > or thrown out.
> 
> Let's stick with the smaller issue first :-) I think Emanuel's idea is
> great for encoding that information in Gopher+ attributes; it just seems
> to leave non-G+ servers out and those aren't just going to disappear.
> If people want an official stopgap-only solution, fine -- I just need to
> know what the bot should support that will work for the widest range of
> servers *now*, since I need to get it running again soon.

OK, let's recap and check what we've got on the table.

1.) Put "pragmas" inside extra fields of gopher lines, i.e.
iFFerror.hostF960F<some pragma here possibly telling robot something>
2.) Create extra attribute entries for robots, in gopher+ attributes,
ala VIEWS and ABSTRACT
3.) Add a robots.txt file to the server.

Please respond if I'm missing something...there are lots of possible
variations on the theme of (1) though.

1 and 2 both seem to rely on gopher+ in that they use either extra
fields past what gopher uses, or the attribute lists.  3 pretty much
sucks because of issues you raised earlier with caching multiple sites
worth of data.

About backward support for non-gopher+ compliant servers - since we're
talking about a new feature, we have two choices if we want to make it
useful.  We can say either "This is the way it will be, and you need
to change your software to make it so" or we can just hack the way
it's currently done, i.e. overload the meaning of something that
already has a specific meaning under gopher.  (To avoid using anything
related to gopher+)  The first is undesireable because it's forcing
people to change, and the 2nd is undesireable because it's ugly, and
things like that often cause inconsistencies between clients.

It seems that either way there is going to be some discomfort.  That's
why I'm suggesting it be done in a clean way, working it into existing
gopher+.  I understand what you're saying about how it's a shame to
require robots or servers to be even minimally gopher+ compliant, but
a change has to be made somewhere to staple this onto gopher.  That's
why I was bringing up the larger issue of what to do with gopher+.

...But I'm not convinced that these 3 are the only options.  What else
is possible?

-- 
David Allen
http://opop.nols.com/
----------------------------------------
"I told them kids to keep their arms inside the ride.
Damnedest thing I ever saw."