Complete.Org: Mailing Lists: Archives: gopher: January 2001:
[gopher] Re: Gopher "robots.txt"
Home

[gopher] Re: Gopher "robots.txt"

[Top] [All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index] [Thread Index]
To: gopher@xxxxxxxxxxxx
Subject: [gopher] Re: Gopher "robots.txt"
From: Cameron Kaiser <spectre@xxxxxxxxxxxxxxxxxxxx>
Date: Sun, 14 Jan 2001 21:32:45 -0800 (PST)
Reply-to: gopher@xxxxxxxxxxxx

> It wouldn't be that much of a performance hit.  Rather than fetching the
> directory with a normal gopher request, then fetching the attributes for
> each entry of the directory, the robot could make a request something
> like this:
> 
>       1/directoryF$+VIEWS+ABSTRACT+ROBOTS
> 
> which would fetch only the VIEWS, ABSTRACT, and ROBOTS attributes of
> every item in the directory (in addition to the INFO attribute, which
> contains the usual Gopher directory entry).  A fully Gopher+ supporting
> robot would need to get the VIEWS and ABSTRACT to build a complete index
> anyway.

This is a excellent idea, but I don't think we want to require all robots or
robot-like things to be gopher+ compliant. It would be nice, but I don't
think it should be mandatory.

Also, this doesn't solve the problem adequately for those *servers* which are
not gopher+ compliant, or certain subsets of indexed search servers that can't
or don't make gopher+ compliant responses (I can think of several immediately).

-- 
----------------------------- personal page: http://www.armory.com/~spectre/ --
 Cameron Kaiser, Point Loma Nazarene University * ckaiser@xxxxxxxxxxxxxxxxxxxx
-- Adore, v.: To venerate expectantly. -- Ambrose Bierce ----------------------



[Prev in Thread] Current Thread [Next in Thread]