[gopher] Re: Veronica-2 and robot exclusion
[Top] [All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index] [Thread Index]
> In any case, itemtype=0 sel=robots.txt is what the crawler will ask for.
> (Note no leading slash, since Gopher is supposed to be independent of the
> host filesystem.)
(i.e., the request will be "robots.txt\r\n", just to make it mega-obvious.)
--
---------------------------------- personal: http://www.armory.com/~spectre/ --
Cameron Kaiser, Floodgap Systems Ltd * So. Calif., USA * ckaiser@xxxxxxxxxxxx
-- Birth, n.: The first and direst of all disasters. -- Ambrose Bierce --------
|
|
