Complete.Org: Mailing Lists: Archives: gopher: January 2001:
[gopher] Gopher "robots.txt" (was Re: New V-2 WAIS database) 		Home

[gopher] Gopher "robots.txt" (was Re: New V-2 WAIS database)

[Top] [All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index] [Thread Index]
To: gopher@xxxxxxxxxxxx
Subject: [gopher] Gopher "robots.txt" (was Re: New V-2 WAIS database)
From: Cameron Kaiser <spectre@xxxxxxxxxxxxxxxxxxxx>
Date: Sat, 13 Jan 2001 12:37:17 -0800 (PST)
Reply-to: gopher@xxxxxxxxxxxx 
>  >New indexing will resume Sunday evening.

> How can I tell it to skip some parts of my server?
> I don't want it to download the whole 19 GB debian archive, or run some
> thousand times the NNTP gateway.

Good point. I am actually trying to think of a way like the HTTP robots.txt
that can more or less transparently tell V-2 what to stay out of. Suggestions?

-- 
----------------------------- personal page: http://www.armory.com/~spectre/ --
 Cameron Kaiser, Point Loma Nazarene University * ckaiser@xxxxxxxxxxxxxxxxxxxx
-- Die, v.: To stop sinning suddenly. -- Elbert Hubbard -----------------------
[Prev in Thread] Current Thread [Next in Thread]