[gopher] Re: Gopher "robots.txt" (was Re: New V-2 WAIS database)
[Top] [All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index] [Thread Index]
On Sat, Jan 13, 2001 at 04:28:12PM -0800, Cameron Kaiser wrote:
>
> >>Good point. I am actually trying to think of a way like the HTTP robots.txt
> >>that can more or less transparently tell V-2 what to stay out of.
> >>Suggestions?
>
> > I don't know well gopher yet, but you'd better find one before starting
> > to index my site, or your database will be filled with crap.
>
> I'm sure Dave and John will have some ideas, but for now just mail me offlist
> with some regexes that are off-limits and I'll hardcode them for the present.
> I appreciate it :-)
Personally, I don't see any reason not to just lift the robots.txt
verbatim and add it. Minus the User-Agent part, which gopher doesn't
really support. (Or, it could always be '*' in case web agents for
some reason ended up reading the file) So maybe we could do something
like this:
Disallow: some_directory
Disallow: another_directory
Questions on these items though:
1.) Should "some_directory" be a selector string, or the portion of the URL
after the host? I.e. on my system, I have a selector,
"1/Python Stuff". Should it be listed as that, or as the portion of
the URL after the host, which would be: "/11/Python%20Stuff"?
2.) Should gopher servers hide files by the name of robot.txt from the
view of the client? (i.e. should it be possible for a human user to
come into a directory and see a robots.txt entry, or should it be
automagically hidden?)
--
David Allen
http://opop.nols.com/
----------------------------------------
Atlee is a very modest man. And with reason.
- Winston Churchill
- [gopher] New V-2 WAIS database, Cameron Kaiser, 2001/01/13
- [gopher] Re: New V-2 WAIS database, Marco d'Itri, 2001/01/13
- [gopher] Gopher "robots.txt" (was Re: New V-2 WAIS database), Cameron Kaiser, 2001/01/13
- [gopher] Re: Gopher "robots.txt" (was Re: New V-2 WAIS database), Marco d'Itri, 2001/01/13
- [gopher] Re: Gopher "robots.txt" (was Re: New V-2 WAIS database), Cameron Kaiser, 2001/01/13
- [gopher] Re: Gopher "robots.txt" (was Re: New V-2 WAIS database),
David Allen <=
- [gopher] Re: Gopher "robots.txt" (was Re: New V-2 WAIS database), Cameron Kaiser, 2001/01/13
- [gopher] Re: Gopher "robots.txt" (was Re: New V-2 WAIS database), David Allen, 2001/01/14
- [gopher] Re: Gopher "robots.txt", emanuel at heatdeath organisation, 2001/01/14
- [gopher] Re: Gopher "robots.txt", David Allen, 2001/01/14
- [gopher] Re: Gopher "robots.txt", emanuel at heatdeath organisation, 2001/01/14
- [gopher] Re: Gopher "robots.txt", Cameron Kaiser, 2001/01/14
- [gopher] Re: Gopher "robots.txt", David Allen, 2001/01/14
- [gopher] Re: Gopher "robots.txt", Cameron Kaiser, 2001/01/15
- [gopher] Re: Gopher "robots.txt", David Allen, 2001/01/15
- [gopher] Re: Gopher "robots.txt", emanuel at heatdeath organisation, 2001/01/15
|
|