[gopher] Re: Cicada Incomplete Gopher Census

[Tim Fraser <tfraser@xxxxxxxxxx>]

ck> Actually, you can see the Floodgap census here

Thanks for updating the floodgap directory!  It was browsing through
this directory and cools sites like quux.org (to name just one) that
got me interested in Gopher again.  I think the "new gopher servers
since 1999" directory is an especially interesting feature, since it
highlights new growth.

ck> After the V-2 cleanup this weekend, it has pared itself down to
ck> 255 unique hosts and a database of about 1.8 million selectors.

OK, I found only 154, so I clearly have a bug.  My selector counts
seem very low, too.  I'm not sure it's worth debugging given that the
floodgap index is updating again, but just in case I get bored: my
spider is supposed to follow only selectors with type 1 or 11.  Are
there other directory types that I should follow?

tf> my primitive spider had been automatically banned
ck> It was? I don't remember blocking any IP addresses ...

Perhaps I was mistaken.  After using another machine to read point 4
in the floodgap terms of service (the one about automatically blocking
the netblocks of spiders and robots), I just assumed that was the
cause without any real proof and left it at that.

How does floodgap's Veronica-2 spider limit the load it places on
sites?  Does it check for a robots.txt file, or some similar
mechanism?

- Tim Fraser