[gopher] Gopher Protocol Issue

[David Allen <s2mdalle@xxxxxxxxxxxxx>]

Hello Guys:

I've been hacking on UMN gopherd for a while now, and I've come across
a bit of a sticky issue that I'm hoping somebody else will have some
input on.  I posted this to comp.infosystems.gopher, which seems to
never have any takers on questions related to protocol or the way
gopher guts work.  Hopefully somebody on this list knows something.
:)

I've got a copy of the Gopher RFC, and the Gopher+ specification, but
I'm having a hard time figuring out the behavior of the UMN gopherd.

The reason I'm asking this is because I'm hacking together my own
gopher client just for kicks, while at the same time writing patches
for UMN gopherd through debian.

UMN gopherd when sending a file out to the client removes anything at
the end of the line (carriage returns, line feeds) and then adds
carriage return, linefeed to the end of the line.  What sucks about
this is that if your line is just terminated with a linefeed, ala
UNIX, the file you're sending  actually gets BIGGER because of the
extra carriage returns.  That's a real problem, because if the server
sends a gopher+ header saying how many bytes it's going to send
according to the filesize, and then ADDS carraige returns, the client
has a problem. It can either ignore the server's suggested number of
bytes and read till the socket closes, or it  can read just the number
of bytes specified, and not get the whole thing.  Here's an example
file: (quotes are not part of the file) and \r is carriage return, \n
linefeed. 

"\n
Foo"

Now this file is 4 bytes long.  So the Gopher+ header should be:

"+4"

telling the client to read 4 bytes.  But under the UMN gohperd rules,
this would be translated into 

"\r\n
Foo"

(because it translates all line endings into "\r\n") which means that
unless the client doesn't follow the server's instruction, it will
read this data:

"\r\n
Fo"

Multiply that by 400 lines, and suddenly, you're client is missing the
last 400 bytes of the file, because they were taken up by appended
"\r"'s instead of the data that should have been sent.  NOTE:  I have
tested this with UMN gopher, lynx and netscape, and even though they
talk Gopher+, they seem to ignore the Gopher+ size header, since they
deliver the full file.

So how do you reconcile this?  Is it "correct" behavior for a gopherd
to terminate ALL lines with "\r\n" even if that's not part of the
original data it's sending?  (In the original gopher RFC, it
specifically says that for directory listings, each line should be
terminated with "\r\n".  But for regular files, it doesn't say)  This
would be intensely lousy, since it would mean that the server wouldn't
know how many bytes it was going to send unless it looked through the
files and counted the number of occurances of that pattern before
hand, which isn't very efficient.

Is there a rule to any subset of the gopher protocol on how to resolve
issues  such as when "\r\n.\r\n" occurs in a file you're sending?
(One way I know of is to send -2 as the file size (special code) and
let the client worry about it) but is there any other way?  

Any help would be appreciated.  Pointers to documents on the protocol
other than the gopher RFC and the gopher+ spec would be greatly
appreciated.  For those of you who want a copy of the original gopher
RFC or the gopher+ protocol information, drop me a line, and I'll send
you a text copy.

-- 
David Allen
http://opop.nols.com/
----------------------------------------
"The human brain is a wonderful thing. It starts working the moment you
are born, and never stops until you stand up to speak in public. " 
        -- Sir George Jessel