[Freeciv-Dev] Re: network send/growing buffers

[Vasco Alexandre Da Silva Costa <vasc@xxxxxxxxxxxxxx>]

On Sun, 29 Jul 2001, Justin Moore wrote:

First of all sorry for the long reply, but i've replied to all your
comments and explain the rationale of the current code. :-)

>    In a particularly large game (200x100 map, 10 AI) and fog of war
> disabled, I kept getting my connection cut due to a large send buffer.
> Digging through common/connection.c, I see some potential problems in the
> connection flushing.  Namely that the return value of write_socket_data is
> never checked (lines 240 and 250).  Shouldn't this value propagate back up
> to send_connection_data, especially since the timeout value in the select

It makes no difference.

> call is set to 0?  What was happening was the large updates would bog down
> the connection, causing the buffer to grow.  If the server CPU is much

Yes this happens. In a past version i made, the buffer didn't have a
maximum size limit, but when i saw the size it was growing into i decided
it was better to cut the client.

> faster than the connection between it and a client, this could be
> problematic for larger games.  I'd much rather have my data take a while

Actually this happens alot because the client is very slow at displaying
updates. The server can send tiles much faster than the client can process
them and draw for e.g.

> to get there than to be completely disconnected.

You would but some people wouldn't. Sometimes people with very bad
connections get lag over 10 seconds. Sometimes this gets to the ridicule
of having several minutes of lag! In my opinion, if the connection is that
bad, the player isn't following the game at all so its better for him to
reconnect. If we stop the game for everyone else we are penalizing
everyone for one person with a bad connection.

> Two potential work-arounds:
> 1. A small (non-zero) timeout value in select to let the buffer drain, OR

I tried that before. Then i got complains that the server would grind to a
halt for 15 minutes and more. It was then i realized that function is
called several hundreds of times sometimes just to flush the buffer of one
player! Even if i put a timeout of 1 second people would wait 5 minutes or
more because of someone with a lousy connection. Players would wait and
wait and then the offending player was disconnected and then everything
was fine, until he connected again :-)

> 2. return error codes through the flush functions (-1 => try again, -2 =>
>     permanent error), or through an fc_errno-type variable

=== Rationale

Its like this: we now have the server variables tcptimeout and netwait for
these cases. You can thank Lauri Tarkkala for this. I made some minor
changes to his code and added the ping packet. These are new features of
the future freeciv 1.20.0 which are on CVS since September.

Tcptimeout works like this: if the server has been trying to write data to
a client for X seconds and it doesn't suceed the client gets cut.

Netwait does this: in certain, let's say, strategic places (after the
AI does its updates, after we send the whole map, etc) we call the
function flush_packets(). What this function does is it tries to flush the
send buffers for all the players for X seconds.

I may have been overzealous on setting the default values for these
variables, but the only way to know that for sure is to test it on real
games. I suggest on the next game you participate you increase the
'netwait' variable and see if you still get disconnected due to a huge
send buffer. If you still get disconnected for this reason please report
what caused it (usually its some global event that causes major packet
spam, or sending the whole map when connecting, or the AIs doing unit
moves. the AIs can move units much faster than the client can display the
moves, especially if you have smooth unit moves on. which is the default).
We need the player's help in order to make the network code better in the
future. The current code was done based on the comments of several 
players on #freeciv.


In the end it all boils down to this: we have all this data to send that
is pilling up in the buffer because the guy on the other side doesn't
read it. I have chosen 512Kb as the maximum size which is a pretty large
number. Remember every player has a buffer like this! On a game with 20
players in the worst case you are using 10Mb just for these silly buffers!

=== What do we do?

Stop? We can do that but then we are penalizing the people with good
connections. We do this on ocasion for a short period.

We cut the player's connection? Kind of extreme but does the trick. The
buffer gets flushed and the player can reconnect and with a bit of luck he
gets a more stable connection this time. I don't like doing this too much
but i think its reasonable as long as on most cases we get superior
performance.

We start losing packets? Well we could just empty the buffer and pretend
nothing happened. This is hard to do because we don't keep track of the
packet boundaries in the send buffer and we can't risk sending half a
packet to a client. In the past the server did this and then the client
crashed. Or even worse the client munched the invalid data and sent the
server an invalid packet as a response which made the server crash.
Unfortunately the server trusts a bit too much and doesn't check well
the input it gets from the clients.
Even if we made sure we didn't send incomplete packets would you want to
continue playing without knowing if the ironclads you have standing close
to your enemy's capital are still alive or not? Without knowing that an
enemy nuked your capital, sent a paratrooper and captured it and you are
now with half your empire because you have suffered civil war? Your
client hapilly sending invalid packets of orders you give to units that
are no longer yours?

In my opinion on a strategy game like freeciv we can't lose packets and
get into a state that is completely different from the one on the server.
This can be done. But it would require a total rewrite of the
network code and its way of doing things. And you must make a much bigger
effort to ensure coherency between the server and the clients.


So we end up with the wait or cut options. Maybe we are cutting too much?
Perhaps. This is where your experience as a player would help!

Thanks for reading this longwinded reply,

---
Vasco Alexandre da Silva Costa @ Instituto Superior Tecnico, Lisboa