[Freeciv-Dev] packet batches? (was: [Patch] Making city report list faster)

[Reinier Post <rp@xxxxxxxxxx>]

On Mon, Jun 10, 2002 at 03:47:10AM -0400, Jason Short wrote:
> Raimar Falke wrote:
> > This patch should avoid make the city report dialog a bit faster.
> 
> Does this really do what it looks like it does?  <checks>  That's a 
> *great* improvement to something that has always bugged me (since it 
> seems so fixable).  In a test, it reduced the lag in a large-scale 
> change of city production by 50% (4 seconds to 2).

> I think in the future an even better solution to this problem is possible.
> 
> Problem: when you change stuff on the client, it has to "request" the 
> change from the server; the change usually doesn't even happen until the 
> server responds confirming the change.  Thus when the user makes a 
> change in the GUI, all that really happens is we call some function to 
> send a message to the server.

That is not a problem, it is a solution: that way you can send off
100 requests without hanging the interface.

> When the server sends a response, it 
> updates whatever-it-is-we're-talking-about and the client/GUI make the 
> actual change.
> 
> This is a particular problem when talking about the city report dialog. 
>   In theory this dialog can be used to easily control massive number of 
> cities, but anyone who has tried to do this quickly realizes that it is 
> actually pretty slow.  The main reason for this is that when you change 
> a bunch of cities, a bunch of change requests get sent to the server and 
> the update confirmations come in one-at-a-time - and are all handled 
> individually.  Thus the update takes a really long time to draw.

Yes.

I used the GTK 1.12.0 client on a slow machine yesterday and a Change
All operation took over half a minute.  I could see the targets being
changed in the dialog one by one, with about half a second in between.
It was so slow I thought a round trip to the server was being made for
every individual city.  But that is not how the code works.

> This patch fixes the second problem: when you change something in the 
> city report, the report is "frozen" until the last change is confirmed. 
>   Thus (to take an extreme, but possible example) if you change 100 
> cities all at once, instead of having 100 graphical updates there will 
> just be one (this is important since graphical updates are quite slow). 

Change All in the Xaw client is fast, so I think you have the cause here.

>   (It looks like it does this by monitoring the request ID, which is a 
> direct result of work for the CMA.  This is also a bit of a hack, and 
> probably a bit unstable in the long term as well.)

Requests may fail.  Raimar sends failure notifications.
But what happens if CMA is disabled?

With all respect I have for Raimar and his design desision to use
request IDs, I think Freeciv is worse off with them, because they are
fundametally a hack.

> However, there is still lag: the change cannot be made until the last 
> packet from the server is received.

I think that is acceptable.

> I think there is a better, and more 
> general solution that is possible.  When a change is made by the user in 
> the GUI, the client should change that data internally _as well as_ send 
> the request to the server.  When the server responds, the client again 
> updates.  If the change was approved, everything is easy and no action 
> is needed.  If the change was not approved, it must be reverted at the 
> client end.

That would be nice.

Another thing that would help is batch processing for packets.

Packets would contain 1 or more requests/notifications (the existing
packets).  The send_packet routines would not send requests/notifications
directly, but save them up; a 'flush' operation would send them off
in a packet.  Packet processing would just iterate over the received
batch of requests.  You send the 'change all' in a batch, and the server
responds by sending the city info notifications in a batch.

Then you can be smart about city info updates by looking at the batch.
It still introduces state, but at least the state doesn't cross
over to the server.

Of course you have to remember to flush in time.

It also introduces the problem of possible cheating: a client can
batch up its requests indefinitely, to have all of them processed
without intervention.  So the server would need to impose
some requirements on what kinds of requests can be batched together.

On the positive side, implementing turn-based mode becomes trivial,
and things like client side goto become feasible on high-latency
connections.

[...]

> There are several drawbacks of such a system.  One, the behavior when a 
> change is rejected by the server will be a bit chaotic: the user will 
> see the data changed and then reverted.

If lag is the problem, that can be addressed by greying the new values
out initially.  If screen updating is the problem, this will only
make it worse.

> Two, the data packet sent to 
> the client will have to be changed so that when a change is made 
> (whether it is accepted or rejected) the server needs to send the full 
> new state.  For instance, if the state is rejected the server needs to 
> tell the client what the state was before the change was made (i.e. what 
> it should be).

Aaaaargh!

This is broken.  What if something else changed the value
in between?  (E.g. your fellow player on a different connection.)

If you really want to remember what the value was at the time
the change request was made, you can keep it at the client side,
and use request IDs to look it up.

> But it will still be necessary to tell the client what 
> the state was if the change is accepted (consider the case where two 
> changes are made before the server responds: the first is accepted and 
> the second rejected).

Raimar implemented failure notifications for CMA.

You can use a combination of Raimars patch and your idea: set the
requested values, remember the old values may need correction,
remember the list of cties with tentative values, process each
incoming change notification, and at the last request ID, reset
all cities in the list to the old values.

> But the advantages quite outweigh these disadvantages, IMO.  First, as 
> I've said the update latency is dropped to nearly zero (you never have 
> to wait on a network connection).  Second, the system itself is both 
> more elegant and more stable.

Hm.

> It can be used for any update made 
> client-side (in fact, I'd say everything should be changed to keep 
> things consistent).

You mean, maintain two game structures,
'confirmed' and 'requested'?  That would be nice.

[...]

> Has such a thing been discussed before?

There was some discussions about requests when Raimar proposed them.
Some info updates are already being sent in batches.

> jason

-- 
Reinier