[Freeciv-Dev] Re: Network bandwidth, and other things.

[Raimar Falke <rf13@xxxxxxxxxxxxxxxxx>]

On Tue, Nov 05, 2002 at 11:05:16AM +0100, Juan Uriarte wrote:
> Hello,
> 
> 
> On Tue, 5 Nov 2002, Raimar Falke wrote:
> > On Tue, Nov 05, 2002 at 08:06:02AM +0100, Juan Uriarte wrote:
> > > Hello,
> > >
> > >  seems Per/Raimar are psychics. After reading Rainar's his last post on
> > > network bandwidth it looks like he's hacked my computer ;-)
> > >
> > > diff.compress11++ on ftp.freeciv.org/.../incoming  contains what I call
> > > caching and he calls 'sending deltas' for the most sent packet types.
> > > An updated zlib compression of the tcp stream is included. Please, read
> > > the README in the patch. It is for a  CVS of around a week ago, I dont
> > > have internet at home, so I suck cvs and send this email at the same time.
> > > Results seem promissing (14kbytes/update, instead of 100+ for some
> > > scenarios). It is not fully debugged, CMA still crashes (usually when
> > > starting a revolution).
> >
> > I read your previous posts and thought "another one who implemented
> > zlib compression for the TCP stream and failed to provide numbers". So
> 
>  Numbers are provided after each read(2) chunk (running with -d 2) and
> on the server console when a player disconnects. Typical compression
> is less than 50% of the original size (between 30-40 in my experience),
> around 20% for dead players.

These numbers are just for the zlib and not the delta?!

> > I didn't looked at the patch. Now looking in the current patch I see
> > that we both had indead the same idea. I avoided zlib compression till
> > now since I thing it can be done on top of the deltas
> > transparently.
> 
>  This is what is implemented. All 6 features (zlib and 5 new packet types)
> are optional and indepent, backward compatibility is a goal.

If we implement delta we should implement it in all packets. So
backward compatibility isn't a goal for my patch.

> > Also till now I didn't know a good way to merge the
> > packets in chunks (to yield better compression ratio). Using
> > PACKET_PROCESSING_{STARTED,FINISHED} is nice.
> 
>  That is nice for the 'during the turn' packets, which get those 6 bytes
> of easily compressable data (0/3/3 and 0/3/2, IIRC).

6 bytes?

> More important is using the turn activities (FREEZE/THAW) as
> markers, as turn updates is my main cause of frustration.
> 
>  Also, STARTED/FINISHED are only sent to the player originating the
> action, so if you have shared vision with someone, those packets wont
> get merged. A possible solution is keeping track of the guys with shared
> vision with the player originating the command, set their per connection
> 'dont_flush' flag copying the shared_vision flags (in case the action
> just resets it) and  on FINISHED test which connection got packets at all
> (another per conn flag) and sending a PACKET_DUMMY to trigger
> the flushing on those connection (maybe there is no need to send a dummy
> packet and it is possible to just flush the compression context, but that
> would be easiest to implement (gotta read zlib.h again)). But I'm not sure
> this is worth it, as 'in-turn' actions usually dont generate that much
> data, so router queue build up should not be an issue.
> 
> 
> > Per will be pleased that
> > you produced a patch which only needs the cpp. Let me finish my delta2
> > and than we can compare features, performance and elegance.
> 
>  Well, elegance is not one of the main features of my patch ;-)
> Maybe renaming those cpp macros would improve readability.
> 

> Speaking of elegance: my patch implements caches for the data, that
> is, it duplicates some of the data that is already on the client

Same here. It is IMHO the sanest approach since it is fully
transparent and so doesn't require to change any of the other code.

> (on the server we must keep a cache per client anyway).

You have to do this per connection.

> I guess the map cache
> could be folded anyway on the existing map data, and maybe the
> game/player/unit cache too, but at least the city cache should be kept
> separate, 

> as I've noticed the CMA *writing* on the local city data kept by the
> client. As I dont know the rest of the code that well, I cant really
> tell if this makes sense or is a bug (there is a cma triggered bug
> im patch, which is a pain to debug, I've chasing it for some time
> now).

CMA is writing to the local city struct. I know. I coded it this
way. Take a look at client/agents/cma_core.c:real_fill_out_result.

> Readjusting some flags on SHORT_CITY_INFO could improve compression,
> the patch prints some flag statistics, to help improve the ratio.
> Actually a 32 bit flag field is sent always, with an optional 8bit
> flag field. Maybe splitting this into 16/32 would give better
> average packet size, but a careful study of flag usage is needed. I
> doubt there is much of a point, with zlib on top of it.

There is another way to encode which fields have changed and so are
transfered: instead of using the bit i as a flag you send the number
i. So if we have

packet_tile_info; PACKET_TILE_INFO
        COORD x,y; key

        UINT8 type, known;
        UINT16 special;
end

we need a value for type, known, special and one for
END-OF-HEADER. These are four values and we need 2bits for each. This
gives a variable header size in addition to the already variable body
size of the packet.

  size_for_fixed = no_of_fields
  size_for_variable = log_2(no_of_fields + 1) * (number_of_fields_changed + 1)

If no_of_fields = 15 we get:
  size_for_fixed = 15 bits
  size_for_variable = 4 + 4 * number_of_fields_changed
  variable is better if number_of_fields_changed <= 2

If no_of_fields = 31 we get:
  size_for_fixed = 31 bits
  size_for_variable = 5 + 5 * number_of_fields_changed
  variable is better if number_of_fields_changed <= 5

To bring this to the extreme you can now even compress the variable
header with a simple Huffman code. So if known does change a lot more
than type you assign "0" to known, "11" to type, "100" to special and
"101" to END-OF-HEADER. And as a nice extra you can do this adaptive
at runtime for optimal performance. All this is possible but this
won't give a result as adding zlib on top of the delta. Or given nice
hints to the zlib to give it bigger chunks to compress.

> Also I suggest running the patch with '-d 2' and scrolling a bit on
> your {x,w,a,e}term.

I will do it.

        Raimar

-- 
 email: rf13@xxxxxxxxxxxxxxxxx
  A supercomputer is a computer running an endless loop in just a second