[Freeciv-Dev] zlib

[Lauri Tarkkala <ltarkkal@xxxxxxxxxxxxxxx>]

Hello,

I did some simple experiments to see what kind of savings could be
attained in bandwith by using simple compression in the protocol.

I dumped the server->client protocol (application level only)
using tcpflow from a game played by an AI. The file was approximately
~5 MB in size. 

I added some code to the connection.c and sernet.c, which calculated
the min, max and avg size of the send_buffer when *do_unbuffer() 
attempts to flush the connection.  Some numbers:

> list clistat
Nr: FD: User:
00   4  Elros <Elros@localhost> Idle: 59 s  Lost: 0 B Sent: 157230 B Buf: 0 B
00   4  Elros <Elros@localhost> Packets: 106 Min: 27 B Max: 3448 B Avg 
612.849060 B

> list clistat
Nr: FD: User:
00   4  Elros <Elros@localhost> Idle: 1246 s  Lost: 0 B Sent: 3245961 B Buf: 0 B
00   4  Elros <Elros@localhost> Packets: 1614 Min: 9 B Max: 3708 B Avg 
527.174744 B

> list clistat
Nr: FD: User:
00   4  Elros <Elros@localhost> Idle: 1652 s  Lost: 0 B Sent: 4750200 B Buf: 0 B
00   4  Elros <Elros@localhost> Packets: 5135 Min: 9 B Max: 3708 B Avg 
251.081787 B

The avg "flushed buffer size" decreases when the game progresses (as witnessed
by some more "list clistats") during the test game. This obviously may
differ from game to game.

I then wrote a simple program to calculate the entropy of the tcpflow dump,
assuming 8-bit characters and then assuming 16-bit characters. For
8-bit chars this was approx ~3.3 bits / char and for 16-bit
chars this was approx ~5.2 bits / char.

Then i wrote a simple proggie (attached as ztest.c), which uses
zlib to do a data compression and decompression. The ZBUFSIZE
macro can be used to tune how big blocks deflate() gets called with
(quite important when compressing).

The  file sizes, and the entropy assuming 8 or 16-bit characters
was:

No compression: 5173372 B Entropy: 8-bit: ~3.32 16-bit:  ~5.24
ZBUFSIZE 256:   1766000 B Entropy: 8-bit: ~7.65 16-bit: ~14.00
ZBUFSIZE 512:   1528898 B Entropy: 8-bit: ~7.87 16-bit: ~15.00

So assuming that zlib is used to compress the data, the bandwidth 
used may be expected to drop to below 1/3, on the other hand, a simple 
huffman encoder might not actually do much worse.

Basically, I figure one probably ought to use zlib in freeciv
to compress atleast the server->client communication.

Lauri

-- 
"The credit belongs to the man in the arena whose face is marred by dust and
sweat and blood, who strives valiantly, who errs, and who comes up short again
and again, who knows the great enthusiasms, the great devotions, and spends
himself in a worthy cause. The man who at best knows the triumph of high
achievement and who at worst, if he fails, fails while daring greatly, so that
his place will never be with those cold timid souls who never knew victory or
defeat." - Theodore Roosevelt