[Freeciv-Dev] Re: (PR#3516) Save files to large and redundant

["Reinier Post" <rp@xxxxxxxxxx>]

On Wed, Feb 26, 2003 at 02:04:31AM -0800, ChrisK@xxxxxxxx wrote:
> On Tue, Feb 25, 2003 at 04:38:41PM -0800, Erik Johansson wrote:
> > 
> > > If the program compresses them anyway, what does it matter?
> > 
> > To clarify, yes  I am complaining about the uncompressed size..
> > 
> > If no format change is made, perhaps it is a good idea to change to 
> > bzip2, which cuts more than half the size. (or custom pipe filters, 
> > which probably would be better)

It is not hard to add bzip2 support to Freeciv.
Some meddling with its configure.am file and those of
a package that already supports it.

I just ran a test and it turns out that, as usual, bzip2 isn't all that
much better than gzip - until it is applied to a whole sequence
of savegames!  gzip doesn't take advantage of the fact that subsequent
savegames are so similar, but bzip2 does.  My evidence is the set of
savegames from pubserver game 86503:

# the number of turns played:
sh> ls *.sav.gz | wc
    136     136    2720

# the total size of the savegames produced by civserver:
sh> cat *.sav.gz | wc
   7554   39881 1748971

# using -9 doesn't wouldn't really help:
sh>  for f in *.sav.gz; do gunzip -c $f | gzip -9 -c; done | wc
   8075   41713 1667174

# recompressing the whole sequence at once doesn't help, either:
sh> gunzip -c *.sav.gz | gzip -c | wc
   8077   41086 1742541
sh> gunzip -c *.sav.gz | gzip -9c | wc
   7947   40709 1671700

# making civserver support bzip2 wouldn't help only a little:
sh> for f in *.sav.gz; do gunzip -c $f | bzip2 -c; done | wc
   4625   25969 1230378

# but look at what happens what we bzip2 the whole sequence:
sh> gunzip -c *.sav.gz | bzip2 -c | wc
    752    4709  214177

So civserver must learn to read and write archives of savegames.
We can't just run tar on the compressed savegames:

sh> tar cf - *.sav.gz | bzip2 -c | wc
   4430   25147 1117958

But if civserver gets tar support it can be made to compress and
uncompress as appropriate so it would perform the equivalent of this:

sh> gunzip *.sav.gz
sh> tar cf - *.sav | bzip2 -c | wc
    709    5030  217681

We can probably use "libtar":

  http://www-dev.cites.uiuc.edu/libtar/

-- 
Reinier